Provide metrics to measure the amount and measure the type of work done by JBI

[mostlygeek]

I'm interesting in being able to measure how much work manual JBI is automating away for us.

I believe the numbers I need for this are:

• number of bugzilla operations done, segmented by type and status (success, failed, etc)
• number of jira operations done, segmented by type and status (success, failed, etc)
• I would also like to sum up those values into a monthly count

Questions:

• how much effort would it take to make that happen?
• any recommendations on different metrics?

[leplatrem]

Since v3.1.0 we have these metrics:

• jbi.bugzilla.ignored.count

• jbi.bugzilla.processed.count

• jbi.action.execution.timer

• jbi.jira.methods.update_issue_field.count

• jbi.jira.methods.set_issue_status.count

• jbi.jira.methods.issue_add_comment.count

• jbi.jira.methods.create_issue.count

• jbi.jira.methods.update_issue_field.timer

• jbi.jira.methods.set_issue_status.timer

• jbi.jira.methods.issue_add_comment.timer

• jbi.jira.methods.create_issue.timer

• jbi.bugzilla.methods.getbug.count

• jbi.bugzilla.methods.get_comments.count

• jbi.bugzilla.methods.update_bugs.count

• jbi.bugzilla.methods.getbug.timer

• jbi.bugzilla.methods.get_comments.timer

• jbi.bugzilla.methods.update_bugs.timer

Most of the measurements you describe should be doable with Grafana (we don't count failed calls, but since a Sentry is sent each time, we expect them to be 0).

In case Grafana is not adequate, I think that the application logs are also sent to BigQuery. So in theory, with the appropriate permissions, we could also have a Redash sheet that queries logs to count operations (we log them consistently)

[mostlygeek]

What I would like to do is be able to get a sum of the change operations JBI has done over a specific time.

In pseudo SQL:

SELECT
  jbi_operation_name,
  year(timestamp), 
  month(timestamp),
  sum( if status="success", 1, 0 ) success,
  sum( if status="failed", 1, 0 ) failed,
FROM jbi_logs
GROUP BY 
  jbi_operation_name,
  year(timestamp), 
  month(timeatmp)

To produce a table like:

jbi.jira.set_issue_status    | 2022 | 8 | 100 | 0 |
jbi.jira.update_issue_field  | 2022 | 8 |  23 | 4 |
jbi.jira.create_issue        | 2022 | 8 |   5 | 0 |
jbi.jira.update_issue_field  | 2022 | 8 | 125 | 5 |

It'll be a query I run monthly/quarterly.

Is this something that I can easily accomplish with the statsd metrics? I think the data in BQ would be easier to run a query like that. Though, I'm not sure if the data I need is in BQ or which log fields I would need to look at.

[leplatrem]

The following query should provide something similar.

SELECT
  extract(YEAR FROM timestamp) AS year,
  extract(MONTH FROM timestamp) AS month,
  jsonPayload.fields.operation AS operation,
  COUNT(*) AS total
FROM `moz-fx-jbi-prod-0397.log_storage.stdout_*` 
WHERE
  resource.labels.container_name = 'jbi' AND
  jsonPayload.Logger = 'jbi' AND
  jsonPayload.fields.operation IS NOT NULL
GROUP BY 1, 2, 3
ORDER BY 1, 2, 4 DESC

For some reason, I don't see (yet?) the operations we introduced in v3.2.X (link, comment, ...). I assume BigQuery are replicated on a daily basis and may require a bit of time.

@mostlygeek Do you really need detail at the "method" level? Or, the enum that is here would be enough?

jira-bugzilla-integration/jbi/__init__.py

Lines 10 to 22 in fcb7390

	class Operation(str, Enum):
	"""Enumeration of possible operations logged during WebHook execution."""
	HANDLE = "handle"
	EXECUTE = "execute"
	IGNORE = "ignore"
	SUCCESS = "success"
	CREATE = "create"
	UPDATE = "update"
	DELETE = "delete"
	COMMENT = "comment"
	LINK = "link"

[mostlygeek]

That's perfect! Thank you!
I ran the query in BQ and it gave what I needed.

[mostlygeek]

Follow up:

For some reason, I don't see (yet?) the operations we introduced in v3.2.X (link, comment, ...). I assume BigQuery are replicated on a daily basis and may require a bit of time.

In #157 the jira attribute changed from a string to an object. The BQ schema didn't match so a lot of logs couldn't be parsed. I asked @kkleemola to update the BQ schema to match the change in #157