Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial metrics ingestion, processing, write to influxdb, and grafana dashboards. #1159

Merged
merged 3 commits into from Oct 6, 2017

Conversation

@jberry-suse
Copy link
Contributor

jberry-suse commented Oct 4, 2017

A lot of room for improvement and additional metrics that can be extracted. Including non-final state requests would allow for analyzing the current staging state instead of only historical state. Additionally, the current state can be used to present an activity log.

Handling incremental updates is non-trivial given the deltas are evaluated and stored in sum state. A few possible approaches, but likely not worth the hassle given the relatively short processing time and infrequent desire to update data (daily at minimum).

One of the biggest things missing is walking the history of the dashboard container to know the state of config options and psuedo-ignore list so those requests can be separated out from backlog.

For simplicity the non-final requests are not included at all which means that for openSUSE:Factory the totals for backlog and such will not reflect currently open request.

Additionally, I have it set to infinitely cache request results until I decide how to handle only looking for new requests. Removing the cache directory (~/.cache/osc-plugin-factory-metrics) or the last few pages will handle looking for new ones. Perhaps the easiest tweak would be sorting the search results on last update for requests to make it easy to append newly changed requests.

The overall structure is to loop over all request creating measurement points, many of which are deltas (like add one to this bucket, or minus one from another). After all requests are process the points are walked to evaluate the deltas and re-create the state at that time.

Due to a number of issues with the data returned from OBS the Factory graphs will show negative values at a couple locations due to completely incorrect dates. The final numbers zero out correctly since the incorrect dates are still within dataset start and end dates. One caveat is that Grafana seems to not always grab the last record and thus shows a trailing one, but the data generated correctly ends at zero. The following are the OBS related issues:

I also have an experimental version of #730 being generated by Grafana. With a few tweaks and including open requests (at least for the log) I expect this to provide a rather nice activity log.

Lastly, a test could be added in a several ways, but this it seems like overkill to add to in-depth of a test for this. The simplest approach would be a test against a finished project using live data and confirm some points of interest and total records created and such. Alternatively, a hand-crafted set of requests could be returned using the current mock setup...or a full OBS setup and release requests setup to be queried. The live approach seems better since the data should not change, provides a large and unique enough data-set to be relevant, and does not requiring adding large XML dumps to this repository.

Finally, ingesting the annotation events from an external data source would definitely be preferable if one exists. Additionally, ingesting upstream release dates of interest or annotating version changes of certain packages signifying major updates may also be interesting.

Since open-build-service team did not indicate interest in hosting the tools I filed an IT ticket to have a publicly facing machine so the metrics can be available to anyone. May also encourage others to expand what is extracted.

@jberry-suse

This comment has been minimized.

Copy link
Contributor Author

jberry-suse commented Oct 4, 2017

Some interesting trends were certainly notable. See a snapshot of graphs generated using everything in this PR.

https://imgur.com/a/kKxnb

@jberry-suse jberry-suse force-pushed the jberry-suse:metrics-v1 branch from f45453d to a156b29 Oct 4, 2017
@openSUSE openSUSE deleted a comment from coveralls Oct 4, 2017
@jberry-suse jberry-suse force-pushed the jberry-suse:metrics-v1 branch from a156b29 to 34aa7e9 Oct 4, 2017
@openSUSE openSUSE deleted a comment from coveralls Oct 4, 2017
@openSUSE openSUSE deleted a comment from coveralls Oct 4, 2017
@jberry-suse jberry-suse force-pushed the jberry-suse:metrics-v1 branch from 183f75b to fdb0e4b Oct 4, 2017
@openSUSE openSUSE deleted a comment from coveralls Oct 4, 2017
@jberry-suse jberry-suse force-pushed the jberry-suse:metrics-v1 branch from fdb0e4b to efe2969 Oct 4, 2017
@jberry-suse

This comment has been minimized.

Copy link
Contributor Author

jberry-suse commented Oct 4, 2017

If anyone plans to review and would like more time please comment, otherwise I'll merge this tomorrow.

jberry-suse added 3 commits Oct 4, 2017
… dashboards.

A lot of room for improvement and additional metrics that can be extracted.
Including non-final state requests would allow for analyzing the current
staging state instead of only historical state. Additionally, the current
state can be used to present an activity log.

Handling incremental updates is non-trivial given the deltas are evaluated
and stored in sum state. A few possible approaches, but likely not worth
the hassle given the relatively short processing time and infrequent desire
to update data (daily at minimum).
Excludes large JSON files from main package as most users will not need.
@jberry-suse jberry-suse force-pushed the jberry-suse:metrics-v1 branch from efe2969 to 298ca5e Oct 6, 2017
@jberry-suse

This comment has been minimized.

Copy link
Contributor Author

jberry-suse commented Oct 6, 2017

rebased post other merges

@openSUSE openSUSE deleted a comment from coveralls Oct 6, 2017
@jberry-suse jberry-suse merged commit e8e1a3d into openSUSE:master Oct 6, 2017
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@coveralls

This comment has been minimized.

Copy link

coveralls commented Oct 6, 2017

Coverage Status

Coverage remained the same at 47.41% when pulling 298ca5e on jberry-suse:metrics-v1 into 9621116 on openSUSE:master.

@jberry-suse jberry-suse deleted the jberry-suse:metrics-v1 branch Oct 6, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.