New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics access log parsing tool for ingesting download.o.o logs. #1578

Merged
merged 5 commits into from Jun 19, 2018

Conversation

Projects
None yet
1 participant
@jberry-suse
Collaborator

jberry-suse commented Jun 15, 2018

  • 26ab99b:
    metrics/access: add summary README.md.

  • baad36e:
    dist: package metrics/access tool and related.

  • 8490877:
    metrics/grafana: add access dashboard and data source.

    Utilized by metrics/access tool.

  • 93530a7:
    metrics/access: initial aggregate script.

  • d073daf:
    metrics/access: initial ingest script.

I need to confirm the distribution bits work (likely will fail on travis) and setup the vhost on pontifex so I can run the most recent data. Overall, this should be rather close and will provide time for anyone interested in reviewing. A better solution for determining real products would be nice as there is a lot of completely garbage data and the current approach requires manually update when new product is released. However, it does not require re-parsing data and as such is not time pertinent.

The generated graphs on my local setup can be seen below.

image

@jberry-suse

This comment has been minimized.

Collaborator

jberry-suse commented Jun 15, 2018

I plan to do a write-up on all the interest data I discovered and possibly some room to run further analysis looking for such data. So far all the data marked invalid by the tool that I reviewed is exactly that...invalid. The graph will make it clear if the parsing becomes broken in the future as there would be a huge spike.

@jberry-suse

This comment has been minimized.

Collaborator

jberry-suse commented Jun 15, 2018

I'll also add a bit more to README.

@jberry-suse

This comment has been minimized.

Collaborator

jberry-suse commented Jun 15, 2018

If it's no clear from picture the "Aggregation Frequency" can be changed to:

  • day
  • week
  • month

As the data has been pre-processed in those intervals. All graphs are thus refreshed from the new source.

@jberry-suse

This comment has been minimized.

Collaborator

jberry-suse commented Jun 15, 2018

A tried a variety of parsing approaches, but as I've seen before and come to expect regex turned out to be fastest...likely becomes of native implementation instead of string operations done in scripting language.

@jberry-suse

This comment has been minimized.

Collaborator

jberry-suse commented Jun 19, 2018

Made a few small tweaks, including dist fix and DNS name suggested in progres.o.o issue. Added a couple extra sections to README.

@jberry-suse

This comment has been minimized.

Collaborator

jberry-suse commented Jun 19, 2018

Added annotations for the last three Leap releases based on the data already imported from schedules. This should be helpful in answering the question, "does anyone Beta test Leap?" as @lnussel was curious.

@jberry-suse

This comment has been minimized.

Collaborator

jberry-suse commented Jun 19, 2018

For example, with 42.2 and 42.3 annotations enabled.

image

@jberry-suse

This comment has been minimized.

Collaborator

jberry-suse commented Jun 19, 2018

Or 42.2 show by itself we see 6.7k users by Beta 1:

image

@jberry-suse

This comment has been minimized.

Collaborator

jberry-suse commented Jun 19, 2018

No one seems interested in reviewing.

@jberry-suse jberry-suse merged commit d41cbda into openSUSE:master Jun 19, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@jberry-suse jberry-suse deleted the jberry-suse:metrics-access branch Jun 19, 2018

@jberry-suse

This comment has been minimized.

Collaborator

jberry-suse commented Jun 19, 2018

Deployed and metrics.o.o and completed initial data import with pontifex ingest disabled (pending vhost setup). https://metrics.opensuse.org/d/osrt_access/osrt-access

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment