Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add metrics for fuzzy-finder filtering time #375

Merged
merged 4 commits into from Mar 29, 2019

Conversation

Projects
None yet
2 participants
@rafeca
Copy link
Contributor

commented Mar 29, 2019

Summary

In order to understand better the real world performance of the fuzzy-finder and allow us to take better decisions, we want to start logging the time it takes to execute a filtering action.

Along with the time taken to do the filtering, we want to log some additional metadata:

  • Number of files in the repository: the time will vary a lot depending on the project size so we need to be able to filter results by similar project sizes.
  • Scoring system: Our hypothesis is that the new fast scoring system implemented here will have an important impact on the filtering time, so we want to log which is the scoring type that's being used.

Since the standard scoring system is not controlled by the fuzzy finder (but via atom-select-list) we cannot easily log it from here. That's not a big deal since we're planning on deprecating that system.

Schema of the data

  • eventType: fuzzy-finder-v1

  • metadata

    field value
    ec time-to-filter
    el Scoring type (alternate or fast)
    ev Number of files in the project

Sample JSON payload

{
  "eventType": "fuzzy-finder-v1",
  "durationInMilliseconds": 539,
  "metadata": {
    "ec": "time-to-filter",
    "el": "fast",
    "ev": 789,
    "cd2": "x64",
    "cd3": "x64",
    "cm1": 180,
    "cm2": 83,
    "sr": "1680x1050",
    "vp": "1680x591",
    "aiid": "dev"
  },
  "date": "2019-03-21T17:05:32.706Z"
}

Verification process

  • Check that Atom is sending the expected JSON payload to the GitHub backend systems by enabling the metrics debug mode.
  • Check that the the events get stored on our pipelines correctly.

//cc @telliott27

@rafeca rafeca requested a review from jasonrudolph Mar 29, 2019

@rafeca

This comment has been minimized.

Copy link
Contributor Author

commented Mar 29, 2019

Something worth to notice is that this is going to create an event for every single keypress on the fuzzy finder. I'm going to add some logic to throttle the number of events so we send maximum one e.g every minute or so.

@jasonrudolph
Copy link
Member

left a comment

Overall: 👍

I'm going to add some logic to throttle the number of events so we send maximum one e.g every minute or so.

This sounds like a good idea. ⚡️😄

@@ -22,3 +22,18 @@ Currently the Fuzzy finder does not log any counter events.
## Standard events

Currently the Fuzzy Finder does not log any standard events.

This comment has been minimized.

Copy link
@jasonrudolph

jasonrudolph Mar 29, 2019

Member

I think we want to remove this ## Standard Events section here, since it's included at the bottom of this file.

This comment has been minimized.

Copy link
@rafeca

rafeca Mar 29, 2019

Author Contributor

ups, my mistake... copy pasta 😅

rafeca added some commits Mar 29, 2019

⬆️ underscore-plus@1.7.0
`underscore-plus@1.7.0` contains `underscore@1.9.1` which has support
for cancelling throttled functions.

@rafeca rafeca force-pushed the filtering-metrics branch from fc053c8 to 58424ac Mar 29, 2019

Add throttling to the filter metric events
This will ensure that we don't send every single filter event happening
on each keystroke from the user on the fuzzy finder, which would be too
much data.

Adding some throttling could cause some bias (as opposed to do standard
sampling), since we're going to send the first and last filter events
from a search than the intermediate ones, but that should not be an
issue since all these searches should perform equally (and the last one
is actually the most important one for the user).

@rafeca rafeca force-pushed the filtering-metrics branch from 58424ac to 41ff6e7 Mar 29, 2019

@rafeca rafeca merged commit 511f091 into master Mar 29, 2019

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@rafeca rafeca deleted the filtering-metrics branch Mar 29, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.