New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Collect statistics for gauge commands #658

Closed
zabil opened this Issue May 25, 2017 · 7 comments

Comments

3 participants
@zabil
Copy link
Member

zabil commented May 25, 2017

If a user opts-in to send usage information.

Collect statistics of gauge commands execution to improve or deprecate command line options.
The statistics will be collected via google analytics using measurement protocol

This must exclude user identifiable information like ip and user specific parameters like spec folders

$ gauge myspecfolder

Must not record myspecfolder.

or

gauge --dir="../"

Must exclude directory information.

@zabil zabil added the telemetry label May 25, 2017

@sriv

This comment has been minimized.

Copy link
Member

sriv commented Jun 2, 2017

The default behaviour shall be "opted in", with options for the users to disable these pings. This is so that unattended execution of gauge is not left out.

@sriv sriv added the in progress label Jun 2, 2017

@sriv sriv self-assigned this Jun 2, 2017

sriv added a commit that referenced this issue Jun 2, 2017

add flag to opt out from analytics, #658
also add a flag to log requests made to analytics engine

sriv added a commit that referenced this issue Jun 3, 2017

sriv added a commit that referenced this issue Jun 3, 2017

sriv added a commit that referenced this issue Jun 8, 2017

add flag to opt out from analytics, #658
also add a flag to log requests made to analytics engine

sriv added a commit that referenced this issue Jun 9, 2017

send single execution event, #658
comma separated execution flags

rename package analytics -> track

sriv added a commit that referenced this issue Jun 9, 2017

sriv added a commit that referenced this issue Jun 13, 2017

send single execution event, #658
comma separated execution flags

rename package analytics -> track

sriv added a commit that referenced this issue Jun 13, 2017

sriv added a commit that referenced this issue Jun 13, 2017

sriv added a commit that referenced this issue Jun 13, 2017

add flag to opt out from analytics, #658
also add a flag to log requests made to analytics engine

sriv added a commit that referenced this issue Jun 13, 2017

sriv added a commit that referenced this issue Jun 13, 2017

sriv added a commit that referenced this issue Jun 13, 2017

send single execution event, #658
comma separated execution flags

rename package analytics -> track

sriv added a commit that referenced this issue Jun 13, 2017

sriv added a commit that referenced this issue Jun 13, 2017

kashishm added a commit that referenced this issue Jun 13, 2017

add flag to opt out from analytics, #658
also add a flag to log requests made to analytics engine

kashishm added a commit that referenced this issue Jun 13, 2017

kashishm added a commit that referenced this issue Jun 13, 2017

kashishm added a commit that referenced this issue Jun 13, 2017

send single execution event, #658
comma separated execution flags

rename package analytics -> track

kashishm added a commit that referenced this issue Jun 13, 2017

kashishm added a commit that referenced this issue Jun 13, 2017

@sriv

This comment has been minimized.

Copy link
Member

sriv commented Jun 14, 2017

Information captured:

  • IP address is anonymized

  • No Personally Identifyable Information is collected.

  • A Client ID (unique UUID)

  • Parameters Tracked:

    • Manifest (Language + Plugins)
    • Execution (tagged, parallel, sort, simple-console, verbose flags). No project specific values are captured.
    • Refactor (Rename) (only usage, no parameters)
    • Refactor (Extract To Concept) (only usage, no parameters)
    • Format (only usage, no parameters)
    • List Templates
    • Plugin Install
    • Plugin Update
    • Plugin Install-All
    • Plugin Update-All
    • Plugin Install from Zip
    • Project Init
    • Daemon execution
    • Docs generation (ex spectacle)

The raw requests that is sent can be traced by setting gauge_ telemetry_log_enabled=true in gauge.properties.

sriv added a commit that referenced this issue Jul 3, 2017

@sriv

This comment has been minimized.

Copy link
Member

sriv commented Jul 3, 2017

should be available in nightly >= 03-jul-2017

@sriv sriv removed the in progress label Jul 3, 2017

@sriv sriv added the ready for QA label Jul 3, 2017

@sguptatw

This comment has been minimized.

Copy link
Contributor

sguptatw commented Jul 4, 2017

  • gauge install event without any parameters on a project is not getting logged.
  • Capture the language, template being used while init.
  • Fixed - gauge uninstall <plugin> is not getting logged.
  • Should a failed request get logged - example failed refactoring, init, add etc?
  • Events which are fast are not getting logged - example format, docs spectacle with one file.
  • gauge update -c is not getting logged

  • The event label of install-all should contain the list of all plugins installed.
  • The event label of update-c should contain the list of all plugins being checked for update.
  • If the event label for an event cannot be logged because we do not capture project specific information the let the event name be the label. This being empty is an issue when custom reports are created.
    e.g: gauge daemon
    gauge format
    gauge refactor
    template list
    gauge validate
  • The analytics of gauge refactor is getting logged. However the action is getting hung.
  • event category should be based on the new subcommands
    e.g: gauge init --templates is getting listed as event category templates event label list. However, this should be listed as event category init and event label templates. Please refer gauge help --legacy and update all such usages.
  • When gauge run is used the event category is execution. On drilling down one level the label and category is serial. In this case doesn't it make sense to capture it as serial-?
  • simple console is captured only for a parallel run. This should get captured for a serial run as well
  • when data network is off. The message Unable to send analytics data, Post https://www.google-analytics.com/collect: dial tcp: lookup www.google-analytics.com on [::1]:53: read udp [::1]:62454->[::1]:53: read: connection refused is printed on the console. Can we move this to log or debug info.

@sguptatw sguptatw added ready and removed ready for QA labels Jul 7, 2017

@sriv sriv added in progress and removed ready labels Jul 10, 2017

sriv added a commit that referenced this issue Jul 10, 2017

@sriv

This comment has been minimized.

Copy link
Member

sriv commented Jul 10, 2017

Fix available in nightly >= 10-Jul-2017

gauge install event without any parameters on a project is not getting logged.

Happened because of fast events, fixed.

Capture the language, template being used while init

Fixed.

gauge uninstall is not getting logged.

Happened because of fast events, fixed.

Should a failed request get logged - example failed refactoring, init, add etc?

Telemetry now captures arguments to init and add subcommands. What is expected to be captured from failed refactoring? Telemetry does not track refactoring text.

Events which are fast are not getting logged - example format, docs spectacle with one file.

Fixed

gauge check-updates is not getting logged.

Incorrect command. Try gauge updates -c (or gauge --check-updates for legacy syntax).

@sriv sriv added ready for QA and removed in progress labels Jul 10, 2017

sriv added a commit that referenced this issue Jul 10, 2017

@sguptatw

This comment has been minimized.

Copy link
Contributor

sguptatw commented Jul 13, 2017

Please find the test results inline in my above comment.

@sguptatw sguptatw added ready and removed ready for QA labels Jul 13, 2017

@sriv

This comment has been minimized.

Copy link
Member

sriv commented Jul 13, 2017

The event label of install-all should contain the list of all plugins installed.

Requires track to happen post the event. deferred.

The event label of update-c should contain the list of all plugins being checked for update.

Requires track to happen post the event. deferred.

If the event label for an event cannot be logged because we do not capture project specific information the let the event name be the label. This being empty is an issue when custom reports are created.
e.g: gauge daemon
gauge format
gauge refactor
template list
gauge validate

Does not make sense to do this, except when there is a default implicit behaviour. changed for gauge install

The analytics of gauge refactor is getting logged. However the action is getting hung.

Cannot replicate.

event category should be based on the new subcommands
e.g: gauge init --templates is getting listed as event category templates event label list. However, this should be listed as event category init and event label templates. Please refer gauge help --legacy and update all such usages.

changed gauge init --templates -> event category="init", event action = "templates"

When gauge run is used the event category is execution. On drilling down one level the label and category is serial. In this case doesn't it make sense to capture it as serial-?

removed logging of "serial" in both action and label.

simple console is captured only for a parallel run. This should get captured for a serial run as well

label now captures simple v/s rich console.

Addressed via b4ea028

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment