New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Research item: anonymous usage analytics #108

Open
alexellis opened this Issue Sep 17, 2017 · 12 comments

Comments

Projects
None yet
3 participants
@alexellis
Member

alexellis commented Sep 17, 2017

Expected Behaviour

Track usage of commands (verbs), parameters (used), build times and other anonymous data that can help us understand usage and improve the CLI.

Remit:

  • This would only cover the CLI usage - this would not instrument the back-end services.

  • Opt-in by default, opt-out of collection via environmental variable in a similar fashion to VSCode and Brew.

.bash_profile

export OPEN_FAAS_TELEMETRY=0
  • We must explain how data is used, how long it is kept for etc and make sure it's in keeping with data protection.

  • Data must be anonymised

Current Behaviour

Relying on unsolicited feedback / word of mouth

Possible Solution

Google Analytics (subject to costing out) - has a Golang library. https://github.com/google/google-api-go-client/blob/master/analytics/v3/analytics-gen.go

Initial work would only be a PoC

@johnmccabe

This comment has been minimized.

Show comment
Hide comment
@johnmccabe

johnmccabe Sep 23, 2017

Member

I'll pick this up, have done something similar in the past.

Member

johnmccabe commented Sep 23, 2017

I'll pick this up, have done something similar in the past.

@johnmccabe

This comment has been minimized.

Show comment
Hide comment
@johnmccabe

johnmccabe Oct 8, 2017

Member

Derek assign: johnmccabe

Member

johnmccabe commented Oct 8, 2017

Derek assign: johnmccabe

@johnmccabe

This comment has been minimized.

Show comment
Hide comment
@johnmccabe

johnmccabe Oct 8, 2017

Member

Have this working locally, will push up a PR asap - heres a screengrab of a Data Studio report, generated with a dev build - thanks to @kinghuang and @burtonr for being my guinea pigs today.

faas-cli-metrics-dashboard-alpha-testdata

Member

johnmccabe commented Oct 8, 2017

Have this working locally, will push up a PR asap - heres a screengrab of a Data Studio report, generated with a dev build - thanks to @kinghuang and @burtonr for being my guinea pigs today.

faas-cli-metrics-dashboard-alpha-testdata

@johnmccabe

This comment has been minimized.

Show comment
Hide comment
@johnmccabe

johnmccabe Oct 8, 2017

Member

@alexellis can you have a think about what else might be useful to capture

Currently have

  • Country
  • Random UUID for telling an existing user from a new one
  • OS
  • CLI version
  • CLI verb used
  • Language when the subcommand uses it, unset otherwise

Other ideas

  • Number of deployed functions
  • Total invocations across all functions
  • Mean invocations across all functions
  • Whether the gateway is running on localhost vs. a remote host (thinking this should just be a bool, as I don't think we want to start doing lookups on IPs).
  • Backend in use, Swarm, Kubernetes, Hyper etc (this one is a no-brainer I think)
Member

johnmccabe commented Oct 8, 2017

@alexellis can you have a think about what else might be useful to capture

Currently have

  • Country
  • Random UUID for telling an existing user from a new one
  • OS
  • CLI version
  • CLI verb used
  • Language when the subcommand uses it, unset otherwise

Other ideas

  • Number of deployed functions
  • Total invocations across all functions
  • Mean invocations across all functions
  • Whether the gateway is running on localhost vs. a remote host (thinking this should just be a bool, as I don't think we want to start doing lookups on IPs).
  • Backend in use, Swarm, Kubernetes, Hyper etc (this one is a no-brainer I think)
@johnmccabe

This comment has been minimized.

Show comment
Hide comment
@johnmccabe

johnmccabe Oct 8, 2017

Member

Also, do you want this opt-in, or opt-out.

Adding a flag and support for setting an envvar that sets your preference wrt letting the project capture the basic usage info above.

Member

johnmccabe commented Oct 8, 2017

Also, do you want this opt-in, or opt-out.

Adding a flag and support for setting an envvar that sets your preference wrt letting the project capture the basic usage info above.

@johnmccabe

This comment has been minimized.

Show comment
Hide comment
@johnmccabe

johnmccabe Oct 8, 2017

Member

Derek add label: hacktoberfest

Member

johnmccabe commented Oct 8, 2017

Derek add label: hacktoberfest

@open-derek open-derek bot added the hacktoberfest label Oct 8, 2017

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Oct 9, 2017

@alexellis @johnmccabe This looks great. Should we also consider:

  • Number of replicas
  • Duration of invocations
  • API connected? Yes/No

In reference to your other ideas section

  • Number of deployed functions != sample functions

ghost commented Oct 9, 2017

@alexellis @johnmccabe This looks great. Should we also consider:

  • Number of replicas
  • Duration of invocations
  • API connected? Yes/No

In reference to your other ideas section

  • Number of deployed functions != sample functions
@alexellis

This comment has been minimized.

Show comment
Hide comment
@alexellis

alexellis Oct 9, 2017

Member

@briansiemens - John's work right now is not around the daemon/server but at the faas-cli level.

I'd say opt out via ENV_VAR to follow the model of Brew and VSCode.

We can report functions deployed / invoked but only really if someone uses faas-cli list

In order to track back-end used, we should return a header via the gateway for system functions - or introduce an info endpoint. People should also be able to turn off the clue being leaked via the gateway for when they don't want to reveal this information. (separate proposal)

Member

alexellis commented Oct 9, 2017

@briansiemens - John's work right now is not around the daemon/server but at the faas-cli level.

I'd say opt out via ENV_VAR to follow the model of Brew and VSCode.

We can report functions deployed / invoked but only really if someone uses faas-cli list

In order to track back-end used, we should return a header via the gateway for system functions - or introduce an info endpoint. People should also be able to turn off the clue being leaked via the gateway for when they don't want to reveal this information. (separate proposal)

@johnmccabe

This comment has been minimized.

Show comment
Hide comment
@johnmccabe

johnmccabe Oct 9, 2017

Member

@briansiemens can you clarify what you're mean by API connected.

Member

johnmccabe commented Oct 9, 2017

@briansiemens can you clarify what you're mean by API connected.

@vegasbrianc

This comment has been minimized.

Show comment
Hide comment
@vegasbrianc

vegasbrianc Oct 9, 2017

@johnmccabe Sorry was logged into the wrong account. I was just brainstorming if we could see if something like Kong or some other service is connected or not.

vegasbrianc commented Oct 9, 2017

@johnmccabe Sorry was logged into the wrong account. I was just brainstorming if we could see if something like Kong or some other service is connected or not.

johnmccabe added a commit to johnmccabe/faas-cli that referenced this issue Oct 24, 2017

Add Opt-out analytics to CLI
This commit implements openfaas#108 adding an opt-out analytics capture using
Google Analytics.

Refer to the analytics.md file for details of what is being captured at
this point and how to opt out from the capture.

*Note* it does not capture function counts etc, pending the merge of
openfaas/faas#315.

@alexellis - you will need to replace the `trackingID` and
`applicationName` in `analytics/google_analytics.go` in order to direct
events to a GA account associated with the OpenFaaS project.

Signed-off-by: John McCabe <john@johnmccabe.net>
@alexellis

This comment has been minimized.

Show comment
Hide comment
@alexellis

alexellis Nov 8, 2017

Member

Additional useful data:

  • Count of URLs in login section and whether they use HTTP/s
  • Are they localhost or remote (don't pick up the actual URL)
  • Is --parallel being used?

Let's checkout the new commands we've added too.

I need to checkout any legal implications or data protection notices we need to include before merging. Conceptually looks great.

On first run - should we write a $HOME/.openfaas/privacy_notice file?

Member

alexellis commented Nov 8, 2017

Additional useful data:

  • Count of URLs in login section and whether they use HTTP/s
  • Are they localhost or remote (don't pick up the actual URL)
  • Is --parallel being used?

Let's checkout the new commands we've added too.

I need to checkout any legal implications or data protection notices we need to include before merging. Conceptually looks great.

On first run - should we write a $HOME/.openfaas/privacy_notice file?

johnmccabe added a commit to johnmccabe/faas-cli that referenced this issue Nov 12, 2017

Add Opt-out analytics to CLI
This commit implements openfaas#108 adding an opt-out analytics capture using
Google Analytics.

Refer to the analytics.md file for details of what is being captured at
this point and how to opt out from the capture.

*Note* it does not capture function counts etc, pending the merge of
openfaas/faas#315.

@alexellis - you will need to replace the `trackingID` and
`applicationName` in `analytics/google_analytics.go` in order to direct
events to a GA account associated with the OpenFaaS project.

Signed-off-by: John McCabe <john@johnmccabe.net>

johnmccabe added a commit to johnmccabe/faas-cli that referenced this issue Nov 24, 2017

Add Opt-out analytics to CLI
This commit implements openfaas#108 adding an opt-out analytics capture using
Google Analytics.

Refer to the analytics.md file for details of what is being captured at
this point and how to opt out from the capture.

*Note* it does not capture function counts etc, pending the merge of
openfaas/faas#315.

@alexellis - you will need to replace the `trackingID` and
`applicationName` in `analytics/google_analytics.go` in order to direct
events to a GA account associated with the OpenFaaS project.

Signed-off-by: John McCabe <john@johnmccabe.net>
@alexellis

This comment has been minimized.

Show comment
Hide comment
@alexellis

alexellis Jul 14, 2018

Member

Bump on this. If we can do this in a GDPR-friendly fashion then I think we should and it will help greatly with understanding usage of the flags/templates etc.

Member

alexellis commented Jul 14, 2018

Bump on this. If we can do this in a GDPR-friendly fashion then I think we should and it will help greatly with understanding usage of the flags/templates etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment