Lakshanf/issue2066/telemetry #2145

LakshanF · Mar 30, 2016

.NET Core Tools Telemetry collection

Overview

In RC2 of the command line tools, we've started collecting usage telemetry. We've made this choice in order to help us better understand the usage patterns of the CLI tools, and will thus allow us to see which parts of the CLI should be improved first

The data collected will not have any personal data, such as usernames or emails or anything that can be used to identify the actual user. We will also not scan the code and we will not extract any project-level data that can be considered sensitive, such as name, repo or author (if you set those in your project.json).

The data that we collect will be shown publicly in due time. Due time here means when we have enough data to show and when we figure out how to make it accessible.

Behavior

The telemetry tracking is "on" by default.

There is a way to opt-out of the telemetry gathering process by setting an environment variable DOTNET_CLI_TELEMETRY_OPTOUT. Doing this will stop the collection process from running. In order to set the environment variable, please use the existing operating system mechanisms for this (e.g. export on OS X/Linux).

What do we collect?

We collect the following pieces of data:

The command being used (e.g. "build", "restore")
Arguments passed to the command
ExitCode of the command
For test projects, the test runner being used
Timestamp of invocation
Details about the project commands are invoked on
- Framework used
- If RIDs are present in the "runtimes" node
The CLI version being used

Questions, feedback, comments

If you have any questions, feedback or comment, please feel free to leave them either on our issues, in our Gitter channel. If you do file an issue for this, please be sure to tag @blackdwarf and @piotrMSFT so we would get notified.

This change is

@LakshanF

Hi @LakshanF, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!

The agreement was validated by .NET Foundation and real humans are currently evaluating your PR.

TTYL, DNFBOT;

dnfclas · Mar 30, 2016

Hi @LakshanF, I'm your friendly neighborhood .NET Foundation Pull Request Bot (You can call me DNFBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!

The agreement was validated by .NET Foundation and real humans are currently evaluating your PR.

TTYL, DNFBOT;

livarcocc · Mar 31, 2016

livarcocc reviewed Mar 31, 2016

View changes

livarcocc · Mar 31, 2016

src/Microsoft.DotNet.Cli.Utils/Telemetry.cs

+
+        public Telemetry()
+        {
+            if (_isInitialized)


Add the brackets.

livarcocc · Mar 31, 2016

livarcocc reviewed Mar 31, 2016

View changes

livarcocc · Mar 31, 2016

src/Microsoft.DotNet.Cli.Utils/Telemetry.cs

+
+            bool optout = Env.GetEnvironmentVariableAsBool(TelemetryOptout);
+
+            if (optout)


Add brackets.

livarcocc · Mar 31, 2016

livarcocc reviewed Mar 31, 2016

View changes

livarcocc · Mar 31, 2016

src/Microsoft.DotNet.Cli.Utils/Telemetry.cs

+
+        public void TrackCommand(string command, IDictionary<string, string> properties = null, IDictionary<string, double> measurements = null)
+        {
+            if (!_isInitialized)


Add brackets.

livarcocc · Mar 31, 2016

livarcocc reviewed Mar 31, 2016

View changes

livarcocc · Mar 31, 2016

src/Microsoft.DotNet.Cli.Utils/Telemetry.cs

+            Dictionary<string, double> eventMeasurements = new Dictionary<string, double>(_commonMeasurements);
+            if (measurements != null)
+            {
+                foreach (var m in measurements)


change the name of m to measurement. There is no need to use these short variable names here.

livarcocc · Mar 31, 2016

livarcocc reviewed Mar 31, 2016

View changes

livarcocc · Mar 31, 2016

src/Microsoft.DotNet.Cli.Utils/Telemetry.cs

+            {
+                foreach (var m in measurements)
+                {
+                    if (eventMeasurements.ContainsKey(m.Key))


livarcocc · Mar 31, 2016

livarcocc reviewed Mar 31, 2016

View changes

livarcocc · Mar 31, 2016

src/Microsoft.DotNet.Cli.Utils/Telemetry.cs

+            Dictionary<string, string> eventProperties = new Dictionary<string, string>(_commonProperties);
+            if (properties != null)
+            {
+                foreach (var p in properties)


p to properties.

livarcocc · Mar 31, 2016

livarcocc reviewed Mar 31, 2016

View changes

livarcocc · Mar 31, 2016

src/Microsoft.DotNet.Cli.Utils/Telemetry.cs

+            {
+                foreach (var p in properties)
+                {
+                    if (eventProperties.ContainsKey(p.Key))


livarcocc · Mar 31, 2016

livarcocc reviewed Mar 31, 2016

View changes

livarcocc · Mar 31, 2016

src/Microsoft.DotNet.Cli.Utils/Telemetry.cs

+
+namespace Microsoft.DotNet.Cli.Utils
+{
+    public class Telemetry


Can you add an ITelemetry interface with the TrackCommand method and then use that in the dotnet Program? Then you can write a unit test that verifies that TrackCommand is being called with the appropriate parameters.

Done in 4d1895d and ffd1c87

eerhardt · Apr 4, 2016

eerhardt reviewed Apr 4, 2016

View changes

eerhardt · Apr 4, 2016

src/dotnet/Program.cs

@@ -45,7 +44,7 @@ public static int Main(string[] args)

        }

-        private static int ProcessArgs(string[] args)
+        public int ProcessArgs(string[] args, ITelemetry telemetryClient)


Why is making this public and non-static needed?

Looks like it is public so it can be unit tested. How about using InternalsVisibleTo instead?

Added InternalsVisibleTo but still verifying whether I did it correctly.

eerhardt · Apr 4, 2016

eerhardt reviewed Apr 4, 2016

View changes

eerhardt · Apr 4, 2016

src/dotnet/Program.cs

@@ -121,22 +120,39 @@ private static int ProcessArgs(string[] args)
                ["test"] = TestCommand.Run
            };

+            int exitCode;            
+            var arguments = string.Empty;


Is arguments used anymore? It appears it is only set and not read.

correct, it was being passed to TrackEvent but now we're giving that no information on arguments so it's never used. We could delete this code but it's going to need to be re-added shortly after we figure out the correct scrubbing of argument values for PII. I could leave it there and add a comment on the TrackEvent call to this effect or I could just delete this stuff if you prefer.

I'd prefer to just remove it for now. Add it back when it is needed/used.

MichaelSimons · Apr 4, 2016

MichaelSimons reviewed Apr 4, 2016

View changes

MichaelSimons · Apr 4, 2016

src/Microsoft.DotNet.Cli.Utils/Telemetry.cs

+        private static TelemetryClient _client = null;
+
+        private static Dictionary<string, string> _commonProperties = null;
+        private static Dictionary<string, double> _commonMeasurements = null;


I am not seeing the point of having _isInitialized, _client, _commonProperties, and _commonMeasurements be static. The appear to be initialized each time the ctor is called and only referenced by instance members. Seems like this code would be holding references unnecessarily and prevent GC.

Done, I think you're right, will change them to instance variables. The Telemetry class is functionally a singleton anyway so there aren't any savings from the statics.

MichaelSimons · Apr 4, 2016

MichaelSimons reviewed Apr 4, 2016

View changes

MichaelSimons · Apr 4, 2016

src/Microsoft.DotNet.Cli.Utils/Telemetry.cs

+            catch (Exception)
+            {
+                // we dont want to fail the tool if telemetry fais. We should be able to detect abnormalities from data 
+                // at the server end


I made a comment in the previous pull request that I didn't see a response on. "I agree with you that this should not fail and have a negative effect on what the end user is trying to achieve. That being said, do you think it would make sense to Debug.Fail or something like that so that there is a better chance issues will be caught during development in case something is wrong?"

Added a simple log statement with Debug.Fail here and to the other empty catch block

Consider including the originating Exception in the message to make diagnostics easier.

MichaelSimons · Apr 4, 2016

MichaelSimons reviewed Apr 4, 2016

View changes

MichaelSimons · Apr 4, 2016

src/Microsoft.DotNet.Cli.Utils/Telemetry.cs

+
+        private Dictionary<string, string> GetEventProperties(IDictionary<string, string> properties)
+        {
+            Dictionary<string, string> eventProperties = new Dictionary<string, string>(_commonProperties);


Is it necessary to declare the new dictionary in the case when no additional properties are specified? It seems heavy handed to be declaring and coping the commonProperties in this case. If I read the code correctly, this is the only code path that is exercised currently.

Makes sense. Now just returns _commonProperties when properties is null, avoiding the allocation.

Did a force push after rebasing as requested. Letting the CI server do its thing now.

danquirk · Apr 5, 2016

Did a force push after rebasing as requested. Letting the CI server do its thing now.

eerhardt · Apr 5, 2016

eerhardt reviewed Apr 5, 2016

View changes

eerhardt · Apr 5, 2016

src/dotnet/Program.cs

@@ -44,7 +43,7 @@ public static int Main(string[] args)

        }

-        private static int ProcessArgs(string[] args)
+        internal int ProcessArgs(string[] args, ITelemetry telemetryClient)


Does this still need to be instance? I think it should be static.

Will do

Edit: pushed

eerhardt · Apr 5, 2016

eerhardt reviewed Apr 5, 2016

View changes

eerhardt · Apr 5, 2016

src/dotnet/Properties/AssemblyInfo.cs

@@ -0,0 +1,3 @@
+using System.Runtime.CompilerServices;
+
+[assembly: InternalsVisibleTo("dotnet.Tests")]


I think you need a Key here, right?

See https://github.com/dotnet/cli/blob/rel/1.0.0/src/Microsoft.DotNet.Compiler.Common/Properties/Properties.cs#L3

Hm, wasn't sure, trying to run it locally now. I thought you only needed that if the target assembly was strong name signed and sn said this one wasn't. I'm probably wrong though.

Edit: looks like the CI server was happy without it

What's with the Ubuntu x64 Debug Build failure? The results don't make much sense to me.

danquirk · Apr 5, 2016

What's with the Ubuntu x64 Debug Build failure? The results don't make much sense to me.

@dotnet-bot

@dotnet-bot test Ubuntu x64 Debug Build

eerhardt · Apr 5, 2016

@dotnet-bot test Ubuntu x64 Debug Build

Different set of CI failures with the last push =\

danquirk · Apr 5, 2016

Different set of CI failures with the last push =\

@danquirk

@danquirk This commit here should fix the failure you're seeing on windows:

068062f

cherry pick it?

brthor · Apr 5, 2016

@danquirk This commit here should fix the failure you're seeing on windows:

068062f

cherry pick it?

@dotnet-bot

@dotnet-bot test RHEL7.2 x64 Release Build
@dotnet-bot test Windows_NT x86 Debug Build

eerhardt · Apr 5, 2016

@dotnet-bot test RHEL7.2 x64 Release Build
@dotnet-bot test Windows_NT x86 Debug Build

When CI is green.

eerhardt · Apr 5, 2016

When CI is green.

@brthor

@brthor do I still need that? The Windows build didn't fail on my previous push. Looks like you guys are used to some flakiness? Just wanted to make sure we didn't introduce anything new.

danquirk · Apr 5, 2016

@brthor do I still need that? The Windows build didn't fail on my previous push. Looks like you guys are used to some flakiness? Just wanted to make sure we didn't introduce anything new.

@danquirk

@danquirk You don't need it, but it will remove the flakiness. I'll make a separate PR if you don't pick it up here.

brthor · Apr 5, 2016

@danquirk You don't need it, but it will remove the flakiness. I'll make a separate PR if you don't pick it up here.

Ok, I'm gonna wait to see what shakes out with release processes and whether I need to merge some of these PRs together. If I do that's probably a good time to cherry pick that. Otherwise I want to make sure this PR continues to look green for anyone looking to sign off.

danquirk · Apr 6, 2016

Ok, I'm gonna wait to see what shakes out with release processes and whether I need to merge some of these PRs together. If I do that's probably a good time to cherry pick that. Otherwise I want to make sure this PR continues to look green for anyone looking to sign off.

@danquirk

@danquirk - I wouldn't mess around trying to merge them together. I would just merge all 3 PRs at the same time. Trying to cherry pick the changes into 1 PR is prone to errors. We have 3 separate PRs all green. Just go through each one and merge.

I've already signed off, but I will again.

eerhardt · Apr 6, 2016

@danquirk - I wouldn't mess around trying to merge them together. I would just merge all 3 PRs at the same time. Trying to cherry pick the changes into 1 PR is prone to errors. We have 3 separate PRs all green. Just go through each one and merge.

I've already signed off, but I will again.

Yes I definitely agree but it is not up to me :) Waiting to hear exactly how to proceed.

danquirk · Apr 6, 2016

Yes I definitely agree but it is not up to me :) Waiting to hear exactly how to proceed.

I don't like the automatic "opt in" policy for this data sharing. I think you should ask the user with a checkbox in the MSI installer in addition to honoring the env var.

With regard to setting the env var, I just want to make it clear to others who stumble on to this (and it should have been in the Annoncements btw) ... set the value of DOTNET_CLI_TELEMETRY_OPTOUT to any of true, 1, or yes in order to opt-out.

guardrex · Apr 8, 2016

I don't like the automatic "opt in" policy for this data sharing. I think you should ask the user with a checkbox in the MSI installer in addition to honoring the env var.

With regard to setting the env var, I just want to make it clear to others who stumble on to this (and it should have been in the Annoncements btw) ... set the value of DOTNET_CLI_TELEMETRY_OPTOUT to any of true, 1, or yes in order to opt-out.

... and I have to ask: Does this apply to servers where the shared framework is installed? The dotnet command will be run on servers, too.

guardrex · Apr 8, 2016

... and I have to ask: Does this apply to servers where the shared framework is installed? The dotnet command will be run on servers, too.

Very bad idea. This hidden "feature" will scare away users and developers. Must be disabled by default!

ghost · May 18, 2016

Very bad idea. This hidden "feature" will scare away users and developers. Must be disabled by default!

leecow added the in progress label Mar 30, 2016

dnfclas added the cla-already-signed label Mar 30, 2016

Use static member

Loading status checks…

0d067be

eerhardt referenced this pull request Apr 5, 2016
Closed
Telemetry PR #2066

leecow removed the in progress label Apr 6, 2016

TheRealPiotrP deleted the lakshanf/issue2066/telemetry branch Apr 6, 2016

guardrex referenced this pull request Apr 11, 2016
Closed
CLI Win32 Installer Fit and Finish #2400

ghost referenced this pull request May 18, 2016
Closed
.NET core should not SPY on users by default #3093

adamgorMSFT referenced this pull request Jun 13, 2016
Closed
Adding machine ids to telemetry #3494

eerhardt referenced this pull request May 2, 2017
Merged
Prime cache for deb "installer" #6480

svick referenced this pull request May 30, 2018
Closed
Telemetry PR link out of date #5697

dotnet/cli

Join GitHub today

Lakshanf/issue2066/telemetry #2145

Conversation

LakshanF commented Mar 30, 2016

.NET Core Tools Telemetry collection

Overview

Behavior

What do we collect?

Questions, feedback, comments

This comment has been minimized.

dnfclas commented Mar 30, 2016

leecow added the in progress label Mar 30, 2016

dnfclas added the cla-already-signed label Mar 30, 2016

livarcocc reviewed Mar 31, 2016 View changes

This comment has been minimized.

This comment has been minimized.

livarcocc reviewed Mar 31, 2016 View changes

This comment has been minimized.

This comment has been minimized.

livarcocc reviewed Mar 31, 2016 View changes

This comment has been minimized.

This comment has been minimized.

livarcocc reviewed Mar 31, 2016 View changes

This comment has been minimized.

This comment has been minimized.

livarcocc reviewed Mar 31, 2016 View changes

This comment has been minimized.

This comment has been minimized.

livarcocc reviewed Mar 31, 2016 View changes

This comment has been minimized.

This comment has been minimized.

livarcocc reviewed Mar 31, 2016 View changes

This comment has been minimized.

This comment has been minimized.

livarcocc reviewed Mar 31, 2016 View changes

This comment has been minimized.

This comment has been minimized.

eerhardt reviewed Apr 4, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

eerhardt reviewed Apr 4, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

MichaelSimons reviewed Apr 4, 2016 View changes

This comment has been minimized.

This comment has been minimized.

MichaelSimons reviewed Apr 4, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

MichaelSimons reviewed Apr 4, 2016 View changes

This comment has been minimized.

This comment has been minimized.

Lakshan Fernando and others added some commits Mar 25, 2016

This comment has been minimized.

danquirk commented Apr 5, 2016

eerhardt reviewed Apr 5, 2016 View changes

This comment has been minimized.

This comment has been minimized.

eerhardt reviewed Apr 5, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

danquirk commented Apr 5, 2016

This comment has been minimized.

eerhardt commented Apr 5, 2016

This comment has been minimized.

danquirk commented Apr 5, 2016

This comment has been minimized.

brthor commented Apr 5, 2016

This comment has been minimized.

livarcocc reviewed Mar 31, 2016

View changes

livarcocc reviewed Mar 31, 2016

View changes

livarcocc reviewed Mar 31, 2016

View changes

livarcocc reviewed Mar 31, 2016

View changes

livarcocc reviewed Mar 31, 2016

View changes

livarcocc reviewed Mar 31, 2016

View changes

livarcocc reviewed Mar 31, 2016

View changes

livarcocc reviewed Mar 31, 2016

View changes

eerhardt reviewed Apr 4, 2016

View changes

eerhardt reviewed Apr 4, 2016

View changes

MichaelSimons reviewed Apr 4, 2016

View changes

MichaelSimons reviewed Apr 4, 2016

View changes

MichaelSimons reviewed Apr 4, 2016

View changes

eerhardt reviewed Apr 5, 2016

View changes

eerhardt reviewed Apr 5, 2016

View changes