[gitpod] enable telemetry for testing #257

jeanp413 · 2021-12-10T13:32:59Z

This PR fixes #

Logs can be found in ~/.vscode-remote/data/logs/<date>/telemetry.log

akosyakov · 2021-12-13T07:26:09Z

@loujaybee with this PR you can see different telemetry events instrumented by MS, could you have a look please what can be interesting for us? We can add also additional instrumentation if needed.

To investigate:

https://gitpod.io#https://github.com/gitpod-io/openvscode-server/pull/257
await till another window on port location 9888 opens with with instrumented dev version, there can be multiple windows opened recursively, just close them
in this window do F1 -> Developer: Open Window Log File (Session)... -> Select current session and then telemetry.log
now you see all telemetry collected
try to perform some interesting actions and see what events are produced

jeanp413 · 2021-12-14T04:09:44Z

Updated steps as I did more changes:

https://gitpod.io#https://github.com/gitpod-io/openvscode-server/pull/257
await till another window on port location 9888 opens with with instrumented dev version, there can be multiple windows opened recursively, just close them
in this window do F1 -> Developer: Set Log Level -> Select trace
Open bottom Panel and click the Output tab, in the dropdown select Log (telemetry)
try to perform some interesting actions and see what events are produced

package.json

src/vs/gitpod/browser/workbench/workbench.ts

src/vs/server/remoteExtensionHostAgentServer.ts

build/.moduleignore

akosyakov · 2021-12-15T05:42:24Z

src/vs/gitpod/node/gitpodInsightsAppender.ts

+	}
+
+	log(eventName: string, data?: any): void {
+		data = mixin(data, this._defaultData);


It will race with preparing api?

src/vs/gitpod/node/gitpodInsightsAppender.ts

loujaybee · 2021-12-15T07:00:11Z

src/vs/server/remoteExtensionHostAgentServer.ts

@@ -290,11 +293,13 @@ export class RemoteExtensionHostAgentServer extends Disposable {

 		let appInsightsAppender: ITelemetryAppender = NullAppender;
 		if (supportsTelemetry(this._productService, this._environmentService)) {
-			if (this._productService.aiConfig && this._productService.aiConfig.asimovKey) {
+			if (this._productService.aiConfig && this._productService.aiConfig.asimovKey !== 'foo') {


Is this left over, or genuine code? 😆

It's intended, I could remove that block of code, but this way I think it would create less merge conflicts

loujaybee · 2021-12-15T07:10:59Z

Wow, this is awesome! Thanks so much @jeanp413 😍

It would be good to get this foundational PR merged, which implements the "framework" for capturing these events, and an easy way to map these events into the tracking event structure (the structure is decided on a case-by-case basis) we send to Segment, which is implemented in analytics.ts [1].

I'd argue that this PR should only put in place that "foundational" code, and then we can specify the exact events we need to track, and what format we need to store them (for our purposes) on the other GitHub issues. e.g. In other words, we add the specifics of which telemetry events to track (and how to map them to the segment structure) on each analytics ticket (gitpod-io/gitpod#6897, gitpod-io/gitpod#6895, gitpod-io/gitpod#6981)

However, for reference, some of events that look useful right now:

telemetry/activityBarAction (id: workbench.view.explorer) - Open file explorer (side bar)
telemetry/filePUT - User has inputted on a file
telemetry/activityBarAction (viewletId: workbench.view.search) - Open file search (side bar)
telemetry/activatePlugin - Installed extension
telemetry/workbenchActionExecuted (id: workbench.files.openFile) - Click to open file
telemetry/fileGet - File request is made
telemetry/workbenchActionExecuted (id: workbench.action.terminal.clear) - Clear terminal output (even with shortcut)

A few events I couldn't trigger yet:

Anything really related to git, I could see some git-triggered events, but nothing super useful
The terminal doesn't really trigger any events it seems (apart from "clear" and/or open new terminal, etc)
I couldn't figure out how to initiate a guide/walkthrough and/or pop-up notifications (yet)

A couple of challenges:

We will need to experiment over time to understand what initiates these events (in some cases it could be multiple sources, which could make our analytics results inaccurate)
We'll need to throttle/debounce some events otherwise we'll get flooded with unnecessary data

Thought:

When we do implement the events, it would be nice if we have a simple collection of maps, that map between these telemetry events, and the analytics ones. e.g.

const maps = [
    (/* The teletry event */) => {
       return // The analytics event
    }, 
    (/* The teletry event */) => {
       return // The analytics event
    }
]

So that it's super easy to add new events (read: I could do it without having to bother you guys 😁 )

akosyakov · 2021-12-15T14:13:05Z

src/vs/gitpod/node/gitpodInsightsAppender.ts

+		data = mixin(data, this._defaultData);
+		data = validateTelemetryData(data);
+
+		if (eventName !== 'workbenchActionExecuted') {


@jeanp413 let's start but tracking first read and write interactions, so we don't produce many data and addressing gitpod-io/gitpod#6981

We can have following event:

{ eventName: 'vscode_file_access`, properties: { // first `editorOpened` or `fileGet` fires `read` // first `filePut` fires `write` kind: 'read' | 'write', // from supervisor info endpoint workspaceId: string, // from supervisor info endpoint workspaceInstanceId: string, // from MS telemetry event sessionId: string, // from MS telemetry event timestamp: Date } }

It is important that we care about only first to avoid spamming segment.

We need also kind of a function which does conversion from MS telemetry event to our event similar how @loujaybee is suggested in #257 (comment) It should help later adding more events for other team members.

It will be all for the scope of PR. After that we can work one by one with concrete issues.

I added the logic to send just the first events but they are misleading as they are triggered when the getting started page or the simple-browser is shown

Is it because editorOpened or fileGet?
Maybe we can filter down editorOpened for files with file scheme?

Will take a look but not sure if that is gonna be reliable enough. either way I think this PR is done and we can polish this event in following PRs related to each gitpod telemetry issue

I ended up creating separate event for editorOpened and filtering by 'workbench.editors.files.fileEditorInput'
fileGet gets fired automatically by the gitpod-web extension when reading the gitpod.yml file.
If this needs more tweaking lets address them in following PRs

yeah it sounds like fileGet and filePut more track fs access by any source (i.e. api and user triggered), we should not rely on it.

so should I delete them? and just send editorOpened?

extensions/gitpod-web/src/extension.ts

akosyakov · 2021-12-15T15:20:46Z

@jeanp413 check that by enabling telemetry for Gitpod we don't send anything to MS even from extensions like built-in for instance git or 3rd party like GHPR extensions.

src/vs/workbench/api/common/extHost.api.impl.ts

jeanp413 · 2021-12-16T23:56:05Z

I'll go ahead and merge this

akosyakov · 2021-12-17T07:27:50Z

src/vs/gitpod/node/gitpodInsightsAppender.ts

+				super(address, protocols, {
+					headers: {
+						'Origin': new URL(gitpodHost).origin,
+						'Authorization': `Bearer ${serverToken}`


@jeanp413 we need to add here User-Agent at least. It should be a requirement for all our clients to Gitpod Server

akosyakov · 2021-12-17T07:34:51Z

src/vs/gitpod/node/gitpodInsightsAppender.ts

+			}
+			filterEvent.add(eventName);
+			return {
+				event: `vscode_${formatEventName(eventName)}`,


it should be vscode_file_access with read kind.

I think it would be better to wait for approval with this PR.

We should be careful with adding new events. Because we need to track them in these doc and make sure that they make sense for @loujaybee and @jakobhero: https://www.notion.so/gitpod/dd900d5177944fc38d22bb8c7d3d2cce?v=54cdae8ff0f54b4d8235429ab9893e49

I see, sorry about that, do you want me to revert it? or just push the changes directly?

It should be enough to make a new PR which properly collects vscode_file_access. And merge it only after approval this time.

jeanp413 requested a review from akosyakov December 10, 2021 13:38

jeanp413 force-pushed the gp-code/main branch 2 times, most recently from 0b66d21 to 4fe2a9f Compare December 11, 2021 09:59

jeanp413 force-pushed the jp/testing-telemetry branch from 2f6c762 to 5efd331 Compare December 11, 2021 20:28

akosyakov changed the title ~~Enable telemetry for testing~~ [gitpod] enable telemetry for testing Dec 13, 2021

akosyakov reviewed Dec 14, 2021

View reviewed changes

package.json Outdated Show resolved Hide resolved

akosyakov reviewed Dec 14, 2021

View reviewed changes

src/vs/gitpod/browser/workbench/workbench.ts Show resolved Hide resolved

akosyakov reviewed Dec 14, 2021

View reviewed changes

src/vs/server/remoteExtensionHostAgentServer.ts Show resolved Hide resolved

This was referenced Dec 14, 2021

Implement analytics on file opening events in VS Code gitpod-io/gitpod#6981

Closed

Implement analytics on generic VS Code pop-up / toast notifications gitpod-io/gitpod#6895

Closed

jeanp413 force-pushed the gp-code/main branch from 4fe2a9f to c53ef1f Compare December 15, 2021 00:27

akosyakov force-pushed the gp-code/main branch from c53ef1f to 5107052 Compare December 15, 2021 00:38

jeanp413 force-pushed the jp/testing-telemetry branch from b8b3838 to ccf3e6a Compare December 15, 2021 00:44

akosyakov reviewed Dec 15, 2021

View reviewed changes

build/.moduleignore Show resolved Hide resolved

akosyakov reviewed Dec 15, 2021

View reviewed changes

src/vs/gitpod/node/gitpodInsightsAppender.ts Outdated Show resolved Hide resolved

loujaybee reviewed Dec 15, 2021

View reviewed changes

akosyakov reviewed Dec 15, 2021

View reviewed changes

extensions/gitpod-web/src/extension.ts Outdated Show resolved Hide resolved

akosyakov force-pushed the gp-code/main branch from 5107052 to fbd9385 Compare December 15, 2021 23:09

jeanp413 force-pushed the jp/testing-telemetry branch from 8f56e45 to 9392504 Compare December 16, 2021 02:12

jeanp413 commented Dec 16, 2021

View reviewed changes

src/vs/workbench/api/common/extHost.api.impl.ts Show resolved Hide resolved

jeanp413 force-pushed the jp/testing-telemetry branch from 0ba1637 to caa452b Compare December 16, 2021 15:46

akosyakov force-pushed the gp-code/main branch from 57e5f82 to 4877cf3 Compare December 16, 2021 23:08

Enable gitpod telemetry

3714d21

jeanp413 force-pushed the jp/testing-telemetry branch from 211a663 to 3714d21 Compare December 16, 2021 23:58

jeanp413 self-assigned this Dec 17, 2021

jeanp413 merged commit 97df074 into gp-code/main Dec 17, 2021

jeanp413 deleted the jp/testing-telemetry branch December 17, 2021 00:02

akosyakov reviewed Dec 17, 2021

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[gitpod] enable telemetry for testing #257

[gitpod] enable telemetry for testing #257

jeanp413 commented Dec 10, 2021 •

edited

Loading

akosyakov commented Dec 13, 2021 •

edited

Loading

jeanp413 commented Dec 14, 2021 •

edited

Loading

akosyakov Dec 15, 2021

loujaybee Dec 15, 2021

jeanp413 Dec 15, 2021

loujaybee commented Dec 15, 2021

akosyakov Dec 15, 2021 •

edited

Loading

jeanp413 Dec 16, 2021 •

edited

Loading

akosyakov Dec 16, 2021

jeanp413 Dec 16, 2021

jeanp413 Dec 16, 2021

akosyakov Dec 17, 2021

jeanp413 Dec 17, 2021 •

edited

Loading

akosyakov commented Dec 15, 2021

jeanp413 commented Dec 16, 2021

akosyakov Dec 17, 2021 •

edited

Loading

akosyakov Dec 17, 2021 •

edited

Loading

jeanp413 Dec 17, 2021

akosyakov Dec 17, 2021

[gitpod] enable telemetry for testing #257

[gitpod] enable telemetry for testing #257

Conversation

jeanp413 commented Dec 10, 2021 • edited Loading

akosyakov commented Dec 13, 2021 • edited Loading

jeanp413 commented Dec 14, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

loujaybee commented Dec 15, 2021

akosyakov Dec 15, 2021 • edited Loading

Choose a reason for hiding this comment

jeanp413 Dec 16, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeanp413 Dec 17, 2021 • edited Loading

Choose a reason for hiding this comment

akosyakov commented Dec 15, 2021

jeanp413 commented Dec 16, 2021

akosyakov Dec 17, 2021 • edited Loading

Choose a reason for hiding this comment

akosyakov Dec 17, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jeanp413 commented Dec 10, 2021 •

edited

Loading

akosyakov commented Dec 13, 2021 •

edited

Loading

jeanp413 commented Dec 14, 2021 •

edited

Loading

akosyakov Dec 15, 2021 •

edited

Loading

jeanp413 Dec 16, 2021 •

edited

Loading

jeanp413 Dec 17, 2021 •

edited

Loading

akosyakov Dec 17, 2021 •

edited

Loading

akosyakov Dec 17, 2021 •

edited

Loading