telemetry in the Go toolchain

[rsc]

Feb 10 5pm US Eastern: Thanks for the lively discussion everyone. It has given me plenty to think about. We are starting to go in circles, and actually new comments are increasingly rare, so I've locked this to give everyone their weekend back. Thanks again.

If anyone feels like there's something important that wasn't said in these 506 comments, please feel free to email me at rsc@golang.org.

Feb 26 The current plan is to use some form of opt-in to collect telemetry.

---

How do software developers understand which parts of their software are being used and whether they are performing as expected? The modern answer is telemetry, which means software sending data to answer those questions back to a collection server.

I believe that open-source software projects need to explore new telemetry designs that help developers get the information they need to work efficiently and effectively, without collecting invasive traces of detailed user activity.

I have written a short series of blog posts about one such design, which I call transparent telemetry, because it collects as little as possible (kilobytes per year from each installation) and then publishes every bit that it collects, for public inspection and analysis.

I’d like to explore using transparent telemetry, or a system like it, in the Go toolchain, which I hope will help Go project developers and users alike. To be clear, I am only suggesting that the instrumentation be added to the Go command-line tools written and distributed by the Go team, such as the go command, the Go compiler, gopls, and govulncheck. I am not suggesting that instrumentation be added by the Go compiler to all Go programs in the world: that’s clearly inappropriate.

Transparent telemetry has the following key properties:

• The decisions about what metrics to collect are made in an open, public process.

• The collection configuration is automatically generated from the actively tracked metrics: no data is collected that isn’t needed for the metrics.

• The collection configuration is served using a tamper-evident transparent log, making it very difficult to serve different collection configurations to different systems.

• The collection configuration is a cacheable, proxied Go module, so any privacy-enhancing local Go proxy already in use for ordinary modules will automatically be used for collection configuration. To further ameliorate concerns about tracking systems by the downloading of the collection configuration, each installation only bothers downloading the configuration each week with probability 10%, so that each installation only asks for the configuration about five times per year.

• Uploaded reports only include total event counts over a full week, not any kind of time-ordered event trace.

• Uploaded reports do not include user IDs, machine IDs, or any other kind of ID.

• Uploaded reports only contain strings that are already known to the collection server: counter names, program names, and version strings repeated from the collection configuration, along with the names of functions in specific, unmodified Go toolchain programs for stack traces. The only types of non-string data in the reports are event counts, dates, and line numbers.

• IP addresses exposed by the HTTP session that uploads the report are not recorded with the reports.

• Thanks to sampling, only a constant number of uploaded reports are needed to achieve a specific accuracy target, no matter how many installations exist. Specifically, only about 16,000 reports are needed for 1% accuracy at a 99% confidence level. This means that as new systems are added to the system, each system reports less often. With a conservative estimate of two million Go installations, about 16,000 reporting each week corresponds to an overall reporting rate of well under 2% per week, meaning each installation would upload a report on average less than once per year.

• The aggregate computed metrics are made public in graphical and tabular form.

• The full raw data as collected is made public, so that project maintainers have no proprietary advantage or insights in their role as the direct data collector.

• The system is on by default, but opting out is easy, effective, and persistent.

Please read the introductory blog post for more background before commenting here. You may also want to check out the detailed design and list of use cases.

[Stolas]

Should be off by default. There is no reason for a development chain to have any kind of telemetry on by default.
Or, as I do understand the need for it to be on by default, it should check what the user wants on first start.

One reason it should be off by default would be cases such as Jenkins farms, Bazel use, and other locations that arnt your regular development environment.

[w3bb]

I think that lowering the barrier to opt-in and share data is a better solution than increasing the barrier to opt out. People who would decline telemetry at a prompt are people who don't want telemetry, making the life of those people harder is really bad whether it's intentional or not.

[padawan]

As soon as there is just one identifiable data collected, opt-out becomes completely illegal WRT GDPR, which applies to any entity processing a European user data regardless where it is collected. Add that the privacy shield is illegal now in Europe, you better watch out for the huge fine if that ships with opt-out to any European user.

This is a legal issue, your personal opinion on the goods and bads of opt-in vs opt-out is irrelevant.

[doggedOwl]

Transparent and "On by default" do not go together. Naming It transparent is like the 1984 moto "war is peace".
It is often said here that just because this is discussed and can be checked publicly is enough and noone should be taken offguard.

“There’s no point in acting surprised about it. All the planning charts and demolition orders have been on display at your local planning department in Alpha Centauri for 50 of your Earth years, so you’ve had plenty of time to lodge any formal complaint and it’s far too late to start making a fuss about it now.... I’m sorry, but if you can’t be bothered to take an interest in local affairs, that’s your own lookout. "

• Vogon Transparency.

[gophun]

I wrote a little about this at https://research.swtch.com/telemetry-design#opt-out.
[...]
In systems that have collection off by default, opt-in rates tend to be very low,

Are there any concrete percentages available from projects who did this before?

You say that about 16,000 samples are needed. There are more than two million Go users. So you would need about 1% of users to opt-in. Is this really that difficult to achieve?

skewing the results toward power users who understand the system well.

One does not have to understand the system well to answer a yes/no question in the installer or before starting a download from the Go website.

This leaves those who install Go via the package manager of their Linux distribution. However, Linux users are more likely to be power users who understand how to set an environment variable to opt-in. And many (most?) Linux distributions would disable telemetry by default anyway.

Second, the existence of an opt-in checkbox is in my opinion too often used as justification for collecting far more data than is necessary.

It's in the hands of the Go project to not use it as a justification for that. Furthermore, it's a question of how attractive you make it to opt-in. More users will opt-in if you can convince them they will benefit. It's a communication and marketing task, just as you need to promote participation in the annual survey.

Also, because the design collects a fixed number of samples, more systems being opted in means collecting less from any given system, reducing the privacy impact to each individual system.

Those who opt-in probably don't care about the collection frequency as long as it doesn't bog down their systems.

[tigrannajaryan]

@rsc I think this is an interesting idea. As an open-source developer (OpenTelemetry) I agree that we need an ethical way to collect usage telemetry about our own products. I will run this idea inside OpenTelemetry project to see what our community thinks.

If you are interested in using an existing metric reporting library it may be helpful to take a look at OpenTelemetry-Go. It supports histograms and can send collected metrics to the backend using a standard OTLP protocol (Protobuf or JSON payloads over HTTP).

[rsc]

It would be interesting if there's an opportunity to collaborate here, although my understanding of OpenTelemetry is that it's architected around an event stream rather than the very coarse week-long counter aggregation in the design. Avoiding an event stream is one of the important ways transparent telemetry reduces its privacy impact.

[tigrannajaryan]

although my understanding of OpenTelemetry is that it's architected around an event stream rather than the very coarse week-long counter aggregation in the design

I think you are right. We can likely make OpenTelemetry Go SDK do what you need but I think it is not worth the effort. Rolling out your own based on the design you posted will likely be simpler.

It would be interesting if there's an opportunity to collaborate here

There probably is. I wonder if your proposal can be generalized to allow any open-source project (and not just to the Go toolchain) to ethically collect usage telemetry. OpenTelemetry may be a good home for this.

[mzhang28]

Even if not using the official OpenTelemetry, it would be nice to respect some the environment variables that people have configured for other tools that use OpenTelemetry, in terms of opt-in / opt-out.

https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/sdk-environment-variables.md

[rsc]

That's quite a few environment variables. Which specifically are you suggesting it look for, and how should it interpret them? I'm not sure I understand why it would make sense to look for OpenTelemetry env variables without using OpenTelemetry. Do other systems do that?

[seankhliao]

Despite sharing the name "telemetry", I don't think this proposal intersects too much with how the OpenTelemetry SDKs or config (env vars) work at this point in time.

[MostAwesomeDude]

What would prevent users from sending garbage to the telemetry server?

[mvdan]

we might add targeted telemetry

I can imagine this bit being misunderstood or read out of context. This targeted telemetry would still be under the constraints of the design: only infrequent sampling of event counters with no sensitive information. I imagine that one example of targetting might be something like: many users on Windows say that their Go builds are slow, so we want to get more counters for build cache hits and misses from Windows users. Perhaps @rsc has other examples to help illustrate.

[MostAwesomeDude]

@svrnm and @lthibault correctly understand how distributed systems work; in particular, they understand that public data-collecting endpoints must accept garbage, and also that the public likes to put trash into such endpoints.

@rsc: The point of the question is not to earn a good answer, but to remind you that the question represents a problem, a fundamental obstacle to your plan and desires. I recommend modifying your desires, rather than trying to administer computers which you do not own. Also, when you say "if a particular statistic is wrong", you are presuming that you can tell whether your data represents a biased sample, which implies that you already have a prior distribution and do not actually need this data collection; or you are trusting priors which do not actually correspond to any real-world data.

The attack that sneaks through your plan is called "boiling the frog" or 'salami-slicing". An attacker would let you set up your system as planned and operate it for several months. During that time, the attacker notes the same trends that you note, including anything from likely IP ranges to expected OS and platform. Then, the attacker slowly starts creating fake reporters, using a variety of cloud services for IP diversity, and programming the reporters to reproduce your existing data distribution through random sampling. After the attacker is confident that they haven't been noticed, they start to systematically bias their reports:

• Find a vulnerability in a popular package? Chaff it out by making other packages look more popular or making it look like people are adopting the patched version.
• Has an entire distro opted out of telemetry? Then the attacker can hallucinate an entire community on that distro, including any weird practices that they might have, like shipping out-of-date packages from the distro's repos.
• Speaking of hallucinations, how does one know that a stack frame is valid? It sure would be awful to receive a bug report generated from crash telemetry which indicates that a non-crashing line of code is crashy.
• It's funny that developers use Macbooks. So, let's just artificially inflate the apparent number of Macbooks.
• It's also funny that new microarchitectures are introduced. So, let's slow down how quickly the Go team prioritizes new microarchitectures by making them appear relatively unpopular and slow to be adopted. This interacts in a positive way with sampling bias!

I read through all four of your posts (the three on telemetry and the one on sampling) and somehow you never used the word "bias". However, even mere opt-out/opt-in creates sampling bias! Please, at the very least, make this consideration part of your design; you should show how your system can fail, not just how it can succeed.

[lthibault]

if we realized Go was misbehaving in a certain way

Also, this isn't quite the scenario we were discussing. We were instead discussing the scenario in which users tried to subvert telemetry. In our example, Go is working just fine.

The difference is important because it goes to show how opt-out telemetry inevitably produces a user-hostile game of cat-and-mouse.

[rsc]

@MostAwesomeDude Taking your argument to its logical conclusion, it seems like the most effective defense against such an attack would be some kind of proof-of-work system that would make it difficult for an attacker to overwhelm the honest participants. The most obvious approch would be to give up on sampling and collect from 100% of the systems, perhaps every day. In that system the "work" is the network traffic and reporting behavior, making it far more costly for attackers to overwhelm the reporting systems. Of course, that would be a very different system, and not one I want to build or use. My point is only that if you focus exclusively on attack mitigation strategies, you end up in a place that most people commenting here probably wouldn't like at all.

[lthibault]

@rsc I don't think this is taking @MostAwesomeDude's argument to its logical conclusion at all. It is rather missing the point by proposing a technical solution that ratchets up the user-hostility another notch; users must now perform work unless they opt out! 😄

To be clear, I get that you are making this proposal in jest, but it goes to show that you've missed the point.

[sulewicz]

I don't think that an opt-out mechanism is compliant with GDPR. Was the proposal reviewed by any Compliance experts?

Personally, I don't like the idea of automatic collection of telemetry pushed on me. Isn't Google tracking enough information about me already?

[padawan]

Also you are moving from an ad hominem to a false comparison naming once again my employer. It becomes hard to believe that this is a simple 'miscommunication'. I gave you facts, please respond on them without attacking or threatening me.

[TedTed]

@padawan I was trying to provide a concrete example that you could go check on. Saying that consent is the only legal way to collect IP addresses under GDPR is plain wrong. Implying that the only exception is mandated traffic logging is also wrong. Legitimate interest can be a legal basis to collect IP addresses, as has been confirmed by the courts already for the case of IP collection for security reasons, or to fight piracy.

@sulewicz I don't know if other legal bases could apply to this case. I'm not saying that they could. I was only saying that this legal determination is certainly not as trivial as has been suggested by multiple people on this thread, and that "GDPR means that you need consent!" is a gross simplification: it's at best not at all clear (I'm not aware of a court decision for a telemetry use case like this proposal), and at worst completely wrong.

[doggedOwl]

@TedTed while you are right that IPs can be collected for legitimate interest without an explicit consent,
Legitimate Interest is clearly defined as "neccessary for the core functionality of the application".
That Argument could be made for the Proxy and sum because it provides core security/redudancy of dependecies but Phoning home for any metric is clearly not a core functionality of a compiler/build tool.

For a case regarding IPs collected without the user consent see the Case of Google Fonts CDN where it was judged that using the CDN in any application requires user consent becuase it provides the IP of the user to the the CDN.

[TedTed]

Legitimate Interest is clearly defined as "neccessary for the core functionality of the application".

I'm not familiar with the GDPR article or DPA interpretation that makes such claims, can you tell more? Article 6.4 says that there are a number of points to take into consideration when making a legitimate interest determination, and this determination is far more subtle and nuanced than "is it part of the core functionality". Data minimization, risks to data subjects, use of technical safeguards, and data sensitivity are all taken into account in this determination. Recital 47 explicitly says that fraud detection or direct marketing can be used to regarded as legitimate interest, and those don't seem to me like "part of the core functionality". Of course neither is directly relevant to this telemetry proposal, but they show that this determination is complex and subtle.

Again, I'm not saying "it would be perfectly fine to use legitimate interest for this use case". This would be wrong. I'm saying that it's complicated, that law is messy, and that absolute statements about compliance are not accurate nor helpful.

Thanks for bringing up the Google Fonts ruling. I agree that it's very relevant, especially since the decision was about a website disclosing IP addresses to Google Fonts, regardless of what Google actually did with these IP addresses. Elsewhere in this thread, @ianlancetaylor says:

No personal data of any sort is collected, no anonymization is required

As the Google Fonts ruling suggests, the compliance story is likely not as simple as that — even without logging IP addresses, the system still relies on processing those in order to receive telemetry reports.

[doggedOwl]

https://commission.europa.eu/law/law-topic/data-protection/reform/rules-business-and-organisations/legal-grounds-processing-data/grounds-processing/what-does-grounds-legitimate-interest-mean_en

Data minimization, risks to data subjects, use of technical safeguards, and data sensitivity are all taken into account in this determination. Recital 47 explicitly says that fraud detection or direct marketing can be used to regarded as legitimate interest

Yes but those are considered legitimate interest only when you "need to process personal data in order to carry out tasks related to your business activities". The compiler's and build tool's activities require nothing of the sort.
GDPR is very hard to interpret correctly but several lawyers have suggested that legitimate interest applies only when a functionality (necessary for the work of the application) can not be reasonably implemented without collecting of PII.
That is why you don't see everyone abusing the "direct marketing legitimate need" to get out of GDPR consent requirements. If the main purpose of you app is direct marketing, sure, collecting PII is legitimate interest. If Direct marketing or Usage statistics are side dishes that help but not essential to the functionality of the offer they need to be disclosed and consented for.

Fraud detection can not apply because without phoning home there is no Fraud to protect From.

I bring the very hard to interpret issue because here in Europe all consulting lawyers are very conservative about these things and even though Google may have the bandwidth to litigate such issues, the workplaces who use go and can't, for any reason, be 100% sure that will be opted out in every case, will need to disclose it and gather consent "just to be safe".
Requiring this consent will be much less work than the headache of GDPR compliance, and this throwing of another burden may very well tip the balance of a company deciding to use Go or not.

[adamsanghera]

I don't feel like I have a stake in this debate, but I want to provide a "steel" man for the counter position.

I think the principal issue for people who are averse to default-opt-in telemetry is that the policy of what data is sent over that connection is mutable. Even if the debates over which telemetry data to publish is public, once we establish the pattern of telemetry, the policy of what is sent through the pipes is subject to change.

Put another way, the logic of "never telemetry" goes something like: if we never have a network call to begin with, we don't have to know the latest version of a policy that's wholly irrelevant to our day-to-day work, but essential to our personal privacy.

I have full faith in the current humans of the Go team to write policy and compose processes that are fair and respect peoples' privacy. However, it's important to recognize that this move opens a hitherto nonexistent door for less-than-scrupulous data collection, and that future members of the Go team (or future versions of ourselves) might not hold privacy to be so sacrosanct.

---

In summary, there's a trade-off to be had here. The proposed metrics to collect would be invaluable to ongoing development of the toolchain. And, to be clear, the technical solution proposed appears to be a sound basis for preserving privacy.

The quandary is whether this value is worth creating space to (eventually/perhaps unwittingly) emit private information?

I think one can argue either side rationally, but it has to be said that making the system unobtrusively opt-in would largely resolve the ethical concerns by introducing a mechanism for informed consent. Even if a default-opt-out solution provides 1/10th the value of opt-in telemetry, it may be worth considering for that reason alone.

[lthibault]

This is true, but it's also true that it's almost as easy to change Go to send that bad telemetry today.

I don't understand this point @rsc. It seems like you have re-articulated the main argument for "no telemetry, ever".

I don't want to read into intent too much, but I am getting the distinct impression that the decision to ship telemetry has already been made, and that we are being asked to rubber stamp it. I sincerely hope that is not the case.

[adamsanghera]

I appreciate your point, which I would summarize to "if Go is sending good telemetry, then it's easy to change it to send bad telemetry".

Yep! I'd say this is an accurate summary.

This is true, but it's also true that it's almost as easy to change Go to send that bad telemetry today.
[...]
To start sending "bad telemetry" would require code changes in the telemetry library itself to send it, as well as server changes to accept it.
[...]
Perhaps having "good telemetry" makes the "bad telemetry" door just a little bit easier to open, but in both cases I expect the open-source community to stand in the way.

Unfortunately, the "no man's land" between "good" and "bad" telemetry is large and its boundaries are fuzzy. I believe the open source community would be able to effectively stand in the way of a direct, unequivocal step from "good" to "bad".

However, I think the primary fear is that the OSS community is not an effective guard rail against a dozen or so tiny steps in a "bad" direction, especially when those steps are each correlated with improving the utility of the collected data. There will be a strong incentive for well-meaning people to propose steps in a "bad" direction.

In a default-opt-out system, the OSS community would have a single, explicit line between consensual and non-consensual telemetry. They can gather their pitchforks around any proposal to change to it.

As long as that consent line stands, the upper-bound on the integral of "harm to the non-consenting user over time" is zero. Zero is a powerful number!

[pierreprinetti]

I can't recall a single situation where the open source community said "no" and Google heeded that opinion. AMP happened. And then AMP4Email happened.

I think it's a mistake to see all of Google as a monolithic entity; "Google" comprises are different groups with different interests and different motives.

When you publicly discuss work from a work platform in your working hours, you represent your company.

[padawan]

All the companies that collect data from me ask if I authorise them to do so, including Google (which is making efforts, e.g. recently they have started to respect GDPR in Maps, YT, they don't show up before asking me to first authorise them).

If this simply opts (pun intended) for opt-in by default, then all my concerns are moot since this becomes legal according to the laws protecting a European user.

[khm]

"The open source community" is a vague and nebulous concept. There was a lot of pushback against Audacity a while ago, but the forks that were predicted to kill off Audacity didn't really go anywhere last I checked, and Audacity essentially just continued as before without much effect. So one must wonder: was there really that much of a pushback from "the open source community" or was it a loud minority of users (some of whom may not even be Audacity users)? You need to be very careful having these kind of discussions in open participation forums like GitHub.

The forks didn't go anywhere because Muse removed the telemetry and left it with optional crash reports and optional update checking. It did not just continue as before; the default telemetry was removed.

Furthermore, I have enough trust in various core developers to "do the right thing", or at least to not "be evil".

I don't, but this isn't about whether or not I have warm feelings toward random Google employees. The current proposal happens to be about Go, but the introductory blog post being titled "Transparent Telemetry for Open Source Projects" instead of "Transparent Telemetry for Go" is probably not accidental. The pushback here is not specific to Go; a larger discussion is happening around whether the "open source community," vague definition notwithstanding, wants baked-in telemetry to be a part of the culture.

It's obvious this stuff is going to get put into the Go programming language. It would look bad for Russ' proposal to get shot down, the Go governance team does not have a history of incorporating external feedback, and half of the Go maintainers are already too emotionally attached to the idea to even consider backtracking now. That's fine; it's Google's sandbox, we're all just playing in it.

The question is whether this behavior convinces people that it's ok to bake telemetry into open-source projects. Previous attempts have been shouted down, but previous attempts were not bankrolled by a corporate behemoth. Google has sufficient influence to swing lots of opinions. Any objection has to be loud or it will not even be noticed. I personally don't want to live in the world where open source projects are rife with baked-in telemetry, as I regard it a technological shortcut: an attempt to bypass the proper community relationship building and trust-fostering that would enable necessary feedback to happen organically. "It's okay, we don't need to talk to anyone, we'll just have their computers report on them for us" is not a healthy community feedback channel, especially with an opt-out flag to ensure it's just a sampling bias factory.

But to bring this back to Go, I don't think "I trust the Go devteam" is a long-term assurance. Google, as diverse a conglomeration of interests and motivations as it might be, is still a faceless megacorporation who could can the whole team at any time, for no reason whatsoever, or even for bad reasons. The fact that the Go dev team has taken zero steps to divest itself of utter reliance on Google dollars and Google services is proof only that the dev team trusts their managers. I do not share that trust, and I think the team would be well served to examine the long-term consequences of both that dependence and large-scale information-gathering exercises which submit the collected data to corporate ownership. It's not that hard to set up an independent body to hold the data in escrow. Hell, a public-benefit corporation dedicated to being a telemetry ombudsman might even be a good business to run. But that's not even being considered here; all the governance is just being taken for granted, and that indicates a lack of depth of consideration beyond the mere technical implementation of the tools.

Content warning for this paragraph: offensive terminology reference
Google has a history of unilateral decisions that have unintended consequences. Consider this document: https://support.google.com/maps/answer/1725632?hl=en Google's position is simple. If you don't want us to geolocate your wifi host, reconfigure your network name to "_nomap." The desires of the network owner, local regulations, and corporate compliance are irrelevant to Google; change your SSID or we will gather your data. We can briefly set aside the hubris of "everyone else in the world has to change but us," to focus on the actual implementation. When this policy was foisted upon the world ten or more years ago, it was merely obnoxious. However, we've had ten or more years of internet trolling since then, and now, unrelated to Google or their wifi data, some trolls have convinced parts of the public that "nomap" is short for "non-offending minor-attracted person" -- i.e., a pedophile. Searching the term on most major search engines brings up this disgusting definition and very little about Google's wifi geolocation. So, here we are, in 2023, and Google still has a policy document instructing people who value their privacy to brand their wifi with an abhorrent label.
Content warning section ends

Obviously nobody deliberately set about to create this situation. Someone at Google just decided to do a thing, and nobody ever revisited that decision. So, when someone at Google sets about to do this thing, they might want to put more thought into what the long-term consequences might be, whether it might be wise to revisit the decision some years down the road, what potential for abuse it might have, and so forth. It's not promising that people who are already asking about abuse vectors are being told that the proposed software is just way too clever to be gamed, even before the software has actually been implemented.

In short, a lot of words to say "there is a lot more to worry about here than Go, but we should worry about Go too."

[bobg]

The controversy here is whether the proposed telemetry should default on or default off. But there's at least one other option: default advertise. When you have neither opted in nor opted out, the Go toolchain could remind you, "There's this telemetry option we'd like you to turn on."

In fact here's another option: time-limited opt-in. "I agree to send telemetry until 31 Dec 2023." That should help some users over the natural resistance barrier to opting in.

[artkiver]

I am a volunteer and I am trying to prevent duplicate discussions. Raising the same point many times will not get you better answers, just more pages of text to sift through. You can vote or thumbs-up comments if you agree with points already made.

I respectfully disagree with the point that any censorship is happening. I didn't even hide a comment here; I just politely asked. And even with hidden comments, you can still read them easily.

It seems as if once a comment has been hidden, or marked as off topic or marked as spam, it is not possible to "vote or thumbs-up comments" so that perspective seems disingenuous, to understate it.

[nemith]

[rsc]

The plan would be that this server (which does not exist yet) would be completely open source, running on Google Cloud, same as pkg.go.dev and go.dev.

(When we did proxy.golang.org, we made the decision to use internal Google infrastructure for key parts, and that unfortunately means that the code is not possible to open source in a meaningful way. It's not worth revisiting that at the moment.)

[rsc]

If you are talking about the specific ideals of having an open-source server, yes, telemetry would use an open-source server.

[saschmit]

for people that don't trust this, how could they verify that the server isn't running a modified build? (or something completely different?)

[rsc]

There is no way to verify what the server is running. That is why so much of the filtering and aggregation is done on the client, which you can verify. Literally the only thing the server needs to do is republish the uploaded record (which you can check) and forget the IP address that uploaded it (which you can't). Literally the only thing you must trust is that the server runs the code we say it does, so that one time per year that the server gets an upload from your system, it discards your IP address. If you don't trust that, it is of course easy to set GOTELEMETRY=off.

[deefdragon]

Edit: I missed the use cases link at the bottom. (I feel one or two examples should have been included in the original post, and left the link as an addendum. the other two blog posts are otherwise sufficiently summarized)

I think my biggest question is; Why?

I'm a fan of looking at pretty graphs, but what I want to know is what purpose does the telemetry serve beyond just pretty graphs? What benifits will the language potentially receive from the collection of these metrics?

Does the team have a concrete example of something that needs the data to act on, or that would have benifited from this data?

[fzipp]

Did you read the list of use cases?

[DeedleFake]

This was answered in the links at the bottom of the initial post, most notably the list of use cases.

[str4d]

The introductory blog post linked in the OP gives multiple concrete examples. To make a sweeping generalisation, the examples basically boil down to "things that the users of Go tools either didn't realise were bugs, or relied on as features", neither of which will motivate the users to open bug reports unprompted (and in the case of bugs, there isn't any way to prompt the user because the bugs aren't intentional).

[DeedleFake]

I'm particularly interested in the numbers for cgo usage. I try to avoid cgo wherever I can, even going so far as to attempt to rewrite wlroots in as pure Go as I could get it, but its usage is just inevitable in an annoying number of cases. I think the discouragement from using it and the awkwardness of getting the garbage collector to play nicely with C has contributed a lot to the lack of, for example, good GUI support in Go, among other things.

Also, if you're going to consider phasing out Fortran support, it might be interesting to see Rust get supported by cgo. Being able to build a mixed Go/Rust system using just go build would be a fascinating plot twist.

[rsc]

I don't believe we'd remove Fortran support outright but it would be interesting to know whether it merits any new attention at all.

[ocdtrekkie]

I have resisted Go for years because it comes from Google. I've started to work with it a little bit, because I've been convinced over time by friends who love building with Go that it really isn't that Googley, and it's just Google funds this thing the Go creators wanted to do their way. The GOPROXY issues were a big scare point, but hey, you can shut that off, I guess. Now I feel just a little bit sold out by my friends who told me Go was okay to invest time in.

Now you guys want to introduce telemetry into your programming language? This is how you drive off any person who even considered giving your project a chance despite the warning signs. Please don't do this, and please issue a public apology for even proposing it. Please leave a blast radius around this idea wide enough that nobody even suggests trying to do this again.

Trust in Google's behavior is at an all time low, and moves like this are a choice to shove what's left of it off the edge of a cliff.

Thank you.

[rsc]

@bronger GOPROXY does not exist to collect data about you, nor does it actually collect data about you. It provides important functionality. I wrote above in #58409 (reply in thread):

For example, Go ships with use of the Go module mirror (proxy.golang.org) enabled by default, so that users get more reliable builds out of the box. Similarly, Go ships with the use of the checksum database also enabled by default, so that users get verified module downloads out of the box. We know that most users don't want to and probably won't spend time reconfiguring the system: they trust us to set it up right instead. Of course, that implies a responsibility to actually look out for users' best interests, and we take that very seriously. There are important privacy concerns about the module mirror and about the checksum database, despite their clear benefits, so we designed those systems to address as many of those concerns as possible. Among the decisions we made to improve privacy there: (1) GOPROXY can proxy both the module mirror and the checksum database, (2) we published a very clear privacy policy (proxy.golang.org/privacy), (3) we introduced the concept of a tiled transparency log to keep log fetches from exposing a potential tracking signal.

You are welcome to turn it off, but understand what you are giving up. Also note that if you set GOPROXY=direct, the go command still uses the checksum database to protect against supply chain attacks. If you really want the go command not to use servers, you also need to set GOSUMDB=off.

[zenhack]

As someone who specifically set GOPROXY=direct in my .bashrc when the feature was announced, I am annoyed to find there is a second variable that needs setting, that I am only just learning about. Opt-out is problematic enough, but a proliferation of variables that need to be set separately for each feature that talks to Google's servers makes it increasingly difficult for the large swath of people who do not want this to be confident that they have successfully disabled it.

If the Go team cannot be convinced to make these features opt-in, I would at least suggest creating a single variable that will shut all of these features off, with a commitment that this will apply to any new features that talk to Google's systems as well. Users should not have to read the release notes to determine whether they need to change how they go about opting out.

[rsc]

The use of the proxy and the checksum database were introduced in the same Go release, and we've been clear to mention both of them from the start, including in the release notes.

[lthibault]

The use of the proxy and the checksum database were introduced in the same Go release, and we've been clear to mention both of them from the start, including in the release notes.

That ... doesn't address the point 😕

[mvdan]

The user was saying these environment variables proliferated, as if they appeared over a period of time without properly announcing them to Go users. There are only two such variables, and they were added and announced at the same time. If a user only knew about one but not the other, it's possible that they heard about GOPROXY from another website or post which was incomplete, not from Go's own release notes, announcements, or documentation.

[noonien]

Quick question. You claim in the article that even though the IP is exposed because of the TCP connection, it is not logged in the reports.

What happens if someone decides to spam the telemetry service with bogus data? Will the data be kept? How will you identify which data came from the spammer's address so that it can be removed?

[rsc]

It remains to be seen what will be necessary to combat potential abuse. I expect that similar to https://proxy.golang.org/privacy there would be the possibility of separate service logs used for running the system that are kept for not more than 30 days. That's fairly standard for running a system. That information would not make its way into the analysis side of the system.

[noonien]

Does this not negate the claim of "This IP address would not be associated with the uploaded reports in any way."?

Even if the IP address and/or association to the IP address to the other service logs are only kept for 30 days (as you claim, I'm guessing laws could impact this retention period, but IANAL), they are still being kept, even if they are "separate" from the reports, they can, and probably will still be used to associate reports to IPs.

Because of this, I don't think it is fair to make the above claim.