Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go: don't change the libraries in 1.18 #48918

Open
robpike opened this issue Oct 12, 2021 · 45 comments
Open

go: don't change the libraries in 1.18 #48918

robpike opened this issue Oct 12, 2021 · 45 comments

Comments

@robpike
Copy link
Contributor

@robpike robpike commented Oct 12, 2021

The 1.18 release of the Go language is likely to include by far the biggest change to the language since its creation: parametric polymorphism, colloquially called generics. There has been much discussion about how the core libraries will adapt, and how to make that adaptation. See #45955 and #48594 for example, and there are others already and sure to be more soon.

How to use these ideas in the standard library requires great thought and planning. Putting them in the library now also adds a significant burden to rolling out the release.

I propose that we do not update the libraries in 1.18.

The reason is simple and compelling: It's too much to do all at once, and we might get it wrong. The language changes have been worked on in some form for over a decade, but the library changes are very new, and we have no experience with the use of the new types in Go on which to base a strong case for their design. Yes, we can reason about them at length and much has been done. Experience with other languages helps, but one thing Go has taught us is that it grows its own ways of doing things.

For generics, we don't know what those new ways are yet. Also, the compatibility promise makes the cost of getting any detail wrong quite high. We should wait, watch, and learn.

Instead, I propose we still design, build, test, and use new libraries for slices, maps, channels, and so on, but start by putting them in the golang/x/exp repository. That way, these new libraries - which are truly experimental at this stage - can be tested in production, but can be changed, adapted, and grown for a cycle or two, letting the whole community try them out, if they are interested and willing to accept a little instability, without requiring every detail of every component to be ready from day one. Once they have soaked a bit, and updated through experience, we move them into the main repo as we have done with other externally-grown packages, but with the confidence that they work well in practice and are deserving of our compatibility promise.

I realize everyone wants to get their hands on the fun of the new language feature, and is looking forward to fixing some of the issues in the core libraries that will be less clumsy once it arrives, but I strongly believe it is best to take it slow for now. Use, learn, study, and move cautiously.

@ALTree
Copy link
Member

@ALTree ALTree commented Oct 12, 2021

Agree. For example, it looked like we were going to add a few new things like constraints.Slice and constraints.Map, which I found really ugly (but thought were necessary), but now after #48424 it looks like they may be not needed (See #48424 (comment)), which is great! And this happened last week (#48424 was accepted 5 days ago).

To me this suggests we don't yet fully understand how the current generics implementation (and possible refinements that may be implemented in the near future) would influence the design of these new libraries; so it may be premature to commit to an API for them. Putting them in x/exp seems like a prudent move.

@ALTree ALTree added this to the Go1.18 milestone Oct 12, 2021
@carlmjohnson
Copy link
Contributor

@carlmjohnson carlmjohnson commented Oct 12, 2021

The slices package has been whittled down to a very functional core. For example, it has no Map or Filter because those weren’t obviously useful. I think it’s good to ship as is. Dropping it now just adds delay for no real reason.

Similarly, if constraints isn’t part of 1.18, there will be a lot of independent redefinitions of orderable types. I don’t think we’re going to learn from experience much that could change that package now.

I agree though that container/set can wait for another cycle (iteration is still up in the air), and probably many or all other changes.

@mvdan
Copy link
Member

@mvdan mvdan commented Oct 12, 2021

This reminds me of the transition done to error values via https://golang.org/x/xerrors, which I think worked really well.

Just one question: would these x packages allow backwards incompatible changes? If so, I think putting in x/exp and clearly documenting them as such would be fine. If they'll follow backwards compatibility guarantees, much like x/xerrors did, then perhaps I'd place them under a new module such as x/generics or x/typeparams.

I'd personally lean towards a new module with backwards compatibility guarantees. Otherwise, it would be risky for Go modules to start using and relying on them, as they may break at any moment. Especially when it comes to libraries, as they can't "pin" a specific version due to MVS. If we learn from past mistakes and want to remove or redesign APIs, we should simply deprecate, and do the backwards incompatible changes in bulk when moving the APIs to the standard library.

@jaloren
Copy link

@jaloren jaloren commented Oct 12, 2021

I concur with this proposal. More concretely, I doubt we should be adding generic libraries or updating existing ones until we have an answer to #48287. And getting an answer to that discussion likely requires time and experimentation from the community .

The one exception may be the constraints library, which strike me as an essential part of the generics design and would severely degrade developer ergonomics without it.

Having x/generics module with backwards compat guarantees sounds like an excellent idea.

@mvdan
Copy link
Member

@mvdan mvdan commented Oct 12, 2021

Another advantage of an external module is that, if a proposed new API gets accepted and implemented, users on Go 1.18 or later can start relying on it the moment it hits the master branch, instead of waiting 3+ months until the next Go release is out with its standard library changes.

@andig
Copy link
Contributor

@andig andig commented Oct 12, 2021

How to use these ideas in the standard library requires great thought and planning. Putting them in the library now also adds a significant burden to rolling out the release.
I propose that we do not update the libraries in 1.18.

I understood rob doesn't want to change the existing libraries. It seems a good idea to introduce new ones and potentially NOT promise to keep them or their api for one or two releases. That will gain more real-world feedback.

@thepudds
Copy link

@thepudds thepudds commented Oct 12, 2021

For example, it looked like we were going to add a few new things like constraints.Slice and constraints.Map, which I found really ugly (but thought were necessary), but now after #48424 it looks like they may be not needed (See #48424 (comment)), which is great! And this happened last week (#48424 was accepted 5 days ago).

Minor: that proposal is in "likely accept", and not yet accepted, though there is a prototype.

Separately, if the new packages do start at x/… (x/generics, x/exp or something else), hopefully they could be tagged to clearly communicate to tools and to humans any backwards compatibility intentions that apply at that import path location (e.g., perhaps something like v0.1.0 ~soon, and v1.0.0 when go1.18 is released, with an option to move to v2 at the x/… location if warranted, which helps with library importers and so on).

When those capabilities land in the standard library, or if there is a v1 to v2 change at x/…, ideally migration for clients could be handled via cmd/go’s hopefully-not-too-distant automatic API migration (initially described here and more recently discussed here).

EDIT: FWIW, I think everything I wrote above is cleaner if the repo is not x/exp, but instead new separate x/something repo(s).

@doggedOwl
Copy link

@doggedOwl doggedOwl commented Oct 12, 2021

Why does it have to be a blanket decision? There are some proposed changes that might need some more work and for those it's reasonable to be in x/* but IMO there are some proposals like the slices package who are reasonably well defined and would be very helpful to have since the start.

@mvdan
Copy link
Member

@mvdan mvdan commented Oct 12, 2021

@doggedOwl one way to look at it is that Go 1.18 is already a huge release, with generics in the language and fuzzing support. And drawing the line around what generic APIs are fully ready to be frozen is hard, especially when we have had close to no experience with them being used in the wild.

@bitfield
Copy link

@bitfield bitfield commented Oct 12, 2021

I made the following comment on #45955, which I think supports Rob's point:

The first step in designing any API... should be to write a number of non-trivial programs using it; ideally, programs that solve real problems. The right design will tend to emerge from looking at those programs (or, at worst, it will save you a few iterations on the design).
I suggest that we find, or try to come up with, some realistic programs involving slice operations, and see what they would look like using the various proposed APIs in the slices library. This is likely to make their advantages and disadvantages much clearer to the casual reader.

Unfortunately, responders to that issue were not able, or not willing, to supply any such programs (so far as I'm aware). I think the best way to get them is to put something in the x tree, as suggested, and let people use it for a while.

The kind of programs I'm talking about are not merely examples cooked up to demonstrate how a certain feature would work. They are demonstrations that such a feature is necessary, widely useful, and worth the cost in additional complexity—a quality bar that has traditionally applied to all Go proposals, not just this one. I see no reason it shouldn't apply here.

Not to labour the point, but it's always surprising to me how long and carefully I can think about some design a priori, and then the moment I actually try to use it, I find it's instantly obvious that I've designed the wrong thing. That's not just me being dumb (though of course that's a factor): it's simply inevitable.

Releasing a draft API in 'experimental' mode is exactly the right thing to do here. When it becomes clear to us all that that API needs to change in several important respects (I'm happy to put money on this outcome if anyone seriously doubts it), it will still be possible to make those changes. The moment it goes into the standard library, that door is closed forever.

Measure twice, cut once.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Oct 12, 2021

@carlmjohnson I agree that container/set should not go into 1.18. I just edited the discussion (#47331) to say as much.

I am also inclined to say that we should not add type parameters to any existing standard library packages for 1.18, especially given #48287.

But I do personally think that we need the constraints package at least in 1.18. Many uses of type parameters are going to need the constraints package. I see no advantage to using an x repo for constraints, which will force us to retain a pair of packages for some time going forward.

So I think that for 1.18 this proposal is really about the slices and maps packages. Just my own opinion, of course.

@neild
Copy link
Contributor

@neild neild commented Oct 12, 2021

Putting the proposed Go 1.13 error changes in golang.org/x/exp/xerrors was very useful for validating the API before committing to it.

However, applying the compatibility guarantee to the x/exp package made it impossible to iterate on the API. The eventual changes to the errors package in Go 1.13 differed from the xerrors prototype in various ways, but xerrors only permitted validating that initial draft. I think this was a mistake.

Testing out new designs in x/exp or some other non-std repo is an excellent idea, but next time let's do it in a way that allows us to iterate on the design, whether that means frequent versioning (set/v1, set/v2, etc.) or accepting that incompatible changes will be made to the experimental package to keep it in sync with the eventual design.

@zxysilent
Copy link

@zxysilent zxysilent commented Oct 12, 2021

Agree. For example,golang.org/x/sync.syncmap => sync.Map and golang.org/x/net/context => context

@carlmjohnson
Copy link
Contributor

@carlmjohnson carlmjohnson commented Oct 12, 2021

Not to labour the point, but it's always surprising to me how long and carefully I can think about some design a priori, and then the moment I actually try to use it, I find it's instantly obvious that I've designed the wrong thing. That's not just me being dumb (though of course that's a factor): it's simply inevitable.

I agree in general, but in the case of slices, the package contains a lot of functions that I and others have already written piecemeal for particular uses, but now it's the generic versions of them. Is the real world really going to change how we feel about those that much when they go from type specific to generic? I have my doubts.

@nevkontakte
Copy link
Contributor

@nevkontakte nevkontakte commented Oct 12, 2021

With my GopherJS maintainer hat on, I would prefer for generics not to appear in the standard library (even in non-public APIs) for one release cycle. If we imagine, for the sake of argument, that generics are used in fmt or io package (which are used by pretty much every Go program) every tool that traverses your package tree and does linting/analysis will need to support generics immediately upon Go 1.18 release, or become broken for everyone. Introducing generics in a new x/... package will make adoption curve smoother and give more time for the ecosystem to catch up. Each user would be able to make a choice to either adopt generics immediately and cope with some tool breakage, or be more conservative and wait for tooling updates first.

Arguably there is a period between an RC and a stable release where APIs are supposed to be mostly stable, but for a feature as complex as generics this may not be enough time for tool developers, especially those working in their spare time.

I suspect that my concern would be shared by most tooling authors who do static checks and linting of various kinds and need to be able to understand the type system well.

@prattmic
Copy link
Member

@prattmic prattmic commented Oct 12, 2021

@nevkontakte it is not a complete solution by any means, but I want to note, in case you haven't seen it, that #48790 is tracking addition of an analyzer to detect use of generics so that tools that aren't yet ready to handle generics may be able to gracefully degrade instead of completely failing.

@borud
Copy link

@borud borud commented Oct 12, 2021

A lot of people look to the standard library for learning idiomatic use of Go. It might be a good idea to think about where one might point people for idiomatic use of parametric polymorphism in code that is....well, used for something. (As opposed to just examples.

@v3n
Copy link

@v3n v3n commented Oct 12, 2021

Another advantage of an external module is that, if a proposed new API gets accepted and implemented, users on Go 1.18 or later can start relying on it the moment it hits the master branch, instead of waiting 3+ months until the next Go release is out with its standard library changes.

100% this. I'm eagerly waiting for generics since it will solve many problems we have, but without standard library we're either committing to a partial rewrite in 3 months when we have the official packages or deferring integration for another 3 months.

@carlmjohnson
Copy link
Contributor

@carlmjohnson carlmjohnson commented Oct 13, 2021

Putting the proposed Go 1.13 error changes in golang.org/x/exp/xerrors was very useful for validating the API before committing to it.

However, applying the compatibility guarantee to the x/exp package made it impossible to iterate on the API. The eventual changes to the errors package in Go 1.13 differed from the xerrors prototype in various ways, but xerrors only permitted validating that initial draft. I think this was a mistake.

I think one important difference between now and then is that Go modules exist and are a standard part of Go (as opposed to the various dependency managers that existed then), so we can just be explicit that this is v0, breaking changes will happen, and if you need stability, either pin a version or just don't use it until it is added to the standard library.

@bigwhite
Copy link

@bigwhite bigwhite commented Oct 13, 2021

Agree! Let's delay the decision making and see what will happen after adding the generics.

@robpike robpike closed this Oct 13, 2021
@crcms
Copy link

@crcms crcms commented Oct 13, 2021

I agree with the genericity, if there is an internal nested, it may be hell, Giving a certain time slowly introducing will be a better choice

@hherman1
Copy link

@hherman1 hherman1 commented Oct 13, 2021

Why was this closed?

@JakubOboza
Copy link

@JakubOboza JakubOboza commented Oct 13, 2021

Most sane approach to new big feature i saw in a long time. I ❤️ the approach of delay stdlib changes, wait and learn.

@mvdan
Copy link
Member

@mvdan mvdan commented Oct 13, 2021

Perhaps this issue was closed by mistake. I'm reopening for now.

@mvdan mvdan reopened this Oct 13, 2021
@robpike
Copy link
Contributor Author

@robpike robpike commented Oct 13, 2021

Yes, it was a mistake, sorry. I was trying to close a different issue and got toomanywindowsbefuddled.

@nieaowei
Copy link

@nieaowei nieaowei commented Oct 13, 2021

If don't change the libraries, what is the significance of 1.18 release? If none of the standard libraries use generics, who wants to use it at risk? And it will generate a lot of third-party libraries.

@DmitriyMV
Copy link

@DmitriyMV DmitriyMV commented Oct 13, 2021

If don't change the libraries, what is the significance of 1.18 release? If none of the standard libraries use generics, who wants to use it at risk? And it will generate a lot of third-party libraries.

It will give to polish it in official "exp" repository before committing to backward-compatibility promises as struct at they are in standard library.

@changkun
Copy link
Contributor

@changkun changkun commented Oct 13, 2021

If the goal of this proposal is to learn more experiences of the usage of type parameters so that we don't make any mistakes in a final release. The easiest intermediate solution of letting the release date of Go 1.18 be delayed.
The delay really just means a delay of the final release, maybe a better wording would be "extend the development cycle" or "reschedule the release date".

As Go 1.1 was released almost a year after Go 1. If Go 1.18 is also a significant change to the language and std library, say Go 2, then Go 1.18 may also be released a year or more after Go 1.17 to gain more experience.

For users who would like to try generics, there may be pre-releases like go1.18-alpha1, go1.18-alpha2, .. go1.18-beta1, ..., go1.18rc1, ... then finally go1.18 until we all are happy.

In this way, we can have those changes together without making mistakes. Wonderful, isn't it?

@mvdan
Copy link
Member

@mvdan mvdan commented Oct 13, 2021

@changkun I don't think delaying 1.18 really helps here; we'd still be getting no feedback from real users of the new generic standard library APIs until they would need to be frozen.

@mvdan
Copy link
Member

@mvdan mvdan commented Oct 13, 2021

allows us to iterate on the design, whether that means frequent versioning (set/v1, set/v2, etc.) or accepting that incompatible changes will be made

I think backwards compatibility is important if we want larger projects to test these APIs properly. Especially for stable libraries, adding dependencies to unstable packages is dangerous, because your library can break at any point via MVS updates.

I was initially envisioning a module like x/generics being bumped to x/generics/v2 if needed, but I think per-package modules like x/set/v2 might work even better - then you can major-version them separately without having to bump everything at once.

@Merovius
Copy link

@Merovius Merovius commented Oct 13, 2021

@nieaowei

If don't change the libraries, what is the significance of 1.18 release?

The language would still support generics, so you'd still be able to write generic code. There are also other changes in go 1.18, of course.

If none of the standard libraries use generics, who wants to use it at risk? And it will generate a lot of third-party libraries.

Presumably, we would release the same libraries, we just wouldn't do it in the stdlib, but in x/. So, there would still be the same set of "blessed" first-party implementation of those, we just would reserve the ability to break them before putting them into the stdlib, if experimentation shows the APIs to be deficient.

@changkun
Copy link
Contributor

@changkun changkun commented Oct 13, 2021

no feedback from real users of the new generic standard library APIs until they would need to be frozen.

Maybe? Maybe not? If there is no feedback from real users, how did the Go 1 was released? Before Go 1 nothing was frozen.

@aarzilli
Copy link
Contributor

@aarzilli aarzilli commented Oct 13, 2021

Maybe? Maybe not? If there is no feedback from real users, how did the Go 1 was released? Before Go 1 nothing was frozen.

Go was public for three years before 1 was released and available internally at google for longer than that.

@changkun
Copy link
Contributor

@changkun changkun commented Oct 13, 2021

Go was public for three years before 1 was released and available internally at google for longer than that.

And? It is a weak argument against Go 1.18 alpha and beta versions may even longer than that.

@Merovius
Copy link

@Merovius Merovius commented Oct 13, 2021

@changkun Generics are not the only thing that is changing for Go 1.18 and it isn't the only change planned over the next years. Many people work on Go, but don't work on generics. The regular release schedule is also determining their work cadence and planning. So I don't think keeping Go 1.18 back for significant amounts of times is at all viable.

@changkun
Copy link
Contributor

@changkun changkun commented Oct 13, 2021

Many people work on Go, but don't work on generics.

That's a very good argument. But does GOEXPERIMENT cover by the compatibility promise? What will happen if the generics feature is not activated by default for years? For example, generics APIs and packages are only useable when GOEXPERIMENT=usegenerics is presented? They still release along with the releases but are only activated by default and finalized when everyone is happy.

@DmitriyMV
Copy link

@DmitriyMV DmitriyMV commented Oct 13, 2021

I think this worked well for golang.org/x/xerrors. With modules it can work even better.

What will happen if the generics feature is not activated by default for years.

There is no reason to disallow language feature just because we don't know if proposed API's will work as we think/expect. Its actually a good reason to move them to /x subrepo and experiment there, since /x is offical enough to be visible to most user, and can break compatibility promises.

GOEXPERIMENT

That's for language features.

@bitfield
Copy link

@bitfield bitfield commented Oct 14, 2021

I agree in general [that real-world use often generates improvements to a design], but in the case of slices, the package contains a lot of functions that I and others have already written piecemeal for particular uses.

Dijkstra reminds us (in an essay aptly called The Humble Programmer):

The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility.

In other words, however smart and capable one or two individual Go programmers might be, they can't necessarily anticipate what kind of programs a million Gophers will want to write, or how they'll write them. Even the beautiful, pragmatic designs of the standard library are occasionally found to be wanting in some minor respect. I think Dijkstra would have approved of Go, because a kind of humility is built into it. We all need reminding of the value of humility once in a while (he said, humbly referencing his own blog post).

@hopehook
Copy link

@hopehook hopehook commented Oct 14, 2021

So many people are looking forward to the feature generics in golang, just be more cautious, and fully validated to better meet the expectations of users.

I personally recommend adding this feature to version 2.x of golang, but not now.

@Sivyer9303
Copy link

@Sivyer9303 Sivyer9303 commented Oct 15, 2021

Agree.It's better to put them into exp package before it's ripe.

@frederikhors
Copy link

@frederikhors frederikhors commented Oct 15, 2021

I agree. Please proceed with caution, Go is great, let's go slow but far!

@MetalRex101
Copy link

@MetalRex101 MetalRex101 commented Oct 15, 2021

I believe the whole go community will start immediately build new or adopt existing libraries (with new major versions) since 1.18 released. I believe it will produce ton of issues, useful and not. And it will show the real problems existing with current generics implementation (if exists). It’s impossible to build so difficult basic functionality from the first try and no real feedback. So in my opinion generics should go with 1.18 as a separate package. Quality over quantity.

@nahwinrajan
Copy link

@nahwinrajan nahwinrajan commented Oct 15, 2021

Agree.

GO is a carefully designed modern language. we take things for granted in GO but it comes from careful thought, plans and execution.

@carlmjohnson
Copy link
Contributor

@carlmjohnson carlmjohnson commented Oct 16, 2021

This issue was put onto social media and the Golang Weekly newsletter, so I think it's getting more feedback than usual. Just as a reminder, if you agree with an argument already made, just thumbs up the person who made the argument before. There's no need to repeat an argument unless you're adding some new point in favor of it. Thanks.

@SekiBetu
Copy link

@SekiBetu SekiBetu commented Oct 17, 2021

There is an old Chinese saying: "you can't eat hot tofu in a hurry"
take your time guys~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet