proposal: Go 2: add mechanism to remove API as a function of caller's go.mod "go 1.xx" version

[bradfitz]

When reviewing a number of the Go2 issues, we keep finding that we're basically unable to clean up past mistakes.

On the language side, the go.mod file's "go 1.xxx" declaration lets users declare their expected Go language semantics, which lets us remove Go language features over time, but we have nothing equivalent for removing standard library symbols, short of simply making new major versions of all packages, which gets contagiously invasive very quickly with the dependencies between the std packages. For redesigns, new major versions works, but it doesn't work well for cleanups.

It would be nice to have a mechanism that's similar (but different) to +build go1.x build tags, but are added implicitly by the caller module module's expected language version.

That'd let a package have different exports depending on who was using it.

Rather than propose an exact solution, this bug is more generally about tracking how (and whether) we can remove things from a package over time.

/cc @ianlancetaylor @griesemer @bcmills

[bcmills]

I don't think this is a problem that actually needs to be solved for package-level identifiers (types, constants, and functions). For those, we still have to provide an implementation anyway, and given that we have that implementation, we may as well just add a // Deprecated comment, and make lint and godoc warn everyone not to use it.

The interesting problem, then, is how to remove methods, because they aren't necessarily visible at the point where they are used.

Suppose I have:

package mine

type Thing struct{ […] }

// Deprecated: use NewThing instead.
func (t *Thing) OldThing() {}

func (t *Thing) NewThing() {}

package mediator

import (
	[…]
)

func […] {
	// This looks like it avoids the deprecated OldThing method...
	var t theirs.NewThinger = new(mine.Thing)
	theirs.Oops(t)
}

package theirs

type NewThinger interface { NewThing() }

func Oops(t NewThinger) {
	if x, ok := t.(interface { OldThing() }); ok {
		// ...but we would need whole-program analysis
		// to realize that OldThing is still in use.
		x.OldThing()
	} else {
		t.NewThing()
	}
}

[bcmills]

I have some ideas about how to address this, but I don't have the time to write them up this week. 😕

[ianlancetaylor]

@bcmills Any ideas on mechanism? Thanks.

[ianlancetaylor]

If we use our standard fallback of a magic comment, then it could look like:

//go:lang < 1.14
type ClientConn struct {
}

This means that ClientConn is normally exported, but it can not be seen when compiling at Go language level 1.14 or higher. This can also apply to struct fields.

type Request struct {
    ...
    //go:lang < 1.14
    Cancel <-chan struct{}
    ...
}

The type Request is visible at all language levels, but it can not be seen at language level 1.14 or higher. (The struct field would still exist at run time and would still show up when using the reflect package.)

The magic comment could be used with package scope types, consts, variables, functions, and methods, and also with fields of struct types and methods of interface types.

It could be used to cover an entire set of types, consts, or variables.

//go:lang < 1.14
const (
    SEEK_SET int = 0
    SEEK_CUR int = 1
    ...
)

In this example SEEK_SET and SEEK_CUR (and any other constants in the group) can not be seen at language level 1.14 or above.

[bcmills]

Yes, that's the thing that works for everything but methods — but it's also mostly redundant with // Deprecated comments.

The problem is, what happens if you do a type-assertion that checks for the removed method in a pre-1.14 module, and then pass the value as an interface{} to some other package that has since been upgraded? When the second package does the same type-assertion it will presumably fail, unless that method has somehow been marked as contaminated by the pre-1.14 package.

[bradfitz]

but it's also mostly redundant with // Deprecated comments.

Deprecated comments have no machine-readable structure, though. Parsing English sentences sounds super gross.

[bcmills]

Why is machine-readable structure important? Either way, the content is merely informational: it's nearly trivial to construct a “backdated” forwarding module to access the removed definitions.

[bradfitz]

Either way, the content is merely informational

That's not true. The whole point of this bug is to remove access to symbols from people. It's not to write English at them. We can already write English at them in comments.

[ianlancetaylor]

The problem is, what happens if you do a type-assertion that checks for the removed method in a pre-1.14 module, and then pass the value as an interface{} to some other package that has since been upgraded? When the second package does the same type-assertion it will presumably fail, unless that method has somehow been marked as contaminated by the pre-1.14 package.

The suggestion is that, if T.M is marked as removed in 1.14, then direct references to T.M will fail. And storing a value of type T in an interface that requires method M will fail. However, storing a value of type T in an interface, and then asserting to a different interface that checks for M will succeed. It may be that this inconsistency is enough to decide against this proposal, or at least to decide against permitting it to be used for methods, but I don't think there is any other plausible approach.

[bcmills]

The whole point of this bug is to remove access to symbols from people. It's not to write English at them.

The go directive doesn't tell us when the code was written, only which version it was written against. Given that, it seems like there is a fundamental tradeoff: we must either make breaking changes (presumably following a deprecation period), or use a mechanism that can be trivially defeated by a backdated intermediary.

Given that we don't want to take the breaking-change approach, I don't see why the defeat mechanism of “ignoring the linter” is all that much worse than “using a backdated trampoline module” — and the comment-and-lint approach has the advantage of already being in wide use.

[bcmills]

Thinking about the linting approach some more: we do have one interesting option, at least. We run vet for tests, but not lint — so perhaps we could adopt some sort of convention that promotes use of a // Deprecated: identifier from a mere lint warning to a stronger vet warning.

For example:
// Deprecated (go 1.14): […]

could be a lint warning if used in a pre-1.14 module, but a vet warning if used in a module with a later go version.

[bradfitz]

The go directive doesn't tell us when the code was written, only which version it was written against.

Yes, but that's fine. We wouldn't do this until modules were on by default for enough time that most the community has go.mod files with a go directive. So future symbol hiding wouldn't affect existing code; only new code, but new code not compiling isn't really a breaking change if it never worked in the first place.

For example:
// Deprecated (go 1.14): […]

We're talking about the language version, not the Go version, though, which that comment doesn't make clear, if that was your intention. That's why the directive Ian wrote above said //go:lang.

[bcmills]

So future symbol hiding wouldn't affect existing code; only new code, but new code not compiling isn't really a breaking change if it never worked in the first place.

Right; my point is that it also wouldn't affect new code written with a backdated go directive. (As long as the older language version continues to work, users can continue to write against it — and they can use packages written against it to defeat whatever symbol-removal we implement. In that sense, it's no more effective than a vet check, albeit a bit more annoying to hack around.)

[bradfitz]

Right; my point is that it also wouldn't affect new code written with a backdated go directive.

That's fine. That's the whole point, actually.

This mechanism says: "You want to to play with the new toys? (e.g. generics) Then first stop using the old deprecated stuff. You can't have both. Choose."

It's not about finding some fool-proof mechanism to stop people from accessing the symbols. (If so, we'd just delete them.) It's a casual nudge, and a way to let us remove godoc clutter. (godoc would stop showing these things, at least on golang.org's default view.)

[ianlancetaylor]

Given that we don't want to take the breaking-change approach

The point of this proposal is to permit breaking changes, without actually breaking existing code. It's adapting the same mechanism that we've decided to use to permit breaking changes in the language.

[bcmills]

This mechanism says: "You want to to play with the new toys? (e.g. generics) Then first stop using the old deprecated stuff. You can't have both. Choose."

Right. The problem is that — unlike language changes — the API check doesn't actually make them choose. A library cannot provide a language feature by definition, but it can easily provide access to a deprecated/removed part of a regular library API.

If it's just a casual nudge either way, then it really seems like a refinement of the comment convention is the way to go — compiler support isn't needed. We can have vet warn about usage, and have go doc (and godoc.org and wherever else) hide the hard-deprecated parts, and so on, without actually removing the symbols at compile time.

[ianlancetaylor]

In order to have vet warn about usage of names that we want to remove, we need a machine parseable comment that clearly says the language versions that should and should not support the name. I don't think we can just use our loosely specified "Deprecated" convention for this. We could use go:lang as suggested above, or something else. Any other suggestions?

[dmitshur]

The go:lang examples above use exported fields/types. I think it'd be helpful to include their documentation to get a better sense of how the go:lang directive would fit in. Does it go on top or below the package comment? To use the httputil.ClientConn type as the example, would it be as follows?

// ClientConn is an artifact of Go's early HTTP implementation.
// It is low-level, old, and unused by Go's current HTTP stack.
// We should have deleted it before Go 1.
//
// Deprecated: Use Client or Transport in package net/http instead.
//go:lang < 1.14
type ClientConn struct { ... }

Or would the //go:lang < 1.14 go on top of the documentation?

The above is not compatible with the current godoc; it doesn't know to not include the //go directive in the documentation. But there is some precedent for this format for unexported identifiers, e.g., see here.

All of the go:lang examples above used < and a Go language version x.y. Is < included because there are expected to be other variants, or is it just for clarity? If the latter...

I don't think we can just use our loosely specified "Deprecated" convention for this.

I agree we can't just use it as it's currently defined:

To signal that an identifier should not be used, add a paragraph to its doc comment that begins with Deprecated: followed by some information about the deprecation, and a recommendation on what to use instead, if applicable.

But perhaps we can use it if it's modified. For example, the equivalent of //go:lang < 1.14 can be expressed as "Deprecated in 1.14: <human readable text>". A full example with ClientConn:

// ClientConn is an artifact of Go's early HTTP implementation.
// It is low-level, old, and unused by Go's current HTTP stack.
// We should have deleted it before Go 1.
//
// Deprecated in 1.14: Use Client or Transport in package net/http instead.
type ClientConn struct { ... }

There are existing tools that parse the current "Deprecated:" convention, so I suspect parsing "Deprecated in x.y:" may also be viable. Having just one place that to tell both humans and machines that an identifier is deprecated has the advantage that they can't fall out of sync.

/cc @FiloSottile @dominikh

Edit: Something I realized just now is that this proposed convention is about removing an identifier, something that may happen after it's been deprecated, not at the same time. An identifier may be deprecated in 1.8, and removed in 1.14. In that case, "Deprecated in 1.14:" phrasing won't work.

Maybe "Deprecated; removed in 1.14:" then. But that's not as clean.