Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/link: lock down future uses of linkname #67401

Open
rsc opened this issue May 15, 2024 · 94 comments
Open

cmd/link: lock down future uses of linkname #67401

rsc opened this issue May 15, 2024 · 94 comments
Assignees
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Milestone

Comments

@rsc
Copy link
Contributor

rsc commented May 15, 2024

Overuse of //go:linkname to reach into Go standard library internals (especially runtime internals) means that when we do change the standard library internals in ways that should not matter, we can end up breaking packages that are depended on by a large swath of the Go ecosystem. For example, https://go.dev/cl/583756 broke github.com/goccy/go-json because it turns out that package copied most of the runtime's internal type API. Now we can't change anything in that list, despite that being an ostensibly internal package, without breaking goccy/go-json. And goccy is used by many packages, including Kubernetes.

This situation is unsustainable. Internals are internal for a reason. We can't keep Go programs working when they create explicit dependencies on details that we have kept internal. But we also care a lot about compatibility: we don't want to break Go programs either. The obvious conclusion is that we have to stop Go programs from being able to create these dependencies on internal details in the first place.

This issue tracks work to prevent new //go:linkname-based dependencies and contain existing ones.

Right now, if package A has a symbol and package B wants to refer to it with //go:linkname, there are three patterns:

  • (Push) Package A uses a //go:linkname to rename one of its own symbols to B.foo, and then B declares func foo() without a body. In this form, A clearly intends for B to use foo, although the compiler cannot quite tell what's going on in B and warns about foo not having a body unless you create an empty dummy.s file.

  • (Pull) Package A defines foo without any annotation, and package B uses //go:linkname to access A.foo. In this form, A may not intend for B to use foo at all. That's a serious problem: when A renames foo and/or changes its type signature, B breaks, and A may never even have heard of B.

  • (Handshake) Package A defines foo with a //go:linkname and package B defines foo also with a //go:linkname, and the two agree on the name (either A.foo or B.foo). This is the ideal form, and it avoids the dummy.s workaround that is needed in the Push case.

The ideal goal state is a world where all //go:linkname usage must be in the Handshake form: both sides must agree to use linkname for a given symbol in order for it to succeed. This will mean that arbitrary packages cannot create new dependencies on runtime internals. At the same time, we realize that the current world is not this ideal world, and we don't want to break all existing uses.

Our plan is as follows.

  1. Introduce a new -checklinkname=1 flag to cmd/link that requires the Handshake form for symbols in the standard library. That flag is already landed in at tip, but it is not the default.

  2. Survey all existing open-source Go packages to find standard library symbols that are being //go:linkname'd (behind our backs!) using the Pull pattern. Add the necessary //go:linkname annotations to the standard library to keep those working, documenting why each exists. The explicit //go:linkname lines and documentation will help avoid accidental breakage in future refactoring. We have done a preliminary survey, but we haven't yet added all the necessary //go:linkname lines.

  3. Make -checklinkname=1 the default for Go 1.23. If this breaks anything, users can use -ldflags=-checklinkname=0 to get unbroken, and we hope they will also file reports letting us know what we missed.

  4. As we get reports of additional breakage we missed, add more //go:linkname annotations to the standard library.

At the completion of that plan, we won't be in the ideal world, but we will have accomplished two important things:

  • We won't have broken anything.

  • We will have stopped new damage from accumulating: there will be no more new references to runtime internals introduced. In particular, new internals we added during the Go 1.23 cycle, like coro and weak pointers, cannot be linknamed, now or ever. And anything that wasn't linknamed yet won't grow new linknames in the future.

Note that anyone who wants to experiment can always build with -ldflags=-checklinkname=0 and linkname whatever they like. That's fine. We like experimenting too. But the fact that the code won't build without special flags should help prevent code that digs into internal details from becoming a core dependency in the Go ecosystem that we end up having to maintain forever.

Note also that for now, //go:linkname can still be used in Pull mode to get at internals of non-standard library packages. We'd like to change that eventually too, insisting on Handshakes everywhere. For now, we are starting with the standard library. If all goes well, we'll circle back and try to devise a plan for the rest of the ecosystem.

@rsc rsc added this to the Go1.23 milestone May 15, 2024
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label May 15, 2024
@gopherbot
Copy link

Change https://go.dev/cl/585820 mentions this issue: runtime/coverage: remove uses of //go:linkname

@Jorropo
Copy link
Member

Jorropo commented May 15, 2024

I think github.com/goccy/go-json pulled internal runtime APIs in a reckless manner and there are safe ways to pull internal packages from the std.
For example quic-go did an equally dangerous thing with crypto/tls by accessing private fields using unsafe.Pointer and maintaining forks of crypto/tls:

However key differences:

As a downstream of quic-go I clearly understood the situation, the worst was the need to wait a couple of days for quic-go to release a new release compatible with the new go release before updating my go toolchain and the inability to run on tip or RCs.


In the case of github.com/goccy/go-json the situation could have been even better than quic-go's as it claims to be compatible with encoder/json:

Fast JSON encoder/decoder compatible with encoding/json for Go

instead of creating a compile time error they could have stubbed their API by forwarding calls to encoding/json when the release of go were to be unknown. (I don't know the details, maybe due to some edge cases or behaviors only go-json implements this wouldn't have been possible)


What I actually propose:

Continue to allow the Pull kind of linkname from std packages when the file is locked with build tags:

//go:build go1.23.4 && !go1.23.5 && !go1.24

To know if a file is properly locked down, the toolchain can evaluate the build tags with the next future release and it MUST fail to be allowed. That means if a file pass the current version but not the next one, then Pull linknames from the std would be allowed.

There is a downside to this approach which is that if there is no change required between two releases you still need a different file per version with the same implementation, to satisfy this constraint. Maybe parsing the build tags would be better.

I am not sure if goX.Y or goX.Y.Z should be used. goX.Y might be more dangerous in case a fix require breaking some internal API, quic-go used that and it was fine and created this couple of days period where you can't use latest go only every 6 months (note that goX.Y.Z build tags didn't existed back then).

@rsc
Copy link
Contributor Author

rsc commented May 15, 2024

I think ... there are safe ways to pull internal packages from the std.

I completely disagree. Quic-go's use of linkname caused all manner of problems for us release after release too, because anyone using quic-go couldn't update to a new Go version until quic-go did.

@gopherbot
Copy link

Change https://go.dev/cl/585916 mentions this issue: internal/coverage/cfile: remove //go:linkname into testing

@gopherbot
Copy link

Change https://go.dev/cl/585915 mentions this issue: internal/coverage/cfile: remove more //go:linkname usage

@ericlagergren

This comment has been minimized.

@randall77
Copy link
Contributor

We can't fix what we don't know about.

  1. As we get reports of additional breakage we missed, add more //go:linkname annotations to the standard library.

We're open to any reports from closed-source packages. It would be particularly useful to hear about these when the release candidate comes out so they can make the .0 release.

@ericlagergren

This comment has been minimized.

@randall77

This comment was marked as outdated.

@ericlagergren

This comment was marked as outdated.

@ruyi789
Copy link

ruyi789 commented May 16, 2024

Misuse is undesirable, disabling is even worse, you want to change you maintain that package (go-json), that doesn't solve it?

@bjorndm
Copy link

bjorndm commented May 16, 2024

I would like to say that //go:linkname is very useful for certain types of low level programming such as Ebitengine , etc. While I agree the situation with goccy/go-json is bad, we should look at how it is being used in detail and think of alternatives for the legitimate uses.

@DmitriyMV
Copy link
Contributor

@bjorndm should't -ldflags=-checklinkname=0 which disables this check be enough for low level programming? And if you can always raise a proposal if something is missing and truly needed without ld flags.

As we get reports of additional breakage we missed, add more //go:linkname annotations to the standard library.

@cherrymui
Copy link
Member

In C, static symbols are not accessible outside of the compilation unit, full stop. There is no way to pull a static symbol from a C library. A number of other languages have similar strict visibility rules. They are very successful languages and are widely used. This suggests that a lot programs can be written and things can go very well without a mechanism to break into a library's internal details.

I don't think Go is fundamentally different. Ideally we could also have strict visibility rules. I would think Go unexported symbols are meant to be similar to C static symbols. In fact, that is what gccgo does. Unfortunately for the gc toolchain it is not the case today. But we can get closer to it. And as we care a lot about compatibility, we'll keep the existing code continue to build in Go 1.23 (Step 2 in the plan). And we have a linker flag to disable the restriction (e.g. for experiments; as far as I know, the C linker doesn't seem to have such an option).

Also, I think the authors of the code should have a way to decide which symbols are visible externally and which are not.

@TotallyGamerJet
Copy link

TotallyGamerJet commented May 16, 2024

Purego which is a dependency of Oto, Beep, and Ebitengine as well as others doesn't just pull symbols it also pushes since it reimplements runtime/cgo package when CGO_ENABLED=0 entirely in Go. Is there any way the symbols defined in the runtime that hook into that package also get comments to avoid breaking us?

Another potential solution for us is to prebuild runtime/cgo into a cgo_GOOS_GOARCH_GOVERSION.syso that ships with purego. That would save us from having to keep up with any changes that package has and allows the Go team the freedom to change it as they like. Is this actually possible? I tried with setting different -buildmode but none of them would link.

Of course, if the Go team wanted to port runtime/cgo to Go that would be optimal.

@cherrymui
Copy link
Member

Is there any way the symbols defined in the runtime that hook into that package also get comments to avoid breaking us?

Push linknames are still allowed. If they are currently pushed from runtime/cgo, I believe you can still push them from Purego. Does Purego push symbols more than runtime/cgo?

Of course, if the Go team wanted to port runtime/cgo to Go that would be optimal.

This might be a possible option, but I think we need to understand the rationales better. The runtime/cgo package is intended to work with cgo, that is, interacting with C code. I'm not sure I understand the use case of runtime/cgo with CGO_ENABLED=0. This is probably better to be a separate discussion. Thanks.

@TotallyGamerJet
Copy link

TotallyGamerJet commented May 16, 2024

Push linknames are still allowed. If they are currently pushed from runtime/cgo, I believe you can still push them from Purego. Does Purego push symbols more than runtime/cgo?

No, Purego pushes the same symbols that runtime/cgo does which means the "Handshake" is already satisfied for those. Step 2 of the suggested plan only mentions surveying for Pull linknames and marking them with comments to avoid future breakage. I'm wondering if there is any plans for Pushes as changes to those would break Purego?

This might be a possible option, but I think we need to understand the rationales better. The runtime/cgo package is intended to work with cgo, that is, interacting with C code. I'm not sure I understand the use case of runtime/cgo with CGO_ENABLED=0. This is probably better to be a separate discussion. Thanks.

Indeed, runtime/cgo is required to allow Go code and C code to play well with each other. Purego provides an entirely Go version of it so that you can call C code using purego.Dlopen and purego.Dlsym without the need of a C compiler so cross-compiling is again possible. We can discuss this further elsewhere.

@cherrymui
Copy link
Member

cherrymui commented May 16, 2024

Thanks.

I'm wondering if there is any plans for Pushes as changes to those would break Purego?

I don't think there is any plan to break the use case of Purego's pushes. If we do anything to restrict push-only ones, they will be equally applied to the ones runtime/cgo pushing to runtime. So we'll need to fix those first (I think many of them are already in handshake form, but it is possible we missed some). And that should make Purego work as well.

@dmitshur dmitshur added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 16, 2024
@gopherbot
Copy link

Change https://go.dev/cl/586259 mentions this issue: runtime: move exit hooks into internal/runtime/exithook

@gopherbot
Copy link

Change https://go.dev/cl/586137 mentions this issue: all: add push linknames to allow legacy pull linknames

@rsc
Copy link
Contributor Author

rsc commented May 23, 2024

@database64128 You got here before I did. I was going to post here once I finished adjusting the CLs.

I took a closer look at tfo-go, and even with the other copies the usage count is still very low, meaning the complexity we take on to support it needs to be similarly low. I'm willing to commit to keeping the portable linknames working (which will suffice for all the Unix platforms), but the Windows-specific linknames are far too invasive. They are exactly the kind of thing we are trying to avoid by locking down linkname in this issue. And since tfo-go has minimal usage overall and therefore probably even more minimal usage on Windows, it seems reasonable to push back on the Windows ones and suggest finding a different way to do that part. It seems like something that cares less about the net package innards should be possible on Windows the same as on Unix systems.

@rsc
Copy link
Contributor Author

rsc commented May 23, 2024

@wwqgtxx In general the bar for adding linknames needs to be a substantial number of dependents, and metacubex/mihomo is barely used. That said, for defaultNS I see a bunch of usage across a variety of packages, and that's probably enough combined with how trivial it will be to keep working. I'll add it to the final CL.

Update: https://go-review.googlesource.com/c/go/+/587756/4/src/net/dnsconfig.go

@database64128
Copy link
Contributor

It seems like something that cares less about the net package innards should be possible on Windows the same as on Unix systems.

Currently there's no way to execute arbitrary async IO with netpoll on Windows, other than linknaming against execIO and friends. To list a few roadblocks:

  • The FileConn, FileListener, FilePacketConn functions in the net package are unimplemented for Windows and return syscall.EWINDOWS.
  • The current syscall.RawConn implementation on Windows cannot be used for any meaningful IO operations. In internal/poll/fd_windows.go, the (*FD).RawRead method does an extra "fake" read, and the (*FD).RawWrite method simply gives up and returns syscall.EWINDOWS.

This deserves a separate issue, and I'll open one later. But working through the problems above is going to take a lot of time. The necessary infrastructure won't be ready until many releases later. And we want tfo-go to keep working on Windows.

So, is there a middle ground where we add the necessary linkname annotations to the standard library, but do not lock down the affected internal API? As you said, there isn't a whole lot of usage of tfo-go. Therefore breaking tfo-go will not have any impact on the overall ecosystem.

@felixge
Copy link
Contributor

felixge commented May 23, 2024

Thank you for tackling this problem. I have three questions:

  1. What about eBPF programs using uprobes to access package internals? I've seen lots of code hooking into scheduler internals like runtime.execute and runtime.casgstatus.
  2. What happens after this initial phase of stabilizing the current situation?
  3. Is it okay to change the performance characteristics of the "frozen internals"? I.e. add allocations to convert from a new internal data structure back to the old one expected by the frozen API?

I really admire the Go team's dedication to compatibility, but I'm worried that freezing internal APIs based on popular usage in the ecosystem is setting a dangerous precedent. What if people find new ways to hack around the visibility rules? Will they also be rewarded with semi-official API access to go internals? What I'd love to see is a clear message that even the "frozen APIs" are just a temporary measure to stabilize the ecosystem, but that library authors should migrate away from them as these guarantees will be removed at some point in the future.

@marten-seemann
Copy link
Contributor

I'm sorry that quic-go's usage of go:linkname caused pain.

I'd love to remove it, but as far as I can tell, there's no other way to configure TLS cipher suites. I'd like to point out that we're only using go:linkname for testing purposes, and we don't expose any API to quic-go users for this. We use it in unit / integration tests and in interop tests os the QUIC interop runner. This is necessary because the QUIC header protection key derivation differs quite a bit depending on whether one of the AES or the ChaCha cipher suite is negotiated, and it would be irresponsible to not test these code paths.

As soon as there's a different way to test these code paths, I'll happily move away from go:linkname. I'm not suggesting reintroducing configurability of TLS cipher suite via the tls.Config. Maybe we could introduce a GODEBUG flag, or is this too much of a fringe use case?

@thaJeztah
Copy link
Contributor

We use it in unit / integration tests

My comment is probably very off-topic for this discussion, but for such cases I'd still love to see the ability to import exported test functions from other test code. Currently such functions tend to land in "non-_test.go" files if they're needed from different packages, which now (implicitly) makes them "production" code, and at risk of being consumed in situations where they shouldn't be. (I guess internal/ and possibly build-flags can aleviate some of this, but of course wouldn't be able to cross module boundaries, and wouldn't be easily usable to expose package internals for testing). (I'm pretty sure this has been proposed before, but my search-foo is failing me 🙈)

@wwqgtxx
Copy link

wwqgtxx commented May 23, 2024

I think @felixge's question involves a basic contradiction. The author of the library needs some means to access the unopened internal implementations in the golang standard library to achieve some functions that cannot be achieved based only on the public API. But after "lock down future uses of linkname", these visits were limited to a subset designed by the official based on popularity statistics. This resulted in the shift of maintainers compatible with Golang's internal changes from third-party library authors to standard library maintainers.

In the past, since there were no restrictions on linkname, third-party library authors could adjust the code in time based on user feedback. Although this would prevent users from updating to the latest golang tool chain, generally popular projects would not need to wait too long.

After implementing strict restrictions, the authors of third-party libraries need to wait for the official to open a hole, and do not know when this hole will be closed. Standard library maintainers also need to pay a huge price for this. Once the frozen structure needs to be modified, they will be under tremendous pressure, because this may be the only way for a third-party library to implement a function. (Even if other methods are technically feasible, third-party libraries cannot access it due to the new restrictions on linkname.)

linkname is indeed ugly, but when a large number of third-party libraries are used, it actually shows that there is a lack of formal and legal means in the standard library to implement it. For example, the question mentioned by the author of tfo-go above:

Currently there's no way to execute arbitrary async IO with netpoll on Windows, other than linknaming against execIO and friends.

Waiting for the standard library to complete the public API may take a lot of time. In the meantime, third-party library authors using some magic means to achieve creative work should not be regarded as a violation of compatibility. Further up, some members said that the very successful C language strictly limits access to symbols, but as another popular language, Python widely supports monkey patch methods to supplement the missing functions of the standard library.

It is conceivable that after golang1.23 is released, some private libraries that rely on linkname are not counted, and some new functions can still only be implemented through linkname, and the targets of these linknames are not included in the whitelist. As a result, a large number of build scripts were forced to add -checklinkname=0 to bypass these restrictions. Eventually, as time goes by, this tag may become a common way of writing for more and more people. By then, will a new issue be raised here to discuss whether the checklinkname tag needs to be abolished to protect the future go ecosystem?

@rsc
Copy link
Contributor Author

rsc commented May 23, 2024

@database64128, re:

So, is there a middle ground where we add the necessary linkname annotations to the standard library, but do not lock down the affected internal API? As you said, there isn't a whole lot of usage of tfo-go. Therefore breaking tfo-go will not have any impact on the overall ecosystem.

The middle ground is that users of tfo-go on Windows can still build their programs with -ldflags=-checklinkname=0. Since there isn't a whole log of usage of tfo-go, that shouldn't affect too many people. But it also stops most of the Windows-using Go ecosystem from accidentally taking a dependency on tfo-go.

@rsc
Copy link
Contributor Author

rsc commented May 23, 2024

@felixge, re:

Thank you for tackling this problem. I have three questions:

  1. What about eBPF programs using uprobes to access package internals? I've seen lots of code hooking into scheduler internals like runtime.execute and runtime.casgstatus.

I'm mainly concerned with widespread usage by programs that don't realize they are depending on packages that depend on packages that depend on packages that depend on packages that depend on //go:linknames into Go internals. I don't think there are many such situations where programs don't know they depend on eBPF programs, so breaking these eBPF programs would not cause the kind of damage we are trying to avoid.

That said we're also not planning gratuitous changes, and execute and casgstatus are fairly stable fundamental operations, so probably such eBPF programs will keep working reasonably well.

  1. What happens after this initial phase of stabilizing the current situation?

The main thing that happens is no new //go:linkname usage can be introduced as a far-off dependency of large fractions of the Go ecosystem. I don't anticipate significant cleanup or removal of the now-documented linknames if that's what you meant. I have been fairly careful about what I was willing to add linknames for (such as pushing back against tfo-go's Windows changes) and I feel okay about keeping the vast majority of those working. That said, if we did manage to cut usage of a linkname down to the level where next to nothing needs it anymore, then we might remove it.

  1. Is it okay to change the performance characteristics of the "frozen internals"? I.e. add allocations to convert from a new internal data structure back to the old one expected by the frozen API?

Yes, that is okay. Of course we don't want to do that gratuitously, but one example where that might be necessary is if we change to a different map representation, and the map iterator struct that people have copied and used with mapiterinit needs to be bigger or have more pointers. We might have the "real" map iterator API mapiterinit2, mapiternext2, etc., and then implement the linknamed old routines by allocating a new iterator state in mapiterinit and storing it in a pointer-typed word of the old iterator struct, leaving the rest of the old struct unused.

I really admire the Go team's dedication to compatibility, but I'm worried that freezing internal APIs based on popular usage in the ecosystem is setting a dangerous precedent. What if people find new ways to hack around the visibility rules?

Do you know of any? I'd be happy to lock them down. It's not like the existence of //go:linkname was a surprise to us: it was an intentional feature, just one that we didn't realize would be quite so widely used.

Will they also be rewarded with semi-official API access to go internals?

It depends on how much of the Go ecosystem breaks without locking the access in. I agree that it's not ideal, but it's better than breaking users who have no idea they are even depending on these things.

What I'd love to see is a clear message that even the "frozen APIs" are just a temporary measure to stabilize the ecosystem, but that library authors should migrate away from them as these guarantees will be removed at some point in the future.

I'd love to see that too, but the fact is that getting packages updated and then getting usage updated takes a very long time. Also as evidenced by the other discussion on this issue, in some cases we need to provide alternatives first, and getting public APIs we are happy with will also be a challenge. I can see these being temporary, but more on the scale of a decade than on the scale of a year.

@gopherbot
Copy link

Change https://go.dev/cl/587795 mentions this issue: cmd/go,testdeps: move import of internal/coverage/cfile to testmain

gopherbot pushed a commit that referenced this issue May 23, 2024
Instead of having testing/internal/testdeps import the
internal/coverage/cfile package directly, have the code in testmain
pass in pointers to cfile functions during setup in the case that
we're running a "go test -cover" binary. This reduces the size of
regular non-coverage test binaries back to what they were before CL
585820.

Updates #67401.
Fixes #67588.

Change-Id: Iaf1a613bc7d3c9df9943189065d0161ca9120d34
Reviewed-on: https://go-review.googlesource.com/c/go/+/587795
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Russ Cox <rsc@golang.org>
@rsc
Copy link
Contributor Author

rsc commented May 23, 2024

@wwqgtxx, re:

It is conceivable that after golang1.23 is released, some private libraries that rely on linkname are not counted, and some new functions can still only be implemented through linkname, and the targets of these linknames are not included in the whitelist. As a result, a large number of build scripts were forced to add -checklinkname=0 to bypass these restrictions. Eventually, as time goes by, this tag may become a common way of writing for more and more people. By then, will a new issue be raised here to discuss whether the checklinkname tag needs to be abolished to protect the future go ecosystem?

I don't think this is at all likely. The vast majority of the ecosystem will refuse to add -ldflags=-checklinkname=0 to their go install invocations. Most major projects in particular will not adopt it, and that should create significant backpressure to prevent widespread use of packages that need the flag.

I do agree with you that the linkname 'heap map' is a good indicator of where we are missing important APIs. Addressing that is a longer term project.

@gopherbot
Copy link

Change https://go.dev/cl/587918 mentions this issue: syscall: rm go:linkname from origRlimitNofile

@database64128
Copy link
Contributor

@wwqgtxx proposed a change (MetaCubeX/tfo-go@7577e13) for tfo-go that would reduce the number of necessary linknames for Windows. Here's the new list:

//go:linkname sockaddrToTCP net.sockaddrToTCP
func sockaddrToTCP(sa syscall.Sockaddr) net.Addr

//go:linkname execIO internal/poll.execIO
func execIO(o *operation, submit func(o *operation) error) (int, error)

//go:linkname fdInit internal/poll.(*FD).Init
func fdInit(fd *pFD, net string, pollable bool) (string, error)

@rsc Can you please take another look and see if it's now eligible for inclusion? Thanks.

@xiaokangwang
Copy link

xiaokangwang commented May 24, 2024

I fully understand that you are intending to avoid unnecessary pulling of dependencies that breaks when go language standard libraries updates. I agree that is something that would be very welcomed in the community and I personally sometimes feels sad when a several libraries I imported don’t like each other as they depend on different versions of the same library, not to say different versions of the standard libraries. However, without viable alternatives to the link name as a means of accessing private functions that would be absolutely necessary to implement some features, libraries authors would need to rely on more unsupported means to accessing these functions like https://github.com/spance/go-callprivate.

There is a way to fix this with a one time effort that is to separate Go standard library from Go runtime, by making sure standard libraries itself only use exported public functionalities of the Go runtime, thus allow any developers to implement whatever functionality they needed for any particular system without having to resort to use golang’s standard library and create a level playing field for all Go developers. By doing so, there will be a finite amount of public runtime API that will need to be maintained, without further limiting what user could do with go programming languages.

The current way of creating an allowlist of functions that could be called is unfair and won’t work: it is unfair that Google affiliated project like gvisor could just add whatever they needed to the allowlist, while other developers are forced to remove functionalities from their projects because they are underprivileged; and it won’t work because these restrictions will be workaround easily with some engineering trick and reflect+unsafe already allow projects to break when private field of struct updates.

@DmitriyMV
Copy link
Contributor

@xiaokangwang use -ldflags=-checklinkname=0 and raise issue for missing APIs.

Russ Cox said about addressing that:

Addressing that (missing APIs) is a longer term project.

@xiaokangwang
Copy link

@xiaokangwang use -ldflags=-checklinkname=0 and raise issue for missing APIs.

Russ Cox said about addressing that:

Addressing that (missing APIs) is a longer term project.

As we have already seen, they are not adding every API that are necessary to implement the features users wants, as they are prioritising projects with elevated privileges and refuse to make changes and maintain API for under privileged projects. This is not a solution that would work for every projects.

@DmitriyMV
Copy link
Contributor

As we have already seen, they are not adding every API that are necessary to implement the features users wants, as they are prioritising projects with elevated privileges and refuse to make changes and maintain API for under privileged projects.

No language ever does. The problem with separating standard library and runtime API is that you are forced to support it in future and stuck with the semantics you defined. I don't think any language allows full separate std implementation.

This is not a solution that would work for every projects.

While project is getting traction -ldflags=-checklinkname=0 should be enough. I agree that this is not ideal, but if my dep modules are trying to get into new or unopened internals I want to know why.

@bjorndm
Copy link

bjorndm commented May 25, 2024

@dmitshur Actually for C it is trivial to switch out the standard library which is why we have projects like musl. One downside of the current Go compiler is that that standard library and the compiler are very tightly coupled. This makes it harder to write an alternative standard library, and leads to people using this linkname hack to bypass the limitations of the standard library. In the long run it would be better to try and decouple the compiler and std library as much as possible.

@DmitriyMV
Copy link
Contributor

DmitriyMV commented May 25, 2024

@bjorndm

Actually for C it is trivial to switch out the standard library which is why we have projects like musl.

Almost zero runtime. I think Rust tries to go there with no_std but I don't know how much progress they made.

One downside of the current Go compiler is that that standard library and the compiler are very tightly coupled

Go is garbage collected language with custom lightweight threads implementation. Both of those require very complicated runtime. Comparing with C is not exactly fair here.

@bjorndm
Copy link

bjorndm commented May 25, 2024

I admit the Go runtime is way more useful than and hence way more complicated than that of C. Still I am surprised how many projects are using linkname to hack the runtime. This seems to indicate certain needs of Go developers are not being met.

@cherrymui
Copy link
Member

There are other implementations of Go, e.g. gccgo, gollvm, TinyGo, and perhaps GopherJS. (They may have limitations, may be incomplete, or special purposed.) They have different implementation details. Using linkname to access internal details likely don't work with them. If one wants to write portable Go code, one should avoid that. Unfortunately, many people don't care.

@bjorndm
Copy link

bjorndm commented May 26, 2024

Yes, and I do appreciate the efforts of those who are making the alternative Go compilers, including your work on GCC go. What I am getting at is that it seems we need some portable way to do certain kinds of low level programming in Go.

@AsterDY
Copy link

AsterDY commented May 27, 2024

As the maintainer of Sonic (github.com/bytedance/sonic), I am deeply concerned about the proposal to discard go:linkname. Sonic relies heavily on go:linkname for implementing critical optimizations and features, such as JIT compilation and fast Go-C intercalls. These enhancements significantly boost the performance of applications developed using Go.

If this proposal is adopted, it will adversely affect one of the fastest JSON libraries available for Go. More importantly, it will dampen the enthusiasm of seasoned developers who leverage these advanced techniques to push the boundaries of Go's performance.

@iDigitalFlame
Copy link

As much as I understand the reasoning behind this. It completely goes against the promise of "not breaking things". Is go:linkname the best way to handle things? No, but it should be left to the developers of the broken packages to fix them, not Golang.

The main reason I've seen (and used) this directive is to gain access to functions that I'd argue shouldn't be hidden behind the runtime or change the behavior of the runtime from some undesirable behavior.

I could totally understand the vet check. That'd keep it as a hey you better know what you're doing but allow advanced users to do what they need to do. I understand the "opt-in" nature of the suggestion also, but it would cause multiple projects and workflows to now be reconfigured (yes I know it's small) over some issues with a small amount of packages.

The vet check should stay, the compile check nope.

@bjorndm
Copy link

bjorndm commented May 27, 2024

The go compatibility promise https://go.dev/doc/go1compat has an explicit exception for undefined and unspecified behavior. The //go:linkname directive is not specified by the Go spec, so it falls under this. Conformant Go compilers may simply consider this as a comment. While I agree that there is a need for low level access to the runtime, using undefined behavior to do this is a recipe for disaster. The Go developers are not bound to keep this working.

@HeRaNO
Copy link

HeRaNO commented May 27, 2024

I'm not quite sure if it is valid. If A.foo and B.foo agree with the name A.foo under the Handshake situation, and someday A.foo changes to A.bar. Now B should change to A.bar. But under the Pull situation, B should also change to bar to keep compatibility with A. It has no difference with the Handshake situation. So what's the point of making this change? Only for the internal-external concern?

If I have any mistake, pls comment and let me know. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Projects
Status: In Progress
Development

No branches or pull requests