[BUG] go build errors after upgrading to 1.59.0

[pdeva]

I upgraded from 1.58.1 to 1.59.0

Doing so results in tons of errors when running go build

system info:

Debian GNU/Linux 12 (bookworm)
go version go1.21.5 linux/amd64

[RomainMuller]

Hey @pdeva,

Can you also provide the output of go env (feel free to redact any value that you don't feel comfortable sharing)?

[RomainMuller]

Seems like hit isn't just with dd-trace-go so it could be a broken purego version (see ebitengine/purego#189). Can you please look in your go.sum file and tell us what version of github.com/ebitengine/purego is currently resolved?

We tested against v0.5.1 and suspect the v0.6.0-alpha is currently somehow broken.

[pdeva]

seeing this for purego:

github.com/ebitengine/purego v0.6.0-alpha.2 h1:lYSvMtNBEjNGAzqPC5WP7bHUOxkFU3L+JZMdxK7krkw=
github.com/ebitengine/purego v0.6.0-alpha.2/go.mod h1:ah1In8AOtksoNK6yk5z1HTJeUkC1Ez4Wk2idgGslMwQ=

[pdeva]

output of go env

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/pdeva/.cache/go-build'
GOENV='/home/pdeva/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/pdeva/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/pdeva/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.5'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/pdeva/code/monorepo/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2973484752=/tmp/go-build -gno-record-gcc-switches'

[RomainMuller]

Okay, so it would appear the version of github.com/ebitengine/purego is not to blame here, and from tracking ebitengine/purego#189 it seems like a plausible cause for this issue could be that your package or one of its dependencies has CGO pragmas that add linker flags (e.g. //#cgo LDFLAGS: <...>) which end up breaking the linkage.

It would probably be a good starting point to grep your codebase & dependencies for //#cgo and see if anything stands out there... I also definitely would recommend following the issue on purego, as it might yield a workaround faster to get you un-blocked.

Also -- if you are not interested in Datadog's Application Security Management features, you should be able to unblock yourself (at least until we have a clean way to address this problem) by adding the datadog.no_waf build tag.

[pdeva]

the datadog.no_waf tag stuff works. but now this is an arg we need to add everywhere. why is this the default? by default everything should compile without any flags or anything. the waf stuff should be opt-in considering how fragile it is. i dont know any cgo stuff in our codebase and i shouldnt have to go on a hunt cause you decided to turn on a feature by default that i dont want.

[RomainMuller]

We're in complete agreement that things should work "out of the box" without any special configuration. We had extensively tested (including on Datadog's own services) and have not encountered any issue, which somewhat contradicts the WAF being "fragile".

I understand you have better things to do with you time than chase through your dependencies for something possibly incompatible. We're trying to understand what the problem is here so that we hopefully can address it quickly. We should be able to survey your dependencies if you're able to provide us the contents of the go.sum file.

We've also already started discussing alternative approaches here that would entirely remove this particular problem.

---

In therms of a workaround in the short term... you could also try to set CGO_ENABLED=0 in your build environments -- this can be easier to perform at scale than adding a go build tag, although there as well, we absolutely agree you should not normally have to do any of this.

[RomainMuller]

Seems like purego released a fixed version, can you try to run go get -u again and see if the issue is fixed?

[eliottness]

The new versions are not tagged yet but you can run go get github.com/ebitengine/purego@main or wait until they release/backport it. It should not take more than a day or two.

[hajimehoshi]

PureGo v0.5.2 was recently released. Please try this.

[eliottness]

The definitive fix for this issue is to upgrade to v1.59.1