/lib64/libc.so.6: version `GLIBC_2.32' not found when building on linux

[pohzipohzi]

I was running into this error when I followed the instructions in the README:

/var/task/main: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /var/task/main)

I managed to fix it via by setting CGO_ENABLED=0 when building on linux, similar to the instructions for windows, ie

GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o main main.go

Should that be included in the README, or is there a better way to fix this?

[bmoffatt]

yeah, I can add a note about CGO in the README. Amazon Linux 1/2 provides glibc 2.17/2.26, so I can see how this'd be a problem depending on a developer's Linux distro.

[mrwormhole]

I have witnessed this one today, seems like it is just a problem of using CGO when the C library you have under the hood is not compatible with glibc 2.32 or there is no C library but you are trying to build with CGO, therefore, it can not detect/find the correct version for .so files. It took the production down for 1 hour, just because of CGO_ENABLED=1

CGO should never be used and it should be avoided as much as possible just like the reflection

https://dave.cheney.net/2016/01/18/cgo-is-not-go

[LeonardoFurtado]

It works. @bmoffatt Is this issue still required to be open?

[HSchmale16]

Affected by the same issue. Trying to use SQLite with go on lambda is painful.

[ccampo133]

Just ran into this myself also trying to use go-sqlite3. @HSchmale16 did you ever find a way to get it to work?

[HSchmale16]

Just ran into this myself also trying to use go-sqlite3. @HSchmale16 did you ever find a way to get it to work?

I ended up building and deploying from cloudshell. Had to move the GOPATH to outside of the home directory. I still haven't fully understood what lambda is doing with it.

[ccampo133]

Gotcha - in that case the solution is probably to build the binary on some form of Amazon Linux (VM, container, etc).

[ccampo133]

Just to follow up - confirmed at least in my case if I build the Go binary (which requires cgo, particularly using go-sqlite3) on Amazon Linux, it will execute successfully. Specifically, I used the amazonlinux:2 Docker image, installed Go on it, and built the binary in that.

[HSchmale16]

@ccampo133 I've documented my fix on my blog. See link for details.

https://www.henryschmale.org/2022/09/29/go-in-cloudshell.html

[gauntface]

I ran into this while using the SAM CLI.

The fix was ultimately the suggestion to use CGO_ENABLED=0 but I had to take some additional steps to make this work with sam, see aws/aws-sam-cli#3894 for the breadcrumbs.

Documented the approach for my project here: https://www.gaunt.dev/blog/2022/glibc-error-with-aws-sam-and-go/

[ccampo133]

@gauntface that will only work if your project does not depend on cgo. In my (and others here) case, I required cgo for go-sqlite3, which requires special compilation specifically for the Lambda Go runtime environment.

[thenom]

I dont really understand this much but CGO_ENABLED=0 did not work for me as the package i was using explicitly required CGO_ENABLED=1
https://github.com/confluentinc/confluent-kafka-go#librdkafka

In my scenario i am building in an amd64 linux golang:latest docker image (GOLANG_VERSION='1.19.3' at time) environment to run in a lambda. This built binary is getting:

/var/task/component-test-proxy: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /var/task/component-test-proxy)
/var/task/component-test-proxy: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /var/task/component-test-proxy)

in cloudwatch logs when invoked in Lambda. Do i have to just start getting older images till i eventually get one that works with Lambda or is there any checks that i can do to see which glibc is in Lambda and then find a docker image to match?

[thenom]

As a test i used an amazonlinux image as the build environment as i thought this should match the lambda env more than anything and just installed golang on it, inc all 138 dependancies which included glibc:
glibc-devel aarch64 2.34-40.amzn2022.0.2. # ignore the arch, this is just from my laptop (m1 mac) as the build env was silent.

Its now worse:

/var/task/component-test-proxy: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /var/task/component-test-proxy)
/var/task/component-test-proxy: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /var/task/component-test-proxy)
/var/task/component-test-proxy: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /var/task/component-test-proxy)
/var/task/component-test-proxy: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /var/task/component-test-proxy)

@HSchmale16 this was based of your blog post so not sure if i missed anything

[HSchmale16]

This seems super weird could something be broken with your Amazon Linux image. this is related to go being very weird about libc stuff. Can you try it on cloudshell? As that's been where I've had the most luck running this stuff.

As a test i used an amazonlinux image as the build environment as i thought this should match the lambda env more than anything and just installed golang on it, inc all 138 dependancies which included glibc:
glibc-devel aarch64 2.34-40.amzn2022.0.2. # ignore the arch, this is just from my laptop (m1 mac) as the build env was silent.

Its now worse:

/var/task/component-test-proxy: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /var/task/component-test-proxy)
/var/task/component-test-proxy: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /var/task/component-test-proxy)
/var/task/component-test-proxy: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /var/task/component-test-proxy)
/var/task/component-test-proxy: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /var/task/component-test-proxy)

@HSchmale16 this was based of your blog post so not sure if i missed anything

[ccampo133]

@thenom I built my binary (which required CGo) using the amazonlinux:2 as my build container, and it worked no problem. See: #340 (comment)

Not sure if you use a different Amazon Linux image.

[thenom]

Sorry, just to highlight and clarify my original message, the builds runs fine. The problem is when its invoked in Lambda, these are the cloudwatch logs from those invokes.

@HSchmale16 this is a stock fresh amazonlinux image that i install golang on then a couple of configurations for git and junit testing, thats it.
@ccampo133 cheers, i will try the alternative image tag. tbh i was looking for an amazonlinux2:latest originally but will try that.

[thenom]

@ccampo133 amazonlinux:2 is better actually. The lambda invoke now only fails with 1 dependancy missing:

/var/task/component-test-proxy: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /var/task/component-test-proxy)

[thenom]

Finally got it working but my process of trial and error doesnt seem that elegant. It works with this image:
public.ecr.aws/lambda/go:1.2022.11.03.18
which installed
glibc-devel-2.17-324.189.amzn1.x86_64
as part of the golang install.

Open to suggestions\advice on why this is happening or anything i can do to improve it. Thanks.

[seantcanavan]

This was absolutely terrifying to discover last minute. I recently switched all 4 of my PCs from Ubuntu to Manjaro and right as I was deploying some new code this issue popped up. I'm extremely lucky all I had to do was add CGO_ENABLED=0 to the build flags to fix it but this is a last minute dependency I didn't even know existed in my build pipeline.

I have no idea why Manjaro needs a new build flag versus Ubuntu unfortunately - that's definitely not an area of expertise for me.

[acidjazz]

does anyone here know how i can find out what package that got included uses this?

[seantcanavan]

@smark-d here's my build command from my makefile. i build on manjaro and deploy through the serverless framework to lambda and it works fine. go is v1.20.

build: clean
	env GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags="-s -w" -o bin/api cmd/api/api.go
	zip bin/api.zip bin/api
	env GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags="-s -w" -o bin/async cmd/async/async.go
	zip bin/async.zip bin/async
	env GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags="-s -w" -o bin/schedule cmd/schedule/schedule.go
	zip bin/schedule.zip bin/schedule

clean:
	rm -rfv ./bin

[cajonKA]

The problem is, that when building on Manjaro ( or Arch or another rolling distro) you are building your go file with a newer GLIBC (e.g. 2.32 oder higher)
The docker container used to run the lambda function is using an older glibc and this results in the mentioned error message.
Compiling with CGO_ENABLED=0 is only a solution if your code does not depend on Glibc
Another solution, as previously written, is to build your code on an older system or a container with an older glibc version

[Omkar76]

Hello? 3 years passed and still not resolved?

[AndrewCharlesHay]

Hello? 3 years passed and still not resolved?

Amazon doesn't care about open source

[victor-teran-rs-old]

Hello? 3 years passed and still not resolved?

Amazon doesn't care about open source

It's also happening with python3.11 runtime

[leepa]

Hello? 3 years passed and still not resolved?

Amazon doesn't care about open source

And people don't appear to read the FIRST POST at the top that literally tells you how to solve it. This is a common Linux building issue.

[FirstWhack]

As an FYI since it has not been posted in this thread.

Amazon Linux 2023 provides glibc 2.34, upgrade to al2023 and latest Go.