Can't run fcct 04.0 standalone binary in alpine container

[jensrutschmann]

Hi,

I cannot run the latest release 0.4.0 binary inside an alpine container:

jens:~> docker run --rm -it alpine:3.8 sh
/ # wget -O /usr/local/bin/fcct https://github.com/coreos/fcct/releases/download/v0.4.0/fcct-x86_64-unknown-linux-gnu
Connecting to github.com (140.82.118.4:443)
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (52.216.170.59:443)
fcct                 100% |*******************************************************************************************************************************************************************************************| 10203k  0:00:00 ETA
/ # chmod a+x /usr/local/bin/fcct 
/ # /usr/local/bin/fcct 
sh: /usr/local/bin/fcct: not found

After installing file (using apk --update-cache --upgrade add file) it turns out the 0.4.0 release is a dynamically linked binary, not a static binary.

/usr/local/bin # file fcct
fcct: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, BuildID[sha1]=d9bb68fd48bf269bb0271cfc71545251d8fe1343, for GNU/Linux 3.2.0, stripped

Yet there aren't any linker issues:

/usr/local/bin # ldd fcct 
	/lib64/ld-linux-x86-64.so.2 (0x7f40e5620000)
	libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f40e5620000)
	libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f40e5620000)

Shouldn't the 0.4.0 release also be a statically linked binary? The 0.2.0 release is, and I can run that one just fine.

[tzawlxih]

in the latest alpine 3.12, 6th release of fcct, still not working.

ct-*-x86_64-unknown-linux-gnu was working well

[Okeanos]

So I just stumbled across this as well – for some reason the releases are all dynamically linked instead of statically linked:

$ ldd butane
./butane: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./butane)
	linux-vdso.so.1 (0x00007ffd19c9d000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f575bd9b000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f575bba9000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f575bdd1000)
$ file butane 
butane: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=c987d3e8e9b404120c192a4b8cac59d452e252dd, stripped

This is something of a problem because the required glibc version is not available on for example Ubuntu 20.04 (the latest LTS version) out of the box:

butane: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./butane)

However, when building the project locally (current latest version) on Ubuntu 20.04.1 using the provided build script, the output is a statically linked binary:

$ uname -a
Linux ubuntuvm 5.11.0-27-generic #29~20.04.1-Ubuntu SMP Wed Aug 11 15:58:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ go version
go version go1.16.2 linux/amd64
$ git rev-parse HEAD
97c6866efaafc46729640a24cd2336245890b63d
$ ./build
$ ldd bin/amd64/butane
not a dynamic executable
$ file bin/amd64/butane
bin/amd64/butane: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=Ye0UWPX393ytSs0asjB3/hgozWMFdOTbSH1gaTEl6/5XqYckPOUWu_TSHtzDm6/ZWXu_yBiboj_afAS-L8R, not stripped

While having a dynamically linked binary is not a dealbreaker on its own, it makes using butane considerably more painful than it needs to – specifically on only slightly older but otherwise well supported distributions such as Ubuntu 20.04.

Could a maintainer have another look at the build chain and check whether producing statically linked prebuilt binaries (again) is feasible?

Edit: just saw #221 exists as well but was closed without a fix, saying that running butane via podman is preferred. While that might be true, I don't believe that should be the official fix when building a statically linked butane release looks to be possible/feasible.

[Okeanos]

@bgilbert I really don't want to be a bother about this but having statically linked butane executables would be a huge boon and in case this isn't possible I would very much like to understand why. Could you or another maintainer chime in here and maybe provide some additional information?

[bgilbert]

@Okeanos This hasn't been a priority for us, but if you'd like to dig into the build script and send a PR, or even just a writeup of what you find, that'd be very helpful.

[Okeanos]

As I stated in the previous comment: the included build script appears to output a statically linked binary just fine on its own (having tested it on Ubuntu 20.04.1 with go 1.16.2). I'd be willing to invest some time into this, however, I can't seem to find any reference to the actual build environment used to generate the official releases. There's no CI file or similar that I can identify from a cursorily look.

[bgilbert]

Whoops, yeah, sorry, I know what's going on. For policy reasons, our release binaries are built in Koji from the Fedora specfile, and of course the Fedora build macros produce dynamically-linked binaries. Probably the right approach is to extend the nonlinux subpackage to include static versions of the binary for all architectures we ship.

[Okeanos]

@bgilbert thank you for the pointers; I had another look at this:

The butane.spec relies on go-rpm-macros and specifically uses the gobuild macro to build the linux binaries for each invoked architecture. However, there's no way to set CGO_ENABLED to any value with gobuild aside from abusing the gomodulesmode flag. Also as far as I understand the ldflags and extldflags, they would nullify this. There's a stackoverflow thread proposing a solution and golang/go#41944 but I am not sure that's a maintainable solution here.

The option you proposed appears to be the easiest right now, i.e. add additional gocrossbuild calls for Linux for each arch with CGO_ENABLED=0. "Optimising" the build flags within the macros, potentially affecting lots of other projects, doesn't strike me as a good option. An alternative would be replacing the gobuild call with a plain go build as well; similar to the gocrossbuild – after all butane doesn't appear to be using any fancy CGO or other dependencies that would require this massive list of ldflags and build optimisation?

As I don't have a Fedora account, I am not sure I can contribute these changes right now.

[bgilbert]

Thanks for digging into that.

An alternative would be replacing the gobuild call with a plain go build as well; similar to the gocrossbuild – after all butane doesn't appear to be using any fancy CGO or other dependencies that would require this massive list of ldflags and build optimisation?

%gobuild is specified by the Fedora Go packaging guidelines. The distro needs to be able to modify build flags without changing each package individually. I don't think that would apply to binaries shipped as opaque data, though.

As I don't have a Fedora account, I am not sure I can contribute these changes right now.

If you're interested, you should be able to just sign up for a Fedora account. I don't think you need any special permissions to submit a PR.

[Okeanos]

I can do that – before I do that, I'd like to clarify the scope of the changes to the butane.spec file, though. As far as I understand it the butane.spec changes would be these:

%build
#...

echo "Building Linux Butane with static linking..."
CGO_ENABLED=0 GOARCH=arm64 GOOS=linux %gocrossbuild -o butane-aarch64-unknown-linux-gnu-static internal/main.go
CGO_ENABLED=0 GOARCH=ppc64le GOOS=linux %gocrossbuild -o butane-ppc64le-unknown-linux-gnu-static internal/main.go
CGO_ENABLED=0 GOARCH=s390x GOOS=linux %gocrossbuild -o butane-s390x-unknown-linux-gnu-static internal/main.go
CGO_ENABLED=0 GOARCH=amd64 GOOS=linux %gocrossbuild -o butane-x86_64-unknown-linux-gnu-static internal/main.go

#...

%install
#...
install -p -m 0644 ./butane-aarch64-unknown-linux-gnu-static %{buildroot}%{_datadir}/butane
install -p -m 0644 ./butane-ppc64le-unknown-linux-gnu-static %{buildroot}%{_datadir}/butane
install -p -m 0644 ./butane-s390x-unknown-linux-gnu-static %{buildroot}%{_datadir}/butane
install -p -m 0644 ./butane-x86_64-apple-darwin %{buildroot}%{_datadir}/butane
install -p -m 0644 ./butane-x86_64-pc-windows-gnu.exe %{buildroot}%{_datadir}/butane
install -p -m 0644 ./butane-x86_64-unknown-linux-gnu-static %{buildroot}%{_datadir}/butane

#....
%files
#...
%{_datadir}/butane/butane-aarch64-unknown-linux-gnu-static
%{_datadir}/butane/butane-ppc64le-unknown-linux-gnu-static
%{_datadir}/butane/butane-s390x-unknown-linux-gnu-static
%{_datadir}/butane/butane-x86_64-unknown-linux-gnu-static

#...

Open questions from my side are mainly:

• is the naming fine?
• is the location (datadir instead of bindir) fine?
• are there any other changes I am overlooking?

Technically, I think adding GOARCH=arm64 GOOS=darwin %gocrossbuild -o butane-m1-apple-darwin internal/main.go should also be feasible as of Go 1.16.

A test run with these changes on Fedora 34 using a modified spec file and calling fedpkg install works fine and outputs alle expected statically linked binaries (and can also build the Apple Silicon version as well 🤷‍♂️).

[Okeanos]

@bgilbert I created a pull request similar to what I outlined above shortly after that message. Is there any chance you or someone else from the project has some time to review it and give me some feedback? Is there process that I have to follow and didn't see?

On an unrelated note, I am somewhat irritated by the fedoraproject platform not allowing people to use SSH unless they are … members of a particular group? That really wasn't fun to work around.

[bgilbert]

Yup, it's on my radar and I'll take a look when I can. Thanks for the PR.

[dcode]

Any update on this issue and related PR on Pagure? I ran into today running on the latest version of Proxmox VE, which runs glibc 2.31.

[8ear]

What workaround is possible for proxmox?

[Okeanos]

The workaround is: build it locally using the supplied build script.

[Timoses]

Trying to install butane in an alpine linux image (we require butane specifically within that image). Thus using the release butane container is no option.

Opting for local build within container.

[bgilbert]

Fixed in https://src.fedoraproject.org/rpms/butane/pull-request/5 for the next release. Thanks @Okeanos!