Fails to build offline with vendored dependencies (building package on Linux / openSUSE) #7881

mcepl · 2021-01-26T23:53:41Z

Version information:

0.7.0 tarball

When trying to build package for openSUSE/Factory (complete build instructions are available), which is basically:

export GOCACHE=$(readlink -f vendor/)
BUILDMOD="-buildmode=pie -trimpath -modcacherw"
export CFLAGS="%{optflags}"
make GOFLAGS="-buildmode=pie -trimpath -modcacherw" nofuse

build fails in the openSUSE build system, because it tries to access network (but all build systems of all major Linux distributions don’t have network access, because of security reasons):

[    8s] + make 'GOFLAGS=-buildmode=pie -trimpath -modcacherw' nofuse
[    8s] go: bazil.org/fuse@v0.0.0-20200117225306-7b5117fecadc: Get "https://proxy.golang.org/bazil.org/fuse/@v/v0.0.0-20200117225306-7b5117fecadc.mod": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:56410->[::1]:53: read: connection refused
[    8s] go version go1.14.13 linux/amd64
[    8s] bin/check_go_version 1.14.4
[    8s] plugin/loader/preload.sh > plugin/loader/preload.go
[    8s] go fmt plugin/loader/preload.go >/dev/null
[    8s] go: bazil.org/fuse@v0.0.0-20200117225306-7b5117fecadc: Get "https://proxy.golang.org/bazil.org/fuse/@v/v0.0.0-20200117225306-7b5117fecadc.mod": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:44033->[::1]:53: read: connection refused
[    8s] make: *** [plugin/loader/Rules.mk:9: plugin/loader/preload.go] Error 1
[    8s] error: Bad exit status from /var/tmp/rpm-tmp.zpp5aA (%build)
[    8s]

How can I make go-ipfs build even in the environment without the network access?

(Vendoring was done by running make build && go mod vendor in the unpacked tarball.)

Complete build log with all details of package versions etc.

The text was updated successfully, but these errors were encountered:

welcome · 2021-01-26T23:53:42Z

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment.
Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

"Priority" labels will show how urgent this is for the team.
"Status" labels will show if this is ready to be worked on, blocked, or in progress.
"Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

RubenKelevra · 2021-01-27T15:36:11Z

Hey @mcepl,

I'm the maintainer for the git-package for the Archlinux AUR package.

Have a look at my build instructions here. They should work fine for you as well.

Have a look at the prepare step on how to fetch the dependencies before building offline. :)

mcepl · 2021-01-27T15:58:35Z

Have a look at my build instructions here. They should work fine for you as well.

Have a look at the prepare step on how to fetch the dependencies before building offline. :)

prepare is done while the network is still connected, right?
I am not the biggest go specialist (super understatement), but I thought that go mod vendor is somehow standard in the Go world. Why go-ipfs doesn’t support it?

(99MB tarball of dependencies? Really?)

mcepl · 2021-01-27T16:53:34Z

Hmm, modified SPEC file is still on the same URL, but the build still fails:

[   55s] + make 'GOFLAGS=-buildmode=pie -trimpath -modcacherw' nofuse
[   55s] go version go1.14.13 linux/amd64
[   55s] bin/check_go_version 1.14.4
[   55s] plugin/loader/preload.sh > plugin/loader/preload.go
[   55s] go fmt plugin/loader/preload.go >/dev/null
[   56s] go build -buildmode=pie -trimpath -modcacherw "-asmflags=all='-trimpath=/home/abuild/rpmbuild/BUILD/go'" "-gcflags=all='-trimpath=/home/abuild/rpmbuild/BUILD/go'" -ldflags="-X "github.com/ipfs/go-ipfs".CurrentCommit=" -tags="nofuse" -o "cmd/ipfs/ipfs" "github.com/ipfs/go-ipfs/cmd/ipfs"
[   60s] # plugin
[   60s] cannot load DWARF output from $WORK/b616//_cgo_.o: decoding dwarf section info at offset 0x0: too short
[   61s] # net
[   61s] cannot load DWARF output from $WORK/b081//_cgo_.o: decoding dwarf section info at offset 0x0: too short
[   62s] # os/user
[   62s] cannot load DWARF output from $WORK/b117//_cgo_.o: decoding dwarf section info at offset 0x0: too short
[   69s] make: *** [cmd/ipfs/Rules.mk:22: cmd/ipfs/ipfs] Error 2
[   69s] error: Bad exit status from /var/tmp/rpm-tmp.HAlSPS (%build)

Full log

Any ideas what do I do wrong?

Stebalien · 2021-02-05T16:45:48Z

Did you download the source from https://dist.ipfs.io/go-ipfs/v0.7.0/go-ipfs-source.tar.gz? That should be pre-vendored.

mcepl · 2021-02-05T20:50:03Z

Did you download the source from https://dist.ipfs.io/go-ipfs/v0.7.0/go-ipfs-source.tar.gz? That should be pre-vendored.

Well, the result is not much better:

[   24s] Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.mpSK2I
[   24s] + umask 022
[   24s] + cd /home/abuild/rpmbuild/BUILD
[   24s] + /usr/bin/rm -rf /home/abuild/rpmbuild/BUILDROOT/go-ipfs-0.7.0-0.x86_64
[   24s] ++ dirname /home/abuild/rpmbuild/BUILDROOT/go-ipfs-0.7.0-0.x86_64
[   24s] + /usr/bin/mkdir -p /home/abuild/rpmbuild/BUILDROOT
[   24s] + /usr/bin/mkdir /home/abuild/rpmbuild/BUILDROOT/go-ipfs-0.7.0-0.x86_64
[   24s] + cd go-ipfs-0.7.0
[   24s] + BUILDMOD='-buildmode=pie -trimpath -modcacherw'
[   24s] + export 'CFLAGS=-O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -g'
[   24s] + CFLAGS='-O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables -fstack-clash-protection -Werror=return-type -flto=auto -g'
[   24s] + make 'GOFLAGS=-v -a -x -mod=vendor -buildmode=pie -trimpath -modcacherw' build
[   24s] go: bazil.org/fuse@v0.0.0-20200117225306-7b5117fecadc: Get "https://proxy.golang.org/bazil.org/fuse/@v/v0.0.0-20200117225306-7b5117fecadc.mod": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:55825->[::1]:53: read: connection refused
[   24s] go version go1.14.14 linux/amd64
[   24s] bin/check_go_version 1.14.4
[   24s] plugin/loader/preload.sh > plugin/loader/preload.go
[   24s] go fmt plugin/loader/preload.go >/dev/null
[   24s] go: bazil.org/fuse@v0.0.0-20200117225306-7b5117fecadc: Get "https://proxy.golang.org/bazil.org/fuse/@v/v0.0.0-20200117225306-7b5117fecadc.mod": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:50666->[::1]:53: read: connection refused
[   24s] make: *** [plugin/loader/Rules.mk:9: plugin/loader/preload.go] Error 1
[   24s] error: Bad exit status from /var/tmp/rpm-tmp.mpSK2I (%build)

Complete build log with all details of the effort.

Previously, we avoided this variable because we were storing some flags with spaces (potentially). Unfortunately, that means these flags are not set for some commands (e.g., `go fmt`) leading to issues when building from vendored sources. Now we: 1. Use GOFLAGS where possible. 2. Pass tags and special flags on the commandline where necessary. This should fix #7881.

Stebalien · 2021-02-05T22:31:02Z

Ok, I'm hoping #7903 fixes the issue. My guess is that some go commands were getting executed without GOFLAGS set causing us to ignore the vendor directory.

mcepl · 2021-02-05T22:45:43Z

@Stebalien, better, but not yet:

[    3s] + make 'GOFLAGS=-v -a -x -mod=vendor -buildmode=pie -trimpath -modcacherw' build
[    3s] go: bazil.org/fuse@v0.0.0-20200117225306-7b5117fecadc: Get "https://proxy.golang.org/bazil.org/fuse/@v/v0.0.0-20200117225306-7b5117fecadc.mod": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:58163->[::1]:53: read: connection refused
[    3s] go version go1.14.14 linux/amd64
[    3s] bin/check_go_version 1.14.4
[    3s] plugin/loader/preload.sh > plugin/loader/preload.go
[    3s] go fmt plugin/loader/preload.go >/dev/null
[    4s] go: bazil.org/fuse@v0.0.0-20200117225306-7b5117fecadc: Get "https://proxy.golang.org/bazil.org/fuse/@v/v0.0.0-20200117225306-7b5117fecadc.mod": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:38007->[::1]:53: read: connection refused
[    4s] go build "-asmflags=all='-trimpath=/home/abuild/go'" "-gcflags=all='-trimpath=/home/abuild/go'" -o "cmd/ipfs/" ""
[    4s] WORK=/tmp/go-build558539244
[    4s] go build: no main packages to build
[    4s] make: *** [cmd/ipfs/Rules.mk:22: cmd/ipfs/] Error 1

Stebalien · 2021-02-05T23:36:30Z

Ok, I think I've finally gotten it working. Apparently, go doesn't obey GOFLAGS everywhere, so I needed to pass -mod=vendor to a few specific commands.

NOTE: This fix will only work with the source tarball from dist.ipfs.io as -mod=vendor will not be picked up from GOFLAGS for a few go commands (go list, go fmt, etc.).

mcepl · 2021-02-06T00:03:02Z

Build work well, unfortunately at least one test fails. However, I think that I can build this in my private repo for further playing before I will submit it to the official repos.

THANK YOU VERY MUCH!

Complete build log

Stebalien · 2021-02-06T00:38:39Z

Thanks for your help debugging!

mvdan · 2021-02-06T11:14:44Z

Apparently, go doesn't obey GOFLAGS everywhere, so I needed to pass -mod=vendor to a few specific commands.

Huh - can you share more details? I think that should be a bug to report upstream.

Stebalien · 2021-02-13T01:20:05Z

I'm... not sure? go list was having trouble, but manual testing with GOFLAGS doesn't have the same problem.

Specifically, one of the issues was https://github.com/ipfs/go-ipfs/pull/7903/files#diff-bede62ea1c96a1d7524e6cffe4a7a06a62392a38e1bdad0e8600d07ec06aa130L29.

mcepl added kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization labels Jan 26, 2021

mcepl mentioned this issue Jan 26, 2021

Package as rpm/deb and add it to the offical repos? #5471

Open

Stebalien mentioned this issue Feb 5, 2021

fix(build): preserve the GOFLAGS environment variable #7903

Closed

aschmahmann added need/analysis Needs further analysis before proceeding and removed need/triage Needs initial labeling and prioritization labels Feb 22, 2021

aschmahmann assigned Stebalien Feb 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fails to build offline with vendored dependencies (building package on Linux / openSUSE) #7881

Fails to build offline with vendored dependencies (building package on Linux / openSUSE) #7881

mcepl commented Jan 26, 2021 •

edited

welcome bot commented Jan 26, 2021

RubenKelevra commented Jan 27, 2021 •

edited

mcepl commented Jan 27, 2021 •

edited

mcepl commented Jan 27, 2021

Stebalien commented Feb 5, 2021

mcepl commented Feb 5, 2021

Stebalien commented Feb 5, 2021

mcepl commented Feb 5, 2021

Stebalien commented Feb 5, 2021

mcepl commented Feb 6, 2021

Stebalien commented Feb 6, 2021

mvdan commented Feb 6, 2021

Stebalien commented Feb 13, 2021

Fails to build offline with vendored dependencies (building package on Linux / openSUSE) #7881

Fails to build offline with vendored dependencies (building package on Linux / openSUSE) #7881

Comments

mcepl commented Jan 26, 2021 • edited

Version information:

welcome bot commented Jan 26, 2021

RubenKelevra commented Jan 27, 2021 • edited

mcepl commented Jan 27, 2021 • edited

mcepl commented Jan 27, 2021

Stebalien commented Feb 5, 2021

mcepl commented Feb 5, 2021

Stebalien commented Feb 5, 2021

mcepl commented Feb 5, 2021

Stebalien commented Feb 5, 2021

mcepl commented Feb 6, 2021

Stebalien commented Feb 6, 2021

mvdan commented Feb 6, 2021

Stebalien commented Feb 13, 2021

mcepl commented Jan 26, 2021 •

edited

RubenKelevra commented Jan 27, 2021 •

edited

mcepl commented Jan 27, 2021 •

edited