Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hugo does not build with go install out of the box anymore #5954

Open
bep opened this issue May 12, 2019 · 9 comments

Comments

@bep
Copy link
Member

commented May 12, 2019

I have been doing some work requiring me to clean my modcache several times, and without any GOPROXY set (which I assume is the default), Hugo does not build on my MacBook.

go clean -modcache && go install;  
  • Looking at the log I'm still a bit puzzled by the high number of seemingly unrelated new deps coming from the Google Cloud add-ons. I assume these are indirect test dependencies, but I don't understand why I need to download them to my PC. I guess this ties into golang/go#26913 (comment)
  • With GOPROXY=https://proxy.golang.org it builds fine, which means that we as a minimum need to update mage script + documentation.
  • Consider if we can opt-in to this, somehow. I don't think making a build tag for this new feature will help out.

/cc @vangent

The runs below is master vs 0.55.5:


go: finding golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52
go: finding gonum.org/v1/gonum v0.0.0-20181121035319-3f7ecaa7e8ca
go: error loading module requirements
( go clean -modcache && go install; )   168.28s  user 102.04s system 85% cpu 5:16.02 total
avg shared (code):         0 KB
avg unshared (data/stack): 0 KB
total (sum):               0 KB
max memory:                198000 MB
page faults from disk:     25650
other page faults:         7069035
go: extracting golang.org/x/text v0.3.0
go: extracting github.com/jdkato/prose v1.1.0
( go clean -modcache && go install; )   70.57s  user 36.62s system 142% cpu 1:15.35 total
avg shared (code):         0 KB
avg unshared (data/stack): 0 KB
total (sum):               0 KB
max memory:                478540 MB
page faults from disk:     17659
other page faults:         2800501

With GOPROXY=https://proxy.golang.org:

go: extracting google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19
go: extracting github.com/jdkato/prose v1.1.0
( go clean -modcache && go install; )   32.90s  user 25.99s system 53% cpu 1:50.78 total
avg shared (code):         0 KB
avg unshared (data/stack): 0 KB
total (sum):               0 KB
max memory:                732620 MB
page faults from disk:     5352
other page faults:         640553
go: extracting github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
( go clean -modcache && go install; )   5.65s  user 7.11s system 46% cpu 27.353 total
avg shared (code):         0 KB
avg unshared (data/stack): 0 KB
total (sum):               0 KB
max memory:                111228 MB
page faults from disk:     3439
other page faults:         50679

@bep bep added the Bug label May 12, 2019

@bep bep added this to the v0.56 milestone May 12, 2019

@bep

This comment has been minimized.

Copy link
Member Author

commented May 13, 2019

Now go get -u from fails for me from github.com/gohugoio/hugo even with a PROXY set :

▶ go get -u
go: finding github.com/gregjones/httpcache v0.0.0-20190212212710-3befbb6ad0cc
go: finding github.com/shurcooL/issues v0.0.0-20190120000219-08d8dadf8acb
go: finding github.com/docker/distribution v2.7.1+incompatible
go get: upgrading github.com/ugorji/go/codec@v0.0.0-20181012064053-8333dd449516: unexpected status (https://proxy.golang.org/github.com/ugorji/go/codec/@v/list): 410 Gone
go: github.com/testcontainers/testcontainer-go@v0.0.4: parsing go.mod: unexpected module path "github.com/testcontainers/testcontainers-go"
go: finding github.com/Azure/azure-sdk-for-go v28.1.0+incompatible
go: finding golang.org/x/net v0.0.0-20190322120337-addf6b3196f6
go: sourcegraph.com/sourcegraph/go-diff@v0.5.1: parsing go.mod: unexpected module path "github.com/sourcegraph/go-diff"
go get: error loading module requirements

Curiously enough it fails with different errors when running with GOPROXY=direct go get -u

go: finding google.golang.org/api v0.0.0-20180829000535-087779f1d2c9
go: finding github.com/smartystreets/assertions v0.0.0-20180820201707-7c9eb446e3cf
go: finding golang.org/x/crypto v0.0.0-20190404164418-38d8ce5564a5
go: github.com/golang/lint@v0.0.0-20190409202823-959b441ac422: parsing go.mod: unexpected module path "golang.org/x/lint"
go: finding k8s.io/api v0.0.0-20180806132203-61b11ee65332
go: finding github.com/lyft/protoc-gen-validate v0.0.

I guess this somehow ties into golang/go#26913 (comment)

@larzza

This comment has been minimized.

Copy link

commented May 13, 2019

On master I'm doing this successfully (in a docker build with default GOPROXY-setting):

git clone --branch master --depth 1 https://github.com/gohugoio/hugo.git hugo
go clean -modcache && go install -ldflags 'all=-w -extldflags "-static"' -tags extended

The difference in size of pkg between master (1.9G) and v0.55.5 (260M) is really quite big. Hopefully that isn't a necessary requirement...!?

master /go/pkg# du -hs    
1.9G

v0.55.5:/go/pkg# du -hs
260M
@bep

This comment has been minimized.

Copy link
Member Author

commented May 13, 2019

@larzza what is your default GOPROXY setting? I will have a more detailed look into this problem later, but I suspect most of this issue comes from golang/go#26913 (comment) -- which goes away if you use a proxy server (then you only need to look at the go.mod and only need to download the actual code if you for some reason want to build the relevant package/test).

@larzza

This comment has been minimized.

Copy link

commented May 13, 2019

@bep Ok, I read the proxy-comment. Hopefully it will be better with using a proxy.

My default is GOPROXY=""

What is your? Have you tried the different ways of unsetting the GOPROXY?:

If GOPROXY is unset, is the empty string, or is the string "direct", downloads use the default direct connection to version control systems

From the go docker container:

$ docker run -it --rm  golang:latest bash

# go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build835505629=/tmp/go-build -gno-record-gcc-switches"
@larzza

This comment has been minimized.

Copy link

commented May 13, 2019

which goes away if you use a proxy server (then you only need to look at the go.mod and only need to download the actual code if you for some reason want to build the relevant package/test).

Ok, I see, this already works as expected! By using a locally installed goproxy (Athens) I've verified that the size of the downloaded modules is much smaller when building Hugo from master.

/go/pkg# du -hs
450M	.

bep added a commit that referenced this issue May 28, 2019

deps: Update Viper
This will hopefully improve the dependency/install problems we have seen lately.

See #5954
@hook-s3c

This comment has been minimized.

Copy link

commented Jun 4, 2019

❯ go clean -modcache && go install
go: finding github.com/hashicorp/go-uuid v1.0.0
go: finding github.com/matttproud/golang_protobuf_extensions v1.0.1
go: finding github.com/go-ini/ini v1.25.4
go: finding github.com/kisielk/errcheck v1.1.0
go: pack.ag/amqp@v0.8.0: unrecognized import path "pack.ag/amqp" (parse https://pack.ag/amqp?go-get=1: no go-import meta tags ())
go: finding github.com/gogo/protobuf v1.1.1
go: finding golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09
go: pack.ag/amqp@v0.11.0: unrecognized import path "pack.ag/amqp" (parse https://pack.ag/amqp?go-get=1: no go-import meta tags ())
go: finding github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973
go: finding golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd
go: finding google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb
go: finding github.com/prometheus/client_golang v0.9.1
go: finding github.com/Azure/azure-pipeline-go v0.1.8
go: finding github.com/oklog/ulid v1.3.1
go: finding github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d
go: finding github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515
go: finding gopkg.in/check.v1 v1.0.0-20161208181325-20d25e280405
go: finding github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910
go: finding github.com/go-logfmt/logfmt v0.3.0
go: finding github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce
go: finding github.com/go-kit/kit v0.8.0
go: finding github.com/jmespath/go-jmespath v0.0.0-20160202185014-0b12d6b521d8
go: finding github.com/cespare/xxhash v1.1.0
go: finding github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954
go: error loading module requirements
@bep

This comment has been minimized.

Copy link
Member Author

commented Jun 8, 2019

@hook-s3c I just tried go clean -modcache && go install and it works fine for me now (go1.12.5 darwin/amd64)

@bep

This comment has been minimized.

Copy link
Member Author

commented Jun 8, 2019

An added note, compared to 0.55.6:

~/go/pkg: 174M => 497M
hugo binary: 25M => 39M

@bep bep modified the milestones: v0.56, v0.57 Jun 14, 2019

@bep bep modified the milestones: v0.57, v0.58 Jul 31, 2019

@temoto

This comment has been minimized.

Copy link

commented Aug 4, 2019

Hello. Sorry for maybe unpleasant question. Could you name a recommended last Hugo version before this amqp nonsense started creeping into static website generator? I could git bisect repo for less dependencies, but of course you would know better about bugfixes. Don't need any features above render template, include header/footer and os.Exec after success.

Thanks for work on Hugo.

@bep bep modified the milestones: v0.58, v0.59 Aug 13, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.