Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: using proxy.golang.org directly is incompatible with builder's outbound network restriction #33928

Closed
bradfitz opened this issue Aug 29, 2019 · 7 comments

Comments

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Aug 29, 2019

Check out the failures on e.g. https://go-review.googlesource.com/c/build/+/191577 ...

https://storage.googleapis.com/go-build-log/0b9b152e/linux-amd64_c339c8eb.log

linux-amd64 at 0b9b152ee3c9a2ff079569d8d5a3a6982b1ae91d building build at dee4382cfa88a26484ad9afc880d5e21ca0c110d

:: Running /workdir/go/bin/go with args ["/workdir/go/bin/go" "test" "-short" "golang.org/x/build/..."] and env ["PATH=/workdir/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "HOSTNAME=buildlet-linux-stretch-rn7c6c4c7" "DEBIAN_FRONTEND=noninteractive" "HOME=/root" "USER=root" "GO_STAGE0_NET_DELAY=100ms" "GO_STAGE0_DL_DELAY=0s" "WORKDIR=/workdir" "GOROOT_BOOTSTRAP=/workdir/go1.4" "GO_BUILDER_NAME=linux-amd64" "GOROOT_BOOTSTRAP=/go1.4" "GO_DISABLE_OUTBOUND_NETWORK=1" "GOROOT=/workdir/go" "GOPATH=/workdir/gopath" "GOPROXY=http://10.240.0.129:30157" "GO111MODULE=on" "TMPDIR=/workdir/tmp" "GOCACHE=/workdir/gocache"] in dir /workdir/gopath/src/golang.org/x/build

...

go: finding grpc.go4.org v0.0.0-20170609214715-11d0a25b4919
go: finding github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1
go: finding github.com/google/go-querystring v1.0.0
go: finding golang.org/x/sys v0.0.0-20190502175342-a43fa875dd82
go: finding golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852
go: finding github.com/coreos/go-systemd v0.0.0-20181012123002-c6f51f82210d
build golang.org/x/build/app/appengine: cannot load google.golang.org/api/compute/v1: google.golang.org/api@v0.4.0: Get "https://storage.googleapis.com/proxy-golang-org-prod/3a4011dc517dc6d3-google.golang.org:api-v0.4.0.zip?Expires=1567105990&GoogleAccessId=gcs-urlsigner-prod%40golang-modproxy.iam.gserviceaccount.com&Signature=H6dQ47hBsB9eSDZcrmF9B%2F4t2v8NvSDIBVL9jwu%2BHDZHxj%2FdVvXRonTHKbF%2B5%2F5XbYTXV8cU%2BwKhEnx%2BfLmxqfcm3CugvFYWmV%2B9qnCENts4aL%2F4a6t3fd2T8%2FwJY75mo4r85%2FIiM0NzpoDUBaef1qPhnVq3u8eegvh5WcZKcXuQb9PXZHKsds%2FTIL2Gy48lGQsg11Gct1H45ycX6Reb2RnzIduOlM%2BVmbrSKB7lQ4MopftS6rlfzhex09p5Kc3AB0eZoUoT27VUE0kXht88dh2GGSirMDVJgD9zbFWmYNMR8OxgC45lyfcuJOAitfMjWL62ijK3sKynaqcRS%2FjV7A%3D%3D": dial tcp 74.125.126.128:443: connect: no route to host

Error: tests failed: exit status 1

That "connect: no route to host is because most of our builders are firewalled off so they don't accidentally depend on the network more than expected.

Did the Go module proxy start returning redirects to GCS?

/cc @andybons @toothrot @dmitshur @katiehockman @ianlancetaylor

@gopherbot gopherbot added this to the Unreleased milestone Aug 29, 2019
@gopherbot gopherbot added the Builders label Aug 29, 2019
@dmitshur
Copy link
Member

@dmitshur dmitshur commented Aug 29, 2019

Did the Go module proxy start returning redirects to GCS?

It seems to for some module versions:

$ curl -i 'https://proxy.golang.org/github.com/google/go-querystring/@v/v1.0.0.zip'
HTTP/2 200
[...]

[... actual zip bytes ...]
$ curl -i 'https://proxy.golang.org/google.golang.org/api/@v/v0.4.0.zip'
HTTP/2 302
[...]
location: https://storage.googleapis.com/proxy-golang-org-prod/3a4011dc517dc6d3-google.golang.org:api-v0.4.0.zip?Expires=1567161567&GoogleAccessId=gcs-urlsigner-prod%40golang-modproxy.iam.gserviceaccount.com&Signature=Tae2DfFrY2UncTN68%2BLskuY%2Bn1oBmYAs%2Fhj4YuNsxew0bmiZCWVqmwdiZchvTazpiT0lC2H7KZNz8yZOQUbqIhnOUyaB4wO7YBNaf3lQQUgIdvN4E%2BPoGsm5wXf7mBwj0PsjkTv5oCzPBsR%2Fek3fHIVgefOogSLVbsyLDt83bgIN11bGmG4pYVHuk4ZdJ2extai92aLiA4Mra2kfomKdrK3GI4lKybuaINfNi%2FBrqC64aR8WiqiYTWGYrFsDRKjvw8cEJhgmOc79iuUtdbk%2F%2Bx0DEZB%2FRyv2Fglc3dz4WQtlHpDhpoe4H0m%2FD8UKCaLY%2FdYEN%2BJH5tcbAolOIa5wDg%3D%3D
[...]

I think we'll need to open up the firewall just enough for GCS to get through.

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Aug 29, 2019

As far as I can tell, it's intended behavior on the proxy's side, it applies to some larger zips.

I think we'll need to open up the firewall just enough for GCS to get through.

This can be a first order fix. Afterwards, if we want to tighten the firewall more, we can go back to having our own builder-specific intermediate proxy to pass everything through that we can whitelist very narrowly.

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 29, 2019

Change https://golang.org/cl/192317 mentions this issue: dashboard: disable outbound network restriction

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Aug 29, 2019

Sent CL 192317 that implements the first step, the smallest change to return builders to a working state. It will give us more time to work on the next steps.

gopherbot pushed a commit to golang/build that referenced this issue Aug 29, 2019
Issue golang.org/issue/30612 implemented a restriction on outbound
network access to help catch unintended uses of internet in tests.

We're currently relying on GOPROXY=https://proxy.golang.org for running
tests in module mode in Go 1.13+, but proxy.golang.org can serve some
module zip files by 302 redirecting to other URLs. Those URLs may not
be accessible due to this builder network restriction.

Disable the outbound network restriction as the first step to return
builders to a working state. The next step will be implement a long
term solution, then we can re-enable the network restriction again.

Updates golang/go#33928
Updates golang/go#30612

Change-Id: I1a87b8d107df8806753117fc60a3e795e1e1b182
Reviewed-on: https://go-review.googlesource.com/c/build/+/192317
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@dmitshur
Copy link
Member

@dmitshur dmitshur commented Aug 29, 2019

The aforementioned CL is deployed, so this issue is mitigated for now. I'll remove the Soon label.

We can leave this issue open to discuss next steps.

This has caused issue #30612 to get re-opened, so we need to find a solution that will allow us to close that again.

@dmitshur dmitshur changed the title x/build: build failures fetching modules x/build: using proxy.golang.org directly is incompatible with builder's outbound network restriction Aug 29, 2019
@gopherbot
Copy link

@gopherbot gopherbot commented Aug 29, 2019

Change https://golang.org/cl/192341 mentions this issue: cmd/coordinator: follow module proxy redirects

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Aug 29, 2019

I sent https://golang.org/cl/192341 but sick baby woke up so I can't test it yet. But the fix will be close to that at least.

codebien added a commit to codebien/build that referenced this issue Nov 13, 2019
Issue golang.org/issue/30612 implemented a restriction on outbound
network access to help catch unintended uses of internet in tests.

We're currently relying on GOPROXY=https://proxy.golang.org for running
tests in module mode in Go 1.13+, but proxy.golang.org can serve some
module zip files by 302 redirecting to other URLs. Those URLs may not
be accessible due to this builder network restriction.

Disable the outbound network restriction as the first step to return
builders to a working state. The next step will be implement a long
term solution, then we can re-enable the network restriction again.

Updates golang/go#33928
Updates golang/go#30612

Change-Id: I1a87b8d107df8806753117fc60a3e795e1e1b182
Reviewed-on: https://go-review.googlesource.com/c/build/+/192317
Reviewed-by: Alexander Rakoczy <alex@golang.org>
codebien added a commit to codebien/build that referenced this issue Nov 13, 2019
And because this uses new-in-Go1.13 http.Header.Clone, restrict all
the files to Go 1.13+ and bump the Dockerfile Go version.

Fixes golang/go#33928

Change-Id: I147cc4465e393d2c34e59c70edf65f3901a5e132
Reviewed-on: https://go-review.googlesource.com/c/build/+/192341
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.