Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: enable builder-specific GOPROXY settings on reverse builders #42971

Closed
bcmills opened this issue Dec 3, 2020 · 16 comments
Closed

x/build: enable builder-specific GOPROXY settings on reverse builders #42971

bcmills opened this issue Dec 3, 2020 · 16 comments
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@bcmills
Copy link
Member

bcmills commented Dec 3, 2020

Looking at the builder log in https://build.golang.org/log/26ea2fd1426a51a224b5d321915717e906ded84a, I see the following line. Note that it lists GOPROXY three different times and GOROOT_BOOTSTRAP twice:

:: Running /tmp/farm/go/bin/go with args ["/tmp/farm/go/bin/go" "test" "-short" "golang.org/x/crypto/..."] and env ["LANG=zh_CN.UTF-8" "PATH=/tmp/farm/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "HOME=/home/gopher" "LOGNAME=gopher" "USER=gopher" "SHELL=/bin/bash" "INVOCATION_ID=fe969678216d4cceaebbdc436df708c2" "JOURNAL_STREAM=8:4142335" "GO_BUILDER_ENV=host-linux-amd64-wsl" "WORKDIR=/tmp/farm" "HTTPS_PROXY=http://proxy:8123" "HTTP_PROXY=http://proxy:8123" "GOPROXY=https://goproxy.io,direct" "WSL_DISTRO_NAME=Ubuntu" "WSL_INTEROP=/run/WSL/20_interop" "WSLENV=BASH_ENV/u" "GO_STAGE0_NET_DELAY=700ms" "GO_STAGE0_DL_DELAY=600ms" "GOROOT_BOOTSTRAP=/tmp/farm/go1.4" "GO_BUILDER_NAME=linux-amd64-wsl" "GO_BUILDER_FLAKY_NET=1" "GOROOT_BOOTSTRAP=/usr/lib/go" "GOROOT=/tmp/farm/go" "GOPATH=/tmp/farm/gopath" "GOPROXY=http://10.240.0.3:30157" "GOPROXY=https://proxy.golang.org" "TMPDIR=/tmp/farm/tmp" "GOCACHE=/tmp/farm/gocache"] in dir /tmp/farm/gopath/src/golang.org/x/crypto

The second GOROOT_BOOTSTRAP seems to come from here, and might be working as intended: https://github.com/golang/build/blob/a9f75ee9992b28221ceec0c56af51265ea857ce2/cmd/buildlet/buildlet.go#L263

The second GOPROXY setting comes from here: https://github.com/golang/build/blob/5f1e306b98a112c236997d393adc5c11405377ca/cmd/coordinator/coordinator.go#L2702

The third GOPROXY setting comes from here: https://github.com/golang/build/blob/b6220de1adc000cad31eadf59558e1d3a8e32205/dashboard/builders.go#L921-L925


Given that one of the tests in that log failed with an EOF error communicating with proxy.golang.org, I suspect that it really should be using the local GOPROXY default (https://goproxy.io,direct) instead of the cmd/go default. I'm not sure what needs to happen to fix that configuration.

CC @mengzhuo @cagedmantis @golang/release

@bcmills bcmills added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Dec 3, 2020
@gopherbot gopherbot added this to the Unreleased milestone Dec 3, 2020
@bcmills bcmills changed the title x/build: environment configuration issue for linux-amd64-wsl builder x/build: GOPROXY environment configuration issue with linux-amd64-wsl builder Dec 3, 2020
@mengzhuo
Copy link
Contributor

mengzhuo commented Dec 4, 2020

Please assign this issue to me. @bcmills
p.s. Why can't I assign this issue to myself?

@mengzhuo
Copy link
Contributor

mengzhuo commented Dec 4, 2020

To other builders maintainer, linux-amd64-wsl using systemd to set the enviroment.

[Unit]
Description=Go Builder For WSL
After=network-online.target

[Service]
ExecStartPre=/bin/bash -c 'chmod -R u+w /tmp/farm; rm -rf /tmp/farm; /bin/mkdir -p /tmp/farm; true'
ExecStart=/home/gopher/go/bin/stage0
Environment=GO_BUILDER_ENV=host-linux-amd64-wsl
Environment=WORKDIR=/tmp/farm
Environment=HTTPS_PROXY=http://proxy:8123
Environment=HTTP_PROXY=http://proxy:8123
Environment=GOPROXY=https://goproxy.io,direct

User=gopher
WorkingDirectory=/home/gopher

Restart=always
RestartSec=5
StartLimitInterval=0

[Install]
WantedBy=multi-user.target

@dmitshur
Copy link
Contributor

dmitshur commented Dec 4, 2020

p.s. Why can't I assign this issue to myself?

@mengzhuo Take a look at https://golang.org/wiki/GitHubAccess#assignees.

@gopherbot
Copy link

gopherbot commented Dec 4, 2020

Change https://golang.org/cl/275332 mentions this issue: buildlet: check $workdir/go1.4 existance before append env

@gopherbot
Copy link

gopherbot commented Dec 4, 2020

Change https://golang.org/cl/275412 mentions this issue: cmd/coordinator,dashboard: add privateGoProxy setting

@oiooj
Copy link
Member

oiooj commented Jan 22, 2021

If anything else need to be change, I'd like to push it.

@bcmills bcmills changed the title x/build: GOPROXY environment configuration issue with linux-amd64-wsl builder x/build: enable builder-specific GOPROXY settings on reverse builders Feb 4, 2022
@bcmills
Copy link
Member Author

bcmills commented Feb 4, 2022

greplogs --dashboard -md -l -e 'https://proxy\.golang\.org.*: Service Unavailable' --since=2021-01-01

2022-02-04T14:01:19-77f5fa5-4afcc9f/linux-amd64-wsl
2022-02-02T23:37:18-6944b10-a9384ee/linux-amd64-wsl
2022-02-01T21:46:52-d1c84af-36b81ac/linux-riscv64-unmatched
2022-02-01T19:09:20-6944b10-e22a14b/linux-amd64-wsl
2022-01-27T16:35:59-03fcf44-6cbcf58/linux-amd64-wsl
2022-01-25T22:04:10-03971e3-38729cf/linux-mips64le-mengzhuo
2022-01-18T16:01:17-e26b9f0-378766a/linux-amd64-wsl
2022-01-05T23:17:27-416c73c-2b39d86/linux-amd64-wsl
2022-01-05T09:50:29-f2171f2-2c58bb2/linux-riscv64-unmatched
2022-01-04T21:07:03-c09c1d3-95b240b/linux-riscv64-unmatched
2022-01-04T16:08:05-cd4a9ba-95b240b/linux-riscv64-unmatched
2021-12-30T12:17:33-53d65af-6178d25/linux-riscv64-unmatched
2021-12-30T12:11:21-084fe85-6178d25/linux-riscv64-unmatched
2021-12-20T22:25:28-30823e0-7c94355/linux-amd64-wsl
2021-12-16T22:15:57-b76863e-d01cd8f/linux-amd64-wsl
2021-12-15T15:54:46-d92f387-de690c2/linux-amd64-wsl
2021-12-04T11:59:35-c33205f-1876b38/linux-amd64-wsl
2021-12-02T05:26:57-ae814b3-1b2930d/linux-amd64-wsl
2021-12-01T15:58:46-ae814b3-029dfbc/linux-amd64-wsl
2021-12-01T15:58:46-6944b10-029dfbc/linux-amd64-wsl
2021-12-01T13:12:25-eb34ac4-b7651e5/linux-amd64-wsl
2021-11-22T03:24:07-03fcf44-a287c4a/linux-amd64-wsl
2021-11-20T08:47:36-ae814b3-91abe4b/linux-amd64-wsl
2021-11-19T21:36:24-b76863e-aec5c2e/linux-amd64-wsl
2021-11-10T02:26:41-03fcf44-02d7eab/linux-riscv64-unmatched
2021-11-02T06:25:39-4a448f8-088bb4b/linux-mips64le-mengzhuo
2021-09-30T01:32:54-22bfdb6-6656269/linux-riscv64-unmatched
2021-09-06T14:02:24-ddb58f2-ecfff58/linux-amd64-wsl
2021-09-02T12:49:05-027fa76-08e2519/linux-amd64-wsl
2021-08-17T04:37:32-b64cdd1-a304273/linux-amd64-wsl
2021-08-16T18:44:48-5d84be3-d1ba047/linux-amd64-wsl
2021-08-16T18:44:48-0a44fdf-d1ba047/linux-amd64-wsl
2021-04-02T17:58:59-cd4b3ce-254948a/linux-amd64-wsl

@bcmills
Copy link
Member Author

bcmills commented Feb 4, 2022

@golang/release: this is a frequent failure mode on multiple reverse builders, and there has been a fix CL (CL 275412) out for review since December 2020. Can we prioritize a review for that change at some point in the near future?

@bcmills
Copy link
Member Author

bcmills commented Apr 18, 2022

@golang/release, any progress to share on this issue? As far as I can tell the CL from 2020 still hasn't been picked up, and I'm still seeing a lot of errors on the builders that look like they could be resolved by this:

greplogs --dashboard -md -l -e 'https://proxy\.golang\.org.*: Service Unavailable' --since=2022-02-04

2022-04-18T12:04:50-9780585-91b9915/linux-amd64-wsl
2022-04-18T12:04:50-46bc274-91b9915/linux-amd64-wsl
2022-04-18T12:04:50-39d8c21-91b9915/linux-amd64-wsl
2022-04-18T12:04:50-290c469-91b9915/linux-amd64-wsl
2022-04-16T11:17:39-506dc81-5e4543c/linux-mips64le-mengzhuo
2022-04-15T17:07:27-807ebe9-df2421d/linux-amd64-wsl
2022-04-14T15:54:48-fc48f9f-cb702a2/linux-mips64le-mengzhuo
2022-04-13T18:05:38-7838acb-346b18e/linux-amd64-wsl
2022-04-12T23:50:58-99f80d0-65558a4/linux-amd64-wsl
2022-04-12T23:24:32-fb223f7-9c7f0b1/linux-amd64-wsl
2022-04-12T18:02:22-caaab50-0b0d2fe/linux-amd64-wsl
2022-04-12T16:47:01-64c5db5-5bf6c97/linux-amd64-wsl
2022-04-12T15:47:54-64c5db5-45c3387/linux-amd64-wsl
2022-04-12T14:45:32-99f80d0-84264fc/linux-amd64-wsl
2022-04-12T14:42:58-e5f449a-2116d60/linux-amd64-wsl
2022-04-12T13:54:06-99f80d0-9ccf5b8/linux-amd64-wsl
2022-04-12T12:59:27-99f80d0-d6320f1/linux-amd64-wsl
2022-04-12T03:13:39-e5f449a-f2d9ab2/linux-amd64-wsl
2022-04-12T03:13:28-fc48f9f-e299381/linux-amd64-wsl
2022-04-12T02:13:10-99f80d0-a362d54/linux-amd64-wsl
2022-04-11T22:24:03-e5f449a-11c450f/linux-amd64-wsl
2022-04-11T16:34:30-1b505d1-1930977/linux-mips64le-mengzhuo
2022-04-07T20:39:32-5424468-6f6942e/linux-amd64-wsl
2022-04-06T22:27:40-03fcf44-9a6acc8/linux-amd64-wsl
2022-04-06T03:13:34-5424468-5bb2628/linux-amd64-wsl
2022-04-05T22:09:11-03fcf44-dbb52cc/linux-amd64-wsl
2022-04-05T17:00:25-a8550c1-65153e4/linux-amd64-wsl
2022-04-05T01:54:27-1983b93-07b0c57/linux-mips64le-mengzhuo
2022-04-05T01:32:49-1983b93-4aacb7f/linux-amd64-wsl
2022-04-04T04:01:17-03fcf44-a92ca51/linux-mips64le-mengzhuo
2022-04-03T15:43:15-5424468-85b5f86/linux-mips64le-mengzhuo
2022-03-31T15:59:22-92d54ef-5138401/linux-amd64-wsl
2022-03-19T23:49:55-723b81c-fa8efc1/linux-amd64-wsl
2022-03-19T23:49:55-3147a52-fa8efc1/linux-amd64-wsl
2022-03-19T23:49:55-3147a52-fa8efc1/linux-mips64le-mengzhuo
2022-03-15T00:19:35-b76863e-41a82aa/linux-mips64le-mengzhuo
2022-03-14T17:45:32-9d39a96-88be85f/linux-amd64-wsl
2022-03-08T21:35:38-03fcf44-dbbe4cc/linux-mips64le-mengzhuo
2022-03-07T23:22:26-9b9b3d8-079a027/linux-amd64-wsl
2022-02-15T01:01:15-9d39a96-dd7194b/linux-riscv64-unmatched
2022-02-11T00:30:56-9d39a96-30501bb/linux-amd64-wsl
2022-02-11T00:26:29-9d39a96-ca3fae1/linux-amd64-wsl
2022-02-11T00:26:18-9d39a96-11788aa/linux-amd64-wsl
2022-02-11T00:26:18-6944b10-11788aa/linux-amd64-wsl
2022-02-10T23:04:01-3450d37-cd6e0d7/linux-amd64-wsl
2022-02-10T22:20:18-d5e7b87-c4b87b8/linux-mips64le-mengzhuo
2022-02-10T22:20:18-cd36cc0-c4b87b8/linux-amd64-wsl
2022-02-10T22:20:18-bff2e91-c4b87b8/linux-mips64le-mengzhuo
2022-02-10T22:20:18-9d39a96-c4b87b8/linux-amd64-wsl
2022-02-10T22:20:18-9d39a96-c4b87b8/linux-mips64le-mengzhuo
2022-02-10T22:20:18-1a07ae6-c4b87b8/linux-amd64-wsl
2022-02-10T22:20:18-03fcf44-c4b87b8/linux-amd64-wsl
2022-02-10T21:40:49-9d39a96-99b61be/linux-amd64-wsl
2022-02-10T21:40:49-6944b10-99b61be/linux-amd64-wsl
2022-02-10T21:40:12-f4118a5-3b8c716/linux-amd64-wsl
2022-02-10T21:40:12-03971e3-3b8c716/linux-amd64-wsl
2022-02-10T19:08:19-d5e7b87-8ba3ad9/linux-mips64le-mengzhuo
2022-02-10T19:08:19-c0b9fb5-8ba3ad9/linux-mips64le-mengzhuo
2022-02-10T19:08:19-9d39a96-8ba3ad9/linux-mips64le-mengzhuo
2022-02-10T01:08:52-d3ed0bb-656d3f4/linux-riscv64-unmatched
2022-02-09T16:29:28-9d39a96-0a6cf87/linux-amd64-wsl
2022-02-04T14:01:19-77f5fa5-4afcc9f/linux-amd64-wsl

@dmitshur dmitshur moved this from Planned to In Progress in Go Release Team Apr 29, 2022
@dmitshur dmitshur self-assigned this Apr 29, 2022
gopherbot pushed a commit to golang/build that referenced this issue Apr 29, 2022
Builders behind firewall may not access to proxy.golang.org but has
its own private Go module proxy

linux-mips64le-mengzhuo
linux-amd64-wsl
linux-riscv64-unmatched

NeedsGoProxy is for long test used which we should not change.

Update golang/go#42971

Change-Id: If300b751487ae08e05f5200581eb4c2719706a56
Reviewed-on: https://go-review.googlesource.com/c/build/+/275412
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Lee Baokun <bk@golangcn.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
@dmitshur
Copy link
Contributor

dmitshur commented Apr 30, 2022

CL 275412 has now landed. Comparing a recent before/after log shows it's working as intended: there's no more unwanted "GOPROXY=https://proxy.golang.org" entry.

We can check again in a few weeks to make sure the same problem doesn't happen again for another reason, but at least so far there aren't any new hits (other than one yesterday, before the fix):

greplogs --dashboard -md -l -e 'https://proxy\.golang\.org.*: Service Unavailable' --since=2022-04-28
2022-04-28T16:00:39-dcaea06-17371ee/linux-amd64-wsl

@bcmills
Copy link
Member Author

bcmills commented May 2, 2022

Unfortunately, goproxy.io doesn't seem to be entirely reliable either. 😞

https://build.golang.org/log/58e02729ea56983be383b850ca96f268ba258e4b:

:: Running /tmp/workdir-host-linux-amd64-wsl/go/bin/go with args ["/tmp/workdir-host-linux-amd64-wsl/go/bin/go" "test" "-short" "golang.org/x/website/..."] and env ["LANG=zh_CN.UTF-8" "HOME=/home/gopher" "LOGNAME=gopher" "USER=gopher" "SHELL=/bin/bash" "INVOCATION_ID=4b8fac3e13e44f2ca8c6759ce9cfce12" "JOURNAL_STREAM=8:47829246" "GO_BUILDER_ENV=host-linux-amd64-wsl" "HTTPS_PROXY=http://uplink.golangcn.org:8123" "HTTP_PROXY=http://uplink.golangcn.org:8123" "GOPROXY=https://goproxy.io,direct" "WSL_DISTRO_NAME=Ubuntu" "WSL_INTEROP=/run/WSL/20_interop" "WSLENV=BASH_ENV/u" "GO_STAGE0_NET_DELAY=400ms" "GO_STAGE0_DL_DELAY=1.3s" "WORKDIR=/tmp/workdir-host-linux-amd64-wsl" "GO_BUILDER_NAME=linux-amd64-wsl" "GO_BUILDER_FLAKY_NET=1" "GOROOT_BOOTSTRAP=/usr/lib/go" "GOROOT=/tmp/workdir-host-linux-amd64-wsl/go" "GOPATH=/tmp/workdir-host-linux-amd64-wsl/gopath" "GO111MODULE=on" "TMPDIR=/tmp/workdir-host-linux-amd64-wsl/tmp" "GOCACHE=/tmp/workdir-host-linux-amd64-wsl/gocache" "PATH=/tmp/workdir-host-linux-amd64-wsl/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "PWD=/tmp/workdir-host-linux-amd64-wsl/gopath/src/golang.org/x/website"] in dir /tmp/workdir-host-linux-amd64-wsl/gopath/src/golang.org/x/website

…
cmd/events/main.go:16:2: github.com/microcosm-cc/bluemonday@v1.0.2: Get "https://proxy.golang.com.cn/github.com/microcosm-cc/bluemonday/@v/v1.0.2.zip": Service Unavailable

@mengzhuo
Copy link
Contributor

mengzhuo commented May 3, 2022

cc @oiooj as goproxy.io maintainer

@oiooj
Copy link
Member

oiooj commented May 3, 2022

I tested this request and there is no problem so far.

curl -v https://proxy.golang.com.cn/github.com/microcosm-cc/bluemonday/@v/v1.0.2.zip

and goproxy.io use Tencent Cloud global CDN service. 😢

@bcmills
Copy link
Member Author

bcmills commented May 3, 2022

greplogs -l -e 'https://(?:goproxy\.io|proxy.golang.com.cn).*(?:unexpected EOF|Service Unavailable|connection reset)' --since=2022-04-29
2022-05-03T12:34:17-fb9be63-a41e37f/linux-amd64-wsl
2022-05-02T19:53:56-fb9be63-3b01a80/linux-amd64-wsl
2022-04-29T18:37:11-fb9be63-8c5917c/linux-amd64-wsl

@bcmills
Copy link
Member Author

bcmills commented May 9, 2022

At any rate: it looks like the builder-specific GOPROXY settings are working as intended, so I'm going to close this issue.

I'll file a separate issue for the connection errors on the builders using goproxy.io.

@bcmills bcmills closed this as completed May 9, 2022
Go Release Team automation moved this from In Progress to Done May 9, 2022
@bcmills
Copy link
Member Author

bcmills commented May 9, 2022

(Filed the goproxy.io connectivity issue as #52793.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Archived in project
Development

No branches or pull requests

5 participants