Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proxy.golang.org: fails to fetch github.com/gmlewis/go-fonts #67996

Closed
gmlewis opened this issue Jun 14, 2024 · 8 comments
Closed

proxy.golang.org: fails to fetch github.com/gmlewis/go-fonts #67996

gmlewis opened this issue Jun 14, 2024 · 8 comments
Assignees
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. proxy.golang.org
Milestone

Comments

@gmlewis
Copy link
Contributor

gmlewis commented Jun 14, 2024

What is the URL of the page with the issue?

https://pkg.go.dev/github.com/gmlewis/go-fonts@v0.16.0

What is your user agent?

[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36](https://explore.whatismybrowser.com/useragents/parse/?analyse-my-user-agent=yes)

Screenshot

bad-latest-2024-06-14_06-50-56

What did you do?

I visited https://pkg.go.dev/github.com/gmlewis/go-fonts@v0.16.0 multiple times over the course of about 15 minutes, each time asking for an update.
Eventually, that URL no longer said that that version couldn't be found, but instead started redirecting to the page with the old version.

What did you expect to see?

The latest version.

What did you see instead?

The old version.

@gabyhelp
Copy link

@seankhliao seankhliao changed the title x/pkgsite: https://pkg.go.dev/github.com/gmlewis/go-fonts@v0.16.0 x/pkgsite: newer versioned path redirects to older unversioned path Jun 14, 2024
@hyangah
Copy link
Contributor

hyangah commented Jun 18, 2024

According to https://proxy.golang.org/github.com/gmlewis/go-fonts/@v/list
proxy.golang.org knows only up to v0.0.13. Pkg.go.dev only learns about module versions through proxy.golang.org, so it is working as intended.

From the proxy's log, I see that proxy.golang.org keeps failing on this repo for unknown reasons.

cc @suzmue @golang/tools-team

@hyangah hyangah changed the title x/pkgsite: newer versioned path redirects to older unversioned path proxy.golang.org: fails to fetch github.com/gmlewis/go-fonts Jun 18, 2024
@hyangah hyangah added proxy.golang.org NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed pkgsite labels Jun 18, 2024
@adonovan
Copy link
Member

adonovan commented Jun 18, 2024

Indeed, it seems to be a problem with the origin server.

xtools$ GOPROXY=direct go mod download -x github.com/gmlewis/go-fonts@v0.16.0
mkdir -p /Users/adonovan/go/pkg/mod/cache/vcs # git3 https://github.com/gmlewis/go-fonts
# lock /Users/adonovan/go/pkg/mod/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51.lock
# /Users/adonovan/go/pkg/mod/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51 for git3 https://github.com/gmlewis/go-fonts
cd /Users/adonovan/go/pkg/mod/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git tag -l
0.016s # cd /Users/adonovan/go/pkg/mod/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git tag -l
cd /Users/adonovan/go/pkg/mod/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git ls-remote -q origin
0.128s # cd /Users/adonovan/go/pkg/mod/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git ls-remote -q origin
cd /Users/adonovan/go/pkg/mod/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2 --
0.011s # cd /Users/adonovan/go/pkg/mod/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2 --
cd /Users/adonovan/go/pkg/mod/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git -c protocol.version=2 fetch -f --depth=1 origin refs/tags/v0.16.0:refs/tags/v0.16.0
50.136s # cd /Users/adonovan/go/pkg/mod/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git -c protocol.version=2 fetch -f --depth=1 origin refs/tags/v0.16.0:refs/tags/v0.16.0
cd /Users/adonovan/go/pkg/mod/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' refs/tags/v0.16.0 --
0.011s # cd /Users/adonovan/go/pkg/mod/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' refs/tags/v0.16.0 --
cd /Users/adonovan/go/pkg/mod/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git cat-file blob 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2:go.mod
0.010s # cd /Users/adonovan/go/pkg/mod/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git cat-file blob 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2:go.mod
cd /Users/adonovan/go/pkg/mod/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git cat-file blob 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2:go.mod
0.009s # cd /Users/adonovan/go/pkg/mod/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git cat-file blob 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2:go.mod
cd /Users/adonovan/go/pkg/mod/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git cat-file blob 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2:go.mod
0.010s # cd /Users/adonovan/go/pkg/mod/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git cat-file blob 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2:go.mod
# get https://sum.golang.org/lookup/github.com/gmlewis/go-fonts@v0.16.0
# get https://sum.golang.org/lookup/github.com/gmlewis/go-fonts@v0.16.0: 500 Internal Server Error (58.136s)
go: github.com/gmlewis/go-fonts@v0.16.0: verifying go.mod: github.com/gmlewis/go-fonts@v0.16.0/go.mod: reading https://sum.golang.org/lookup/github.com/gmlewis/go-fonts@v0.16.0: 500 Internal Server Error

Disabling the sumdb, it runs to completion, albeit slowly:

xtools$ time GOMODCACHE=$(mktemp -d) GOSUMDB=off GOPROXY=direct go mod download -x github.com/gmlewis/go-fonts@v0.16.0
mkdir -p /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs # git3 https://github.com/gmlewis/go-fonts
# lock /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51.lock
mkdir -p /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51 # git3 https://github.com/gmlewis/go-fonts
cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git init --bare
0.028s # cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git init --bare
cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git remote add origin -- https://github.com/gmlewis/go-fonts
0.009s # cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git remote add origin -- https://github.com/gmlewis/go-fonts
cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git tag -l
0.008s # cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git tag -l
cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git ls-remote -q origin
0.152s # cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git ls-remote -q origin
cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2 --
0.013s # cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2 --
cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git -c protocol.version=2 fetch -f --depth=1 origin refs/tags/v0.16.0:refs/tags/v0.16.0
49.405s # cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git -c protocol.version=2 fetch -f --depth=1 origin refs/tags/v0.16.0:refs/tags/v0.16.0
cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' refs/tags/v0.16.0 --
0.017s # cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' refs/tags/v0.16.0 --
cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git cat-file blob 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2:go.mod
0.016s # cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git cat-file blob 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2:go.mod
cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git cat-file blob 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2:go.mod
0.011s # cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git cat-file blob 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2:go.mod
cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git cat-file blob 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2:go.mod
0.010s # cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git cat-file blob 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2:go.mod
cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git cat-file blob 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2:go.mod
0.011s # cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git cat-file blob 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2:go.mod
cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git cat-file blob 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2:go.mod
0.012s # cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git cat-file blob 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2:go.mod
cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git -c core.autocrlf=input -c core.eol=lf archive --format=zip --prefix=prefix/ 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2
34.982s # cd /var/folders/fy/dn6v01n16zjdwsqy_8qfbbxr000_9w/T/tmp.nEs0ebhRxI/cache/vcs/82e0d29a7e1123c37c06e94c9f9310af2c093c0854570f189fbc3358dc25cb51; git -c core.autocrlf=input -c core.eol=lf archive --format=zip --prefix=prefix/ 2c7b6b4f53ba603b21877e400d2b2c60fbcd2ef2

So it seems the upstream git fetch and git archive operations for this repo are so slow that they exceed the timeouts of the sumdb and the proxy. FWIW, the repo contains almost 1GB of font data and very little Go code.

@adonovan adonovan reopened this Jun 18, 2024
@seankhliao
Copy link
Member

is this the 500MB limit for proxy.golang.org?

@adonovan
Copy link
Member

Seems like sumdb is the bottlenneck, but it could be for a similar reason. I don't know, but I bet @suzmue does.

@suzmue
Copy link
Contributor

suzmue commented Jun 22, 2024

sum.golang.org and proxy.golang.org have the same limits and issues.

It appears that the proxy is not able to even successfully download the module in order to check the 500MB limit, but does seem that this has to do with the size as the fetch uses a lot of memory / takes a long time in the proxy's fetch environment.

@rsc
Copy link
Contributor

rsc commented Jun 28, 2024

I looked into this.

The repo itself is 1.2 GB after git clone --depth=1; 385 MB is the .git directory and 815 MB is the actual content.

The limit on the uncompressed size of an individual module is 500 MB (500<<20 bytes).

I see that there are many modules in this repo and each is small, but we also have to impose some limit on how large a git repo we are willing to download to get a module. It seems reasonable to me to say that this repo is too large. The solution would be to break it up into multiple modules.

Apologies, @gmlewis.

@gmlewis
Copy link
Contributor Author

gmlewis commented Jun 28, 2024

Thank you, everyone, who looked into this issue!

I really appreciate your effort and I'm sorry for the enormity of the repo.
I will break it up. That might be a cleaner solution anyway.

Closing with apologies.

@gmlewis gmlewis closed this as completed Jun 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. proxy.golang.org
Projects
None yet
Development

No branches or pull requests

8 participants