Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/buildlet: trim go1.4 tarballs #9797

Closed
bradfitz opened this issue Feb 7, 2015 · 6 comments
Closed

x/build/buildlet: trim go1.4 tarballs #9797

bradfitz opened this issue Feb 7, 2015 · 6 comments
Assignees
Milestone

Comments

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Feb 7, 2015

Not high priority, but there's a lot of stuff in our go1.4 tarballs that are untarred on builds which can be removed to reduce network & disk I/O and thus improve build latency:

api
misc
test
doc
**/testdata
**/*_test.go
bin/gofmt*  (exe or not)

Biggest things:

1388    ./pkg/tool/windows_amd64/6l.exe
1548    ./pkg/windows_amd64/net.a
1548    ./pkg/windows_amd64/runtime.a
1560    ./pkg/windows_amd64/cmd/internal/rsc.io
1604    ./pkg/tool/windows_amd64/6c.exe
1652    ./pkg/windows_amd64/net/http.a
1668    ./pkg/windows_amd64/text
1816    ./pkg/windows_amd64/debug
1924    ./pkg/obj/windows_amd64/libgc.a
1996    ./pkg/windows_amd64/cmd/internal
2016    ./pkg/windows_amd64/cmd/pprof/internal
2020    ./pkg/windows_amd64/cmd/pprof
2344    ./pkg/tool/windows_amd64/pack.exe
2712    ./pkg/windows_amd64/crypto
2784    ./pkg/windows_amd64/go
2808    ./pkg/tool/windows_amd64/6g.exe
2956    ./pkg/windows_amd64/encoding
3048    ./src/syscall
3436    ./pkg/tool/windows_amd64/yacc.exe
3528    ./pkg/tool/windows_amd64/fix.exe
3672    ./bin/gofmt.exe
3716    ./src/runtime
3788    ./pkg/obj/windows_amd64
3792    ./pkg/obj
3972    ./pkg/windows_amd64/net
4020    ./pkg/windows_amd64/cmd
4220    ./pkg/tool/windows_amd64/addr2line.exe
4236    ./pkg/tool/windows_amd64/nm.exe
4420    ./pkg/tool/windows_amd64/objdump.exe
4584    ./pkg/tool/windows_amd64/cgo.exe
5820    ./src/cmd
9396    ./bin/go.exe
9828    ./pkg/tool/windows_amd64/pprof.exe
13072   ./bin
21036   ./src
36408   ./pkg/windows_amd64
43444   ./pkg/tool/windows_amd64
43448   ./pkg/tool
83652   ./pkg
118472  .

Probably don't need pprof, objdump, nm, yacc, etc to bootstrap Go 1.5.

Do we even need the source? I forget when/why the go tool consults it. Did we ever add a mode to have it not look for source?

/cc @adg

@bradfitz bradfitz added the builder label Feb 7, 2015
@minux
Copy link
Member

@minux minux commented Feb 7, 2015

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Feb 7, 2015

There's a cost to just opening, writing, and closing a file, however small, since it needs to do discontiguous reads/writes for the inode metadata, etc. Eliminating a few thousand of them seems worthwhile.

@minux
Copy link
Member

@minux minux commented Feb 7, 2015

@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Feb 7, 2015

I did end up baking Go 1.4 into the Linux images, since they were easy to generate (with Docker).

But the VM images are slow enough to generate, I'd rather not bake in Go 1.4 in general, since it gives us more flexibility to not touch the VM image (the buildlet library supports either way, though).

I suppose to eliminate disk I/O (if we don't care about the network transfer speed, which I don't really), we could teach the buildlet to just skip over certain file entries when writing Go 1.4 to disk. That's probably easiest, and fixes the problem for all builders retroactively without regenerating their go1.4.*.tar.gz files on GCS.

@rsc rsc added this to the Unplanned milestone Apr 10, 2015
@rsc rsc changed the title buildlet: trim go1.4 tarballs x/build/buildlet: trim go1.4 tarballs Apr 14, 2015
@rsc rsc modified the milestones: Unreleased, Unplanned Apr 14, 2015
@rsc rsc removed the builder label Apr 14, 2015
@rsc rsc added the Builders label Jun 11, 2015
@bradfitz
Copy link
Contributor Author

@bradfitz bradfitz commented Apr 1, 2016

Notes on how I trimmed a bunch of stuff from the bootstrap.bash result:

rm -rf $(find . -name testdata)
rm -rf test
rm -rf api
rm -rf doc
rm bin/gen
rm bin/gofmt
cd pkg/tool/darwin_amd64 && rm addr2line api cgo cover doc fix nm objdump pack  pprof trace vet yacc 
rm -rf ./darwin_amd64/image
rm -rf ./darwin_amd64/database
rm -rf pkg/darwin_amd64/cmd
rm -rf misc/cgo/test*
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 7, 2016

CL https://golang.org/cl/21657 mentions this issue.

@golang golang locked and limited conversation to collaborators Apr 8, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.