Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: document if the go test and build cache is threadsafe #40461

Open
alvaroaleman opened this issue Jul 28, 2020 · 11 comments
Open

cmd/go: document if the go test and build cache is threadsafe #40461

alvaroaleman opened this issue Jul 28, 2020 · 11 comments

Comments

@alvaroaleman
Copy link

@alvaroaleman alvaroaleman commented Jul 28, 2020

What version of Go are you using (go version)?

$ go version
go version go1.15rc1 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env

What did you do?

go help cache

What did you expect to see?

A sentence telling me if the go build and test cache is safe for parallel use by multiple invocation of the go binary

What did you see instead?

@davecheney
Copy link
Contributor

@davecheney davecheney commented Jul 28, 2020

I don’t think it is

@ALTree
Copy link
Member

@ALTree ALTree commented Jul 29, 2020

I know for sure that cleaning the build cache (go cache -clean) while building something is not safe. This is discussed in #31948, which also functions as a TODO for possibly making it safe.

@ALTree ALTree changed the title Document if the go test and build cache is threadsafe cmd/go: document if the go test and build cache is threadsafe Jul 29, 2020
@ALTree
Copy link
Member

@ALTree ALTree commented Jul 29, 2020

@alvaroaleman
Copy link
Author

@alvaroaleman alvaroaleman commented Jul 29, 2020

I know for sure that cleaning the build cache (go cache -clean) while building something is not safe.

Okay but what about multiple parallel go build or go test invocations?

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Aug 3, 2020

Okay but what about multiple parallel go build or go test invocations?

This is safe on most file systems. Files are added to the cache atomically, and a global file lock is held when files are evicted. If the file system does not support file locking or atomic renaming, then it may not be safe.

@jayconrod jayconrod added this to the Backlog milestone Aug 3, 2020
@bcmills
Copy link
Member

@bcmills bcmills commented Aug 3, 2020

FWIW, the build and test cache does not use file locking, and I don't think it relies on atomic renaming either. (Instead, it writes the expected lengths in metadata files, and overwrites and appends to partial files in-place rather than truncating or deleting them.)

@bcmills
Copy link
Member

@bcmills bcmills commented Aug 3, 2020

I take that back: we use file-locking in one place in the test cache, to store the expiration date for stale tests.

// Read testcache expiration time, if present.
// (We implement go clean -testcache by writing an expiration date
// instead of searching out and deleting test result cache entries.)
if dir := cache.DefaultDir(); dir != "off" {
if data, _ := lockedfile.Read(filepath.Join(dir, "testexpire.txt")); len(data) > 0 && data[len(data)-1] == '\n' {

@networkimprov
Copy link

@networkimprov networkimprov commented Aug 3, 2020

@alvaroaleman are you hoping to do simultaneous go build/go test runs of the same project, or different ones, or partially overlapping ones?

@alvaroaleman
Copy link
Author

@alvaroaleman alvaroaleman commented Aug 3, 2020

@networkimprov the question came up in the context of "We prepopulate and then use one cache per machine that runs many concurrent builds"

@networkimprov
Copy link

@networkimprov networkimprov commented Aug 3, 2020

If prepopulated, is your cache read-only?

@alvaroaleman
Copy link
Author

@alvaroaleman alvaroaleman commented Aug 3, 2020

If prepopulated, is your cache read-only?

Its not my cache and I believe its not

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.