Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: build argument to disable readonly $GOPATH/pkg/mod #31481

Open
Psykar opened this issue Apr 16, 2019 · 9 comments

Comments

Projects
None yet
6 participants
@Psykar
Copy link

commented Apr 16, 2019

What did you do?

This has been discussed previously:
#27161

Most build systems (buildroot et al.) expect to be able to do an rm -rf to do a completely fresh build. However, because the pkg sets it's containing directories to read only, this command will fail.

Requiring a custom command to be added to various Makefile's scattered around the place to clean this up is far from ideal.

A potential solution might be to add an environment variable to disable this read only permission change on the pkg directory.
Open to other alternatives - but ideally there needs to be a command on the BUILD side that allows rm -rf to work correctly.

Psykar added a commit to Psykar/buildroot that referenced this issue Apr 16, 2019

Makefile: add chmod before rm when cleaning.
Some build systems (looking at you golang) create read only directories
as caches.
As such rm -rf will actually fail, causing clean and <pkg>-dirclean to fail.

This patch will cause `make clean` to run chmod -R +w on the relevant
directory first, which will allow rm -rf to work.

This may be resolved if golang/go#31481 is
resolved satisfactorily.

Signed-off-by: Louis des Landes <louis.deslandes@fleet.space>

Psykar added a commit to Psykar/buildroot that referenced this issue Apr 16, 2019

package/pkg-generic: add chmod when cleaning.
Some build systems (looking at you golang) create read only directories
as caches.
As such rm -rf will actually fail, causing clean and <pkg>-dirclean to fail.

This patch will cause `make <pkg>-dirclean` to force chmod -R +w on the relevant
directory first, which will allow rm -rf to work.

This may be resolved if golang/go#31481 is
resolved satisfactorily.

Signed-off-by: Louis des Landes <louis.deslandes@fleet.space>

Psykar added a commit to Psykar/buildroot that referenced this issue Apr 16, 2019

Makefile: add chmod before rm when cleaning.
Some build systems (looking at you golang) create read only directories
as caches.
As such rm -rf will actually fail, causing clean and <pkg>-dirclean to fail.

This patch will cause `make clean` to run chmod -R +w on the relevant
directory first, which will allow rm -rf to work.

This may be resolved if golang/go#31481 is
resolved satisfactorily.

Signed-off-by: Louis des Landes <louis.deslandes@fleet.space>

Psykar added a commit to Psykar/buildroot that referenced this issue Apr 16, 2019

package/pkg-generic: add chmod when cleaning.
Some build systems (looking at you golang) create read only directories
as caches.
As such rm -rf will actually fail, causing clean and <pkg>-dirclean to fail.

This patch will cause `make <pkg>-dirclean` to force chmod -R +w on the relevant
directory first, which will allow rm -rf to work.

This may be resolved if golang/go#31481 is
resolved satisfactorily.

Signed-off-by: Louis des Landes <louis.deslandes@fleet.space>
@agnivade

This comment has been minimized.

Copy link
Member

commented Apr 16, 2019

#27161 clearly mentions that it is intentional and will remain this way.

It would be good if your proposal addresses the issue raised by @rsc here.

@dmitshur dmitshur changed the title Build argument to disable readonly $GOPATH/pkg/mod cmd/go: Build argument to disable readonly $GOPATH/pkg/mod Apr 16, 2019

@Psykar

This comment has been minimized.

Copy link
Author

commented Apr 17, 2019

@agnivade - the default being read only makes complete sense to me, I'm not suggesting changing that.
I would have thought a manual flag to allow the directories to be writeable would be acceptable - to tell golang that I know what I'm doing, it's OK to not lock the directory down?

It's worth noting in this kind of scenario (embedded build systems), tests are not going to be run as part of the build process, and the whole build directory (which includes this cache directory) frequently gets removed entirely.

So we're looking for a way to allow rm -rf to work on the $GOPATH/pkg/mod directory while still blocking tests from being able to write files to this directory? The two seem mutually exclusive to me unfortunately, hence suggesting the manual flag.

(As an aside, happy to do the PR for this if it's acceptable)

@agnivade

This comment has been minimized.

Copy link
Member

commented Apr 17, 2019

Marking it as a proposal.

@agnivade agnivade changed the title cmd/go: Build argument to disable readonly $GOPATH/pkg/mod proposal: cmd/go: build argument to disable readonly $GOPATH/pkg/mod Apr 17, 2019

@gopherbot gopherbot added this to the Proposal milestone Apr 17, 2019

@gopherbot gopherbot added the Proposal label Apr 17, 2019

@andybons andybons added the modules label Apr 17, 2019

@bcmills

This comment has been minimized.

Copy link
Member

commented Apr 19, 2019

You can already run chmod u+w -R $GOPATH/pkg/mod or go clean -modcache explicitly before cleaning up.

Why would an explicit argument be more helpful than an explicit command? Seems like you have to update your configuration either way.

@pbrkr

This comment has been minimized.

Copy link

commented Apr 29, 2019

@bcmills: The problem for us is that we can't modify our CI system to run those commands at the start of a build. The fault here is 100% with the golang compiler, it has no valid case for removing the write bit from files & directories in the build tree.

Running git clean -xffd (which the CI system does) should be sufficient to fully clean up a build directory.

@bcmills bcmills removed the WaitingForInfo label Apr 29, 2019

@bcmills

This comment has been minimized.

Copy link
Member

commented Apr 29, 2019

@paul-betafive, how are you able to add arguments to the go command but not able to add a chmod or go clean command? Could you give some more detail on the specifics of your CI system?

@pbrkr

This comment has been minimized.

Copy link

commented Apr 29, 2019

We can add a chmod command after the go build command but that isn't water-tight as the build may fail or crash during go build after directories have already been marked as read-only. If the build server loses power or has a hard crash during go build then no error handling code will be able to run the chmod command.

We're using GitLab CI which unconditionally uses git clean after checkout to remove old build files. There's no option to add commands between the checkout and the clean.

@bcmills bcmills removed the WaitingForInfo label May 7, 2019

@bcmills

This comment has been minimized.

Copy link
Member

commented May 7, 2019

We're using GitLab CI which unconditionally uses git clean after checkout to remove old build files. There's no option to add commands between the checkout and the clean.

Wait, why is your module cache in the same git repository as the thing you're testing in CI? That seems like a very unusual configuration, and if you split the two it would be a lot easier to decouple the go clean -modcache or chmod from the go build or git clean step.

@Psykar

This comment has been minimized.

Copy link
Author

commented May 9, 2019

@bcmills In my case at least it's not - buildroot (for example) build's it's output in a subdirectory of the repo however (which is under .gitignore)
Builds need to be reproducable without relying on any system packages (part of the build process will download golang source, compile and install it to a subdirectory, then use this installation for compiling the actual go packages which trigger this issue)

Having the module cache exist outside of the build directory means the build can potentially be affected by code outside of the repository.

For CI to a certain extend I agree that having the module cache actually be cached across builds makes a certain amount of sense.
But for buildroot (and similar) use case it does not.

FWIW the other side of the fence doesn't want to have to add a new command to their Makefile's specifically to handle a single package (golang) doing this.
http://lists.busybox.net/pipermail/buildroot/2019-April/248069.html

I'd like to find somewhere in the middle if possible!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.