Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: directories named vendor are ignored in //go:embed #45744

Closed
jakelucas opened this issue Apr 24, 2021 · 5 comments
Closed

cmd/go: directories named vendor are ignored in //go:embed #45744

jakelucas opened this issue Apr 24, 2021 · 5 comments
Labels
Documentation Issues describing a change to documentation. NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@jakelucas
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.16.3 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=D:\projects\go\bin
set GOCACHE=D:\devenv\go\1.16.3\..\cache
set GOENV=C:\Users\\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=D:\projects\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=D:\projects\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=D:\devenv\go\1.16.3
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=D:\devenv\go\1.16.3\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.16.3
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=NUL
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\~1\AppData\Local\Temp\go-build3923578036=/tmp/go-build -gno-record-gcc-switches

What did you do?

Used //go:embed <dir> to recursively embed static website assets into an embed.FS variable, and then ran go install <repo> to install the binary into gobin.

What did you expect to see?

All assets embedded into the embed.FS variable in the final binary in gobin.

What did you see instead?

Only some assets were embedded in the embed.FS variable in the final binary in gobin.

The problem comes from having folders named vendor within the directory that gets embedded.

If you create a project that embeds a directory, for example //go:embed static, which includes some sub-directory named vendor, for example static/vendor/style.css, then the embed behaviour is different depending on how you install the project into gobin.

If you have the entire project source locally and you run go install within the project then the vendor directory is correctly included and embedded into the final binary.

If you push the project to somewhere like GitHub and then try to install via go install <repo> or go get <repo>, then all files are correctly embedded according to the documentation except for the entire vendor folder, which is ignored. You can work around this by setting GO111MODULE=auto. If this is set then the vendor directory is no longer ignored and is correctly embedded in the final binary.

I believe this is just an unintended side-effect of go get/go install and how they deal with vendored dependencies. If it is actually intended then it at the very least needs to be documented and made consistent across the different methods of installation.

@seankhliao seankhliao changed the title Directories named vendor are ignored in //go:embed cmd/go: directories named vendor are ignored in //go:embed Apr 24, 2021
@cherrymui cherrymui added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 26, 2021
@cherrymui cherrymui added this to the Backlog milestone Apr 26, 2021
@cherrymui
Copy link
Member

cc @bcmills @rsc

@rsc
Copy link
Contributor

rsc commented May 4, 2021

We can only embed files that are saved in the packaging of the module, and directories named vendor are excluded from the module. Therefore they need to be excluded from the embed as well.

@patrickbsf
Copy link

patrickbsf commented Aug 27, 2021

Could you just make sure it's at least mentioned in the go:embed documentation at least? This isn't obvious.

@bcmills bcmills added Documentation Issues describing a change to documentation. and removed Documentation Issues describing a change to documentation. labels Nov 1, 2021
@bcmills
Copy link
Contributor

bcmills commented Nov 1, 2021

This is currently mentioned in the embed package does, although it could stand to be clearer for vendor in particular.

Today (https://pkg.go.dev/embed@go1.17.2) it says:

Patterns must not match files outside the package's module, such as ‘.git/*’ or symbolic links. Matches for empty directories are ignored. After that, each pattern in a //go:embed line must match at least one file or non-empty directory.

Perhaps we are also missing a check to cause the build to error out if the pattern does match files in the vendor directory? That should be an explicit error — it shouldn't implicitly drop files that match, although things get a bit tricky when we consider glob patterns (which should just stop at module boundaries without erroring out).

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/633815 mentions this issue: embed: document exclusions more explicitly

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Dec 5, 2024
@dmitshur dmitshur modified the milestones: Backlog, Go1.24 Dec 5, 2024
@dmitshur dmitshur added the Documentation Issues describing a change to documentation. label Dec 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Documentation Issues describing a change to documentation. NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

7 participants