Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

embed: document go.mod exclusion #45197

Open
codefromthecrypt opened this issue Mar 24, 2021 · 8 comments
Open

embed: document go.mod exclusion #45197

codefromthecrypt opened this issue Mar 24, 2021 · 8 comments

Comments

@codefromthecrypt
Copy link

@codefromthecrypt codefromthecrypt commented Mar 24, 2021

We have a use case for serving example go projects via go:embed. This is particularly important as alternatives to go:embed have less maintenance now that they are obviated. The problems is that the file go.mod is special cased and prevents serving a directory that contains it.

@opennota did research here and found this behavior to be intentional, notable "stopping at module boundaries" #41191 (comment)

When serving static assets, module boundaries may not be important and in fact be a surprising limitation, as surprising as files outside the directory or via symbolic links might be. Right now, the latter constraints like symbolic links are documented, but go.mod, and why it doesn't work is not entirely obvious.

Ideally, there could be a way to allow go projects to serve go projects as static assets via go:embed. Until then, it would be handy to change the documentation to note this surprising constraint explicitly, as it would save folks time and head scratching.

Ex a line like this "Matches for a directory containing "go.mod" is invalid and fail to build".

What version of Go are you using (go version)?

1.16.2

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

amd64 darwin

What did you do?

//go:embed /path/to/goproject/*
var exfs embed.FS

What did you expect to see?

I expected to have go build not fail

What did you see instead?

An error like "pattern X: cannot embed directory Y in different module"

@Merovius
Copy link

@Merovius Merovius commented Mar 24, 2021

Note that the primary distribution mechanism of modules are zip-files, which will only include the files that are considered part of the module. As I understand it, the zip you download from the module proxy (as opposed to using the repo) won't contain the nested module. So, module boundaries are very important - you can't embed a file that isn't there.

A workaround for your case is to name the file differently - e.g. go.mod_ - and rename it after the fact by wrapping the embed.FS.

Loading

@jayconrod jayconrod changed the title Support or document go.mod exclusion for go:embed embed: document go.mod exclusion Mar 24, 2021
@jayconrod jayconrod added this to the Go1.17 milestone Mar 24, 2021
@carlmjohnson
Copy link
Contributor

@carlmjohnson carlmjohnson commented Mar 24, 2021

Name rewriting seems to come up a fair amount as a workaround for edge case problems with embedding. Someone should write a module to wrap an io.FS in a name rewriter. I volunteer to click the star button on Github and then link to it in comments when someone brings up an issue solved by rewriting, when someone else creates it.

Loading

codefromthecrypt added a commit to tetratelabs/func-e that referenced this issue Mar 25, 2021
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it.

go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step.

Signed-off-by: Adrian Cole <adrian@tetrate.io>
@cagedmantis
Copy link
Contributor

@cagedmantis cagedmantis commented Mar 25, 2021

/cc @rsc

Loading

codefromthecrypt added a commit to tetratelabs/func-e that referenced this issue Mar 26, 2021
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it.

go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step.

Signed-off-by: Adrian Cole <adrian@tetrate.io>
codefromthecrypt added a commit to tetratelabs/func-e that referenced this issue Mar 26, 2021
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it.

go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step.

Signed-off-by: Adrian Cole <adrian@tetrate.io>
codefromthecrypt added a commit to tetratelabs/func-e that referenced this issue Apr 3, 2021
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it.

go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step.
codefromthecrypt added a commit to tetratelabs/func-e that referenced this issue Apr 6, 2021
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it.

go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step.

Signed-off-by: Adrian Cole <adrian@tetrate.io>
codefromthecrypt added a commit to tetratelabs/func-e that referenced this issue Apr 6, 2021
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it.

go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step.

Signed-off-by: Adrian Cole <adrian@tetrate.io>
@codefromthecrypt
Copy link
Author

@codefromthecrypt codefromthecrypt commented Apr 6, 2021

one problem with the rename go.mod to go.mod_ approach is that running go mod tidy in the root, ends up importing whatever is in go.mod_. I'm not sure how to work around this, but it is annoying.

It would be nice to have a clean way to allow go:embed to opt-out of interpreting go.mod...

Loading

@Merovius
Copy link

@Merovius Merovius commented Apr 6, 2021

It would be nice to have a clean way to allow go:embed to opt-out of interpreting go.mod...

That can't work. It would mean your module can't be build from the zip-file uploaded to the mirror. Third-party embedding solutions don't have this problem only because they generate code which is compiled in - so they effectively create a copy (but in Go syntax) of the files from the nested module in the current module, so it's not a problem if the nested module isn't distributed.

This really can't be solved on the //go:embed side, as far as I can tell. It might be possible to add a way to distribute subdirectories with a go.mod without interpreting them as a module - I don't know. But as long as the files are not part of the module, //go:embed can't use them, full stop.

Loading

@codefromthecrypt
Copy link
Author

@codefromthecrypt codefromthecrypt commented Apr 6, 2021

great explanation, @Merovius. I didn't quite get it last time, but I do now, both the what and the why. awesome!

Loading

codefromthecrypt added a commit to tetratelabs/func-e that referenced this issue Apr 7, 2021
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it.

go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step.

Signed-off-by: Adrian Cole <adrian@tetrate.io>
codefromthecrypt added a commit to tetratelabs/func-e that referenced this issue Apr 7, 2021
…129)

This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it.

go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step.

Co-authored-by: Takeshi Yoneda <takeshi@tetrate.io>
Signed-off-by: Adrian Cole <adrian@tetrate.io>
@ianlancetaylor ianlancetaylor removed this from the Go1.17 milestone Apr 19, 2021
@ianlancetaylor ianlancetaylor added this to the Backlog milestone Apr 19, 2021
@crhntr
Copy link

@crhntr crhntr commented Aug 2, 2021

I ran into this and decided to use go:generate to tar the embedded module and then include the tape-archived file. It worked. https://github.com/crhntr/disaster-recovery-acceptance-tests/blob/embed-fixtures/fixtures/fixtures.go#L22-L28

My solution looked something like this:

repo/
  cmd/
    test-app/
      go.mod
      go.sum
      main.go
  fixtures/
  fixtures.go
  go.mod
  go.sum

In fixtures.go I have something like:

var (
	//go:embed fixtures
	fixtures embed.FS
)

//go:generate tar cf fixtures/test-app.tar ./cmd/test-app

I then commit the fixtures directory.

If there was a zip or txtar FS implementation I would use that instead but tar works. For my use case, I have a helper function to write the tar to the filesystem in a tmp dir for each test that needs the test app.

Loading

@carlmjohnson
Copy link
Contributor

@carlmjohnson carlmjohnson commented Aug 2, 2021

For the record, there are zip and txtar fs.FS implementations.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants