embed: document go.mod exclusion

[codefromthecrypt]

We have a use case for serving example go projects via go:embed. This is particularly important as alternatives to go:embed have less maintenance now that they are obviated. The problems is that the file go.mod is special cased and prevents serving a directory that contains it.

@opennota did research here and found this behavior to be intentional, notable "stopping at module boundaries" #41191 (comment)

When serving static assets, module boundaries may not be important and in fact be a surprising limitation, as surprising as files outside the directory or via symbolic links might be. Right now, the latter constraints like symbolic links are documented, but go.mod, and why it doesn't work is not entirely obvious.

Ideally, there could be a way to allow go projects to serve go projects as static assets via go:embed. Until then, it would be handy to change the documentation to note this surprising constraint explicitly, as it would save folks time and head scratching.

Ex a line like this "Matches for a directory containing "go.mod" is invalid and fail to build".

What version of Go are you using (go version)?

1.16.2

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

amd64 darwin

What did you do?

//go:embed /path/to/goproject/*
var exfs embed.FS

What did you expect to see?

I expected to have go build not fail

What did you see instead?

An error like "pattern X: cannot embed directory Y in different module"

[Merovius]

Note that the primary distribution mechanism of modules are zip-files, which will only include the files that are considered part of the module. As I understand it, the zip you download from the module proxy (as opposed to using the repo) won't contain the nested module. So, module boundaries are very important - you can't embed a file that isn't there.

A workaround for your case is to name the file differently - e.g. go.mod_ - and rename it after the fact by wrapping the embed.FS.

[carlmjohnson]

Name rewriting seems to come up a fair amount as a workaround for edge case problems with embedding. Someone should write a module to wrap an io.FS in a name rewriter. I volunteer to click the star button on Github and then link to it in comments when someone brings up an issue solved by rewriting, when someone else creates it.

[cagedmantis]

/cc @rsc

[codefromthecrypt]

one problem with the rename go.mod to go.mod_ approach is that running go mod tidy in the root, ends up importing whatever is in go.mod_. I'm not sure how to work around this, but it is annoying.

It would be nice to have a clean way to allow go:embed to opt-out of interpreting go.mod...

[Merovius]

It would be nice to have a clean way to allow go:embed to opt-out of interpreting go.mod...

That can't work. It would mean your module can't be build from the zip-file uploaded to the mirror. Third-party embedding solutions don't have this problem only because they generate code which is compiled in - so they effectively create a copy (but in Go syntax) of the files from the nested module in the current module, so it's not a problem if the nested module isn't distributed.

This really can't be solved on the //go:embed side, as far as I can tell. It might be possible to add a way to distribute subdirectories with a go.mod without interpreting them as a module - I don't know. But as long as the files are not part of the module, //go:embed can't use them, full stop.

[codefromthecrypt]

great explanation, @Merovius. I didn't quite get it last time, but I do now, both the what and the why. awesome!

[crhntr]

I ran into this and decided to use go:generate to tar the embedded module and then include the tape-archived file. It worked. https://github.com/crhntr/disaster-recovery-acceptance-tests/blob/embed-fixtures/fixtures/fixtures.go#L22-L28

My solution looked something like this:

repo/
  cmd/
    test-app/
      go.mod
      go.sum
      main.go
  fixtures/
  fixtures.go
  go.mod
  go.sum

In fixtures.go I have something like:

var (
	//go:embed fixtures
	fixtures embed.FS
)

//go:generate tar cf fixtures/test-app.tar ./cmd/test-app

I then commit the fixtures directory.

If there was a zip or txtar FS implementation I would use that instead but tar works. For my use case, I have a helper function to write the tar to the filesystem in a tmp dir for each test that needs the test app.

[carlmjohnson]

For the record, there are zip and txtar fs.FS implementations.