cmd/go: no compilation error for invalid filenames in subdirectories with embed.FS

[dwrz]

What version of Go are you using (go version)?

$ go version

go version go1.18.4 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="auto"

GOARCH="amd64"

GOBIN=""

GOCACHE="/home/dwrz/.cache/go-build"

GOENV="/home/dwrz/.config/go/env"

GOEXE=""

GOEXPERIMENT=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="linux"

GOINSECURE=""

GOMODCACHE="/home/dwrz/.go/pkg/mod"

GONOPROXY=""

GONOSUMDB=""

GOOS="linux"

GOPATH="/home/dwrz/.go"

GOPRIVATE=""

GOPROXY="https://proxy.golang.org,direct"

GOROOT="/usr/lib/go"

GOSUMDB="sum.golang.org"

GOTMPDIR=""

GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"

GOVCS=""

GOVERSION="go1.18.4"

GCCGO="gccgo"

GOAMD64="v1"

AR="ar"

CC="gcc"

CXX="g++"

CGO_ENABLED="1"

GOMOD=""

GOWORK=""

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3921777299=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I am trying to use embed.FS to store photographs for a website. There are photographs with a colon in the filename (an RFC3339 formatted timestamp) which were not appearing on the site. The logs reported the following error:

failed to open: open static/media/1920/image_2020-10-07T12:36:46Z.jpg: file does not exist

On investigation, it appears that files with a colon in the filename are not included in an embed.FS. It appears that an error is generated at compilation if a filename with a colon exists in the embedded root directory, but not if the problematic filename is in a subdirectory. In the latter case, the file is simply not included in the embedded FS.

This came as a surprise to me, since (1) the limitation with colons is not listed in the embed package documentation, (2) fs.ValidPath is documented as follows:

// Paths containing other characters such as backslash and colon
// are accepted as valid, but those characters must never be
// interpreted by an FS implementation as path element separators.

To replicate, I set up a directory like the following:

.
├── go.mod
├── snippet.go
└── static
    ├── 2022-07-22T15:02:45Z.txt
    ├── foo.txt
    └── subdir
        ├── 2022-07-22T15:02:45Z.txt
        └── bar.txt

snippet.go contains the following:

package main

import (
	"embed"
	"fmt"
)

//go:embed static/*
var static embed.FS

func main() {
	dir, err := static.ReadDir("static/subdir")
	if err != nil {
		fmt.Println(err)
		return
	}

	for _, f := range dir {
		fmt.Println(f.Name())
	}
}

When compiled, this returns the following compilation error:

snippet.go:8:12: pattern static/*: cannot embed file static/2022-07-22T15:02:45Z.txt: invalid name 2022-07-22T15:02:45Z.txt

However, if the filename with colons is removed from the top level, the program compiles. It then outputs the following:

bar.txt

subdir/2022-07-22T15:02:45Z.txt is excluded, and no compilation error is reported.

What did you expect to see?

• Ideally, colons are supported in the filename.
• Alternatively, documentation stating that colons are not permitted in filenames for embed.FS.
• A compilation error when files in subdirectories have invalid filenames.

What did you see instead?

• No compilation error when a file with an invalid filename is in a subdirectory for embed.FS. This left me with the expectation that my files were included in the embed.FS, when in fact they were not.

[seankhliao]

Technically the restriction is implied by

Patterns must not match files outside the package's module

and if you try to download a module with a file containing colons it says

create zip: hello:world: malformed file path "hello:world": invalid char ':'

but it could be clearer.

cc @bcmills @matloob