Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: no compilation error for invalid filenames in subdirectories with embed.FS #54003

Open
dwrz opened this issue Jul 22, 2022 · 1 comment
Labels
NeedsInvestigation

Comments

@dwrz
Copy link

@dwrz dwrz commented Jul 22, 2022

What version of Go are you using (go version)?

$ go version

go version go1.18.4 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env

GO111MODULE="auto"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/dwrz/.cache/go-build"
GOENV="/home/dwrz/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/dwrz/.go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/dwrz/.go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18.4"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3921777299=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I am trying to use embed.FS to store photographs for a website. There are photographs with a colon in the filename (an RFC3339 formatted timestamp) which were not appearing on the site. The logs reported the following error:

failed to open: open static/media/1920/image_2020-10-07T12:36:46Z.jpg: file does not exist

On investigation, it appears that files with a colon in the filename are not included in an embed.FS. It appears that an error is generated at compilation if a filename with a colon exists in the embedded root directory, but not if the problematic filename is in a subdirectory. In the latter case, the file is simply not included in the embedded FS.

This came as a surprise to me, since (1) the limitation with colons is not listed in the embed package documentation, (2) fs.ValidPath is documented as follows:

// Paths containing other characters such as backslash and colon
// are accepted as valid, but those characters must never be
// interpreted by an FS implementation as path element separators.

To replicate, I set up a directory like the following:

.
├── go.mod
├── snippet.go
└── static
    ├── 2022-07-22T15:02:45Z.txt
    ├── foo.txt
    └── subdir
        ├── 2022-07-22T15:02:45Z.txt
        └── bar.txt

snippet.go contains the following:

package main

import (
	"embed"
	"fmt"
)

//go:embed static/*
var static embed.FS

func main() {
	dir, err := static.ReadDir("static/subdir")
	if err != nil {
		fmt.Println(err)
		return
	}

	for _, f := range dir {
		fmt.Println(f.Name())
	}
}

When compiled, this returns the following compilation error:

snippet.go:8:12: pattern static/*: cannot embed file static/2022-07-22T15:02:45Z.txt: invalid name 2022-07-22T15:02:45Z.txt

However, if the filename with colons is removed from the top level, the program compiles. It then outputs the following:

bar.txt

subdir/2022-07-22T15:02:45Z.txt is excluded, and no compilation error is reported.

What did you expect to see?

  • Ideally, colons are supported in the filename.
  • Alternatively, documentation stating that colons are not permitted in filenames for embed.FS.
  • A compilation error when files in subdirectories have invalid filenames.

What did you see instead?

  • No compilation error when a file with an invalid filename is in a subdirectory for embed.FS. This left me with the expectation that my files were included in the embed.FS, when in fact they were not.
@seankhliao seankhliao changed the title embed: no compilation error for invalid filenames in subdirectories with embed.FS cmd/go: no compilation error for invalid filenames in subdirectories with embed.FS Jul 23, 2022
@seankhliao
Copy link
Member

@seankhliao seankhliao commented Jul 23, 2022

Technically the restriction is implied by

Patterns must not match files outside the package's module

and if you try to download a module with a file containing colons it says

create zip: hello:world: malformed file path "hello:world": invalid char ':'

but it could be clearer.

cc @bcmills @matloob

@seankhliao seankhliao added the NeedsInvestigation label Jul 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation
Projects
None yet
Development

No branches or pull requests

2 participants