embed: go1.16beta1 http.FileServer runtime error: invalid memory address or nil pointer dereference

[vomnes]

What version of Go are you using (go version)?

$ go version
go version go1.16beta1 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go1.16beta1 env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOOS="darwin"
GOPROXY="https://proxy.golang.org,direct"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOVCS=""
GOVERSION="go1.16beta1"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/fn/61cn31hn7r1bbd_hbpmfjqdm0000gp/T/go-build1615016027=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I'm using the "embed" package available with go1.16beta1.

I'm trying to implement a web server using http.ListenAndServe to render my Angular build Website.

embed was working fine until I replace the http.FS by http.Dir.
Now, http.Dir inside http.FileServer is working fine but http.FSand so embed are not working anymore.
I have a crash http: panic serving [::1]:50241: runtime error: invalid memory address or nil pointer dereference either calling localhost:8080 on my web browser or just need to wait few seconds to receive the crash from the server.

Reproduce the behaviour

Need to have a folder 'a' containing for example a file such as 'index.html'.

Step 1

package main

import (
	"embed"
	"net/http"
)

func main() {
        // go:embed a
	var server embed.FS
        http.ListenAndServe(":8080", http.FileServer(http.FS(server))) // <-- http.FS
}

go1.16beta1 run main.go

Step 2

package main

import "net/http"

func main() {
        http.ListenAndServe(":8080", http.FileServer(http.Dir("a"))) // <-- http.Dir
}

go1.16beta1 run main.go

Step 3

package main

import (
	"embed"
	"net/http"
)

func main() {
        // go:embed a
	var server embed.FS
        http.ListenAndServe(":8080", http.FileServer(http.FS(server))) // <-- http.FS
}

go1.16beta1 run main.go

Step 3

The server should not working anymore and return a crash.
http: panic serving [::1]:50241: runtime error: invalid memory address or nil pointer dereference
See below for more details.

What did you expect to see?

The file rendered in the web browser.

What did you see instead?

2021/01/14 00:08:00 http: panic serving [::1]:50138: runtime error: invalid memory address or nil pointer dereference
goroutine 5 [running]:
net/http.(*conn).serve.func1(0xc00007c000)
        /Users/username/sdk/go1.16beta1/src/net/http/server.go:1824 +0x153
panic(0x125dc80, 0x1446a90)
        /Users/username/sdk/go1.16beta1/src/runtime/panic.go:971 +0x47a
embed.FS.readDir(0x0, 0x12a16a5, 0x1, 0x14484a0, 0x160b280, 0x0)
        /Users/username/sdk/go1.16beta1/src/embed/embed.go:266 +0x26
embed.FS.Open(0x0, 0x12a16a5, 0x1, 0x8, 0x1484768, 0x64, 0x203000)
        /Users/username/sdk/go1.16beta1/src/embed/embed.go:285 +0xce
net/http.ioFS.Open(0x12f58e0, 0x0, 0xc0000160b4, 0x1, 0x24, 0x160b280, 0xc0000579c0, 0x100eedb)
        /Users/username/sdk/go1.16beta1/src/net/http/fs.go:760 +0x68
net/http.serveFile(0x12f8cd0, 0xc00018e000, 0xc000180000, 0x12f5ba0, 0xc000098cd0, 0xc0000160b4, 0x1, 0xc00007a001)
        /Users/username/sdk/go1.16beta1/src/net/http/fs.go:597 +0x84
net/http.(*fileHandler).ServeHTTP(0xc000098ce0, 0x12f8cd0, 0xc00018e000, 0xc000180000)
        /Users/username/sdk/go1.16beta1/src/net/http/fs.go:848 +0x9c
net/http.serverHandler.ServeHTTP(0xc0000fc000, 0x12f8cd0, 0xc00018e000, 0xc000180000)
        /Users/username/sdk/go1.16beta1/src/net/http/server.go:2887 +0xa3
net/http.(*conn).serve(0xc00007c000, 0x12f9100, 0xc00002a080)
        /Users/username/sdk/go1.16beta1/src/net/http/server.go:1952 +0x8ed
created by net/http.(*Server).Serve
        /Users/username/sdk/go1.16beta1/src/net/http/server.go:3013 +0x39b

[ianlancetaylor]

Marking as a release blocker for 1.16, as it seems that it might be a bug in the new go:embed support.

[bcmills]

CC @jayconrod @rsc

[andig]

3 looks identicial to 1. are you saying one works and one doesn‘t?

[vomnes]

3 looks identicial to 1. are you saying one works and one doesn‘t?

Yes, they are the same; It's just the steps to reproduce the bug.
Step 1: Work, Step 2: Work, Step 3: Doesn't work anymore

[jayconrod]

It looks like the embed.FS isn't initialized because there's a space between // and go:embed. They have to be together. This works:

package main

import (
	"embed"
	"net/http"
)

//go:embed a
var server embed.FS

func main() {
	http.ListenAndServe(":8080", http.FileServer(http.FS(server))) // <-- http.FS
}

(Note also that in 1.16, the //go:embed directive can only apply to global variables. Allowing local variables 1.16beta1 led to some unexpected behaviors.)

This seems like it might come up a lot. Maybe http could panic with a clearer message?

[bcmills]

Should it be a compile-time error to instantiate a value of type embed.FS without a corresponding go:embed directive?

If not, should the embed.FS methods explicitly check that the FS has been initialized, or should the zero embed.FS be a valid empty filesystem?

[bcmills]

Ah, I see that's addressed in the documentation:

When declared without a //go:embed directive, an FS is an empty file system.

So the fact that Open is panicking here seems to be out-of-spec.

[bcmills]

That said, I think at the very least go vet should emit a warning for package-level variables of type embed.FS that lack go:embed directives.

[jayconrod]

Should it be a compile-time error to instantiate a value of type embed.FS without a corresponding go:embed directive?

Not necessarily, especially for local variables and fields. It's essentially a reference type, so I could see copying it into a variable that's uninitialized.

So the fact that Open is panicking here seems to be out-of-spec.

That seems like the thing to fix. All the other methods call that.

That said, I think at the very least go vet should emit a warning for package-level variables of type embed.FS that lack go:embed directives.

+1 to that. Should that be a new check or part of something existing? I think other pragmas have the same comment restriction, and we might want to check them all.

[bcmills]

Filed the vet check as proposal #43698.

[gopherbot]

Change https://golang.org/cl/283852 mentions this issue: embed: treat uninitialized FS as empty

[vomnes]

Indeed, it's the space before go:embed that cause the problem without there are no crash anymore.