dev: gen cgo does not make jemalloc.h available

[erikgrinaker]

I'm using gopls via VSCode on a gceworker (Linux). Often, it complains about not finding jemalloc.h, despite having run dev gen cgo go. If I run make buildshort, the problem goes away. Any idea what's going on here?

# github.com/cockroachdb/cockroach/pkg/server/status
pkg/server/status/runtime_jemalloc.go:21:11: fatal error: jemalloc/jemalloc.h: No such file or directory
 // #include <jemalloc/jemalloc.h>
           ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
# github.com/cockroachdb/cockroach/pkg/server/status
pkg/server/status/runtime_jemalloc.go:21:11: fatal error: jemalloc/jemalloc.h: No such file or directory
 // #include <jemalloc/jemalloc.h>
           ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
# github.com/cockroachdb/cockroach/pkg/server/status
pkg/server/status/runtime_jemalloc.go:21:11: fatal error: jemalloc/jemalloc.h: No such file or directory
 // #include <jemalloc/jemalloc.h>
           ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
# github.com/cockroachdb/cockroach/pkg/server/status
pkg/server/status/runtime_jemalloc.go:21:11: fatal error: jemalloc/jemalloc.h: No such file or directory
 // #include <jemalloc/jemalloc.h>
           ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
# github.com/cockroachdb/cockroach/pkg/server/status
pkg/server/status/runtime_jemalloc.go:21:11: fatal error: jemalloc/jemalloc.h: No such file or directory
 // #include <jemalloc/jemalloc.h>
           ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
# github.com/cockroachdb/cockroach/pkg/server/status
pkg/server/status/runtime_jemalloc.go:21:11: fatal error: jemalloc/jemalloc.h: No such file or directory
 // #include <jemalloc/jemalloc.h>
           ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
# github.com/cockroachdb/cockroach/pkg/server/status
pkg/server/status/runtime_jemalloc.go:21:11: fatal error: jemalloc/jemalloc.h: No such file or directory
 // #include <jemalloc/jemalloc.h>
           ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
: packages.Load error

Jira issue: CRDB-18204
Epic CRDB-17171

[rickystewart]

@erikgrinaker Can you cat pkg/server/status/zcgo_flags.go (after dev gen cgo)? There should be an -I flag in that file, can you verify that the directory exists and contains a file jemalloc/jemalloc.h?

[cucaroach]

I'll try to repro this and see if I can make any hay.

[erikgrinaker]

I can reproduce this by running make clean and then running a test via go test, both in VSCode and a regular shell:

$ go test -timeout 5m -run ^TestJemalloc$ github.com/cockroachdb/cockroach/pkg/server/status -count 1 -v -p 1
# github.com/cockroachdb/cockroach/pkg/server/status
pkg/server/status/runtime_jemalloc.go:21:11: fatal error: jemalloc/jemalloc.h: No such file or directory
 // #include <jemalloc/jemalloc.h>
           ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.

There should be an -I flag in that file, can you verify that the directory exists and contains a file jemalloc/jemalloc.h?

// #cgo CPPFLAGS: -I/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libjemalloc_linux/include -I/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libkrb5_linux/include
// #cgo LDFLAGS: -L/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libjemalloc_linux/lib -L/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libproj_linux/lib -L/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libkrb5_linux/lib

The include directory does not exist:

$ ls -l /home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libjemalloc_linux/include
ls: cannot access '/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libjemalloc_linux/include': No such file or directory

It looks like the hashed directory does not exist:

$ ls /home/erik/.cache/bazel/_bazel_erik/
cache  install

In fact, there is no jemalloc.h anywhere in ~/.cache/bazel:

$ find ~/.cache/bazel -name jemalloc.h
$ 

---

Now, after doing this, I pulled master again, ran dev doctor, and then did another dev gen cgo, and now it seems to be working:

// #cgo CPPFLAGS: -I/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libjemalloc_linux/include -I/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libkrb5_linux/include
// #cgo LDFLAGS: -L/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libjemalloc_linux/lib -L/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libproj_linux/lib -L/home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libkrb5_linux/lib

$ ls -l /home/erik/.cache/bazel/_bazel_erik/58b8816bb117734f0a0eaaef7fcef95b/execroot/com_github_cockroachdb_cockroach/external/archived_cdep_libjemalloc_linux/include
total 4
drwxrwxr-x 2 erik erik 4096 Aug  1 19:01 jemalloc

$ go test -timeout 5m -run ^TestJemalloc$ github.com/cockroachdb/cockroach/pkg/server/status -count 1 -v -p 1
=== RUN   TestJemalloc
--- PASS: TestJemalloc (0.00s)
PASS
ok  	github.com/cockroachdb/cockroach/pkg/server/status	0.053s

I don't know specifically how it gets into this broken state, but when it does, running dev gen cgo does not help. Not sure what made it work this time around. I'll poke around a bit the next time it happens, it tends to crop up every now and then.

[cucaroach]

I've seen this and looked at it but can't for the life of me figure it out, one "workaround" is to put "build --force_build_cdeps" in your .bazelrc and build and it will build it instead of downloading it from gs. What I haven't figured out is how the include gets dropped in the first place or how to repair it from the gs archive.

[erikgrinaker]

Thanks for looking into this @cucaroach. I wouldn't spend too much time on it, I can ping you again the next time it happens and we can poke at it live. I should be able to get it working again anyway.