new_http_archive can only find build_file in current workspace

[philsc]

First off, love bazel! Thank you guys for all your hard work. Switched to it at work and I can't believe how much better it is than Makefiles.

I tried using bazel to compile some rust code and followed the instructions from here:
https://github.com/bazelbuild/bazel/blob/master/tools/build_rules/rust/README.md

I cannot get this example to compile. I've got a sample project here: https://github.com/philsc/bazel-rust-tests

$ git clone https://github.com/philsc/bazel-rust-tests brt
Cloning into 'brt'...
remote: Counting objects: 8, done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 8 (delta 0), reused 8 (delta 0), pack-reused 0
Unpacking objects: 100% (8/8), done.
Checking connectivity... done.
$ cd brt/ && ls
hello_lib/  WORKSPACE
$ bazel build //hello_lib
......
ERROR: /home/phil/.bazel/base_workspace/tools/build_rules/rust/BUILD:35:1: no such package '@rust-linux-x86_64//': In new_http_archive rule //external:rust-linux-x86_64 the 'build_file' attribute does not specify an existing file (/home/phil/repos/brt/tools/build_rules/rust/rust.BUILD does not exist) and referenced by '//tools/build_rules/rust:rustlib'.
ERROR: Loading failed; build aborted.
INFO: Elapsed time: 137.540s
minitardis/phil [brt] (master) r1,j0,t138
$ 

This is my bazel verison:

$ bazel version
Build label: 0.1.0
Build target: bazel-out/local_linux-fastbuild/bin/src/main/java/bazel-main_deploy.jar
Build time: Tue Sep 8 23:08:34 2015 (1441753714)
Build timestamp: 1441753714
Build timestamp as int: 1441753714
$ 

Am I misunderstanding something or is there something missing from the README?

[davidzchen]

@kchodorow - Is this also due to load() currently taking local paths only, such as mentioned in #429?

[kchodorow]

You are hitting a bug where new_http_archive only looks in your workspace for the build_file. Workaround: copy /usr/local/lib/bazel/base_workspace/tools/build_rules/rust/rust.BUILD to /home/phil/repos/brt/tools/build_rules/rust/rust.BUILD. You don't have to copy anything else into your tree, just the file used by build_file.

I don't think there's actually an issue filed for this yet, so I'll use this one to track.

[davidzchen]

Thanks for the explanation, Kristina. I have updated the bug title and reassigned it to you.

[achew22]

How are people working around this issue? Even when I copy rules out of the bazel repo and into my tools/build_rules directory I still get the following error:

➜  golang  blaze run //grpc-gateway:main
ERROR: /tmp/golang/tools/build_rules/go/toolchain/BUILD:15:1: no such package '@golang-darwin-amd64//': In new_http_archive rule //external:golang-darwin-amd64 the 'build_file' attribute does not specify an existing file (/tmp/golang/tools/build_rules/go/toolchain/BUILD.go-toolchain does not exist) and referenced by '//tools/build_rules/go/toolchain:toolchain'.
ERROR: Loading failed; build aborted.
INFO: Elapsed time: 0.961s
ERROR: Build failed. Not running target.
➜  golang  cat /tmp/golang/tools/build_rules/go/toolchain/BUILD.go-toolchain
package(
  default_visibility = [ "//visibility:public" ])

filegroup(
  name = "toolchain",
  srcs = glob(["go/bin/*", "go/pkg/**", ]),
)

filegroup(
  name = "go_tool",
  srcs = [ "go/bin/go" ],
)

[AustinSchuh]

We've been running with the patch that Brian posted to revert to the original behavior. I'm sooo looking forwards to the new label based behavior.

[bsilver8192]

@achew22: Did you copy just the *.bzl files or the whole tools/build_rules directory? Did you try a bazel shutdown (Bazel caches things related to external repositories pretty aggressively)?

I'm pretty sure @AustinSchuh is referring to the workaround for #429, which is related but different. This bug is about the build_file referred to by a new_* rule having to come from the main workspace, while #429 is about load statements in those files bringing in Skylark code.

[achew22]

Here is a complete reproduction of my process. Starting in $HOME/Projects/. My ZSH is configured to show the arrow (colored with the last status code) then the name of the folder then the command. So, ➜ Projects echo means I'm in a folder called "Projects" and I'm running the command echo

I created a github repo of the folders so if you're not interested in reading scrollback on this, check that out: https://github.com/achew22/bazelgolang

➜  Projects  git clone https://github.com/bazelbuild/bazel
Cloning into 'bazel'...
remote: Counting objects: 57063, done.
remote: Compressing objects: 100% (94/94), done.
remote: Total 57063 (delta 31), reused 0 (delta 0), pack-reused 56951
Receiving objects: 100% (57063/57063), 89.65 MiB | 8.85 MiB/s, done.
Resolving deltas: 100% (35878/35878), done.
Checking connectivity... done.
➜  Projects  cd bazel
➜  bazel git:(master) git log HEAD^..HEAD | cat -
commit 420804811ddb7798a02c5eb9fd2be1c253799f9d
Author: Michajlo Matijkiw <michajlo@google.com>
Date:   Fri Nov 20 20:37:05 2015 +0000

    Extract Attribute serialization logic

    --
    MOS_MIGRATED_REVID=108368348
➜  bazel git:(master) ./compile.sh
INFO: You can skip this first step by providing a path to the bazel binary as second argument:
INFO:    ./compile.sh build /path/to/bazel
🍃  Building Bazel from scratch............
🍃  Building Bazel with Bazel.
.Extracting Bazel installation...
...........
INFO: Found 1 target...
Target //src:bazel up-to-date:
  bazel-bin/src/bazel
INFO: Elapsed time: 43.075s, Critical Path: 35.55s

🍃  Setting up base workspace.
INFO: You already have a .bazelrc. Make sure it contains the  following package paths:

build --package_path %workspace%:/home/achew22/Projects/bazel/base_workspace
fetch --package_path %workspace%:/home/achew22/Projects/bazel/base_workspace
query --package_path %workspace%:/home/achew22/Projects/bazel/base_workspace

Build successful! Binary is here: /home/achew22/Projects/bazel/output/bazel
➜  bazel git:(master) mkdir /tmp/golang
➜  bazel git:(master) cp tools/build_rules/go/toolchain/WORKSPACE.go-toolchain /tmp/golang/WORKSPACE
➜  bazel git:(master) cat - <<EOF > /tmp/golang/BUILD
heredoc> load("/tools/build_rules/go/def", "go_prefix")
heredoc>
heredoc> go_prefix("github.com/joe/project")
heredoc> EOF
➜  bazel git:(master) mkdir /tmp/golang/lib
➜  bazel git:(master) cat - <<EOF > /tmp/golang/lib/BUILD
heredoc> load("/tools/build_rules/go/def", "go_library")
heredoc>
heredoc> go_library(
heredoc>     name = "go_default_library",
heredoc>     srcs = ["file.go"]
heredoc> )
heredoc> EOF
➜  bazel git:(master) cat - <<EOF > /tmp/golang/lib/file.go
heredoc> package lib
heredoc>
heredoc> func SayHello() string {
heredoc>        return "Hello"
heredoc> }
heredoc> EOF
➜  bazel git:(master) mkdir /tmp/golang/src
➜  bazel git:(master) cat - <<EOF > /tmp/golang/src/BUILD
heredoc> load("/tools/build_rules/go/def", "go_binary")
heredoc>
heredoc> go_binary(
heredoc>     name = "test",
heredoc>     srcs = ["main.go"],
heredoc>     deps = ["//lib:go_default_library"],
heredoc> )
heredoc> EOF
➜  bazel git:(master) cat - <<EOF > /tmp/golang/src/main.go
heredoc> package main
heredoc>
heredoc> import "fmt"
heredoc> import "github.com/joe/project/lib"
heredoc>
heredoc> func main() {
heredoc>        fmt.Println(lib.SayHello())
heredoc> }
heredoc> EOF

Now I will try building without the tools directory copied in

➜  golang  $HOME/Projects/bazel/output/bazel build //src:test
Extracting Bazel installation...
........
ERROR: /home/achew22/tools/bazel/base_workspace/tools/build_rules/go/toolchain/BUILD:24:1: no such package '@golang-linux-amd64//': In new_http_archive rule //external:golang-linux-amd64 the 'build_file' attribute does not specify an existing file (/tmp/golang/tools/build_rules/go/toolchain/BUILD.go-toolchain does not exist) and referenced by '//tools/build_rules/go/toolchain:go_tool'.
ERROR: Loading failed; build aborted.
INFO: Elapsed time: 13.168s

Now just copy the whole tools directory in to see what happens

➜  golang  cp -r $HOME/Projects/bazel/tools .
➜  golang  cat /tmp/golang/tools/build_rules/go/toolchain/BUILD.go-toolchain
package(
  default_visibility = [ "//visibility:public" ])

filegroup(
  name = "toolchain",
  srcs = glob(["go/bin/*", "go/pkg/**", ]),
)

filegroup(
  name = "go_tool",
  srcs = [ "go/bin/go" ],
)
➜  golang  $HOME/Projects/bazel/output/bazel build //src:test
ERROR: /tmp/golang/tools/build_rules/go/toolchain/BUILD:15:1: no such package '@golang-linux-amd64//': In new_http_archive rule //external:golang-linux-amd64 the 'build_file' attribute does not specify an existing file (/tmp/golang/tools/build_rules/go/toolchain/BUILD.go-toolchain does not exist) and referenced by '//tools/build_rules/go/toolchain:toolchain'.
ERROR: Loading failed; build aborted.
INFO: Elapsed time: 1.056s
➜  golang  cat /tmp/golang/tools/build_rules/go/toolchain/BUILD.go-toolchain
package(
  default_visibility = [ "//visibility:public" ])

filegroup(
  name = "toolchain",
  srcs = glob(["go/bin/*", "go/pkg/**", ]),
)

filegroup(
  name = "go_tool",
  srcs = [ "go/bin/go" ],
)

Okay, that didn't work. Let's try removing tools and just bringing in the build_rules directory

➜  golang  rm -rf tools
➜  golang  ls tools
ls: cannot access tools: No such file or directory
➜  golang  mkdir tools
➜  golang  cp -r $HOME/Projects/bazel/tools/build_rules tools/
➜  golang  cat /tmp/golang/tools/build_rules/go/toolchain/BUILD.go-toolchain
package(
  default_visibility = [ "//visibility:public" ])

filegroup(
  name = "toolchain",
  srcs = glob(["go/bin/*", "go/pkg/**", ]),
)

filegroup(
  name = "go_tool",
  srcs = [ "go/bin/go" ],
)
➜  golang  ls tools
build_rules
➜  golang  $HOME/Projects/bazel/output/bazel build //src:test
ERROR: /tmp/golang/tools/build_rules/go/toolchain/BUILD:15:1: no such package '@golang-linux-amd64//': In new_http_archive rule //external:golang-linux-amd64 the 'build_file' attribute does not specify an existing file (/tmp/golang/tools/build_rules/go/toolchain/BUILD.go-toolchain does not exist) and referenced by '//tools/build_rules/go/toolchain:toolchain'.
ERROR: Loading failed; build aborted.
INFO: Elapsed time: 0.935s

Okay, let's try just copying in the golang specific rules

➜  golang  rm -rf tools
➜  golang  mkdir -p tools/build_rules
➜  golang  cp -r $HOME/Projects/bazel/tools/build_rules/go tools/build_rules/
➜  golang  ls tools/build_rules/go
BUILD  def.bzl  README.md  toolchain  tools
➜  golang  cat /tmp/golang/tools/build_rules/go/toolchain/BUILD.go-toolchain
package(
  default_visibility = [ "//visibility:public" ])

filegroup(
  name = "toolchain",
  srcs = glob(["go/bin/*", "go/pkg/**", ]),
)

filegroup(
  name = "go_tool",
  srcs = [ "go/bin/go" ],
)
➜  golang  $HOME/Projects/bazel/output/bazel build //src:test
ERROR: /tmp/golang/tools/build_rules/go/toolchain/BUILD:24:1: no such package '@golang-linux-amd64//': In new_http_archive rule //external:golang-linux-amd64 the 'build_file' attribute does not specify an existing file (/tmp/golang/tools/build_rules/go/toolchain/BUILD.go-toolchain does not exist) and referenced by '//tools/build_rules/go/toolchain:go_tool'.
ERROR: Loading failed; build aborted.
INFO: Elapsed time: 0.899s

Push it to github so you can see the state

➜  golang  git init
Initialized empty Git repository in /tmp/golang/.git/
➜  golang git:(master) ✗ git add .
➜  golang git:(master) ✗ git commit -m "Test go repo"
[master (root-commit) a1f3936] Test go repo
 14 files changed, 866 insertions(+)
 create mode 100644 BUILD
 create mode 100644 WORKSPACE
 create mode 100644 lib/BUILD
 create mode 100644 lib/file.go
 create mode 100644 src/BUILD
 create mode 100644 src/main.go
 create mode 100644 tools/build_rules/go/BUILD
 create mode 100644 tools/build_rules/go/README.md
 create mode 100644 tools/build_rules/go/def.bzl
 create mode 100644 tools/build_rules/go/toolchain/BUILD
 create mode 100644 tools/build_rules/go/toolchain/BUILD.go-toolchain
 create mode 100644 tools/build_rules/go/toolchain/WORKSPACE.go-toolchain
 create mode 100644 tools/build_rules/go/tools/BUILD
 create mode 100644 tools/build_rules/go/tools/generate_test_main.go
➜  golang git:(master) git push --force git@github.com:achew22/bazelgolang.git master
Counting objects: 22, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (20/20), done.
Writing objects: 100% (22/22), 8.08 KiB | 0 bytes/s, done.
Total 22 (delta 1), reused 0 (delta 0)
To git@github.com:achew22/bazelgolang.git
 * [new branch]      master -> master

I'm sort of at a loss for what to do from here. Could it be related to sandboxing or something? How can I help debug this?