Bazel can't exclude directories

[girving]

According to https://groups.google.com/forum/#!topic/bazel-discuss/KBfaySoVEFs bazel has no way to exclude directories from the search for BUILD files. In the case of TensorFlow, the instructions for setting up bazel for Python development call for creating a _python_build file at the top level with a bunch of symlinks to other directories: https://www.tensorflow.org/versions/r0.7/get_started/os_setup.html#setting-up-tensorflow-for-development

Bazel then chokes whenever I try to build from the top level, since the symlinks point to directories with BUILD files. The _python_build directory is supposed to be purely for Python's benefit, so bazel should just ignore it.

This is mainly a feature request for ignoring a directory, but I'd also appreciate any suggestions for dodging around this limitation in bazel in the short term.

[girving]

Looks like protobuf has the same problem: once I remove _python_build I get

bazel build ...
INFO: Found 1066 targets...
INFO: From Compiling google/protobuf/python/google/protobuf/internal/api_implementation.cc:
google/protobuf/python/google/protobuf/internal/api_implementation.cc:31:10: fatal error: 'Python.h' file not found
#include <Python.h>
         ^
1 error generated.
ERROR: /Users/geoffreyi/tensorflow/google/protobuf/BUILD:520:1: C++ compilation of rule '//google/protobuf:internal/_api_implementation.so' failed: osx_gcc_wrapper.sh failed: error executing command external/bazel_tools/tools/cpp/osx_gcc_wrapper.sh '-D_FORTIFY_SOURCE=1' -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety -Wself-assign -fno-omit-frame-pointer '-std=c++0x' -iquote . -iquote ... (remaining 28 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
INFO: Elapsed time: 12.425s, Critical Path: 4.45s

where google/protobuf/python/google/protobuf is presumably another directory purely for Python's benefit that bazel shouldn't be seeing.

[hhclam]

See if this helps: #829

2016-03-24 13:18 GMT-04:00 Geoffrey Irving notifications@github.com:

According to
https://groups.google.com/forum/#!topic/bazel-discuss/KBfaySoVEFs bazel
has no way to exclude directories from the search for BUILD files. In the
case of TensorFlow, the instructions for setting up bazel for Python
development call for creating a _python_build file at the top level with
a bunch of symlinks to other directories:
https://www.tensorflow.org/versions/r0.7/get_started/os_setup.html#setting-up-tensorflow-for-development

Bazel then chokes whenever I try to build from the top level, since the
symlinks point to directories with BUILD files. The _python_build
directory is supposed to be purely for Python's benefit, so bazel should
just ignore it.

This is mainly a feature request for ignoring a directory, but I'd also
appreciate any suggestions for dodging around this limitation in bazel in
the short term.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#1083

[girving]

It doesn't seem to. Both of these commands still produce errors:

(master) shimura:tensorflow% bazel build --deleted_packages=//_python_build ...
ERROR: /Users/geoffreyi/tensorflow/_python_build/tensorflow/models/image/cifar10/BUILD:18:1: Target '//tensorflow/python:framework_test_lib' is not visible from target '//_python_build/tensorflow/models/image/cifar10:cifar10_input_test'. Check the visibility declaration of the former target if you think the dependency is legitimate.
ERROR: /Users/geoffreyi/tensorflow/_python_build/tensorflow/models/image/cifar10/BUILD:18:1: Target '//tensorflow/python:platform_test' is not visible from target '//_python_build/tensorflow/models/image/cifar10:cifar10_input_test'. Check the visibility declaration of the former target if you think the dependency is legitimate.
ERROR: Analysis of target '//_python_build/tensorflow/models/image/cifar10:cifar10_input_test' failed; build aborted.
INFO: Elapsed time: 0.452s

(master) shimura:tensorflow% bazel build --deleted_packages=_python_build ...
ERROR: /Users/geoffreyi/tensorflow/_python_build/tensorflow/models/image/cifar10/BUILD:18:1: Target '//tensorflow/python:framework_test_lib' is not visible from target '//_python_build/tensorflow/models/image/cifar10:cifar10_input_test'. Check the visibility declaration of the former target if you think the dependency is legitimate.
ERROR: /Users/geoffreyi/tensorflow/_python_build/tensorflow/models/image/cifar10/BUILD:18:1: Target '//tensorflow/python:platform_test' is not visible from target '//_python_build/tensorflow/models/image/cifar10:cifar10_input_test'. Check the visibility declaration of the former target if you think the dependency is legitimate.
ERROR: Analysis of target '//_python_build/tensorflow/models/image/cifar10:cifar10_input_test' failed; build aborted.
INFO: Elapsed time: 0.291s

[hhclam]

I'm using Bazel release 0.2 and --deleted_packages seems to work fine for
me.

$ mkdir test

$ cd test

$ touch WORKSAPCE BUILD

$ mkdir broken

$ echo "what" > broken/BUILD

$ bazel build ...

ERROR: /Users/alpha/Projects/test/broken/BUILD:1:1: name 'what' is not
defined.

ERROR: package contains errors: broken.

ERROR: no such package 'broken': Package 'broken' contains errors.

INFO: Elapsed time: 0.138s

$ bazel build ... --deleted_packages broken

INFO: Found 0 targets...

INFO: Elapsed time: 0.811s, Critical Path: 0.00s

2016-03-25 10:41 GMT-04:00 Geoffrey Irving notifications@github.com:

It doesn't seem to. Both of these command still produce errors:

(master) shimura:tensorflow% bazel build --deleted_packages=//_python_build ...
ERROR: /Users/geoffreyi/tensorflow/_python_build/tensorflow/models/image/cifar10/BUILD:18:1: Target '//tensorflow/python:framework_test_lib' is not visible from target '//_python_build/tensorflow/models/image/cifar10:cifar10_input_test'. Check the visibility declaration of the former target if you think the dependency is legitimate.
ERROR: /Users/geoffreyi/tensorflow/_python_build/tensorflow/models/image/cifar10/BUILD:18:1: Target '//tensorflow/python:platform_test' is not visible from target '//_python_build/tensorflow/models/image/cifar10:cifar10_input_test'. Check the visibility declaration of the former target if you think the dependency is legitimate.
ERROR: Analysis of target '//_python_build/tensorflow/models/image/cifar10:cifar10_input_test' failed; build aborted.
INFO: Elapsed time: 0.452s

(master) shimura:tensorflow% bazel build --deleted_packages=_python_build ...
ERROR: /Users/geoffreyi/tensorflow/_python_build/tensorflow/models/image/cifar10/BUILD:18:1: Target '//tensorflow/python:framework_test_lib' is not visible from target '//_python_build/tensorflow/models/image/cifar10:cifar10_input_test'. Check the visibility declaration of the former target if you think the dependency is legitimate.
ERROR: /Users/geoffreyi/tensorflow/_python_build/tensorflow/models/image/cifar10/BUILD:18:1: Target '//tensorflow/python:platform_test' is not visible from target '//_python_build/tensorflow/models/image/cifar10:cifar10_input_test'. Check the visibility declaration of the former target if you think the dependency is legitimate.
ERROR: Analysis of target '//_python_build/tensorflow/models/image/cifar10:cifar10_input_test' failed; build aborted.
INFO: Elapsed time: 0.291s

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#1083 (comment)

[aehlig]

--deleted_packages might ignore packages names starting with _

[payload]

I am currently running into this problem, because I depend on the node package grpc containing BUILD files.

[Globegitter]

I am currently running into the problem also. We have a mono-repository with an experimental folder that we want to exclude per default from everything (including ci/cd). Only if someone specifically runs bazel run/build/test experimental/... this should get run, built or tested.

So we can easily separate stuff that is well, experimental, does not have to be tested necessarily and should not be used outside of that folder and not get auto-deployed either or any other way interfere with any of our tooling. But you still want version control for this code and it may or may not move out of experimental at some stage.

I do not see how such a thing is currently possible without having a potentially huge deleted_packages and also it would be nice to have a possibility to put this into bazelrc (again, in a way where on bazel build ... it should ignore it but if I want to build the package specifically it works).

[Globegitter]

Some progress on this, with a local_repository setup in the main WORKSPACE file this can be worked around so that now I can now run bazel build //... which excludes the experimental folder and run bazel build @experimental//.... But it is a bit cumbersome, as macros/any targets shared between the two repos now need @// in front of any label. Also packages referenecs within the experimental folder itself need to be //path/to/dep:dep rather than //experimental/path/to/dep:dep.

Further the protoc_gen_swagger rule provided by https://github.com/grpc-ecosystem/grpc-gateway/ now fails with output 'external/experimental/.../api.swagger.json' was not created, not all outputs were created or valid. I imagine that is an issue specific to the rule but it nevertheless worked before.

Having something along the lines of #4888 (comment) would work better. Or if it should still go into the WORKSPACE file as @dslomov mentioned, how about just having a manual_directory or excluded_directory rule, that would take the same arguments as the local_repository repository rule, but would not requite another WORKSPACE file being present and also allow the targets still to define dependencies exactly as any other targets in the main repository.

[gholms]

We make heavy use of the experimental directory and have been carrying a patch to restore the original behaviour ever since it stopped being special. The external repository workaround that @Globegitter mentioned partially solves the issue by not making //... attempt to build things under experimental, but it still requires everything to have up-to-date BUILD files before the build graph will load. We cannot require people refactoring the main code base to also refactor everything in experimental.

[dpursehouse]

Can this be closed? See #4888

[jin]

@dpursehouse yes. This feature is now available with .bazelignore. https://docs.bazel.build/versions/master/guide.html#.bazelignore