Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Is a directory" failure #77

Closed
excavador opened this issue Jan 12, 2018 · 19 comments
Closed

"Is a directory" failure #77

excavador opened this issue Jan 12, 2018 · 19 comments

Comments

@excavador
Copy link

Hello,

I tried to use buildfarm for build my project.
With notes from #76 it works, except following problem:

java.io.IOException: Is a directory
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:46)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:197)
	at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:159)
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
	at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
	at com.google.protobuf.ByteString.readChunk(ByteString.java:493)
	at com.google.protobuf.ByteString.readFrom(ByteString.java:467)
	at com.google.protobuf.ByteString.readFrom(ByteString.java:429)
	at build.buildfarm.worker.Worker.execute(Worker.java:247)
	at build.buildfarm.worker.Worker.lambda$start$0(Worker.java:167)
	at build.buildfarm.instance.stub.StubInstance.match(StubInstance.java:330)
	at build.buildfarm.worker.Worker.start(Worker.java:165)
	at build.buildfarm.worker.Worker.main(Worker.java:528)

rules_go declare some output as file (by "ctx.actions.declare_file"), but really returns the directory.
Same problem with our in-house rule for creating virtualenv.

It is hack, but it works locally.

Without this hash would be too complex to support tools like virtualenv, or golang compilter.

Can we allow behavior like this for buildfarm?

@werkt
Copy link
Collaborator

werkt commented Jan 12, 2018

tl;dr: no, dirs != files, but output_directories seem like the way to go

Ah, that's definitely a no-go.

There is, however, this:

https://github.com/googleapis/googleapis/blob/497074446009147599499db18141927bc649a561/google/devtools/remoteexecution/v1test/remote_execution.proto#L306

However its implementation in bazel 0.9 looks like this:

https://github.com/bazelbuild/bazel/blob/f4d58293c69e9359141576d2da88665dc63f2467/src/main/java/com/google/devtools/build/lib/remote/GrpcRemoteCache.java#L219

And I was very excited to find when I went looking for this, that output_directories now has an implementation in the latest unreleased bazel:

https://github.com/bazelbuild/bazel/blob/22c2f9a7722e8c8b7fdf8f5d30a40f1c4118e993/src/main/java/com/google/devtools/build/lib/remote/AbstractRemoteActionCache.java#L133

For which I will happily mirror an implementation in buildfarm. My guess on documentation for how to add an entry to the output_directories list is that it will also be supplied in 0.10. Thanks @ulfjack!

@excavador
Copy link
Author

If I use "declary_directory" instead of "declare_file" I receive following problem with symlink to system python (only under linux, under OSX everything works fine) (just local bazel, not farm)

Is it expected behavior or another bug in bazel?

ERROR: /home/jenkins/backend-ci/plato/BUILD.bazel:45:1: Error while validating output TreeArtifact File:[[/home/jenkins/.cache/bazel/_bazel_jenkins/cff2319697c0ff2c301458e924e0bef7/execroot/XXX]bazel-out/k8-dbg/bin]plato/tests.virtualenv : /home/jenkins/.cache/bazel/_bazel_jenkins/cff2319697c0ff2c301458e924e0bef7/execroot/XXX/bazel-out/k8-dbg/bin/plato/tests.virtualenv/lib/python3.6/config-3.6m-x86_64-linux-gnu (Operation not permitted)
ERROR: /home/jenkins/backend-ci/plato/BUILD.bazel:45:1: not all outputs were created or valid

[jenkins@ci-slave-1:Ireland backend-ci]$ ls -al /home/jenkins/.cache/bazel/_bazel_jenkins/cff2319697c0ff2c301458e924e0bef7/execroot/XXX/bazel-out/k8-dbg/bin/plato/tests.virtualenv/lib/python3.6/
total 44
dr-xr-xr-x.  5 jenkins jenkins  4096 Jan 12 19:27 .
dr-xr-xr-x.  3 jenkins jenkins    23 Jan 12 19:27 ..
lrwxrwxrwx.  1 jenkins jenkins    27 Jan 12 19:27 abc.py -> /usr/lib64/python3.6/abc.py
lrwxrwxrwx.  1 jenkins jenkins    30 Jan 12 19:27 base64.py -> /usr/lib64/python3.6/base64.py
lrwxrwxrwx.  1 jenkins jenkins    30 Jan 12 19:27 bisect.py -> /usr/lib64/python3.6/bisect.py
lrwxrwxrwx.  1 jenkins jenkins    35 Jan 12 19:27 _bootlocale.py -> /usr/lib64/python3.6/_bootlocale.py
lrwxrwxrwx.  1 jenkins jenkins    30 Jan 12 19:27 codecs.py -> /usr/lib64/python3.6/codecs.py
lrwxrwxrwx.  1 jenkins jenkins    32 Jan 12 19:27 collections -> /usr/lib64/python3.6/collections
lrwxrwxrwx.  1 jenkins jenkins    40 Jan 12 19:27 _collections_abc.py -> /usr/lib64/python3.6/_collections_abc.py
lrwxrwxrwx.  1 jenkins jenkins    49 Jan 12 19:27 config-3.6m-x86_64-linux-gnu -> /usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu
lrwxrwxrwx.  1 jenkins jenkins    28 Jan 12 19:27 copy.py -> /usr/lib64/python3.6/copy.py
lrwxrwxrwx.  1 jenkins jenkins    31 Jan 12 19:27 copyreg.py -> /usr/lib64/python3.6/copyreg.py
drwxr-xr-x.  3 jenkins jenkins    65 Jan 12 19:27 distutils
lrwxrwxrwx.  1 jenkins jenkins    37 Jan 12 19:27 _dummy_thread.py -> /usr/lib64/python3.6/_dummy_thread.py
lrwxrwxrwx.  1 jenkins jenkins    30 Jan 12 19:27 encodings -> /usr/lib64/python3.6/encodings
lrwxrwxrwx.  1 jenkins jenkins    28 Jan 12 19:27 enum.py -> /usr/lib64/python3.6/enum.py
lrwxrwxrwx.  1 jenkins jenkins    31 Jan 12 19:27 fnmatch.py -> /usr/lib64/python3.6/fnmatch.py
lrwxrwxrwx.  1 jenkins jenkins    33 Jan 12 19:27 functools.py -> /usr/lib64/python3.6/functools.py
lrwxrwxrwx.  1 jenkins jenkins    34 Jan 12 19:27 __future__.py -> /usr/lib64/python3.6/__future__.py
lrwxrwxrwx.  1 jenkins jenkins    35 Jan 12 19:27 genericpath.py -> /usr/lib64/python3.6/genericpath.py
lrwxrwxrwx.  1 jenkins jenkins    31 Jan 12 19:27 hashlib.py -> /usr/lib64/python3.6/hashlib.py
lrwxrwxrwx.  1 jenkins jenkins    29 Jan 12 19:27 heapq.py -> /usr/lib64/python3.6/heapq.py
lrwxrwxrwx.  1 jenkins jenkins    28 Jan 12 19:27 hmac.py -> /usr/lib64/python3.6/hmac.py
lrwxrwxrwx.  1 jenkins jenkins    30 Jan 12 19:27 importlib -> /usr/lib64/python3.6/importlib
lrwxrwxrwx.  1 jenkins jenkins    27 Jan 12 19:27 imp.py -> /usr/lib64/python3.6/imp.py
lrwxrwxrwx.  1 jenkins jenkins    26 Jan 12 19:27 io.py -> /usr/lib64/python3.6/io.py
lrwxrwxrwx.  1 jenkins jenkins    31 Jan 12 19:27 keyword.py -> /usr/lib64/python3.6/keyword.py
lrwxrwxrwx.  1 jenkins jenkins    32 Jan 12 19:27 lib-dynload -> /usr/lib64/python3.6/lib-dynload
lrwxrwxrwx.  1 jenkins jenkins    33 Jan 12 19:27 linecache.py -> /usr/lib64/python3.6/linecache.py
lrwxrwxrwx.  1 jenkins jenkins    30 Jan 12 19:27 locale.py -> /usr/lib64/python3.6/locale.py
-rw-r--r--.  1 jenkins jenkins     0 Jan 12 19:27 no-global-site-packages.txt
lrwxrwxrwx.  1 jenkins jenkins    30 Jan 12 19:27 ntpath.py -> /usr/lib64/python3.6/ntpath.py
lrwxrwxrwx.  1 jenkins jenkins    32 Jan 12 19:27 operator.py -> /usr/lib64/python3.6/operator.py
-rw-r--r--.  1 jenkins jenkins     4 Jan 12 19:27 orig-prefix.txt
lrwxrwxrwx.  1 jenkins jenkins    26 Jan 12 19:27 os.py -> /usr/lib64/python3.6/os.py
lrwxrwxrwx.  1 jenkins jenkins    33 Jan 12 19:27 posixpath.py -> /usr/lib64/python3.6/posixpath.py
drwxr-xr-x.  2 jenkins jenkins  4096 Jan 12 19:27 __pycache__
lrwxrwxrwx.  1 jenkins jenkins    30 Jan 12 19:27 random.py -> /usr/lib64/python3.6/random.py
lrwxrwxrwx.  1 jenkins jenkins    31 Jan 12 19:27 reprlib.py -> /usr/lib64/python3.6/reprlib.py
lrwxrwxrwx.  1 jenkins jenkins    26 Jan 12 19:27 re.py -> /usr/lib64/python3.6/re.py
lrwxrwxrwx.  1 jenkins jenkins    35 Jan 12 19:27 rlcompleter.py -> /usr/lib64/python3.6/rlcompleter.py
lrwxrwxrwx.  1 jenkins jenkins    30 Jan 12 19:27 shutil.py -> /usr/lib64/python3.6/shutil.py
drwxr-xr-x. 66 jenkins jenkins  4096 Jan 12 19:27 site-packages
-rw-r--r--.  1 jenkins jenkins 27543 Jan 12 19:27 site.py
lrwxrwxrwx.  1 jenkins jenkins    35 Jan 12 19:27 sre_compile.py -> /usr/lib64/python3.6/sre_compile.py
lrwxrwxrwx.  1 jenkins jenkins    37 Jan 12 19:27 sre_constants.py -> /usr/lib64/python3.6/sre_constants.py
lrwxrwxrwx.  1 jenkins jenkins    33 Jan 12 19:27 sre_parse.py -> /usr/lib64/python3.6/sre_parse.py
lrwxrwxrwx.  1 jenkins jenkins    28 Jan 12 19:27 stat.py -> /usr/lib64/python3.6/stat.py
lrwxrwxrwx.  1 jenkins jenkins    30 Jan 12 19:27 struct.py -> /usr/lib64/python3.6/struct.py
lrwxrwxrwx.  1 jenkins jenkins    31 Jan 12 19:27 tarfile.py -> /usr/lib64/python3.6/tarfile.py
lrwxrwxrwx.  1 jenkins jenkins    32 Jan 12 19:27 tempfile.py -> /usr/lib64/python3.6/tempfile.py
lrwxrwxrwx.  1 jenkins jenkins    32 Jan 12 19:27 tokenize.py -> /usr/lib64/python3.6/tokenize.py
lrwxrwxrwx.  1 jenkins jenkins    29 Jan 12 19:27 token.py -> /usr/lib64/python3.6/token.py
lrwxrwxrwx.  1 jenkins jenkins    29 Jan 12 19:27 types.py -> /usr/lib64/python3.6/types.py
lrwxrwxrwx.  1 jenkins jenkins    32 Jan 12 19:27 warnings.py -> /usr/lib64/python3.6/warnings.py
lrwxrwxrwx.  1 jenkins jenkins    31 Jan 12 19:27 weakref.py -> /usr/lib64/python3.6/weakref.py
lrwxrwxrwx.  1 jenkins jenkins    35 Jan 12 19:27 _weakrefset.py -> /usr/lib64/python3.6/_weakrefset.py

@werkt
Copy link
Collaborator

werkt commented Jan 12, 2018

I'm going to have to punt to the bazel project for non-buildfarm behaviors around declare_directory

@excavador
Copy link
Author

@werkt Yes. My latest comment about local bazel (not a farm).
Can you please give me advise - is it expected behavior or bug to bazel which I should report to bazelbuild/bazel?

@werkt
Copy link
Collaborator

werkt commented Jan 12, 2018

Looks like a bazel bug to me.

My clue: Error while validating output TreeArtifact - I usually don't expect to see classname mentions in expected behavior.

@excavador
Copy link
Author

For which I will happily mirror an implementation in buildfarm.

@werkt I am very sorry for probably stupid question...

Why you do not want to use bazel as external repo in farm?

As result you can import libraries from bazel directly to farm and 99% to have correct behavior always

@werkt
Copy link
Collaborator

werkt commented Jan 13, 2018

Not stupid, I would say that there are two reasons:

  1. my implementation of this (or almost any other mechanism) will probably be dissimilar, and would need to accommodate a reflection that will not be quite the same during the workers evaluation to anything implemented in bazel; my guidance for how it should behave had no model in an existing implementation for download, which is why I needed some treatment of it from bazel's side to get started (and I keep forgetting to ask the bazel/remote ex team what their intention was)
  2. the other is that using bazel sources directly without filtering their packaging requires that I pull in dependencies in the groupings that they've defined for them - the remote ex interactions that I'm interested in are not isolated completely in bazel sources. I'd be reliant or committed to maintaining boundaries in their code, and tbh I don't feel we need to be beholden to it. We already share a common endpoint in googleapis, and that has been helpful enough for me to remain consistent with them.

@excavador
Copy link
Author

Thank you!
Wait for fix for double check

@excavador
Copy link
Author

Any updates?

@werkt
Copy link
Collaborator

werkt commented Jan 27, 2018

No movement. Happy to accept a PR if you want this to be accelerated.

@werkt
Copy link
Collaborator

werkt commented Feb 20, 2018

You may be interested in bazelbuild/bazel#4668, which I encountered when trying to demonstrate a solution for this. I'll have to proceed without an integration test on this issue until they fix that.

@excavador
Copy link
Author

@werkt That's exactly my problem.
That's why my virtualenv rule (and rules_go) use "declare_file" and store directory instead of file in result.

If you replace "declare_directory" by "declare_file" everything would works fine.

@werkt
Copy link
Collaborator

werkt commented Feb 20, 2018 via email

@excavador
Copy link
Author

@werkt thank you very much for update!

@werkt
Copy link
Collaborator

werkt commented Mar 16, 2018

https://github.com/werkt/bazel-buildfarm/tree/output-directories provides an implementation for buildfarm, according to the observed behavior in bazel, and includes some tests. Don't know what to say for in-bazel support for legit output directory specification, but at least we'll be ready...

@excavador
Copy link
Author

@werkt good news! I will check in nearest future how buildfarm works now

@pauldraper
Copy link

@werkt
Copy link
Collaborator

werkt commented Apr 25, 2018

Thanks @pauldraper, the PR is only currently held up by me - there is substantially strange behavior that needs to be considered when working with output directories, specifically intersections with other output files/dirs, and other inputs. The initial behavior will be to reject these outliers, and I have to clean things up a little more in the change to reject those conditions as invalid initially.

@werkt
Copy link
Collaborator

werkt commented Jun 16, 2018

Implemented in #128 and landed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants