ctx.actions.declare_directory throws error

[werkt]

Description of the problem / feature request:

ctx.actions.declare_directory throws runtime exceptions after action execution (run_shell used to demonstrate)

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

$ cat BUILD
load(":ext.bzl", "extrule")

extrule(
  name = "test",
)
$ cat ext.bzl
def _extrule(ctx):
  dir = ctx.actions.declare_directory("foo/bar/baz")
  ctx.actions.run_shell(
      outputs = [dir],
      command = "mkdir -p " + dir.path + " && echo wtf > " + dir.path + "/wtf.txt")

extrule = rule(
    _extrule,
    outputs = {
      "out": "foo/bar/baz",
    },
)
$ > WORKSPACE
$ bazel build :test

What operating system are you running Bazel on?

ubuntu 14.04

What's the output of bazel info release?

release 0.10.1

Any other information, logs, or outputs that you want to share?

My output from this build is as follows:

INFO: Analysed target //:test (6 packages loaded).
INFO: Found 1 target...
SUBCOMMAND: # //:test [action 'SkylarkAction foo/bar/baz']
(cd %bazel_workdir%/execroot/__main__ && \
  exec env - \
  /bin/bash -c 'mkdir -p bazel-out/k8-fastbuild/bin/foo/bar/baz && echo wtf > bazel-out/k8-fastbuild/bin/foo/bar/baz/wtf.txt')
Target //:test failed to build
Use --verbose_failures to see the command lines of failed build steps.
Unhandled exception thrown during build; message: Unrecoverable error while evaluating node 'foo/bar/baz //:test c17eebfa9142729699644ad827e2969b (2007417241 898773795)' (requested by nodes 'TargetCompletionKey{configuredTargetKey=//:test c17eebfa9142729699644ad827e2969b (1599517238 898773795), topLevelArtifactContext=com.google.devtools.build.lib.analysis.TopLevelArtifactContext@4ff827c3, willTest=false}')
INFO: Elapsed time: 1.009s, Critical Path: 0.07s
FAILED: Build did NOT complete successfully
java.lang.RuntimeException: Unrecoverable error while evaluating node 'foo/bar/baz //:test c17eebfa9142729699644ad827e2969b (2007417241 898773795)' (requested by nodes 'TargetCompletionKey{configuredTargetKey=//:test c17eebfa9142729699644ad827e2969b (1599517238 898773795), topLevelArtifactContext=com.google.devtools.build.lib.analysis.TopLevelArtifactContext@4ff827c3, willTest=false}')
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:414)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:355)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz ActionExecutionValue{artifactData={File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz/wtf.txt=[%bazel_workdir%/execroot/__main__/bazel-out/k8-fastbuild/bin]/[foo/bar/baz/wtf.txt], RegularFileStateValue{digest=null, size=4, mtime=-1, contentsProxy=ctime of 1519148685775 and nodeId of 57937409}}, treeArtifactData={File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz=TreeArtifactValue{digest=[-70, -88, 30, 63, -73, -73, 15, 64, 123, -82, 16, 22, 40, 56, -93, 25], childData={File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz/wtf.txt=RegularFileArtifactValue{digest=[-27, 126, -23, -15, 80, 52, -61, -103, -12, -32, -37, -112, 110, 121, -15, -107, -84, -67, -52, -97, 48, -88, -124, 103, 50, -22, 43, 39, 29, -23, 30, 19], size=4}}}}, additionalOutputData={File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz/wtf.txt=RegularFileArtifactValue{digest=[-27, 126, -23, -15, 80, 52, -61, -103, -12, -32, -37, -112, 110, 121, -15, -107, -84, -67, -52, -97, 48, -88, -124, 103, 50, -22, 43, 39, 29, -23, 30, 19], size=4}}}
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:1221)
        at com.google.devtools.build.lib.skyframe.ArtifactFunction.createSimpleFileArtifactValue(ArtifactFunction.java:265)
        at com.google.devtools.build.lib.skyframe.ArtifactFunction.compute(ArtifactFunction.java:134)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:340)
        ... 4 more
java.lang.RuntimeException: Unrecoverable error while evaluating node 'foo/bar/baz //:test c17eebfa9142729699644ad827e2969b (2007417241 898773795)' (requested by nodes 'TargetCompletionKey{configuredTargetKey=//:test c17eebfa9142729699644ad827e2969b (1599517238 898773795), topLevelArtifactContext=com.google.devtools.build.lib.analysis.TopLevelArtifactContext@4ff827c3, willTest=false}')
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:414)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:355)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz ActionExecutionValue{artifactData={File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz/wtf.txt=[%bazel_workdir%/execroot/__main__/bazel-out/k8-fastbuild/bin]/[foo/bar/baz/wtf.txt], RegularFileStateValue{digest=null, size=4, mtime=-1, contentsProxy=ctime of 1519148685775 and nodeId of 57937409}}, treeArtifactData={File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz=TreeArtifactValue{digest=[-70, -88, 30, 63, -73, -73, 15, 64, 123, -82, 16, 22, 40, 56, -93, 25], childData={File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz/wtf.txt=RegularFileArtifactValue{digest=[-27, 126, -23, -15, 80, 52, -61, -103, -12, -32, -37, -112, 110, 121, -15, -107, -84, -67, -52, -97, 48, -88, -124, 103, 50, -22, 43, 39, 29, -23, 30, 19], size=4}}}}, additionalOutputData={File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz/wtf.txt=RegularFileArtifactValue{digest=[-27, 126, -23, -15, 80, 52, -61, -103, -12, -32, -37, -112, 110, 121, -15, -107, -84, -67, -52, -97, 48, -88, -124, 103, 50, -22, 43, 39, 29, -23, 30, 19], size=4}}}
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:1221)
        at com.google.devtools.build.lib.skyframe.ArtifactFunction.createSimpleFileArtifactValue(ArtifactFunction.java:265)
        at com.google.devtools.build.lib.skyframe.ArtifactFunction.compute(ArtifactFunction.java:134)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:340)
        ... 4 more

[werkt]

Recent changes by @ola-rozenfeld have resulted in this changing (but still failing). New log is:

$ bazel-master build :test
...........
INFO: Analysed target //:test (4 packages loaded).
INFO: Found 1 target...
Target //:test failed to build
Use --verbose_failures to see the command lines of failed build steps.
Unhandled exception thrown during build; message: Unrecoverable error while evaluating node 'foo/bar/baz //:test com.google.devtools.build.lib.skyframe.BuildConfigurationValue$Key@ebc0bfed false (44055863)' (requested by nodes 'TargetCompletionKey{configuredTargetKey=//:test com.google.devtools.build.lib.skyframe.BuildConfigurationValue$Key@ebc0bfed false (44055863), topLevelArtifactContext=com.google.devtools.build.lib.analysis.TopLevelArtifactContext@4ff827c3, willTest=false}')
INFO: Elapsed time: 1.373s, Critical Path: 0.06s
FAILED: Build did NOT complete successfully
java.lang.RuntimeException: Unrecoverable error while evaluating node 'foo/bar/baz //:test com.google.devtools.build.lib.skyframe.BuildConfigurationValue$Key@ebc0bfed false (44055863)' (requested by nodes 'TargetCompletionKey{configuredTargetKey=//:test com.google.devtools.build.lib.skyframe.BuildConfigurationValue$Key@ebc0bfed false (44055863), topLevelArtifactContext=com.google.devtools.build.lib.analysis.TopLevelArtifactContext@4ff827c3, willTest=false}')
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:419)
          at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:355)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
  Caused by: java.lang.NullPointerException: File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz ActionExecutionValue{artifactData={File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz/wtf.txt=[%bazel_workdir%/execroot/__main__/bazel-out/k8-fastbuild/bin]/[foo/bar/baz/wtf.txt], RegularFileStateValue{digest=null, size=4, contentsProxy=ctime of 1519318866016 and nodeId of 49678102}}, treeArtifactData={File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz=TreeArtifactValue{digest=[-70, -88, 30, 63, -73, -73, 15, 64, 123, -82, 16, 22, 40, 56, -93, 25], childData={File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz/wtf.txt=RegularFileArtifactValue{digest=e57ee9f15034c399f4e0db906e79f195acbdcc9f30a8846732ea2b271de91e13, size=4, proxy=ctime of 1519318866016 and nodeId of 49678102}}}}, additionalOutputData={File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz/wtf.txt=RegularFileArtifactValue{digest=e57ee9f15034c399f4e0db906e79f195acbdcc9f30a8846732ea2b271de91e13, size=4, proxy=ctime of 1519318866016 and nodeId of 49678102}}}
          at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:1221)
          at com.google.devtools.build.lib.skyframe.ArtifactFunction.createSimpleFileArtifactValue(ArtifactFunction.java:262)
          at com.google.devtools.build.lib.skyframe.ArtifactFunction.compute(ArtifactFunction.java:126)
          at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:345)
          ... 4 more
          java.lang.RuntimeException: Unrecoverable error while evaluating node 'foo/bar/baz //:test com.google.devtools.build.lib.skyframe.BuildConfigurationValue$Key@ebc0bfed false (44055863)' (requested by nodes 'TargetCompletionKey{configuredTargetKey=//:test com.google.devtools.build.lib.skyframe.BuildConfigurationValue$Key@ebc0bfed false (44055863), topLevelArtifactContext=com.google.devtools.build.lib.analysis.TopLevelArtifactContext@4ff827c3, willTest=false}')
                  at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:419)
          at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:355)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
  Caused by: java.lang.NullPointerException: File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz ActionExecutionValue{artifactData={File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz/wtf.txt=[%bazel_workdir%/execroot/__main__/bazel-out/k8-fastbuild/bin]/[foo/bar/baz/wtf.txt], RegularFileStateValue{digest=null, size=4, contentsProxy=ctime of 1519318866016 and nodeId of 49678102}}, treeArtifactData={File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz=TreeArtifactValue{digest=[-70, -88, 30, 63, -73, -73, 15, 64, 123, -82, 16, 22, 40, 56, -93, 25], childData={File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz/wtf.txt=RegularFileArtifactValue{digest=e57ee9f15034c399f4e0db906e79f195acbdcc9f30a8846732ea2b271de91e13, size=4, proxy=ctime of 1519318866016 and nodeId of 49678102}}}}, additionalOutputData={File:[[%bazel_workdir%/execroot/__main__]bazel-out/k8-fastbuild/bin]foo/bar/baz/wtf.txt=RegularFileArtifactValue{digest=e57ee9f15034c399f4e0db906e79f195acbdcc9f30a8846732ea2b271de91e13, size=4, proxy=ctime of 1519318866016 and nodeId of 49678102}}}
          at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:1221)
          at com.google.devtools.build.lib.skyframe.ArtifactFunction.createSimpleFileArtifactValue(ArtifactFunction.java:262)
          at com.google.devtools.build.lib.skyframe.ArtifactFunction.compute(ArtifactFunction.java:126)
          at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:345)
          ... 4 more

New rev head is 0af09b7

[Reflexe]

After debugging it a bit: the problem is not with declare_directory (at least not directly). The problem is with your output: bazel thinks that foo/bar/baz would be a file but it ends up being a directory.

Changing the output to foo/bar/baz to foo/bar/baz/wtf.txt and then adding ctx.outputs.out to the run_shell's outputs makes bazel run with no problem:

$ cat ext.bzl
def _extrule(ctx):
  dir = ctx.actions.declare_directory("foo/bar/baz")
  ctx.actions.run_shell(
      outputs = [dir, ctx.outputs.out],
      command = "mkdir -p " + dir.path + " && echo wtf > " + dir.path + "/wtf.txt")

extrule = rule(
    _extrule,
    outputs = {
      "out": "foo/bar/baz/wtf.txt",
    },
)

I don't know if this is a bug or simply not an error. AFAIK, outputs can't be directories so bazel assumptions are true. Please correct me if I am wrong.

[werkt]

This is a bug, and the intention of declare_directory is explicit in order to declare a directory (and all of it's contents, recursively) as an output. The behavior has been confirmed to be the 'correct' way to declare this as an output, and the failure progress and continued stack trace are indications of a software failure.

[Reflexe]

@werkt Thanks for your response. However we still have a problem: what should be returned from ctx.outputs.out? by default, it should have been a file; However, after calling declare_directory it should be a directory.

That's a problem because that the outputs are prebuilt as files before the rule's implementation runs. We could try to delay it and assume an output is a file only if the user calls ctx.output. However, that could lead to very strange bugs (adding ctx.output makes the code fail with no reason, you real way to know if an output is a directory or a file until you read the implementation function).

I could not find any good solution for these problems; Please let me know what you think.

[werkt]

ctx.outputs.out being a File return is acceptable, because at this point no content knowledge exists for the file, and it is essentially a Path container and placeholder artifact. A File is also what declare_directory returns.

Re: "The outputs are prebuilt as files": yes, but they're built as File, which represents an interface into native implementations which do not require their targets to have particular properties or even exist. The 'outputs' are built prior to the rule implementation, but since the contents are neither meaningful, nor could/should any action have been run to produce them, there should be no dependency on them being produced in a particular way.

The out specification in the outputs attribute of extrule is for depending upon an extrule target by other targets, so that an entire directory can be specified as an input by virtue of it being an output itself, with the inputs fully enumerated as directory members.

Note that the precondition failure occurs having not specified wtf.txt except as text in my action command, so this must be occurring after execution and evaluation through enumeration of directory members in my output dir. I also have not connected any other targets to this one, so the exception occurs without 'needing' to enumerate the outputs, except for criticality.

[Reflexe]

@werkt , Thanks for your detailed answer. I was not explaining the problem enough in my previous comment; I will try to do better this time: We currently have Artifact that is, a File object in skylark. However, internally, it could be represented as a few objects, including TreeFileArtifact and 'SpecialArtifact', and more;

According to my debugging, the error has been caused by a path to directory that has been represented as a regular Artifact (and bazel has detected that something is wrong about it).

I have found two ways of implementing output directories properly:

• First method - complicated but meets the current design: We currently required to create some kind of artifact (let's call it UnresolvedArtifact) that could be both a file (Artifact) and a directory (TreeFileArtifact). However, we can't just create one and then change it in the middle (e.g. when the user calls declare_directory; this is the exact problem we're having right now).

  So let's summarize:

  • Every output object is declared as a UnresolvedArtifact.
  • When we know what is it (Artifact by default and TreeFileArtifact if declare_directory has been called), we'll change all the instances of this artifact to the right type.

  As you probably can see, that would require big changes to the Artifact object and may have some overhead. I am not a even a bad java developer (that means, i'm even less than that 😄); however, the only way I could find is to introduce UnresolvedArtifact as an Artifact transparent wrapper that would pass every call to the underling Artifact object. At the start, the underlying object would be a SpecialArtifact with a new UnresolvedArtifact attribute and then it would be updated to the right kind of artifact.

  To make sure we only have one artifact object for each path (so when we change an UnresolvedArtifact, every instance would be updated), we'll require a few changes on CachingAnalysisEnvironment.

• Simpler alternative: leave declare_directory for provides and introduce a way to declare an output directory. Example:

  def _extrule(ctx):
    dir = ctx.actions.declare_directory("foo/bar/baz")
    ctx.actions.run_shell(
        outputs = [dir],
        command = "mkdir -p " + dir.path + " && echo wtf > " + dir.path + "/wtf.txt")
  
  extrule = rule(
      _extrule,
      outputs = {
        "out": outputs.dir(default="foo/bar/baz"),
      },
  )

I hope you understand me, Please write if you have a better design / you don't agree with some of the assumptions I made.

[infojg9]

Hi All,

In short, how we can customize: 1. "bazel-bin" location with "bazel-bin/opt" and 2. "bazel-bin/subpackage-A/" with "bazel-bin/opt/subpackage-A/bin/" by any technical means ?

Similar to "output_to_bindir=1" concept, how any Bazel package at root level, allow any generic ctx.actions/genrule/plugin can be extended/utilized, to define per sub-package specific custom target output path "bazel-bin/opt/subpackage-A/bin/" location ?

While trying to adapt the above mentioned solution, ctx.actions.declare_directory("custom_unix_path") as inline, also facing similar issues as inline:

We are not very sure whether the problem is in our api implementation usage or Bazel specification ( for example: declare_directory, genrule ) limitation ?

root/build_rules.bzl
def _extrule(ctx):
  dir = ctx.actions.declare_directory("opt/child1/bin")
  ctx.actions.run_shell(
    #outputs = [dir, ctx.outputs.out],
    outputs = [dir],
    command = "mkdir -p " + dir.path + "; cp -vf $< $@",
    progress_message="run_shell.....",
    output_to_bindir=1
    )

extrule = rule(
  implementation = _extrule,
  outputs = {
  "out": outputs.dir(default="opt/child1/bin"),
  },
  output_to_genfiles = False,
  attrs = {
    # Do not declare "name": It is added automatically. 
    # https://github.com/bazelbuild/examples/blob/master/rules/attributes/printer.bzl
    "number": attr.int(default = 1),
    "bins": attr.label_list(),
    "deps": attr.label_list(),
    "etcs": attr.label_list(),
  },
)

$ bazel clean; bazel build -s ...
INFO: Starting clean (this may take a while).
ERROR: /home/user/root_pkg/build_rules.bzl:line:xx: name 'outputs' is not defined
ERROR: error loading package '': Extension 'build_rules.bzl' has errors
INFO: Elapsed time: 0.090s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
    currently loading:  ... (2 packages)


$ bazel version
Build label: 0.13.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Mon Oct 18 21:33:40 +50297 (1525078013620)
Build timestamp: 1525078013620
Build timestamp as int: 1525078013620

---

@mhlopko, Bazel users, any idea/hint would be highly appreciated.

[dslomov]

The discussion has been wandering around for a bit, it seems.
The problem with the original example is (re)declaring outputs of a rule - the outputs of a rule are files, and declaring them as directories does not work. Of course, Bazel should not crash, and that is a bug.

We have been discussing adding the ability of declaring direct outputs. We have decided against it so far, since we would like to generally get rid of declared outputs in favor of output groups.

Re #4668 (comment): we do not allow further customization within bazel-bin at the moment. It is bazel-bin/<configuration>/package/name. More details on what are you trying to do might let us suggest some alternatives.