Empty directories inside tree artifacts are not tracked

[tjgq]

Description of the bug:

Skyframe doesn't track the existence of empty directories inside a tree artifact. As a result, if a tree artifact with an empty directory is produced and the empty directory is manually deleted, rebuilding doesn't recreate the empty directory. Likewise, changing a rule implementation to produce an empty directory when it previously didn't doesn't invalidate the build results.

I believe the root cause is that we model a TreeArtifactValue (the SkyValue representing the contents of a tree artifact) as a collection of TreeFileArtifacts, one per regular file inside the tree artifact, ignoring empty directories.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Workspace:

cat > BUILD <<'EOF'
load(":defs.bzl", "my_rule")

my_rule(name = "my")
EOF

touch WORKSPACE

cat > defs.bzl <<'EOF'
def _impl(ctx):
  d = ctx.actions.declare_directory(ctx.label.name + ".dir")
  ctx.actions.run_shell(
    outputs = [d],
    command = "cd %s && mkdir empty" % d.path,
  )
  return DefaultInfo(files = depset([d]))

my_rule = rule(implementation = _impl)
EOF

Then run:

bazel build :my  # clean build
ls bazel-bin/my.dir  # contains empty subdirectory
chmod u+w bazel-bin/my.dir && rmdir bazel-bin/my.dir/empty
bazel build :my  # incremental build
ls bazel-bin/my.dir  # should contain empty subdirectory, but doesn't

Which operating system are you running Bazel on?

Linux, MacOS

What is the output of bazel info release?

development version

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

Built @ 6f0913e (near current head)

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

N/A

Have you found anything relevant by searching the web?

N/A

Any other information, logs, or outputs that you want to share?

N/A

[fmeum]

@tjgq What do you think of #15789 (comment)? I could work on this, I always felt like #15276 wasn't as clean as it could have been due to this missing.

[tjgq]

@fmeum Yes, I think you are correct and we either have to extend the definition of TreeFileArtifact to also cover empty directories, or introduce a new artifact type (TreeDirectoryArtifact?) for them. Then we should be able to track the existence of empty subdirectories correctly in Skyframe. Feel free to work on this.

[tjgq]

My previous comment was a little too optimistic. I do think there are use cases where accurate tracking of tree artifact contents is required, and this probably goes beyond empty subdirectories (for example, it's also impossible for a tree artifact to contain a symlink since we always dereference them; see #15454). But this would be a feature request, not a bug.

I'm reassigning this to the core team because I feel that's where a decision to implement belongs. (I'm not asking for a decision to be made at this time, though.)