Windows: corrupt output files

[laszlocsomor]

Description of the problem / feature request / question:

Sometimes Bazel produces corrupt output files on Windows. This happens rarely and sporadically so it's hard to repro, but it's a serious problem.

This is likely the cause of #2742 and of the errors I've mentioned in #2742 (comment), as well as some of our intermittent CI failures.

If possible, provide a minimal example to reproduce the problem:

Repro-ing this was hard; I had to run a build and keep interrupting and restarting it. We finally figured out that Bazel may not be deleting outputs of killed actions when the build is interrupted, and upon the next build it uses those partial output files. If for example the C++ linking action mmaps the output file and zeroes the memory, and the linking action is killed before it's done, that can explain the all-zero output files that nonetheless have a right size.

And lo and behold here's a simple repro indeed:

genrule(
    name = "slow",
    outs = ["slow.txt"],
    cmd = "echo slow1 > $@ ; sleep 60; echo slow2 >> $@",
)

genrule(
    name = "fast",
    outs = ["fast.txt"],
    cmd = "echo fast > $@",
)

genrule(
    name = "top",
    srcs = [
        "fast",
        "slow",
    ],
    outs = ["top.txt"],
    cmd = "echo top1 > $@ ; cat $(location :fast) >> $@; cat $(location :slow) >> $@ ; echo top2 >> $@",
)

Now run:

bazel clean  # so we have a clean output tree for the demo
bazel build //foo:top -s

Interrupt the build while the slow action is sleeping, then ls bazel-genfiles/foo/.
It'll have slow.txt with contents "slow1".

Now re-run the build and watch as :slow is not rebuilt, and top is built and top.txt contains "slow1" but not "slow2".

Explanation

When a build or test action fails, it fails the whole build, so Bazel interrupts all running actions. All interrupted actions should be considered failed so even if they produced some output the next build should ignore that.

Unfortunately we ignore InterruptedException in StandaloneSpawnStrategy and, on Windows, consider the action succeeded, and use its partial output in the next build.

The fix is to add

        if (Thread.currentThread().isInterrupted()) {
          throw new InterruptedException();
        }

after cmd.execute.

I'll do that tomorrow and also write a test.

[laszlocsomor]

Big thanks to both @philwo and @ulfjack for their help in figuring this out.

[laszlocsomor]

@dslomov , @damienmg : this is a strict blocker for 0.5.0. I'll fix it today.

[damienmg]

Thanks Laszlo!

…

On Tue, Apr 4, 2017, 9:37 AM László Csomor ***@***.***> wrote: @dslomov <https://github.com/dslomov> , @damienmg <https://github.com/damienmg> : this is a strict blocker for 0.5.0. I'll fix it today. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2774 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADjHf0aLRab2UEy-mfzrDYhqZ5_12qyAks5rsfMvgaJpZM4MxwJM> .

[laszlocsomor]

Fixed internally, will be pushed to github later today.

[laszlocsomor]

(Fix is https://bazel-review.googlesource.com/c/9791/)

[hlopko]

This is just awesome. 0.5 will be an awesome release :)