Spurious failure in incremental\cache_file_headers.rs on Windows

[alexcrichton]

I've seen this failure a number of times on AppVeyor at this point:

Check compiletest suite=incremental mode=incremental (x86_64-pc-windows-msvc -> x86_64-pc-windows-msvc)
running 69 tests
F....................................................................
failures:
---- [incremental] incremental\cache_file_headers.rs stdout ----
	
error in revision `rpass2`: compilation failed!
status: exit code: 101
command: PATH="C:\projects\rust\build\x86_64-pc-windows-msvc\stage2\bin;C:\Program Files (x86)\Windows Kits\10\bin\x64;C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\amd64;C:\projects\rust\build\x86_64-pc-windows-msvc\stage2-tools\x86_64-pc-windows-msvc\release\deps;C:\projects\rust\build\x86_64-pc-windows-msvc\stage2-rustc\x86_64-pc-windows-msvc\release\deps;C:\projects\rust\build\x86_64-pc-windows-msvc\stage2-test\x86_64-pc-windows-msvc\release\deps;C:\projects\rust\build\x86_64-pc-windows-msvc\stage2-std\x86_64-pc-windows-msvc\release\deps;C:\msys64\mingw64\bin;C:\msys64\mingw64\bin;C:\msys64\usr\bin;C:\Perl\site\bin;C:\Perl\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Program Files\7-Zip;C:\Program Files\Microsoft\Web Platform Installer;C:\Tools\GitVersion;C:\Tools\PsTools;C:\Program Files\Git LFS;C:\Program Files\Mercurial;C:\Program Files (x86)\Subversion\bin;C:\Program Files\Microsoft SQL Server\120\Tools\Binn;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\110\Tools\Binn;C:\Program Files (x86)\Microsoft SQL Server\120\Tools\Binn;C:\Program Files\Microsoft SQL Server\120\DTS\Binn;C:\Program Files (x86)\Microsoft SQL Server\120\Tools\Binn\ManagementStudio;C:\Program Files\Microsoft Windows Performance Toolkit;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit;C:\Tools\WebDriver;C:\Program Files (x86)\Microsoft SDKs\TypeScript\1.4;C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\PrivateAssemblies;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI\wbin;C:\Ruby193\bin;C:\Tools\NUnit\bin;C:\Tools\xUnit;C:\Tools\MSpec;C:\Tools\Coverity\bin;C:\Program Files (x86)\CMake\bin;C:\go\bin;C:\Program Files\Java\jdk1.8.0\bin;C:\Python27;C:\Program Files\erl7.3\bin;C:\Program Files\nodejs;C:\Program Files (x86)\iojs;C:\Program Files\iojs;C:\Users\appveyor\AppData\Roaming\npm;C:\Program Files\Microsoft SQL Server\130\Tools\Binn;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit;C:\Program Files (x86)\MSBuild\14.0\Bin;C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\120;C:\Tools\NuGet;C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow;C:\Program Files\Amazon\AWSCLI;C:\Windows\SysWOW64\WindowsPowerShell\v1.0\Modules\TShell\TShell;C:\Program Files\Microsoft DNX\Dnvm;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\130\Tools\Binn;C:\Program Files\Git\cmd;C:\Program Files\Git\usr\bin;C:\Program Files (x86)\Microsoft SQL Server\130\Tools\Binn;C:\Program Files (x86)\Microsoft SQL Server\130\DTS\Binn;C:\Program Files\Microsoft SQL Server\130\DTS\Binn;C:\Program Files (x86)\Microsoft SQL Server\110\DTS\Binn;C:\Program Files (x86)\Microsoft SQL Server\120\DTS\Binn;C:\Program Files (x86)\Apache\Maven\bin;C:\Program Files\LLVM\bin;C:\ProgramData\chocolatey\bin;C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code;C:\Program Files\Microsoft SDKs\Service Fabric\Tools\ServiceFabricLocalClusterManager;C:\Python27\Scripts;C:\Program Files (x86)\Yarn\bin;C:\Tools\NUnit3;C:\Program Files\dotnet;C:\Program Files (x86)\nodejs;C:\Users\appveyor\.dnx\runtimes\dnx-clr-win-x86.1.0.0-rc1-update2\bin;C:\Users\appveyor\AppData\Local\Yarn\.bin;C:\Users\appveyor\AppData\Roaming\npm;C:\Program Files\AppVeyor\BuildAgent;C:\projects\rust\sccache2" C:\projects\rust\build\x86_64-pc-windows-msvc\stage2\bin\rustc.exe C:\projects\rust\src/test\incremental\cache_file_headers.rs -L C:\projects\rust\build\x86_64-pc-windows-msvc\test\incremental --target=x86_64-pc-windows-msvc --cfg rpass2 -Z incremental=C:\projects\rust\build\x86_64-pc-windows-msvc\test\incremental\cache_file_headers.inc --error-format json -L C:\projects\rust\build\x86_64-pc-windows-msvc\test\incremental\cache_file_headers.stage2-x86_64-pc-windows-msvc.incremental.libaux -C prefer-dynamic -o C:\projects\rust\build\x86_64-pc-windows-msvc\test\incremental\cache_file_headers.stage2-x86_64-pc-windows-msvc.exe -Crpath -O -Lnative=C:\projects\rust\build\x86_64-pc-windows-msvc\rust-test-helpers -Zincremental-info
stdout:
------------------------------------------
incremental: session directory: 4 files hard-linked
incremental: session directory: 0 files copied
incremental: ignoring cache artifact `dep-graph.bin`: Different compiler version
incremental: re-using 0 out of 1 modules
------------------------------------------
stderr:
------------------------------------------
{"message":"linking with `link.exe` failed: exit code: 1104","code":null,"level":"error","spans":[],"children":[{"message":"\"C:\\\\Program Files (x86)\\\\Microsoft Visual Studio 14.0\\\\VC\\\\bin\\\\amd64\\\\link.exe\" \"/LIBPATH:C:\\\\Program Files (x86)\\\\Microsoft Visual Studio 14.0\\\\VC\\\\lib\\\\amd64\" \"/LIBPATH:C:\\\\Program Files (x86)\\\\Windows Kits\\\\10\\\\lib\\\\10.0.14393.0\\\\ucrt\\\\x64\" \"/LIBPATH:C:\\\\Program Files (x86)\\\\Windows Kits\\\\10\\\\lib\\\\10.0.14393.0\\\\um\\\\x64\" \"/NOLOGO\" \"/NXCOMPAT\" \"/LIBPATH:C:\\\\projects\\\\rust\\\\build\\\\x86_64-pc-windows-msvc\\\\stage2\\\\lib\\\\rustlib\\\\x86_64-pc-windows-msvc\\\\lib\" \"C:\\\\projects\\\\rust\\\\build\\\\x86_64-pc-windows-msvc\\\\test\\\\incremental\\\\cache_file_headers.cache_file_headers.o\" \"/OUT:C:\\\\projects\\\\rust\\\\build\\\\x86_64-pc-windows-msvc\\\\test\\\\incremental\\\\cache_file_headers.stage2-x86_64-pc-windows-msvc.exe\" \"/OPT:REF,ICF\" \"/DEBUG\" \"/LIBPATH:C:\\\\projects\\\\rust\\\\build\\\\x86_64-pc-windows-msvc\\\\test\\\\incremental\" \"/LIBPATH:C:\\\\projects\\\\rust\\\\build\\\\x86_64-pc-windows-msvc\\\\test\\\\incremental\\\\cache_file_headers.stage2-x86_64-pc-windows-msvc.incremental.libaux\" \"/LIBPATH:C:\\\\projects\\\\rust\\\\build\\\\x86_64-pc-windows-msvc\\\\rust-test-helpers\" \"/LIBPATH:C:\\\\projects\\\\rust\\\\build\\\\x86_64-pc-windows-msvc\\\\stage2\\\\lib\\\\rustlib\\\\x86_64-pc-windows-msvc\\\\lib\" \"/LIBPATH:C:\\\\projects\\\\rust\\\\build\\\\x86_64-pc-windows-msvc\\\\stage2\\\\lib\\\\rustlib\\\\x86_64-pc-windows-msvc\\\\lib\" \"std-5a02bf719865b808.dll.lib\" \"C:\\\\projects\\\\rust\\\\build\\\\x86_64-pc-windows-msvc\\\\stage2\\\\lib\\\\rustlib\\\\x86_64-pc-windows-msvc\\\\lib\\\\libcompiler_builtins-b05c732613224206.rlib\" \"advapi32.lib\" \"ws2_32.lib\" \"userenv.lib\" \"shell32.lib\" \"msvcrt.lib\"","code":null,"level":"note","spans":[],"children":[],"rendered":null},{"message":"LINK : fatal error LNK1104: cannot open file 'C:\\projects\\rust\\build\\x86_64-pc-windows-msvc\\test\\incremental\\cache_file_headers.stage2-x86_64-pc-windows-msvc.exe'\r\n","code":null,"level":"note","spans":[],"children":[],"rendered":null}],"rendered":null}
{"message":"aborting due to previous error","code":null,"level":"error","spans":[],"children":[],"rendered":null}
------------------------------------------
thread '[incremental] incremental\cache_file_headers.rs' panicked at 'explicit panic', src\tools\compiletest\src\runtest.rs:2427
note: Run with `RUST_BACKTRACE=1` for a backtrace.
failures:
    [incremental] incremental\cache_file_headers.rs
test result: FAILED. 68 passed; 1 failed; 0 ignored; 0 measured

where the actual error is:

LINK : fatal error LNK1104: cannot open file 'C:\\projects\\rust\\build\\x86_64-pc-windows-msvc\\test\\incremental\\cache_file_headers.stage2-x86_64-pc-windows-msvc.exe

So far I've been unable to isolate what's happening here much less reproduce it, unfortunately

Other examples are:

• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.1245/job/ntewu3y9r37b0a3a
• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.1260/job/0y9ep7dnl2rnd13x
• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.1227/job/f59ruuh32pei993k
• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.1279/job/pcc6w2w30v8hosck
• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.1278/job/9eh3n1k3qlbetxif
• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.1291/job/9nm55n3mc6e93rcg
• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.1301/job/8vx1eofjeqh9x65e
• https://ci.appveyor.com/project/rust-lang/rust/build/1.0.1310/job/j5ddadcicguw3puo

Conclusions so far is that:

• This is not MSVC or GNU specific
• This is not 32 or 64-bit specific
• It's always the cache_file_headers.rs test
• This test is always the first failure in the incremental test suite

[alexcrichton]

cc @michaelwoerister

[michaelwoerister]

This test relies on an environment variable being set correctly by compiletest. Might that be a hint?

[alexcrichton]

Hm I think that may be a red herring. Presumably the env var is being set correctly because the compiler is recompiling, right?

Digging more into this this morning I came across https://bugzilla.mozilla.org/show_bug.cgi?id=509960, a similar error, but no resolution there. One possibility is disk space running out but this is such a deterministic error that seems unlikely.

[alexcrichton]

My guess is that this is failing because something else has the file open. On Windows at least if there's an open file handle you're unable to delete that file. This is happening on both MSVC and GNU so I don't think it's linker specific (e.g. no problems like with mspdbsrv.exe I believe).

@michaelwoerister you wouldn't happen to have any ideas about when we'd open this file, would you? IIRC the compiler never actually opens the output file, it just relies on the linker to do so. I also couldn't find anything in the compiletest incremental pieces as well...

[brson]

Why would this only show up on AppVeyor? Is the Windows job object set up differently in rustbuild than in rust-buildbot?

[alexcrichton]

That's a good question! (I'm not sure why it's only on AppVeyor). We aren't actually using job objects on AppVeyor like we were on rust-buildbot, notably we aren't using rustjob. The rustbuild job object (which we do use), however, should be the same across rust-buildbot/AppVeyor.

Between AppVeyor and rust-buildbot we've had a ton of changes, though. AppVeyor has likely a newer install of Windows and also a newer version of Visual Studio.

[brson]

It doesn't look like compiletest is doing anything itself that would hold the exe open.

[michaelwoerister]

I can't think of anything special that this test case. It will try to overwrite the executable for each revision, but all the other incremental test cases do that too.

[alexcrichton]

This hasn't showed up in ~2 months, so presumed fix by... something!