rustc 1.30 beta 12 incremental compilation hangs on win 7

[sinesc]

I'm seeing frequent rustc freezes (0% CPU) with incremental compilation enabled on Windows 7. This happens about every 2nd or 3rd build attempt. Once this happens (and after killing the frozen rustc), any further attempts hang forever as well (i.e. I waited 5 minutes). cargo clean resolves the issue until it happens again a few builds later, setting CARGO_INCREMENTAL=0 appears to completely prevent it.

This might be a duplicate of/related to #54627.

I have attached a debugger to get the traces, but I don't have any symbols for rustc, so this is probably useless:

>	ntdll.dll!NtWaitForKeyedEvent�()
 	ntdll.dll!RtlSleepConditionVariableSRW�()
 	kernel32.dll!SleepConditionVariableSRW�()
 	std-955ac6734338b235.dll!000007fedb3c7fa4()
 	std-955ac6734338b235.dll!000007fedb3a7781()
 	rustc_codegen_llvm-llvm.dll!000007fed71f49d2()
 	rustc_codegen_llvm-llvm.dll!000007fed71c8283()
 	rustc_codegen_llvm-llvm.dll!000007fed718739c()
 	rustc_codegen_llvm-llvm.dll!000007fed71c2720()
 	rustc_driver-7ae0c7f4e267e2e6.dll!000007fedc915108()
 	rustc_driver-7ae0c7f4e267e2e6.dll!000007fedc9384ac()
 	rustc_driver-7ae0c7f4e267e2e6.dll!000007fedc8d8fb3()
 	rustc_driver-7ae0c7f4e267e2e6.dll!000007fedc8d4e36()
 	rustc_driver-7ae0c7f4e267e2e6.dll!000007fedc987961()
 	rustc_driver-7ae0c7f4e267e2e6.dll!000007fedc9e27e0()
 	rustc_driver-7ae0c7f4e267e2e6.dll!000007fedc92e728()
 	rustc_driver-7ae0c7f4e267e2e6.dll!000007fedc9c7908()
 	rustc_driver-7ae0c7f4e267e2e6.dll!000007fedc92c325()
 	rustc_driver-7ae0c7f4e267e2e6.dll!000007fedc9c6588()
 	rustc_driver-7ae0c7f4e267e2e6.dll!000007fedc92b665()
 	std-955ac6734338b235.dll!000007fedb3da2a2()
 	rustc_driver-7ae0c7f4e267e2e6.dll!000007fedc9c37b3()
 	rustc_driver-7ae0c7f4e267e2e6.dll!000007fedc9d5f8e()
 	rustc.exe!000000013f071056()
 	std-955ac6734338b235.dll!000007fedb3a8477()
 	std-955ac6734338b235.dll!000007fedb3da2a2()
 	std-955ac6734338b235.dll!000007fedb3ba4c3()
 	rustc.exe!000000013f07104a()
 	rustc.exe!000000013f071299()
 	kernel32.dll!BaseThreadInitThunk�()
 	ntdll.dll!RtlUserThreadStart�()

>	ntdll.dll!NtWaitForMultipleObjects�()
 	KernelBase.dll!WaitForMultipleObjectsEx()
 	kernel32.dll!WaitForMultipleObjects�()
 	rustc-cfefa64b59573cce.dll!000007fedbea2476()
 	rustc-cfefa64b59573cce.dll!000007fedbea2768()
 	std-955ac6734338b235.dll!000007fedb3da2a2()
 	rustc-cfefa64b59573cce.dll!000007fedbe9e662()
 	std-955ac6734338b235.dll!000007fedb3d8672()
 	std-955ac6734338b235.dll!000007fedb3b48e9()
 	kernel32.dll!BaseThreadInitThunk�()
 	ntdll.dll!RtlUserThreadStart�()

>	ntdll.dll!NtWaitForKeyedEvent�()
 	ntdll.dll!RtlSleepConditionVariableSRW�()
 	kernel32.dll!SleepConditionVariableSRW�()
 	std-955ac6734338b235.dll!000007fedb3c7fa4()
 	std-955ac6734338b235.dll!000007fedb3a7781()
 	rustc_codegen_llvm-llvm.dll!000007fed70cfb2a()
 	rustc_codegen_llvm-llvm.dll!000007fed71c78f8()
 	rustc_codegen_llvm-llvm.dll!000007fed70c9384()
 	rustc_codegen_llvm-llvm.dll!000007fed70e3346()
 	std-955ac6734338b235.dll!000007fedb3da2a2()
 	rustc_codegen_llvm-llvm.dll!000007fed70d4e59()
 	std-955ac6734338b235.dll!000007fedb3d8672()
 	std-955ac6734338b235.dll!000007fedb3b48e9()
 	kernel32.dll!BaseThreadInitThunk�()
 	ntdll.dll!RtlUserThreadStart�()

>	ntdll.dll!ZwWaitForSingleObject�()
 	ntdll.dll!RtlpWaitOnCriticalSection�()
 	ntdll.dll!RtlEnterCriticalSection�()
 	rustc_codegen_llvm-llvm.dll!000007fed8af53a0()
 	rustc_codegen_llvm-llvm.dll!000007fed8af564b()
 	rustc_codegen_llvm-llvm.dll!000007fed8af554e()
 	ntdll.dll!RtlProcessFlsData�()
 	ntdll.dll!LdrShutdownThread�()
 	ntdll.dll!RtlExitUserThread�()
 	kernel32.dll!BaseThreadInitThunk�()
 	ntdll.dll!RtlUserThreadStart�()

>	ntdll.dll!ZwWaitForSingleObject�()
 	ntdll.dll!RtlpWaitOnCriticalSection�()
 	ntdll.dll!RtlEnterCriticalSection�()
 	ntdll.dll!LdrGetProcedureAddressEx�()
 	ntdll.dll!LdrGetProcedureAddress�()
 	KernelBase.dll!GetProcAddress()
 	rustc_codegen_llvm-llvm.dll!000007fed8af6236()
 	rustc_codegen_llvm-llvm.dll!000007fed8af6bdc()
 	rustc_codegen_llvm-llvm.dll!000007fed8af2e9a()
 	rustc_codegen_llvm-llvm.dll!000007fed8ad3d7d()
 	rustc_codegen_llvm-llvm.dll!000007fed73bc2fe()
 	rustc_codegen_llvm-llvm.dll!000007fed73bc203()
 	rustc_codegen_llvm-llvm.dll!000007fed73bb9e1()
 	rustc_codegen_llvm-llvm.dll!000007fed752c6e7()
 	rustc_codegen_llvm-llvm.dll!000007fed7c20cbc()
 	rustc_codegen_llvm-llvm.dll!000007fed7c19b56()
 	rustc_codegen_llvm-llvm.dll!000007fed7c239d3()
 	rustc_codegen_llvm-llvm.dll!000007fed733a810()
 	rustc_codegen_llvm-llvm.dll!000007fed72469f9()
 	rustc_codegen_llvm-llvm.dll!000007fed70f40bf()
 	rustc_codegen_llvm-llvm.dll!000007fed70cd5d2()
 	rustc_codegen_llvm-llvm.dll!000007fed70e3399()
 	std-955ac6734338b235.dll!000007fedb3da2a2()
 	rustc_codegen_llvm-llvm.dll!000007fed70d4b9a()
 	std-955ac6734338b235.dll!000007fedb3d8672()
 	std-955ac6734338b235.dll!000007fedb3b48e9()
 	kernel32.dll!BaseThreadInitThunk�()
 	ntdll.dll!RtlUserThreadStart�()

>	ntdll.dll!ZwWaitForSingleObject�()
 	ntdll.dll!RtlpWaitOnCriticalSection�()
 	ntdll.dll!RtlEnterCriticalSection�()
 	ntdll.dll!LdrShutdownThread�()
 	ntdll.dll!RtlExitUserThread�()
 	kernel32.dll!BaseThreadInitThunk�()
 	ntdll.dll!RtlUserThreadStart�()

>	ntdll.dll!ZwWaitForSingleObject�()
 	ntdll.dll!RtlpWaitOnCriticalSection�()
 	ntdll.dll!RtlEnterCriticalSection�()
 	ntdll.dll!LdrShutdownThread�()
 	ntdll.dll!RtlExitUserThread�()
 	kernel32.dll!BaseThreadInitThunk�()
 	ntdll.dll!RtlUserThreadStart�()

>	ntdll.dll!ZwWaitForSingleObject�()
 	ntdll.dll!RtlpWaitOnCriticalSection�()
 	ntdll.dll!RtlEnterCriticalSection�()
 	ntdll.dll!LdrShutdownThread�()
 	ntdll.dll!RtlExitUserThread�()
 	kernel32.dll!BaseThreadInitThunk�()
 	ntdll.dll!RtlUserThreadStart�()

>	ntdll.dll!ZwWaitForSingleObject�()
 	ntdll.dll!RtlpWaitOnCriticalSection�()
 	ntdll.dll!RtlEnterCriticalSection�()
 	ntdll.dll!LdrShutdownThread�()
 	ntdll.dll!RtlExitUserThread�()
 	kernel32.dll!BaseThreadInitThunk�()
 	ntdll.dll!RtlUserThreadStart�()

>	ntdll.dll!ZwWaitForSingleObject�()
 	ntdll.dll!RtlpWaitOnCriticalSection�()
 	ntdll.dll!RtlEnterCriticalSection�()
 	ntdll.dll!LdrpInitializeThread�()
 	ntdll.dll!_LdrpInitialize�()
 	ntdll.dll!LdrInitializeThunk�()

>	ntdll.dll!ZwWaitForSingleObject�()
 	ntdll.dll!RtlpWaitOnCriticalSection�()
 	ntdll.dll!RtlEnterCriticalSection�()
 	ntdll.dll!LdrpInitializeThread�()
 	ntdll.dll!_LdrpInitialize�()
 	ntdll.dll!LdrInitializeThunk�()

>	ntdll.dll!ZwWaitForSingleObject�()
 	ntdll.dll!RtlpWaitOnCriticalSection�()
 	ntdll.dll!RtlEnterCriticalSection�()
 	ntdll.dll!LdrpInitializeThread�()
 	ntdll.dll!_LdrpInitialize�()
 	ntdll.dll!LdrInitializeThunk�()

>	ntdll.dll!ZwWaitForSingleObject�()
 	ntdll.dll!RtlpWaitOnCriticalSection�()
 	ntdll.dll!RtlEnterCriticalSection�()
 	rustc_codegen_llvm-llvm.dll!000007fed8af53e0()
 	rustc_codegen_llvm-llvm.dll!000007fed8af5533()
 	rustc_codegen_llvm-llvm.dll!000007fed8af5926()
 	rustc_codegen_llvm-llvm.dll!000007fed8af13c9()
 	rustc_codegen_llvm-llvm.dll!000007fed8ad0bf6()
 	rustc_codegen_llvm-llvm.dll!000007fed8ad0471()
 	rustc_codegen_llvm-llvm.dll!000007fed8ad06a1()
 	ntdll.dll!LdrpInitializeThread�()
 	ntdll.dll!_LdrpInitialize�()
 	ntdll.dll!LdrInitializeThunk�()

>	ntdll.dll!ZwWaitForSingleObject�()
 	ntdll.dll!RtlpWaitOnCriticalSection�()
 	ntdll.dll!RtlEnterCriticalSection�()
 	rustc_codegen_llvm-llvm.dll!000007fed8af53a0()
 	rustc_codegen_llvm-llvm.dll!000007fed8af564b()
 	rustc_codegen_llvm-llvm.dll!000007fed8af554e()
 	ntdll.dll!RtlProcessFlsData�()
 	ntdll.dll!LdrShutdownThread�()
 	ntdll.dll!RtlExitUserThread�()
 	kernel32.dll!BaseThreadInitThunk�()
 	ntdll.dll!RtlUserThreadStart�()

>	ntdll.dll!ZwWaitForSingleObject�()
 	ntdll.dll!RtlpWaitOnCriticalSection�()
 	ntdll.dll!RtlEnterCriticalSection�()
 	ntdll.dll!LdrpInitializeThread�()
 	ntdll.dll!_LdrpInitialize�()
 	ntdll.dll!LdrInitializeThunk�()

>	ntdll.dll!ZwWaitForSingleObject�()
 	ntdll.dll!RtlpWaitOnCriticalSection�()
 	ntdll.dll!RtlEnterCriticalSection�()
 	ntdll.dll!LdrpInitializeThread�()
 	ntdll.dll!_LdrpInitialize�()
 	ntdll.dll!LdrInitializeThunk�()

>	ntdll.dll!ZwWaitForSingleObject�()
 	ntdll.dll!RtlpWaitOnCriticalSection�()
 	ntdll.dll!RtlEnterCriticalSection�()
 	ntdll.dll!LdrpInitializeThread�()
 	ntdll.dll!_LdrpInitialize�()
 	ntdll.dll!LdrInitializeThunk�()

[ghost]

I had this issue too, your fix solved the issue. Hope this gets fixed for others.

[pietroalbini]

Was this working fine on Rust 1.29?

[Mark-Simulacrum]

Cc @michaelwoerister

[sinesc]

@pietroalbini I never experienced the issue with 1.29

There is also a post on reddit about the it, in case it might be helpful: https://www.reddit.com/r/rust/comments/9xseoa/rustc_hangs_on_windows_7_anyone_else/

[pietroalbini]

Ok, marking this as a stable-to-stable regression.

[michaelwoerister]

Did anyone ever see this on other platforms too?

[pnkfelix]

triage: Marking as P-high for now, but am not opposed to downgrading to P-medium if others think that more appropriate.

[pnkfelix]

discussed at T-compiler meeting. Based on the End-of-Life schedule for Windows 7, we assume it is not a tier 1 platform for Rust, and therefore we are downgrading this to P-medium.

[nikomatsakis]

@sinesc My guess is that this is not related #54627, which @nagisa ultimately blamed on an LLVM bug. Given that your backtrace seems to include blocking on a monitor, my guess would be that it has something to do with our job queue support. I'm trying to remember if there is a way to disable the job-queue by default -- @alexcrichton do you recall?

[nikomatsakis]

(I think disabling incremental might be doing that as a side-effect)

[pnkfelix]

filed #56344 to ask the question about whether we are justified in treating Windows 7 as non-tier-1.

[retep998]

Has anyone been able to reproduce this issue on newer versions of Windows?

[ghost]

I posted a quick stawpoll on r/rust: https://old.reddit.com/r/rust/comments/a1jskw/what_os_do_you_build_rust_on_os_census/

Probably right to determine that Win7 isn't a high enough priority.
Just to point out though, another Win7 user did comment they were having the same issue.

[sinesc]

I updated to rustc 1.31.0-beta.19 today and so far could not reproduce the issue with it. It is probably to early to pronounce it fixed since I only had half an hour but 1.30 beta 12 would have already frozen a couple of times. I'll make another comment when or if it breaks again.

As a side note, working without incremental for a while really makes you appreciate it all the more afterwards...

[nikomatsakis]

We unearthed some numbers suggesting Win7 was 5% of the documentation downloads, which is non-trivial. Regardless, I would like to get to the bottom of this -- for all we know it's a subtle bug that's just not manifesting elsewhere. I guess the next step is to try installing Win 7 in a VM.

[nikomatsakis]

But then again if it won't manifest, I guess we'll just have to wait =)

[rodrimati1992]

I posted a quick stawpoll on r/rust: https://old.reddit.com/r/rust/comments/a1jskw/what_os_do_you_build_rust_on_os_census/

This poll suffers from only allowing one OS,meaning that if people use Linux as their main development OS,but also use Windows,it will be biased against it.