Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

STATUS_ILLEGAL_INSTRUCTION exception in win-x86 wpfgfx_cor3.dll running on x64 Win Server 2012 R2 #2259

Open
vatsan-madhavan opened this issue Nov 28, 2019 · 15 comments
Assignees
Milestone

Comments

@vatsan-madhavan
Copy link
Member

@vatsan-madhavan vatsan-madhavan commented Nov 28, 2019

Hi I have the same issue

Problem signature:
Problem Event Name: APPCRASH
Application Name: RocketM.exe
Application Version: 2.0.0.0
Application Timestamp: 5d7bb0cb
Fault Module Name: wpfgfx_cor3.dll
Fault Module Version: 4.800.19.46238
Fault Module Timestamp: 5d7ab8ec
Exception Code: c000001d
Exception Offset: 00158e19
OS Version: 6.3.9600.2.0.0.272.7
Locale ID: 5129
Additional Information 1: 5861
Additional Information 2: 5861822e1919d7c014bbb064c64908b2
Additional Information 3: 6fbe
Additional Information 4: 6fbe6bde2701766d81cbca0597a5fa35

Read our privacy statement online:
http://go.microsoft.com/fwlink/?linkid=280262

If the online privacy statement is not available, please read our privacy statement offline:
C:\Windows\system32\en-US\erofflps.txt

OS is

image

app is x86

Originally posted by @thudugala in #2057 (comment)

@vatsan-madhavan

This comment has been minimized.

Copy link
Member Author

@vatsan-madhavan vatsan-madhavan commented Nov 28, 2019

/cc @thudugala FYI

@thudugala

This comment has been minimized.

Copy link

@thudugala thudugala commented Nov 28, 2019

@vatsan-madhavan This only happens in our internal server. It works in our cloud server (also Windows Server 2012 R2 Standard). Windows update are up to date. in both servers.

@vatsan-madhavan

This comment has been minimized.

Copy link
Member Author

@vatsan-madhavan vatsan-madhavan commented Nov 29, 2019

If you can generate a crash dump and share it here, that could help us.

https://docs.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps

@thudugala

This comment has been minimized.

Copy link

@thudugala thudugala commented Nov 29, 2019

Will do next Monday 👍

@thudugala

This comment has been minimized.

Copy link

@thudugala thudugala commented Dec 1, 2019

@arpitmathur arpitmathur added this to the 5.0 milestone Dec 3, 2019
@thudugala

This comment has been minimized.

Copy link

@thudugala thudugala commented Dec 3, 2019

Any plan to release a fix for this ?

@vatsan-madhavan

This comment has been minimized.

Copy link
Member Author

@vatsan-madhavan vatsan-madhavan commented Dec 4, 2019

Any plan to release a fix for this ?

After we figure out whats wrong 😁. From what I can tell so far, this may be not be within the power of WPF team alone to solve, but if we find that there is something reasonable we can do, then we will try.

--

This is the crash stack I see:

[0x0]   wpfgfx_cor3!_ftoui3 + 0x9   
[0x1]   wpfgfx_cor3!CDisplaySet::ReadDefaultDisplaySettings + 0x107   
[0x2]   wpfgfx_cor3!CDisplaySet::Init + 0xbd   
[0x3]   wpfgfx_cor3!CDisplayManager::CreateNewDisplaySet + 0xbc   
[0x4]   wpfgfx_cor3!CDisplayManager::DangerousGetLatestDisplaySet + 0xb9   
[0x5]   wpfgfx_cor3!CMILFactory::GetCurrentDisplaySet + 0x73   
[0x6]   wpfgfx_cor3!CMILFactory::UpdateDisplayState + 0x57   
[0x7]   wpfgfx_cor3!CComposition::ProcessComposition + 0x52   
[0x8]   wpfgfx_cor3!CComposition::Compose + 0x41   
[0x9]   wpfgfx_cor3!CPartitionThread::RenderPartition + 0x31   
[0xa]   wpfgfx_cor3!CPartitionThread::Run + 0x57   
[0xb]   wpfgfx_cor3!CPartitionThread::ThreadMain + 0x4c   
[0xc]   kernel32!BaseThreadInitThunk + 0x24   
[0xd]   ntdll!__RtlUserThreadStart + 0x2f   
[0xe]   ntdll!_RtlUserThreadStart + 0x1b  

I'm guessing that _ftoi3 may be a compiler generated or possibly coming from a compiler supplied library. This is what I'm seeing in its disassembled definition:

    wpfgfx_cor3!_ftoui3:
6e6d8e10 833d54cd726e06   cmp     dword ptr [wpfgfx_cor3!__isa_available (6e72cd54)], 6
6e6d8e17 7c07             jl      wpfgfx_cor3!_ftoui3+0x10 (6e6d8e20)
6e6d8e19 62f17e0878c0     vcvttss2usi eax, xmm0
6e6d8e1f c3               ret     
6e6d8e20 660f7ec0         movd    eax, xmm0
6e6d8e24 d1e0             shl     eax, 1
6e6d8e26 721b             jb      wpfgfx_cor3!_ftoui3+0x33 (6e6d8e43)
6e6d8e28 3d0000009e       cmp     eax, 9E000000h
6e6d8e2d 7305             jae     wpfgfx_cor3!_ftoui3+0x24 (6e6d8e34)
6e6d8e2f f30f2cc0         cvttss2si eax, xmm0
6e6d8e33 c3               ret     
6e6d8e34 3d0000009f       cmp     eax, 9F000000h
6e6d8e39 730f             jae     wpfgfx_cor3!_ftoui3+0x3a (6e6d8e4a)
6e6d8e3b c1e007           shl     eax, 7
6e6d8e3e 0fbae81f         bts     eax, 1Fh
6e6d8e42 c3               ret     
6e6d8e43 3d0000007f       cmp     eax, 7F000000h
6e6d8e48 72e5             jb      wpfgfx_cor3!_ftoui3+0x1f (6e6d8e2f)
6e6d8e4a f30f2c0d202f706e cvttss2si ecx, dword ptr [wpfgfx_cor3!_NaN (6e702f20)]
6e6d8e52 f5               cmc     
6e6d8e53 1bc0             sbb     eax, eax
6e6d8e55 c3               ret   

The crash is happening on vcvttss2usi. I found at least one other reference to vcvttss2usi crash on Xeon processors - see https://developercommunity.visualstudio.com/content/problem/832044/vcvttss2usi-crash-at-server-cpu.html.

@thudugala, Do you have all relevant updates installed on the OS, esp. any Intel microcode updates that may be needed?

@tgani-msft, can you help take a peek at this (or perhaps redirect to someone apt) ? It looks like (a) this issue has been (minimally) known in the ecosystem and to us and (b) it seems to have been solved on some OS versions (like Windows Server 2016).

Is there something that can be done to solve this on Windows Server 2012 R2 + Xeon ? BTW, if you have trouble with symbols, please let me know - I can help build some private symbols that you can force windbg to load using .reload /f /i

@thudugala

This comment has been minimized.

Copy link

@thudugala thudugala commented Dec 4, 2019

@vatsan-madhavan

Yes I think all updates are installed.

image

image

app works in below PC

image

app does not work in below PC

image

only difference (I think) is

app works in : Intel(R) Xeon(R) CPU E5-2620
app does not work in: Intel(R) Xeon(R) Silver 4110

@vatsan-madhavan

This comment has been minimized.

Copy link
Member Author

@vatsan-madhavan vatsan-madhavan commented Dec 4, 2019

app works in : Intel(R) Xeon(R) CPU E5-2620
app does not work in: Intel(R) Xeon(R) Silver 4110

Good to know.

At this point, we need some guidance from a C++ compiler expert (@tgani-msft).

@tgani-msft

This comment has been minimized.

Copy link

@tgani-msft tgani-msft commented Dec 4, 2019

@thudugala the instructions before the offending instruction are like this from CRT code:

        cmp       __isa_available, __ISA_AVAILABLE_AVX512
        jl        _ftoui3_default       ; If VCVTTSS2USI instruction not available

        vcvttss2usi eax, xmm0

So, it could be that the variable __isa_available variable is lying if the instruction is not supported. This variable is set in vcruntime.dll startup code in the function __isa_available_init(). I would set a breakpoint there and step through the function on the failing machine to see why it's failing. This function uses the result of the CPUID instruction so it's hard to see why this is failing.

Is this happening on just 1 machine or can you repro it on other machines as well?

@tgani-msft

This comment has been minimized.

Copy link

@tgani-msft tgani-msft commented Dec 4, 2019

@vatsan-madhavan

This comment has been minimized.

Copy link
Member Author

@vatsan-madhavan vatsan-madhavan commented Dec 4, 2019

@tgani-msft @stwish-msft Can we built a targeted test app that @thudugala can build and run on the offending machine to generate useful debug logs/output for us?

Otherwise doing what you suggest may require someone to (a) installing debugging tools on a server and (b) know to use something like windbg to debug disassembly and get us the right information. Possible assuming their policies would allow this sort of flexibility, but I worry that its error prone and not within the comfort-zone of everyone.

@tgani-msft

This comment has been minimized.

Copy link

@tgani-msft tgani-msft commented Dec 4, 2019

Added @stwish-msft from the libs team who can help more.

@tgani-msft

This comment has been minimized.

Copy link

@tgani-msft tgani-msft commented Dec 4, 2019

@tgani-msft Can we built a targeted test app that @thudugala can build and run on the offending machine to generate useful debug logs/output for us?

Otherwise doing what you suggest may require someone to (a) installing debugging tools on a server and (b) know to use something like windbg to debug disassembly and get us the right information. Possible assuming their policies would allow this sort of flexibility, but I worry that its error prone and not within the comfort-zone of everyone.

I doubt very much that the libs team has the kind of logging this would require but @stwish-msft should be able to tell us more.

@vatsan-madhavan

This comment has been minimized.

Copy link
Member Author

@vatsan-madhavan vatsan-madhavan commented Dec 4, 2019

I doubt very much that the libs team has the kind of logging this would require

https://docs.microsoft.com/en-us/cpp/intrinsics/cpuid-cpuidex?redirectedfrom=MSDN&view=vs-2019 seems to have a sample that can detect supported proc. features, including AVX512* extensions. Could this be of any help as a quick tool to run on the seemingly-bad server?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.