Maybe wrong optimization

[chenbaiyu0414]

I tried this code:

fn main() {
    println!("test_func returns: {}", test_func());
}

fn test_func() -> bool {
    unsafe {
        let hwnd = windows::Win32::Foundation::HWND(0x2006c as *mut std::ffi::c_void); // here is a valid hwnd from my os

        let mut cloaked = false;

        let is_ok = windows::Win32::Graphics::Dwm::DwmGetWindowAttribute(
            hwnd,
            windows::Win32::Graphics::Dwm::DWMWA_CLOAKED,
            &mut cloaked as *mut _ as *mut _,
            size_of::<windows::core::BOOL>() as u32,
        ).is_ok();

        println!("is_ok: {is_ok}, cloaked: {cloaked}");

        if is_ok && cloaked {
            false
        } else {
            true
        }
    }
}

I expected to see this happen:

is_ok: true, cloaked: true
test_func returns: false

Instead, this happened:

is_ok: true, cloaked: true
test_func returns: true

Meta

rustc --version --verbose:

rustc 1.87.0 (17067e9ac 2025-05-09)
binary: rustc
commit-hash: 17067e9ac6d7ecb70e50f92c1944e545188d2359
commit-date: 2025-05-09
host: x86_64-pc-windows-msvc
release: 1.87.0
LLVM version: 20.1.1

Cargo.toml

[package]
name = "abc"
version = "0.1.0"
edition = "2024"

[dependencies]

[dependencies.windows]
version = "0.61.1"
features = [
    "Win32_System_LibraryLoader",
    "Graphics_Capture",
    "Win32_System_WinRT",
    "Win32_System_WinRT_Graphics_Capture",
    "Win32_Graphics_Gdi",
    "Win32_UI_WindowsAndMessaging",
    "UI_WindowManagement",
    "Win32_System_Diagnostics_ToolHelp",
    "Win32_Graphics_Dwm"
]

Summary

Totally, I defined a bool variable cloaked and cast to raw mutable pointer and passed to WIN32 API DwmGetWindowAttribute, the API maybe write true or false to cloaked, finally returns a bool. As you can see, if is_ok && cloaked is true, returns false, otherwise true. But the actual result is test_func returns true when both is_ok and cloaked are true. So I guess maybe cast cloaked to raw pointer lost ownership and cause compiler makes wrong optimization?

[RalfJung]

Looking at https://learn.microsoft.com/en-us/windows/win32/api/dwmapi/ne-dwmapi-dwmwindowattribute, the value for "cloaked" can be 0, 1, 2, or 4. A bool must be 0 or 1 (see the reference). Therefore your code has UB, and that's why you are seeing strange effects.

The value also seems to be 4 byte large, so there's further UB due to an out-of-bounds access (so other parts of the stack can get clobbered). The fact that you had to cast the raw pointer type around should be a hint that something is probably not quite right.

[chenbaiyu0414]

Thanks, I forgotten check the value size