runtime: high CPU usage and hang in syscall_windows.go after recent Windows 10 updates

[tmm1]

What version of Go are you using (go version)?

$ go version
go version go1.19.6 windows/amd64

Does this issue reproduce with the latest release?

Have not tried go 1.20

What operating system and processor architecture are you using (go env)?

Windows 10

What did you do?

Using go-ole or gopsutil which make calls into wmi

import "github.com/yusufpapurcu/wmi"

	// gwmi -namespace root\standardcimv2 -Query "Select * from MSFT_NetFirewallRule WHERE DisplayName LIKE 'Channels DVR%'"
	var list []MSFT_NetFirewallRule
	q := wmi.CreateQuery(&list, "WHERE DisplayName LIKE 'Channels DVR%'")
	err := wmi.QueryNamespace(q, &list, "root/StandardCimv2")

What did you expect to see?

A couple weeks ago the same code and builds would return results.

What did you see instead?

A hang in syscall.SyscallN

goroutine 24 [syscall, 1765 minutes, locked to thread]:
syscall.SyscallN(0xd2ced0?, {0xc00049f858?, 0x0?, 0x0?})
	runtime/syscall_windows.go:557 +0x109
syscall.Syscall9(0x0?, 0x0?, 0xc000125380?, 0x200000003?, 0xc000125380?, 0xc000056f00?, 0x9d4eaa?, 0xc00049f910?, 0xc00049f938?, 0xc00049fa08, ...)
	runtime/syscall_windows.go:507 +0x78
github.com/go-ole/go-ole.invoke(0x18f4657b4d8, 0x1, 0x1, {0x0, 0x0, 0x18f465aaff0?})
	github.com/go-ole/go-ole@v1.2.6/idispatch_windows.go:175 +0x14e5
github.com/go-ole/go-ole.(*IDispatch).Invoke(...)
	github.com/go-ole/go-ole@v1.2.6/idispatch.go:27
github.com/go-ole/go-ole.(*IDispatch).InvokeWithOptionalArgs(0x2801e8e?, {0x27d303d?, 0x736c61663a226573?}, 0x7d65?, {0x0, 0x0, 0x0})
	github.com/go-ole/go-ole@v1.2.6/idispatch.go:69 +0x67
github.com/go-ole/go-ole/oleutil.CallMethod(...)
	github.com/go-ole/go-ole@v1.2.6/oleutil/oleutil.go:51
github.com/yusufpapurcu/wmi.(*Client).coinitService(0x2465880?, {0x0, 0x0, 0x0})
	github.com/yusufpapurcu/wmi@v1.2.2/wmi.go:175 +0x2da
github.com/yusufpapurcu/wmi.(*Client).Query(0x6172546e49222c32?, {0xc000175b00, 0x87}, {0x2428e80?, 0xc0006a7020?}, {0x0, 0x0, 0x0})
	github.com/yusufpapurcu/wmi@v1.2.2/wmi.go:245 +0x296
github.com/yusufpapurcu/wmi.Query({0xc000175b00?, 0x0?}, {0x2428e80?, 0xc0006a7020?}, {0x0?, 0x0?, 0x0?})
	github.com/yusufpapurcu/wmi@v1.2.2/wmi.go:77 +0x85
github.com/shirou/gopsutil/v3/internal/common.WMIQueryWithContext.func1()
	github.com/shirou/gopsutil/v3@v3.21.10/internal/common/common_windows.go:209 +0x3e
created by github.com/shirou/gopsutil/v3/internal/common.WMIQueryWithContext
	github.com/shirou/gopsutil/v3@v3.21.10/internal/common/common_windows.go:208 +0x1e5

goroutine 356 [syscall, 121 minutes, locked to thread]:
syscall.SyscallN(0x7ffc7e1988a0?, {0xc0007babc0?, 0x3?, 0x1b?})
	runtime/syscall_windows.go:557 +0x109
syscall.Syscall6(0x697263536d656257?, 0x57532e676e697470?, 0x7461636f4c6d6562?, 0x726f?, 0xc0007bac98?, 0x2ef0e5?, 0x6300000053?, 0xc0007bacb0?)
	runtime/syscall_windows.go:501 +0x50
golang.org/x/sys/windows.(*Proc).Call(0x2ef3e7?, {0xc0007c3ce0?, 0x1e61d20?, 0xc000107c01?})
	golang.org/x/sys@v0.2.0/windows/dll_windows.go:178 +0x233
golang.org/x/sys/windows.(*LazyProc).Call(0xc000107bf0, {0xc0007c3ce0, 0x5, 0x5})
	golang.org/x/sys@v0.2.0/windows/dll_windows.go:348 +0x50
github.com/go-ole/go-ole.CreateInstance(0xc000e6a380, 0xc0002a4620)
	github.com/go-ole/go-ole@v1.2.6/com.go:184 +0xab
github.com/go-ole/go-ole/oleutil.CreateObject({0x21da157?, 0xc0007bad80?})
	github.com/go-ole/go-ole@v1.2.6/oleutil/oleutil.go:21 +0x31
github.com/StackExchange/wmi.(*Client).coinitService(0x1e2e940?, {0xc000d71f00, 0x2, 0x2})
	github.com/StackExchange/wmi@v1.2.1/wmi.go:162 +0x209
github.com/StackExchange/wmi.(*Client).Query(0x30003?, {0xc0007d8aa0, 0x9f}, {0x1deca20?, 0xc001448ee8?}, {0xc000d71f00, 0x2, 0x2})
	github.com/StackExchange/wmi@v1.2.1/wmi.go:245 +0x296
github.com/StackExchange/wmi.Query({0xc0007d8aa0?, 0x12?}, {0x1deca20?, 0xc001448ee8?}, {0xc000d71f00?, 0x0?, 0xc0007bb238?})
	github.com/StackExchange/wmi@v1.2.1/wmi.go:77 +0x85
github.com/StackExchange/wmi.QueryNamespace(...)
	github.com/StackExchange/wmi@v1.2.1/wmi.go:59
main.checkFirewall()
	github.com/fancybits/channels-server/troubleshoot_windows.go:140 +0x105
main.troubleshooter()
	github.com/fancybits/channels-server/troubleshoot.go:65 +0x99
main.httpMiscTroubleshoot.func1(0x21a5799?)
	github.com/fancybits/channels-server/http_misc.go:135 +0x1e

[tmm1]

We have multiple users reporting issues this week. I'm collecting system information to find any commonalities.

22H2 19045.2604
KB5022834 Cumulative Update

Edition Windows 10 Home
Version 22H2
Installed on ‎28/‎01/‎2021
OS build 19045.2486
Experience Windows Feature Experience Pack 120.2212.4190.0
Last Windows Update 20/02/2023

[prattmic]

We'll need more information of course, but it is not clear to me if this is a problem with Go, with the use of the Windows APIs in github.com/go-ole/go-ole, or in the libraries themselves (bug in the Windows update).

At this point, one of the latter two looks most likely. Are these API calls allowed to be blocking? It isn't clear to me, though I will note that https://learn.microsoft.com/en-us/windows/win32/api/combaseapi/nf-combaseapi-cocreateinstance says "In particular, background tasks should limit the objects they communicate with to avoid hangs or other complications due to connected stand-by."

[tmm1]

I tried on my own Windows 10 and Windows 11 machines after updating to latest and could not reproduce.

We ended up removing our usage of wmi package and the problem went away. We still use gopsutil which also uses wmi but that works fine.