[service] Unpredictable init_once_begin_initialize link errors in Visual Studio 16 agent

[SpaceIm]

Visual Studio 16 agent randomly fails to link symbols like init_once_begin_initialize (should live in kernel32: https://docs.microsoft.com/en-us/windows/win32/api/synchapi/nf-synchapi-initoncebegininitialize). We have to close/open PR until it works.

Affected PR:
#4810
#4798
#4752 (/cc @mathbunnyru)

[ericLemanissier]

Maybe related, I am getting some unexpected error LNK2019: unresolved external symbol __dyn_tls_on_demand_init and error LNK2001: unresolved external symbol __tls_guard on visual studio 16 too: #4818 (comment)

I found the following piece of information https://developercommunity.visualstudio.com/t/-dyn-tls-on-demand-init-and-tls-guard-related-weir/1010557 which indicates that there could be a mismatch beween the compiler and library ?

[SpaceIm]

Yes, it's exactly the issue we have I think.

And init_once_begin_initialize is just one of the unresolved symbols I see. For example in #4810, __dyn_tls_on_demand_init is also a missing symbol (actually it comes from protobuf dependency):

libprotobuf.lib(arena.obj) : error LNK2019: unresolved external symbol __dyn_tls_on_demand_init referenced in function "public: void __cdecl google::protobuf::internal::ArenaImpl::AddCleanup(void *,void (__cdecl*)(void *))" (?AddCleanup@ArenaImpl@internal@protobuf@google@@QEAAXPEAXP6AX0@Z@Z) [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(arena.obj) : error LNK2019: unresolved external symbol __tls_guard referenced in function "public: void __cdecl google::protobuf::internal::ArenaImpl::AddCleanup(void *,void (__cdecl*)(void *))" (?AddCleanup@ArenaImpl@internal@protobuf@google@@QEAAXPEAXP6AX0@Z@Z) [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(text_format.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_begin_initialize [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(dynamic_message.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_begin_initialize [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(any.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_begin_initialize [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(implicit_weak_message.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_begin_initialize [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(descriptor.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_begin_initialize [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(extension_set_heavy.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_begin_initialize [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(map_field.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_begin_initialize [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(generated_message_reflection.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_begin_initialize [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(message.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_begin_initialize [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(reflection_ops.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_begin_initialize [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(wire_format.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_begin_initialize [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(text_format.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_complete [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(dynamic_message.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_complete [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(any.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_complete [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(implicit_weak_message.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_complete [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(descriptor.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_complete [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(extension_set_heavy.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_complete [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(map_field.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_complete [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(generated_message_reflection.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_complete [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(message.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_complete [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(reflection_ops.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_complete [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
libprotobuf.lib(wire_format.obj) : error LNK2001: unresolved external symbol __imp___std_init_once_complete [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]
C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\bin\test_package.exe : fatal error LNK1120: 4 unresolved externals [C:\J\w\BuildSingleReference@3\conan-center-index\recipes\onnx\all\test_package\build\4f6198648ec03c7d41d9e22055e19eaa74b31eea\test_package.vcxproj]

[anton-danielsson]

Yeah, there was an ABI break in 16.5, see: https://docs.microsoft.com/en-us/cpp/overview/cpp-conformance-improvements?view=msvc-160#tls-guard-changes and since Conan only considers major MSVC versions this becomes a bit fragile.

There is a longer discussion that touches upon this here: conan-io/conan#3311
TLDR: It will be fixed by the new msvc compiler in settings.yml.

[bredej]

Here's a workaround and here's some background information.

The TLDR

#if _MSC_VER >= 1932 // Visual Studio 2022 version 17.2+
#    pragma comment(linker, "/alternatename:__imp___std_init_once_complete=__imp_InitOnceComplete")
#    pragma comment(linker, "/alternatename:__imp___std_init_once_begin_initialize=__imp_InitOnceBeginInitialize")
#endif

[jgsogo]

Hi! I think the best thing we can do is to ensure that all windows servers are using exactly the same VS version (first be deterministic, then the rest). Now I can see that two versions are reported for VS 2017 (and VS 2019):

OK  : Windows | ansiblewindows6 | Visual Studio 15/2017/19.1x - Found 19.16.27048
OK  : Windows | ansiblewindows6 | Visual Studio 16/2019/19.2x - Found 19.29.30143

vs

OK  : Windows | ansiblewindows4 | Visual Studio 15/2017/19.1x - Found 19.16.27045
OK  : Windows | ansiblewindows4 | Visual Studio 16/2019/19.2x - Found 19.29.30141

The only sustainable policy can be to "install latest update" and I think it is what we should commit to. Whenever we provision a new Windows server we need to either update all of them or provision all instances and decommission previous ones.

Of course there is still an underlying issue, if that last bit is not included in the packageID (given we are using compiler Visual Studio, not msvc) we might have binaries that will fail to link 😓 , but we can only build them again.

Does it sound good?

[mathbunnyru]

@jgsogo sounds good to me.
I think it's the best solution at the moment.

[jgsogo]

It's been working like this for a while now. Whenever we deploy a new server... we recreate all of them.