body track DLL fails to manage dependencies: ignores DLL load path, probably missing manifest,

[diablodale]

The body tracking dll k4abt.dll is improperly checking-for and loading dependent DLLs. This results in a failure to load k4abt.dll and cascades up to parent DLLs/EXEs.

It appears k4abt.dll lacks code to manage its own DLL load path and/or a manifest. Without these, Windows follows a well defined series of steps to find body tracking dependent DLLs. And can easily lead to failures.

Many customers do not create single monolithic applications. Therefore, we create DLLs...just like K4A created two DLLs. The K4A DLLs need to play well in these scenarios.

I believe I have isolated the first errant code at D:\a\1\s\src\TrackerHost\TrackerHost.cpp (469): VerifyDependencies(). It is my suspicion that the code there is not respecting DLL load paths and flags. Instead, it is blindly loading from current_working_directory.

Setup

• Windows 10 current updates
• K4a SDK v1.4.1
• K4bt SDK v1.0.1
• Visual Studio 2019 Community v16.8.4

Repro and Description

1. Create a Windows program EXE
2. Enhance that EXE so it can have new features through dynamically loading DLLs. (This is the common "plugin" scenario). Ensure this DLL feature is loaded dynamically...not static...not at program start. It is loaded only through LoadLibrary().
3. Create a DLL to be the Kinect Body Tracking feature.
4. Install your EXE in a directory like "c:\program files\myprogram"
5. Install your plugin in a different directory like "c:\myplugins"
6. Copy all body tracking dependent DLLs into c:\myplugins. These DLLs are: k4a.dll, k4bt.dll, onnxruntime.dll, cudnn64_7.dll, cublas64_100.dll, etc.
7. Run your program and dynamically load your feature

Result

The DLL feature fails to load. The cause is it can not find any of the body tracking depedent DLLs.
Using Sysinternals process monitor, I can see the loader trying to find these DLLs and it can not. It searchs the windows os folders, path, etc.... can't be found.

The feature DLL can successfully load k4a.dll and k4abt.dll by using its own manifest and/or hooking it via a delayload dll. Yet, even though the feature DLL loaded k4a.dll and it is in memory...that doesn't equate to the k4a.dll that is needed by k4abt.dll. When I use Sysinternals process monitor, I see the feature DLL successfully load k4.dll, then I see k4abt.dll load, then I see k4a.dll fail to load because the body tracking dll is separately loading it and fails due ot above issues.

I was able to capture some traces in the console window. Perhaps part of the issue is in your function VerifyDependencies(). Are you blindly using currentworkingdirectory? Or the directory of the master EXE? Ignoring flags set by SetDefaultDllDirectories() or directories set by SetDllDirectory()/AddDllDirectory()? All those are errant approaches and create the problem.

k4abt.dll   <-- incoming delay dll name string
C:\myplugins\k4abt.dll   <--- calculated full qualified pathname
Loaded 'C:\myplugins\k4abt.dll'. 
[2021-02-21 21:13:00.411] [error] [t=4308] [K4ABT] D:\a\1\s\src\TrackerHost\TrackerHost.cpp (469): VerifyDependencies(). Cannot locate cudnn64_7.dll; Please download and install cuDNN 7 at https://developer.nvidia.com/rdp/cudnn-archive
[2021-02-21 21:13:00.411] [error] [t=4308] [K4ABT] D:\a\1\s\src\TrackerHost\TrackerHost.cpp (129): Create(). At least one dependent library is missing!
[2021-02-21 21:13:00.411] [error] [t=4308] [K4ABT] D:\a\1\s\src\sdk\k4abt.cpp (38): tracker->Create(sensor_calibration, config) returned failure in k4abt_tracker_create()
exception in xxxxxxxxxx() Failed to create k4abt tracker!

In general, I think a function like VerifyDependencies() is problematic. It is the apps responsibility to check/manage dependencies with various APIs and tools like Windows Manifests. I suspect that k4abt.dll itself doesn't use a Windows manifest, therefore the windows loader doesn't look in the current or any other specific directory. In addition k4abt.dll doesn't call SetDllDirectoryW(the_k4abt.dll_dir). If it did, it would look in the same directory as itself.

Further...I was able to workaround this problematic VerifyDependencies() by surrounding k4abt::tracker::create() with...

cwd = GetCurrentDirectoryW();
SetCurrentDirectoryW(myplugin_directory);
k4abt::tracker::create()
SetCurrentDirectoryW(cwd);

When I used that hack, your VerifyDependencies() succeeded and afterwards the windows loader loaded the need DLLs.

Expected

Feature loads. No errors.

Workarounds

Hack the current working directory as above. And then use various APIs like SetDefaultDllDirectories(), SetDllDirectoryW(), and LoadLibraryExW(dllname, NULL, LOAD_LIBRARY_SEARCH_DEFAULT_DIRS | LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR) to actually load the DLL.

Note...

• The feature DLL can not list these dependent DLLs in its manifest. Why? Because the feature manifest only handles loads for its direct dependencies (e.g. k4a.dll). It is not used for its dependencies dependencies (e.g. cudnn64_7.dll).
• The feature DLL can not call SetDllDirectoryW(the_k4abt.dll_dir). That API only changes the search path for its own dependencies. It is not used for its dependencies dependencies.

Logs

Here is a log from sysinternals process monitor. This was run with my own test code which does the following in specific order

1. linked specifically with /DELAYLOAD:k4a.dll /DELAYLOAD:k4abt.dll
2. delay load hook does: SetDllDirectoryW(plugin_dll_dir); LoadLibraryExW(fulldelayloadeddllpath, NULL, LOAD_LIBRARY_SEARCH_DEFAULT_DIRS | LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR)
3. when feature loaded...
4. SetDefaultDllDirectories(LOAD_LIBRARY_SEARCH_DEFAULT_DIRS | LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR);
5. hardcoded for testing SetDllDirectoryW(L"C:\\myplugins");
6. LoadLibraryW(L"onnxruntime.dll");
7. k4abt::tracker::create()

Some interesting things to notice

• Delayload of k4a.dll and k4abt.dll themselves is successful
• My manual forced load of onnxruntime.dll from the plugin directory is successful
• Further down, it appears that k4abt.dll loads onnxruntime.dll in a way that ignores the previously loaded copy and it loads it from the Windows/System32 directory instead of the desired plugin directory.
• The two copies of onnxruntime.dll each load different CUDA dlls. The version from the k4abt distribution loads cudart64_100.dll. The version from the windows directory loads cudnn64_7.dll. 🤔 I suspect that VerifyDependencies() reads the DLLs in a different order than the Windows loader.

Operation	Path	Result
CreateFile	c:\program files\myprogram\onnxruntime.dll	NAME NOT FOUND
CreateFile	c:\myplugins\onnxruntime.dll	SUCCESS
QueryBasicInformationFile	c:\myplugins\onnxruntime.dll	SUCCESS
CloseFile	c:\myplugins\onnxruntime.dll	SUCCESS
CreateFile	c:\myplugins\onnxruntime.dll	SUCCESS
CreateFileMapping	c:\myplugins\onnxruntime.dll	FILE LOCKED WITH ONLY READERS
CreateFileMapping	c:\myplugins\onnxruntime.dll	SUCCESS
Load Image	c:\myplugins\onnxruntime.dll	SUCCESS
CreateFile	c:\myplugins\onnxruntime.dll	SUCCESS
CloseFile	c:\myplugins\onnxruntime.dll	SUCCESS
CloseFile	c:\myplugins\onnxruntime.dll	SUCCESS
CreateFile	c:\program files\myprogram\cudart64_100.dll	NAME NOT FOUND
CreateFile	c:\myplugins\cudart64_100.dll	SUCCESS
QueryBasicInformationFile	c:\myplugins\cudart64_100.dll	SUCCESS
CloseFile	c:\myplugins\cudart64_100.dll	SUCCESS
CreateFile	c:\myplugins\cudart64_100.dll	SUCCESS
CreateFileMapping	c:\myplugins\cudart64_100.dll	FILE LOCKED WITH ONLY READERS
CreateFileMapping	c:\myplugins\cudart64_100.dll	SUCCESS
Load Image	c:\myplugins\cudart64_100.dll	SUCCESS
CloseFile	c:\myplugins\cudart64_100.dll	SUCCESS
CreateFile	c:\myplugins\k4abt.dll	SUCCESS
QueryBasicInformationFile	c:\myplugins\k4abt.dll	SUCCESS
CloseFile	c:\myplugins\k4abt.dll	SUCCESS
CreateFile	c:\myplugins\k4abt.dll	SUCCESS
CreateFileMapping	c:\myplugins\k4abt.dll	FILE LOCKED WITH ONLY READERS
CreateFileMapping	c:\myplugins\k4abt.dll	SUCCESS
Load Image	c:\myplugins\k4abt.dll	SUCCESS
CreateFile	c:\myplugins\k4abt.dll	SUCCESS
CloseFile	c:\myplugins\k4abt.dll	SUCCESS
CloseFile	c:\myplugins\k4abt.dll	SUCCESS
CreateFile	c:\program files\myprogram\onnxruntime.dll	NAME NOT FOUND
CreateFile	c:\program files\myprogram\onnxruntime.dll	NAME NOT FOUND
CreateFile	C:\Windows\System32\onnxruntime.dll	SUCCESS
QueryBasicInformationFile	C:\Windows\System32\onnxruntime.dll	SUCCESS
CloseFile	C:\Windows\System32\onnxruntime.dll	SUCCESS
CreateFile	c:\program files\myprogram\onnxruntime.dll	NAME NOT FOUND
CreateFile	c:\program files\myprogram\onnxruntime.dll	NAME NOT FOUND
CreateFile	C:\Windows\System32\onnxruntime.dll	SUCCESS
QueryBasicInformationFile	C:\Windows\System32\onnxruntime.dll	SUCCESS
CloseFile	C:\Windows\System32\onnxruntime.dll	SUCCESS
CreateFile	c:\program files\myprogram\cudnn64_7.dll	NAME NOT FOUND
CreateFile	c:\program files\myprogram\cudnn64_7.dll	NAME NOT FOUND
CreateFile	C:\Windows\System32\cudnn64_7.dll	NAME NOT FOUND
CreateFile	C:\Windows\System\cudnn64_7.dll	NAME NOT FOUND
CreateFile	C:\Windows\cudnn64_7.dll	NAME NOT FOUND
CreateFile	C:\Windows\System32\wbem\cudnn64_7.dll	NAME NOT FOUND
CreateFile	C:\Windows\System32\WindowsPowerShell\v1.0\cudnn64_7.dll	NAME NOT FOUND
...	..and it continues to check in every location of PATH and eventually fails causing program failure	...

[vladalx]

Configuring the dll search path(s) is the responsibility of the programmer not of the library.
The VerifyDependencies functions only checks for the installation of two critical components, CUDA and CUDNN.

[diablodale]

Hello. You seem to be a new user or perhaps a bot.
I believe you are errant in your post. I must point out that you did not read my information above as I documented correct usage of DLL search paths and that VerifyDependencies() is incorrectly looking in CWD rather than the DLL search path.

Unless you can provide specific information, I believe your post is not relevant to this conversation.

[scotthsu98052]

@qm13 this is a feature request. Current BTSDK doesn't support application isolation (side-by-side assemblies). The standard search order for Windows applications is used for loading linked dlls.

[diablodale]

@qm13, that is misleading. While LoadLibrary might be using standard path, the function VerifyDependencies IS NOT.
This is distinct.
Two actions.

1. VerifyDependencies is a separate and independent function in the k4abt that checks for the presence of certain files.
2. Later, various DLLs are loaded by k4abt using Windows standard paths

It is (1) of which I'm reporting this bug. Not (2). This is a recurring misunderstanding by your team.

[scotthsu98052]

Hi @diablodale,
Re: Perhaps part of the issue is in your function VerifyDependencies(). Are you blindly using currentworkingdirectory? Or the directory of the master EXE?

No, VerifyDependencies is not doing what you thought. It is just a wrapper call of Windows API SearchPath.
return SearchPath(nullptr, fileName.data(), ".dll", 0, nullptr, nullptr) != 0;

The lpPath parameter is NULL, so the default search order is similar to how system loads DLLs.
If you explicitly use LoadLibrary to load dlls as you described, then this check won't work well for you, but this is what current design is.

[diablodale]

Ah ha! You have admitted the problem.

SearchPath api does not follow the same rules and have the same functionality of the DLL load APIs. 👎

The SearchPath function is not recommended as a method of locating a .dll file if the intended use of the output is in a call to the LoadLibrary function. This can result in locating the wrong .dll file because the search order of the SearchPath function differs from the search order used by the LoadLibrary function. If you need to locate and load a .dll file, use the LoadLibrary function.

It follows different paths, ignores apis like SetDllDirectory() and AddDllDirectory(), and as officially written by your own corporation...should not be used to find DLLs, resources, etc.

This is related to https://docs.microsoft.com/en-us/azure/kinect-dk/troubleshooting#using-body-tracking-sdk-with-unreal and its general topic of "finding required resources (dll,onnx,etc.) for k4a and k4abt SDKs.

[qm13]

Done

[diablodale]

@qm13, Done in what version? For customers to verify unseeable code changes, we need to know the exact version in which is was potentially fixed so that customers can verify, escalate non-fix, etc.