Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Loading onnxruntime.dll from path crashes on Windows build 19025 #932

Closed
conatsera opened this issue Nov 22, 2019 · 17 comments
Closed

Loading onnxruntime.dll from path crashes on Windows build 19025 #932

conatsera opened this issue Nov 22, 2019 · 17 comments
Assignees
@yijiew
Labels
Body Tracking Issue related to the Body Tracking SDK Bug Something isn't working Triage Approved The Issue has been approved by an Azure Kinect team member.

Comments

@conatsera
Copy link

conatsera commented Nov 22, 2019 •
edited
Loading

Describe the bug
On windows insider build 19025 (slow ring), tests fail if loading onnxruntime.dll from the path. This bug is very likely in Windows and not the body tracking SDK, but I don't see any other way to report this without knowing how k4abt.dll is loading onnxruntime.dll. Other applications and even Cuda/CuDNN loaded by k4abt.dll do not seem to be affected when loaded from the path.

To Reproduce
Run any of the body tracking samples on build 19025 with onnxruntime.dll in the path but not the application directory.

Expected behavior
onnxruntime.dll should load correctly wherever it is located.

Logs
Trace level logging does not provide any useful info.
Stack trace:
KernelBase.dll!RaiseException() (Code= 0xC06D007F: Procedure not found)
k4abt.dll@0x‭18025AFEB‬
k4abt.dll@0x‭1802583A6‬
k4abt.dll@0x‭180248AE5‬
k4abt.dll@0x‭‭180240D28‬
k4abt.dll@0x‭1800BC265‬
k4abt.dll@0x‭1800C7F19‬
k4abt.dll@0x‭18008C0F2‬

Desktop (please complete the following information):

  • OS with Version: 20H1 (19025.vb_release.191112-1414)
  • SDK Version: 1.3.0 (BT 0.9.4)
  • Firmware version: N/A

Additional context
None needed.
@conatsera conatsera added Body Tracking Issue related to the Body Tracking SDK Bug Something isn't working Triage Needed The Issue still needs to be reviewed by Azure Kinect team members. labels Nov 22, 2019
@yijiew
Copy link
Contributor

yijiew commented Nov 25, 2019 •
edited
Loading

Hi @conatsera : thanks for reporting this issue. k4abt.dll uses a standard way to delay load onnxruntime.dll. Have you tried it before your OS got updated to the latest one? Did it work before?

@yijiew
Copy link
Contributor

yijiew commented Nov 25, 2019

@conatsera Based on this thread, loading onnxruntime.dll from path seemed to work before:
#709.

@conatsera
Copy link
Author

That's correct, the same setup was working in build 19013. I have actually noticed some strange behavior of LoadLibraryW in another project that also had functioned before and I will investigate further. A new slow ring build has been released today, I will check for changes.

@yijiew
Copy link
Contributor

yijiew commented Nov 27, 2019 •
edited
Loading

@conatsera Thanks for your confirmation! I got a repro of this issue on our latest internal build. It fails in the delayhlp.cpp file (the file is included in the Windows SDK):
image

The error code 182 means "ERROR_INVALID_ORDINAL". I am not an expert on dll loading in Windows... Any help from you to debug this issue would be welcomed :)

@tesych tesych removed the Triage Needed The Issue still needs to be reviewed by Azure Kinect team members. label Dec 2, 2019
@conatsera
Copy link
Author

Sorry, I'm not expert in that either. That error doesn't make sense to me in this context, but the bug appears to have been fixed in 19037 anyways.

@conatsera
Copy link
Author

It appears to have only temporarily worked until after a rebuild with 0.9.5. I suppose I now have something to work off of for debugging this issue.

@conatsera conatsera reopened this Dec 8, 2019
@yijiew
Copy link
Contributor

yijiew commented Dec 9, 2019

@conatsera , thanks for checking and verifying this issue. For the 0.9.5 release, we didn't change anything related to the dll loading. Do you happen to get a new windows update in between? If you go back to the 0.9.4 version, do you still see this problem?

@qm13 qm13 assigned yijiew and unassigned yijiew and qm13 Dec 9, 2019
@qm13 qm13 added the Investigating Dev team needs to Investigate label Dec 9, 2019
@qm13 qm13 closed this as completed Jan 6, 2020
@yijiew yijiew reopened this Feb 6, 2020
@yijiew yijiew mentioned this issue Feb 6, 2020
Closed
@microsoft microsoft deleted a comment from qm13 Feb 6, 2020
@yijiew
Copy link
Contributor

yijiew commented Feb 10, 2020

I got in touch with the OS team about this issue. The latest windows includes an onnxruntime.dll in the system32 path. However, the onnxruntime.dll we are using in the body tracking SDK is not the same one the windows system is using. Unfortunately they are sharing the same name now. Then it depends on from which folder the executable loads the dll first. Normally, it loads dll first at the same folder as the executable. That is why you see the Body Tracking Viewer works without problem if you put onnxruntime.dll at the same folder as the exe. If the dll is not there, then it normally start to search from system32 path. In this case, it found the wrong system onnxruntime.dll and started to load from there. Since it is the wrong one, it failed to load correctly and then crash. (For your reference, you can find the dll search order here )
Thanks for reporting this issue. We will come up with a solution in next release as soon as possible.

@mromeolisten
Copy link

Hi,
Seeing the same issue. Our app crashes utilizing the below SDK version but the Azure Kinect Body Tracking viewer app runs. The system info is Windows 10 1903v2 english x64. Azure Kinect Body Viewer 0.9.5 SDK 1.3.0. One solution was to place the onnxruntime.dll files with the files that were previously working but when running the app that utilizes the Body Tracking SDK it looks to be that it replaces the onnxruntime.dll with a new version.

@qm13 qm13 added Triage Approved The Issue has been approved by an Azure Kinect team member. and removed Investigating Dev team needs to Investigate labels Feb 25, 2020
@qm13
Copy link
Collaborator

qm13 commented Feb 25, 2020

We are working on a fix for the problem.

@yijiew
Copy link
Contributor

yijiew commented Feb 25, 2020

Hi @mromeolisten , could you clarify a bit on your "One solution"? The sentence sounds confusing to me... What do you mean by saying "when running the app that utilizes the Body Tracking SDK it looks to be that it replaces the onnxruntime.dll with a new version." What does "it" refer to?

@mromeolisten
Copy link

@yijiew Apologies. I meant to say that I had attempted to find a working solution by replacing the onnxruntime.dll files with older onnxruntime.dll files I had backed up. This revealed something interesting that could be useful in finding a fix. It appears that the onnxruntime.dll files get updated when running our app that utilizes the Body Tracking SDK, although I'm just making an observation I don't have any debug info to describe what I'm seeing.

@yijiew
Copy link
Contributor

yijiew commented Feb 25, 2020

@mromeolisten I see. Thanks for trying to debug this issue. I think we have a good understanding of the problem now. The issue is the naming conflict between our onnxruntime.dll and the onnxruntime.dll that located in system32 folder. We will launch a fix that tries to dynamically load the onnxruntime.dll in the SDK folder first. Please stay tuned.

If you want some quick workaround, you will need to delete the onnxruntime.dll in the system32 folder (C:\Windows\System32). I have confirmed with Windows team that this dll is not used in any system process yet. It is safe to delete for now.
Here is the tutorial about how to get permission to delete a file in system32 folder: https://www.drivereasy.com/knowledge/you-require-permission-from-trustedinstaller-error-fixing-guide/

@yijiew
Copy link
Contributor

yijiew commented Mar 10, 2020

The latest 1.0.1 release has addressed this issue.

@yijiew yijiew closed this as completed Mar 10, 2020
@rfilkov
Copy link

rfilkov commented Mar 10, 2020

The latest 1.0.1 release has addressed this issue.

@yijiew Is v1.0.1 already released? If yes, could you please provide a link.

@yijiew
Copy link
Contributor

yijiew commented Mar 10, 2020

@rfilkov I just realized that the official documentation is not updated yet. You can access the 1.0.1 version SDK here: msi, nuget

@rfilkov
Copy link

rfilkov commented Mar 10, 2020

@yijiew Thank you!

@RamType0 RamType0 mentioned this issue Mar 23, 2020
Closed
@benMOOVENCY benMOOVENCY mentioned this issue Oct 12, 2020
Closed
@irexyc irexyc mentioned this issue Apr 2, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yijiew yijiew

Labels
Body Tracking Issue related to the Body Tracking SDK Bug Something isn't working Triage Approved The Issue has been approved by an Azure Kinect team member.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@yijiew @rfilkov @conatsera @qm13 @mromeolisten @tesych