awt.dll Can't find dependent libraries on windows server 2019 (JDK 11)

[edonin]

Hi,

I am trying to run the adopt openjdk 11 (Hotspot + only JRE) on windows server 2019.
The first time i am trying to use awt, i got an exception :

Caused by: java.lang.UnsatisfiedLinkError: F:\XXXX\jre\bin\awt.dll: Can't find dependent libraries
at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
at java.base/java.lang.ClassLoader$NativeLibrary.load(Unknown Source)
at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(Unknown Source)
at java.base/java.lang.ClassLoader.loadLibrary0(Unknown Source)
at java.base/java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.base/java.lang.Runtime.loadLibrary0(Unknown Source)
at java.base/java.lang.System.loadLibrary(Unknown Source)
at java.desktop/java.awt.Toolkit$3.run(Unknown Source)
at java.desktop/java.awt.Toolkit$3.run(Unknown Source)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.desktop/java.awt.Toolkit.loadLibraries(Unknown Source)
at java.desktop/java.awt.Toolkit.(Unknown Source)
at java.desktop/java.awt.Color.(Unknown Source)

It is reproducable with jdk-11.0.7+10.2 and jdk-11.0.6+10 (both hotsport and JRE)

It is a fresh and up-to-date windows server 2019.
Manually installing the microsoft Visual C++ redistributable for Visual Studio 2015, 2017 and 2019 fixes the issue, but I prefer to have a JDK that does not have any pre-requesites.

[aahlenst]

Related: #32

@edonin Can you please share a sample project or a class that reproduces the problem? This would make it significantly easier for me to work on that.

[aahlenst]

@karianna In #32 (same problem, but on JDK 8) you indicated the problem should be resolved by moving up to VS 2017. AdoptOpenJDK 11 is on VS 2017 and the problem persists, so it does not seem that this is the solution. Having to install the redistributables indicates that we either don't bundle some DLLs that we should or we're linking against the wrong versions. Do you have an expert handy that can look into this or at least talk me through diagnosing this? The latter would have the advantage that I'd learn something which could come handy in the future.

[karianna]

I've asked @d3r3kk to look into this whole stack this quarter - but he'll be weighing this up against his other priorities :-). Might be worth starting a discussion on slack with all of us.

[edonin]

Hi, I am digging a bit inside the problem:
it does not occur when launching the java.exe directly, but when I am using a java wrapper using the server\jvm.dll directly.
Only this code
System.out.println(java.awt.Color.BLACK.toString());
could cause the error.

[edonin]

And I have the same issue with Zulu JDK 11 and Oracle JDK 11.

[aahlenst]

@edonin Super helpful, thanks a lot.

[aahlenst]

This seems to turn into another wild goose chase 😞. I could not reproduce the behaviour on Windows Server 2019 with almost nothing installed, neither with AdoptOpenJDK 11 nor AdoptOpenJDK 8. Any ideas? Full details below.

AdoptOpenJDK 11 JRE:

PS C:\Users\Administrator\Desktop> systeminfo /fo csv | ConvertFrom-Csv | select OS*, System*, Hotfix* | Format-List

OS Name             : Microsoft Windows Server 2019 Standard
OS Version          : 10.0.17763 N/A Build 17763
OS Manufacturer     : Microsoft Corporation
OS Configuration    : Standalone Server
OS Build Type       : Multiprocessor Free
System Boot Time    : 7/2/2020, 6:55:54 AM
System Manufacturer : OpenStack Foundation
System Model        : OpenStack Nova
System Type         : x64-based PC
System Directory    : C:\Windows\system32
System Locale       : en-us;English (United States)
Hotfix(s)           : 6 Hotfix(s) Installed.,[01]: KB4532947,[02]: KB4462930,[03]: KB4512577,[04]: KB4516115,[05]:
                      KB4523204,[06]: KB4534273

PS C:\Users\Administrator\Desktop> Get-WmiObject win32_product

IdentifyingNumber : {8F984728-BDA5-4135-99FC-A4F14F918B9C}
Name              : Spice webdavd 2.2 (64-bit)
Vendor            : The Spice Project
Version           : 2.2.
Caption           : Spice webdavd 2.2 (64-bit)

IdentifyingNumber : {9B265631-958D-415B-9925-53DEEC43E31D}
Name              : QEMU guest agent
Vendor            : RedHat
Version           : 100.0.0
Caption           : QEMU guest agent

IdentifyingNumber : {04D1CEE8-553B-4762-A9FC-463851C0DD35}
Name              : AdoptOpenJDK JRE with Hotspot 11.0.7.10 (x64)
Vendor            : AdoptOpenJDK
Version           : 11.0.7.10
Caption           : AdoptOpenJDK JRE with Hotspot 11.0.7.10 (x64)

IdentifyingNumber : {ED85F19F-057A-4EE6-BC8D-F576DEACE78D}
Name              : Cloudbase-Init 0.9.11
Vendor            : Cloudbase Solutions Srl
Version           : 0.9.11.0
Caption           : Cloudbase-Init 0.9.11

PS C:\Users\Administrator\Desktop> java -version
openjdk version "11.0.7" 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+10)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.7+10, mixed mode)
PS C:\Users\Administrator\Desktop> java AwtTest
java.awt.Color[r=0,g=0,b=0]

Same machine, AdoptOpenJDK 8 JRE:

PS C:\Users\Administrator\Desktop> Get-WmiObject win32_product


IdentifyingNumber : {8F984728-BDA5-4135-99FC-A4F14F918B9C}
Name              : Spice webdavd 2.2 (64-bit)
Vendor            : The Spice Project
Version           : 2.2.
Caption           : Spice webdavd 2.2 (64-bit)

IdentifyingNumber : {9B265631-958D-415B-9925-53DEEC43E31D}
Name              : QEMU guest agent
Vendor            : RedHat
Version           : 100.0.0
Caption           : QEMU guest agent

IdentifyingNumber : {C7AF7F2D-AF0F-48AA-8238-A715A73142C7}
Name              : AdoptOpenJDK JRE with Hotspot 8.0.252.09 (x64)
Vendor            : AdoptOpenJDK
Version           : 8.0.252.09
Caption           : AdoptOpenJDK JRE with Hotspot 8.0.252.09 (x64)

IdentifyingNumber : {ED85F19F-057A-4EE6-BC8D-F576DEACE78D}
Name              : Cloudbase-Init 0.9.11
Vendor            : Cloudbase Solutions Srl
Version           : 0.9.11.0
Caption           : Cloudbase-Init 0.9.11



PS C:\Users\Administrator\Desktop> java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.252-b09, mixed mode)
PS C:\Users\Administrator\Desktop> java AwtTest
java.awt.Color[r=0,g=0,b=0]

The test program:

public class AwtTest {
    public static void main(String[] args) {
        System.out.println(java.awt.Color.BLACK.toString());
    }    
}

[edonin]

Thanks for your time @aahlenst.
I am not reproducing the issue when launching the java.exe directly, but when I use a java wrapper (in C code) that use JNI for launching the JVM (through JNI_CreateJavaVM call). Curently, I tested only jdk 11, not jdk 8.

It seems that the awt.dll was well loaded when I asked to load the jvm.dll (where the JNI_CreateJavaVM is located), but not some of its dependent library.
I will investigate a bit more on this code, and try to give you a C code test case.

[aahlenst]

@edonin Can you produce something like https://github.com/aahlenst/datetimeformatter-parse-bug for us? That would be most helpful.

[edonin]

@aahlenst : I will try to take a look, after.

Neverthless, I could give you my finds:

Our C java wrapper for windows is doing the following (very simplified):

// Add the jre\bin directory in the dll lookup
SetDllDirectory("jre\bin");
//Load the jvm.dll
HMODULE jvmDLL = LoadLibrary("jre\bin\server\jvm.dll");
//Reset the dll lookup to the standard one.
SetDllDirectory(0);
// Then instanciate the JNI_CreateJavaVm
JniCreateVM jniCreateVM = (JniCreateVM)GetProcAddress(jvmDLL, "JNI_CreateJavaVM");
(...)
jint result = jniCreateVM(&jvm, (void**)&env, &args);

It used to work with Oracle JDK 8, but it is no more working with openJDK 8 or 11 (I have just tested openjdk8).
Neverthless, if you remove the line SetDllDirectory(0), the code is now working:

// Add the jre\bin directory in the dll lookup
SetDllDirectory("jre\bin");
//Load the jvm.dll
HMODULE jvmDLL = LoadLibrary("jre\bin\server\jvm.dll");
// Then instanciate the JNI_CreateJavaVm
JniCreateVM jniCreateVM = (JniCreateVM)GetProcAddress(jvmDLL, "JNI_CreateJavaVM");
(...)
jint result = jniCreateVM(&jvm, (void**)&env, &args);

What I understood (sorry I am not a C/C++ dev, so I might be wrong), is when the JVM is initializing the java.awt.Toolkit class, we have some System.loadLibrary("awt") to execute. That will trigger the code in hotspot/share/runtime/os.cpp native_java_library()

    if (dll_locate_lib(buffer, sizeof(buffer), Arguments::get_dll_dir(),
                       "java")) {
      _native_java_library = dll_load(buffer, ebuf, sizeof(ebuf));
    }

And the windows implementation of the dll_load function is localised in hotspot/os/windows/os_windows.cpp

void * os::dll_load(const char *name, char *ebuf, int ebuflen) {
  void * result = LoadLibrary(name);

So, from my understanding, hotspot is able to to figure out that the awt.dll is localized in my jre\bin directory and is able to load it, even in the jre\bin path is no more in the search dll path.
Nevertheless, It forget that this dll may required other dlls (the visual studio redistribuable dlls) are also located in this directory. It seems to be why we having such error.

It explains:

• why we have no error when launching the java.exe : the directory where the executable is located is inside the default dll path.
• why we have no error, when I install manually the visual studio redistribuable. The default dll path contains the system library directory where we have the redistribuable dll.
• It does not seem to be a wrong versio of the visual studio dll or a missing dll.

For me, I have a workaround by do not calling SetDllDirectory(0) after loading the jvm.dll library, but a more lasting solution could be to execute SetDllDirectory(dll_path) inside the os_windows.cpp at startup to be sure that the dll directory will be present in the default dll path.

Moreover, here is the windows documentation about the dll path search : https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-setdlldirectorya

What do you think ?

[aahlenst]

This is not my area of expertise. I haven't found any recommendations, either.

For me, I have a workaround by do not calling SetDllDirectory(0) after loading the jvm.dll library, ...

I'm curious: Why are you doing it in the first place?

... but a more lasting solution could be to execute SetDllDirectory(dll_path) inside the os_windows.cpp at startup to be sure that the dll directory will be present in the default dll path.

Reading the Microsoft docs, that would not help: If you pass NULL, the default search order will be restored and that does not include jre/bin. See https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-setdlldirectorya#remarks. Java would have to re-insert it upon every call to LoadLibrary, so you cannot remove it from the search path. This may not be desirable.

[edonin]

I'm curious: Why are you doing it in the first place?

It is a old code, and I was not responsible of it :) The only info I could find is the commit messageand some comments: fix bugs on windows // The library jvm.dll requires the MSVC runtime, present under jre/bin.`

Java would have to re-insert it upon every call to LoadLibrary, so you cannot remove it from the search path. This may not be desirable.

I agree. But i do not understand how it was working with Oracle JDK 8...

[aahlenst]

But i do not understand how it was working with Oracle JDK 8...

A lot of things were different back then.

Are you happy with closing the issue or is there anything left to do?

[edonin]

It is ok for me to close it.

[Mike4Online]

I was getting this error trying to start a Java application wrapped in an Apache Commons Daemon Procrun exe using Zulu Community JDK 11.0.11+9 x86_64 (or amd_64) edition.

My first work-around was to open a Command Prompt first, manually reset the PATH so it included my application's jre/bin folder (and no other jre/bin folder), and then run my exe, e.g. "C:\Program Files\MyApp\MyJavaServiceWrapper.exe".

I was able to resolve this issue by upgrading my Apache Commons Daemon Procrun EXE from version 1.0.15.0 to version 1.2.4.0.