Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jansi in GraalVM native images #162

Open
remkop opened this issue Oct 21, 2019 · 2 comments

Comments

@remkop
Copy link

@remkop remkop commented Oct 21, 2019

TL;DR - for Jansi Users

Jansi by itself is insufficient to show colors in Java applications compiled to GraalVM native images for Windows. This is partly because GraalVM requires configuration and partly because Jansi internally depends on non-standard system properties, without a graceful fallback if these properties are absent (as is the case in GraalVM).

Users may be interested in combining Jansi with picocli-jansi-graalvm until this issue is fixed.

For the Jansi Maintainers

Background

I'm working on picocli support for Graal native images. Building native images for Windows is still experimental, and it's not perfect but it works.

I would like to provide support for colored output on Windows console when executing a native image. Using Jansi for this is the obvious choice. (We don't need to worry about other OS-es.)

Would you be interested in helping to provide Jansi support for GraalVM native images on Windows?

Objective

Easily create a single Windows executable that shows colors on the console.

Problem Description

We need two configuration files to make Jansi work in a native image. If we can include these in the Jansi JAR file, it becomes very easy for developers to create native images.

  • JNI - Jansi uses JNI, and all classes, methods, and fields that should be accessible via JNI must be specified during native image generation in a configuration file
  • resources - to get a single executable we need to bundle the jansi.dll in the native image. We need some configuration to ensure the jansi.dll is included as a resource.

Also, there is a problem extracting the jansi.dll from the native image. This should work similarly to extracting it from the jansi JAR, but there is some difference and org.fusesource.hawtjni.runtime.Library (in jansi 1.18) is unable to extract jansi.dll from the native image.

What I've done so far

Created the below GraalVM configuration files:

  • jni-config.json
  • resource-config.json

I've created a jni-config.json file for all classes, methods and fields in org.fusesource.jansi.internal.CLibrary and org.fusesource.jansi.internal.Kernel32. (See attached jni-config.json.txt file.)

(The jni-config.json config file can be re-generated with the below command if necessary: )

java -cp picocli-4.0.4.jar;jansi-1.18.jar;picocli-codegen-4.0.5-SNAPSHOT.jar ^
  picocli.codegen.aot.graalvm.JniConfigGenerator ^
  org.fusesource.jansi.internal.CLibrary ^
  org.fusesource.jansi.internal.Kernel32 ^
  -o=.\jni-config.json

Secondly, we need to ensure that the jansi.dll is included in the native image, just like it is included in the Jansi JAR. To do this, we register it as a resource with GraalVM using configuration. The Jansi JAR file includes native libraries for many platforms, but we only need the jansi.dll for 64-bit Windows. We can ensure this DLL is included in the native image by supplying this resource-config.json file:

{
  "resources": [
    {"pattern": "META-INF/native/windows64/jansi.dll"}
  ]
}

What I need from you

Summary:

  • Make things easy for developers by including GraalVM config in Jansi in future releases
  • There is a problem in the hawtjni Library logic that extracts the jansi.dll, when running as a native image. There is a workaround, but it would be great if the Library logic itself could be fixed so that the workaround is not necessary.

Please include GraalVM configuration in Jansi distribution going forward

Developers can specify the above two configuration files on the command line when creating a native image, but this is cumbersome.

If Jansi can include these two configuration files in the jansi-x.x.jar in the following location, then the GraalVM native-image generator tool will pick up the configuration automatically:

/META-INF/native-image/jansi/jni-config.json
/META-INF/native-image/jansi/resource-config.json

Extraction Issue in hawtjni Library

The configuration alone is not sufficient to get colored output from a native image. When I create a native image for a sample program that runs AnsiMain after AnsiConsole.systemInstall(), I get the following error:

Jansi null (Jansi native null, HawtJNI runtime null)

library.jansi.path=
library.jansi.version=
Exception in thread "main" java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [java.lang.LinkageError: Unable to load library jansi]
        at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:233)
        at org.fusesource.hawtjni.runtime.Library.load(Library.java:185)
        at org.fusesource.jansi.AnsiMain.main(AnsiMain.java:63)
        at App.main(App.java:8)

Setting the library.jansi.path system property to a writable directory did not help, resulting in a similar but longer error message:

...
Exception in thread "main" java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [java.lang.LinkageError: Unable to load library from C:\Users\remko\IdeaProjects\native-java-cli-demo\build\graal\windows-1\amd64\jansi.dll, java.lang.LinkageError: Unable to load library from C:\Users\remko\IdeaProjects\native-java-cli-demo\build\graal\windows-1\jansi.dll, java.lang.LinkageError: Unable to load library from C:\Users\remko\IdeaProjects\native-java-cli-demo\build\graal\windows\jansi.dll, java.lang.LinkageError: Unable to load library from C:\Users\remko\IdeaProjects\native-java-cli-demo\build\graal\.\jansi.dll, java.lang.LinkageError: Unable to load library jansi]
        at org.fusesource.hawtjni.runtime.Library.doLoad(Library.java:233)

Cause: problem in extraction logic when running in native image

The problem does not manifest when the application extracts jansi.dll before calling AnsiConsole.systemInstall(): there is no UnsatisfiedLinkError, and the console shows colors! See the workaround below for details.

So there is some problem in the hawtjni Library extraction logic that makes it fail when run in a GraalVM native image. I have not been able to determine what that problem is exactly.

Would you be interested in helping me figure out where the problem is, and fixing the hawtjni Library extraction logic?

Steps to reproduce:

choco install windows-sdk-7.1 kb2519277

Then (from the cmd prompt), activate the sdk-7.1 environment:

call "C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd"

This starts a new Command Prompt, with the sdk-7.1 environment enabled. Run all subsequent commands in this Command Prompt window.

Example app:

import org.fusesource.jansi.AnsiConsole;
import org.fusesource.jansi.AnsiMain;
import java.io.IOException;

class App {
    public static void main(String[] args) throws IOException {
        AnsiConsole.systemInstall();
        AnsiMain.main(args);
        AnsiConsole.systemUninstall();
    }
}

Here is the command to create the native image:

javac -cp jansi-1.18.jar App.java

C:\apps\graalvm-ce-19.2.1\bin\native-image -H:JNIConfigurationFiles=jni-config.json ^
  -H:ResourceConfigurationFiles=resource-config.json ^
  -cp .;jansi-1.18.jar ^
  App myapp

This will create a native image myapp.exe in the current directory.
Executing this native image will show the UnsatisfiedLinkError.

Workaround: Extract jansi.dll in the application

To fix the UnsatisfiedLinkError and show colors, replace the main method in App with the below.

Here is the code to “manually” extract the jansi.dll from the native image resource and add it to the java.library.path in the application (instead of relying on Library:

    public static void main(String[] args) {
        URL url = org.fusesource.jansi.internal.CLibrary.class
                .getResource("/META-INF/native/windows64/jansi.dll");
        File lib = new File(System.getProperty("java.io.tmpdir"), "jansi.dll");
        if (!lib.getParentFile().exists() && !lib.getParentFile().mkdirs()) {
            throw new IOException(lib.getParentFile() +
                    " does not exist and could not be created");
        }
        try (InputStream in = url.openStream()) {
            Files.copy(in, lib.toPath(), StandardCopyOption.REPLACE_EXISTING);
        }
        String libPath = System.getProperty("java.library.path");
        if (libPath != null && libPath.length() > 0) {
            libPath += File.pathSeparator;
        }
        libPath += lib.getParentFile().getAbsolutePath();
        System.setProperty("java.library.path", libPath);

        AnsiConsole.systemInstall();
        AnsiMain.main(args);
        AnsiConsole.systemUninstall();
    }

With this workaround, colors are shown on the console when running as a native image. It would be great if the Library logic itself could be fixed so that this workaround is not necessary.

Sorry for the very long issue.

@remkop

This comment has been minimized.

Copy link
Author

@remkop remkop commented Oct 22, 2019

In the cold light of the next morning I realize that a potentially much simpler solution would be to link jansi.dll statically with the native image at compile time. I’ll investigate this option next.

@remkop

This comment has been minimized.

Copy link
Author

@remkop remkop commented Oct 23, 2019

Cause of the dll extraction issue: getBitModel is broken

I believe I found the reason why org.fusesource.hawtjni.runtime.Library cannot extract jansi.dll from the native image: its implementation of the getBitModel method depends on non-standard system properties which are not present in SubstrateVM (the native image JVM).

In a GraalVM native image, the getBitModel method returns -1, and the extractAndLoad logic will try the following locations, all of which will fail:

/META-INF/native/windows-1/amd64/jansi.dll
/META-INF/native/windows-1/jansi.dll
/META-INF/native/windows/jansi.dll

FIX: make getBitModel work in SubstrateVM

To fix this, I propose we change the getBitModel implementation to this:

public static int getBitModel() {
    String prop = System.getProperty("sun.arch.data.model");
    if (prop == null) {
        prop = System.getProperty("com.ibm.vm.bitmode");
    }
    if (prop != null) {
        return Integer.parseInt(prop);
    }
    // No 100% certainty... take an educated guess
    // https://stackoverflow.com/questions/10846105/all-possible-values-os-arch-in-32bit-jre-and-in-64bit-jre
    prop = System.getProperty("os.arch");
    if (prop.endsWith("64")) {
        return 64;
    } else if (prop.endsWith("86")) {
        return 32;
    }
    return -1; // we don't know..
}

UPDATE:

If there is a concern that this may give incorrect results on other platforms, we can limit this to GraalVM native images only:

    ...
    prop = System.getProperty("os.arch");
    if (prop.endsWith("64") && "Substrate VM".equals(System.getProperty("java.vm.name"))) {
        return 64;
    }
    return -1; // we don't know...

Thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.