GraalVM Native Image does not include MixIn options

[ngeor]

I'm very new in both picocli (fantastic tool!) and GraalVM. I'm using a Mixin to reuse options across multiple sub-commands. This works as expected when running the app with Java. When packaging the app with GraalVM native image, the Mixin options are missing.

• Expected result: my CLI app should offer the same options, regardless of whether the app is run via Java or packaged via GraalVM

• Actual result: the app does not offer reusable mixin options when packaged via GraalVM

Some versions:

Java: 1.8
OS: Fedora 30
Building with: Gradle 5.6.3
Picocli: 4.0.4
GraalVM: 19.0.2.1

This is the main Command Line (just pulls-in various sub-commands):

@Command(
    synopsisSubcommandLabel = "COMMAND",
    subcommands = {
        CreateCommand.class,
        DeleteCommand.class,
        ListCommand.class
    }
)
public class App implements Callable<Integer> {
    /**
     * Runs the application.
     * @param args Command line arguments.
     */
    public static void main(String[] args) {
        System.exit(
            new CommandLine(new App())
                .setCaseInsensitiveEnumValuesAllowed(true)
                .execute(args)
        );
    }

    @Override
    public Integer call() {
        System.out.println("Missing sub-command");
        return -1;
    }
}

This is the list command:

@Command(
    name = "list",
    description = {"Lists the available repositories"}
)
class ListCommand implements Callable<Integer> {
    @Mixin
    private ProviderMixin providerMixin;

    @Override
    public Integer call() throws IOException {
        System.out.println(
            String.format(
                "owner %s username %s password %s provider %s",
                providerMixin.getOwner(),
                providerMixin.getUsername(),
                providerMixin.getPassword(),
                providerMixin.getProvider()
            )
        );
        return 0;
    }
}

and this is the mixin:

public class ProviderMixin {
    @CommandLine.Option(
        names = {"--owner"},
        required = true,
        description = {"The owner of the repository"}
    )
    private String owner;

    @CommandLine.Option(
        names = {"--username"},
        required = true,
        description = {"The username to access the git provider API"}
    )
    private String username;

    @CommandLine.Option(
        names = {"--password"},
        required = true,
        description = {"The password to access the git provider API"}
    )
    private String password;

    @CommandLine.Option(
        names = {"--provider"},
        required = true,
        description = {"The provider of the git repository (${COMPLETION-CANDIDATES})"}
    )
    private GitProvider provider;

    public String getOwner() {
        return owner;
    }

    public void setOwner(String owner) {
        this.owner = owner;
    }

    public String getUsername() {
        return username;
    }

    public void setUsername(String username) {
        this.username = username;
    }

    public String getPassword() {
        return password;
    }

    public void setPassword(String password) {
        this.password = password;
    }

    public GitProvider getProvider() {
        return provider;
    }

    public void setProvider(GitProvider provider) {
        this.provider = provider;
    }
}

The GitProvider is an enum.

I have Gradle configured regarding the annotation processor:

dependencies {
    // This dependency is used by the application.
    implementation "info.picocli:picocli:4.0.4"
    annotationProcessor "info.picocli:picocli-codegen:4.0.4"
}

compileJava {
    options.compilerArgs += ["-Aproject=${project.name}"]
}

Note that I had to remove from compilerArgs the ${project.group}/ that is mentioned in the documentation.

When I build the project, I see that I have a reflect-config.json which indeed lacks the mixin fields.

  {
    "name" : "instarepo.ListCommand",
    "allDeclaredConstructors" : true,
    "allPublicConstructors" : true,
    "allDeclaredMethods" : true,
    "allPublicMethods" : true
  }

Direct option fields (not via a mixin) are included:

{
    "name" : "instarepo.CreateCommand",
    "allDeclaredConstructors" : true,
    "allPublicConstructors" : true,
    "allDeclaredMethods" : true,
    "allPublicMethods" : true,
    "fields" : [
      { "name" : "description" },
      { "name" : "language" },
      { "name" : "name" }
    ]
  }

This is the output of my cli app generated by GraalVM (it lacks the mixin options):

[ngeor@localhost instarepo]$ ./build/graal/instarepo create --help
Missing required options [--name=<name>, --description=<description>, --language=<language>]
Usage: <main class> create --description=<description> --language=<language>
                           --name=<name>
Creates a new git repository
      --description=<description>
                      The description of the repository
      --language=<language>
                      The language of the repository
      --name=<name>   The name of the repository

and this is the output when I run it with Java (it has the extra options that are offered via the mixin):

[ngeor@localhost instarepo]$ gradle run --args="create --help"

Missing required options [--name=<name>, --owner=<owner>, --username=<username>, --password=<password>, --provider=<provider>, --description=<description>, --language=<language>]
Usage: <main class> create --description=<description> --language=<language>
                           --name=<name> --owner=<owner> --password=<password>
                           --provider=<provider> --username=<username>
Creates a new git repository
      --description=<description>
                        The description of the repository
      --language=<language>
                        The language of the repository
      --name=<name>     The name of the repository
      --owner=<owner>   The owner of the repository
      --password=<password>
                        The password to access the git provider API
      --provider=<provider>
                        The provider of the git repository (GITHUB, BITBUCKET)
      --username=<username>
                        The username to access the git provider API

[remkop]

Thanks for raising this and the many details!That doesn’t sound good. I’ll look at this as soon as I get to my PC.

[ngeor]

Thank you for the fast reply! The code is available here https://github.com/ngeor/instarepo/tree/java-gradle-rewrite if you want to have a look (work in progress).

[remkop]

@ngeor Thanks again for raising this. I was able to reproduce it and it was a real bug.
I've pushed a fix to master. Can you verify?

Please clone the project, and build it with gradlew clean build publishToMavenLocal.
Then change the dependencies in your project to use version 4.0.5-SNAPSHOT and build your project.

[ngeor]

Hi @remkop ,

I just did it and now the options appear as expected. Great work!

[remkop]

@ngeor Glad to hear that!
Will you be able to make progress with the local snapshot version for a while? I don’t know yet when I’ll be able to do the next release.

[ngeor]

@remkop oh don't worry about it at all. I'm totally new to GraalVM. After this fix you provided I got further exceptions originating either in okhttp or my local openJDK and I gave up with GraalVM for now :D

[remkop]

I see. Hopefully picocli is still useful to you even without GraalVM. :-)

[remkop]

Actually I’m writing an article on the feasibility of using picocli + GraalVM for writing native CLI apps and I’m interested in the difficulties you encountered. Could you share some details? Can I reproduce them with your project?

[remkop]

@ngeor Something I noticed yesterday while I looked at your project: you might be interested in using the @Command(name = "...", mixinStandardHelpOptions = true, version = "...") annotation to give your commands --help and --version options with minimal fuss.

See the mixinStandardHelpOptions docs and version docs. All strings may contain system properties and other variables.

[ngeor]

So I think picocli works fine, my code crashes further down where it tries to use okhttp to make a call to GitHub's REST API.

As soon as I run into an error, I google about it and apply whatever workaround I find.

The first problem was that the latest okhttp (v4) didn't work because (according to the internets) it's in Kotlin. I downgraded to version 3 (still Java) and my code still compiled.

The second problem was something about missing a charset UTF_32 or so. The workaround was to add this option to the gradle-graal plugin: option "-H:+AddAllCharsets"

And now it's something about TLS not present:

Exception in thread "main" java.lang.AssertionError: No System TLS
	at okhttp3.internal.Util.platformTrustManager(Util.java:648)
	at okhttp3.OkHttpClient.<init>(OkHttpClient.java:228)
	at okhttp3.OkHttpClient.<init>(OkHttpClient.java:202)
	at instarepo.ListCommand.call(ListCommand.java:41)
	at instarepo.ListCommand.call(ListCommand.java:21)
	at picocli.CommandLine.executeUserObject(CommandLine.java:1781)
	at picocli.CommandLine.access$900(CommandLine.java:145)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2139)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2106)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1973)
	at picocli.CommandLine.execute(CommandLine.java:1902)
	at instarepo.App.main(App.java:37)
Caused by: java.security.NoSuchAlgorithmException: PKIX TrustManagerFactory not available
	at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
	at javax.net.ssl.TrustManagerFactory.getInstance(TrustManagerFactory.java:139)
	at okhttp3.internal.Util.platformTrustManager(Util.java:638)
	... 11 more

and some suggested workaround said to copy over certificates from Oracle into the OpenJDK files (at which point I gave up).

[ngeor]

Ok... with the help of this article I was able to solve the TLS issue:

• add option "--enable-https" to the configuration of the graal-gradle plugin
• copy the libsunec.so from the GraalVM distribution into my app (which means my app is no longer a single executable, but it needs this shared library as well)

[remkop]

Nice! Any reason you're not using the latest version of GraalVM (19.2.1)?

[ngeor]

I missed that one, thanks :-) I tried it and it works the same.

[remkop]

FYI: There is some discussion here and here on how to avoid shipping libsunec.so with your native image and still have SSL support.

Specifically:

Update 3: The solution we've settled on for now is to remove the SunEC provider from the java.security file. In this case, none of the mentioned workarounds is necessary, and HTTPS still works for us.

[remkop]

I'm closing this ticket because there is no more work remaining for the "Mixins on GraalVM" problem, but we can continue to discuss further here or on a new ticket if you like.

[ngeor]

Awesome, thank you for fixing this!

[nicobao]

In case it can help somebody, I am going to leave this here as I stumbled upon this thread when googling my problem.

I had a similar issue with GraalVM native-image when trying to use Picocli to make a CLI that will serialize objects to YAML files using Jackson (an OpenAPI document).
It was working great for the JVM target but not for native-image.

The Jackson MixIn registered within the swagger-core lib didn't seem to be taken into consideration, and it was leaking exampleSetFlag in the output file, which, to make things more messy, happen to have been a recent bug in swagger-core as shown here swagger-api/swagger-core#3637 (comment).
It turns out it had nothing to do with this bug though, and this thread helped me understand it.
I fixed the problem by properly registering the MixIn classes for GraalVM reflection (I use Quarkus with Kotlin):

import io.quarkus.runtime.annotations.RegisterForReflection
import io.swagger.v3.core.jackson.mixin.ExampleMixin
import io.swagger.v3.core.jackson.mixin.MediaTypeMixin
import io.swagger.v3.core.jackson.mixin.SchemaMixin
import io.swagger.v3.oas.models.examples.Example
import io.swagger.v3.oas.models.media.MediaType
import io.swagger.v3.oas.models.media.Schema

@RegisterForReflection(targets=[
    MediaType::class,
    MediaTypeMixin::class,
    Example::class,
    ExampleMixin::class,
    Schema::class,
    SchemaMixin::class,
   // ... more classes...
])
class Configuration {
}