automatic type stub (pyi) generation for java classes

[michi42]

I'm working on making a rather big, domain specific java API available through JPype. One of our main issues is the complete lack of type information during static analysis - and therefore no linting or IDE auto-completion.

PEP 484 and 561 support providing type information in separate .pyi "stub" files. The mypy project provides stubgen, a tool to automatically create these from python or C modules.
Since Java is strongly typed, all the information necessary to generate these kind of stub files is in principle available in the class files and can be obtained e.g. through reflection.

I implemented a first version: https://gist.github.com/michi42/2110a8615a0e917a13ec6748c6168735
EDIT: Now a standalone repo: https://gitlab.cern.ch/scripting-tools/stubgenj

The module is supposed to be called from a simple script that sets up the JVM, e.g.

import jpype
import stubgenj

# fire up the JVM with the necessary JARs

stubgenj.generate_java_stubs(["java", "cern.lsa"], "pyi")

This is still very limited, just a starting point that handles the "most common" cases. In particular, it does not support Generics (should be translated to python type vars), Constructors (should be translated to init methods) yet, and the types of constants and enum constants are not yet detected.

[Thrameos]

Interesting. I look forward to some additional information.

Can this be connected to the existing annotation API that we are using? We currently support Jedi auto-completion which allows linting of returns types if the overload is not mixed. I haven't tried with Kite.

Here is my recommendation as an implementation. Just like Java docs we can pick up Java information though the resource API. Therefore, if you run a preprocessor that converts the Java jar file (org-mystuff.jar) into a second jar with the needed annotation org-mystuff-py.jar. Then when I load a class using org.mystuff.MyClass.class I can also fetch the org.mystuff.MyClass.annot file which can then provide stubs.

[michi42]

Hi,

Thanks a lot the quick response!

Indeed, it goes into the same direction as the annotation API (and __docs__) that JPype is already populating at run-time (which is great for interactive applications like IPython or Jupyter), however it provides static information that is pre-generated in once and can then be used without executing any of the code (as it is typically the case for IDEs like PyCharm).

Also, this supports overloads, even when the signatures are completely different and return types depend on the signatures, through the overload decorator: https://docs.python.org/3/library/typing.html#typing.overload - also it should be possible to represent generics and type arguments, although my first version does not handle that. I frankly don't know if the annotation API supports all of these.

As for distribution, I'm afraid it can't be (just) a JAR file, as the whole point behind this is that python IDEs (and possible linting tools) can read the type stubs in a purely static way, without having to run any of our code. Therefore, the type stub "shadow packages" have to be packaged and installed in a way common python IDEs support, typically as an egg/wheel. The structure has to match the Java class hierarchy, for instance (after generating "java.*" and some private APIs):

Note that there are only __init__.py files - because Java does not have the concept of modules, only packages and contained classes.

A few results (in PyCharm):

[Thrameos]

Okay that makes sense. Please keep me posted on any hooks that you require in terms of support.

https://www.jetbrains.com/help/pycharm/stubs.html

I guess the correct solution would be to make a jpype-java-v8, jpype-java-v11, etc. package that gets posted to pypi then? We can then have those packages depend on JPype1 to reduce the need to install both separately. Is that what you envision?

This is very similar to what I am working on for the reverse bridge in which Python generates a package with nothing more than interfaces for each Python class. That allows Java to have an interface with each required method exposed in Python.

[michi42]

Okay that makes sense. Please keep me posted on any hooks that you require in terms of support.

Sure, thanks a lot!

I guess the correct solution would be to make a jpype-java-v8, jpype-java-v11, etc. package that gets posted to pypi then? We can then have those packages depend on JPype1 to reduce the need to install both separately. Is that what you envision?

That would indeed be an option to ease the working with java standard library classes (like https://github.com/python/typeshed does for python standard libraries).
Also, I think it could be useful to ship the stub generator along with JPype to allow users to generate stubs for their own domain-specific Java packages.

One potential (minor) issue with static stub generation is that java.lang.String auto-conversion can be switched on and off at run-time. For the time being I assumed it to be on, and therefore java.lang.String becomes str in my first version of the stub generator. Perhaps it would be more correct to treat it as Union[java.lang.String, str] (= either of these).
Still, a static type checker will not be able to detect if the conversion flag is going to be on or off at run-time.

[Thrameos]

The default for JPype 0.8 on will be conversion is false. The on mode is a bit of a misfeature as there are a few functions in Java where chaining a string is needed. (It is non obvious if you call the Java string constructor and get back a Python string). Though we may be able to splice some aliases into the java.lang.String so that it works a bit closer to Python strings. I remember there was a conflict on a method contract a while back. But I should probably refresh my memory on that topic.

We should definitely ship the stub generator as part of JPype so that someone can run it as needed in the field.

[Thrameos]

I reviewed the contract for string and Java string. There is one strong conflict, two weak conflicts and 2 near conflicts.

The strong conflict is Java and Python format. They are similar in design but one is a static method and the other operates on a string. Split is very similar but Java uses regexp rather than simple text on the split, Replace is almost the same but Python adds an extra parameter. endswith and endsWith are near miss. startswith and startsWith are near miss.

I could submit a PR with Java String completing almost the same contract as Python. The return type will always be Java strings for consistency. Give then we are switching to no longer convert strings, this would in principle save a few people, but as overloading strings completely is not possible (though maybe I should review this as this was last reviewed in Python 2 age. The other thing that will not pass is of course isinstance. Hard to dodge that one without deriving the type.

@marscher would this be worth the effort to put in as a PR?

[michi42]

Thanks a lot for checking. I agree that conceptually this feature is not very nice, although in many common use cases it saves a lot of explicit type conversion clutter. Checking how this is done in PySide (Qt), they do the same for QString ...
Actually I think there are two use cases for the automatic conversion - from str to java.lang.String (e.g. arguments of Java methods) and the other way around (e.g. return values of Java methods).

For the return types, yes, we may get away with making java.lang.String "look like" python str, and I agree it could be a nice feature, although as you noticed it will not cover all use cases.

On the other hand, what is the plan for str to java.lang.String conversion? This one seems "harmless" to me. With convertStrings=False, will you have to pass explicitly constructed java.lang.String objects to Java methods which take strings as arguments?

Anyway, I'm going to make that the stub generator honors the flag at generation time, so it is up to the user to decide. "Official" stubs would be generated assuming convertStrings=False, but for those using the feature, they can build their own stub tree with convertStrings=True. In the end it's a simple type replacement at the level of the stubs.

Edit: is there any foreseen way to detect from Python if JPype was started with convertStrings=True? Currently I'm using this, but it is somewhat artificial...

def convert_strings() -> bool:
    if convert_strings.jpype_flag is None:
        from java.lang import String  # noqa
        convert_strings.jpype_flag = isinstance(String().trim(), str)
    return convert_strings.jpype_flag


convert_strings.jpype_flag = None

[Thrameos]

Convert strings only affects the return path. We always implicitly convert string as arguments.

The argument path for stubs likely needs some extra hooks. We have quite a few automatic forward conversions like string, path, date, and list. I dont think there is currently a way to get a list of implicit conversions. Some of these are hard types, and others are duck types. What level of detail do you need for stubs? Do you need a method to extract the conversion rule by Java class?

[michi42]

Yes, then indeed I would need a way to extract a list of the types accepted by the implicit conversion for a particular java class in a method argument.
For the accepted hard types, it's straightforward, I would just need them as python type objects or strings.
For the duck types, we could define Protocol types - or if there is one, use one of the pre-defined protocols (e.g. Iterable, Mapping, ...)

Another question that goes in the same direction: is there a hook to get the "mangling" jpype does to identifiers when they match a python keywords (adding '_')?

[Thrameos]

The implicit rules currently recognize three types of conversions. The first is exact in which a conversion is only applied if it exactly matches, but this one is only there to trigger fast logic. The others are in the form of a list of types that will be taken or a list of attributes that will be probed. I suppose the best way to do this is to get you an API call that returns those two lists as a tuple so that we can construct Protocol types for each.

Name mangling is handled by the pysafe function in jpype._pykeywords. It is applied by classes to methods and fields, to accessors in beans, and to packages and imports.

[Thrameos]

Okay looking it over there is one special exception in the type system. Python strings are Sequences but JPype flatly refuses to recognize them. There is almost no case in which the user wants to pass a string to a list object and make it break it into chars.

[michi42]

Ok, I can't see how this special exception could be easily represented in the type annotation system, as if we e.g. allow Iterable, it will include str. But I would say, we can leave that aside for a first version.

[Thrameos]

"Never let perfect be the enemy of good."

There will also be an edge case or two. But nothing here gives me any pause. I will work on the probe API. Hopefully completed by the end of the weekend.

[michi42]

Perfect, thanks a lot, that would allow me to get rid of this hardcoded mess: https://gist.github.com/michi42/2110a8615a0e917a13ec6748c6168735#file-stubgenj-py-L212 ;-)

For objects returned by java to python, are there any implicit conversions (apart from String -> str if convertStrings = True)?
From what I see, boxed types are returned as such.
Primitives become JShort, JInt, ... - but since these implement the python int, it is probably safe to treat them as int (or bool, float, ... respectively)
Arrays become something that implements the List contract?

[Thrameos]

There are a few implicit conversions

• null goes to type None. So anything returning an object type will get None as well as the object type. Not sure stubs will care about this.
• Void type will eventually go to JVoid which will be magical type that refuses to do anything. This will prevent someone from accidentally using a void in an if statement.
• Java boolean goes to Python bool type (True, False)
• Java string goes to Python str when convertString is True
• Primitives each return the JPrimitive wrappers which all derive from the corresponding type.
• Boxes types act like primitives with the minor exception that they can be null objects. They do have some special methods so you may need to leave them as is.
• Java char goes to a str with length 1. This is a weird special case because I can't make str and int types play nice.
• Most containers will implement the corresponding contract and satisfy the collections.abc. This is still a work in progress. Best to leave them as the Java type then make the stub show it implements the contract.
• The other special case is the JObject cast which forces the type to match. Unfortunately it takes string arguments so we can really see the contract. The new casting operator fixes this problem. (Please comment on that thread if you have an operator preference).

[michi42]

Indeed, not being able to enumerate all overloads from JClass was one of the reasons why I used the Java reflection API. It is not the only reason though - not all typing information is available currently from __annotations__, e.g. generic types, wildcard types (with or without bounds, e.g. List<? extends T> for a complex example), or inner types of arrays.

Of course we could aim at putting some or all of the functionality stubgenj currently does to map these to the python typing system directly into JPype. However, this is probably a good piece of work and may not be straightforward in all cases, in particular if old python versions need to be supported.

The big advantage I see in doing this mapping independently from Java Reflection is that it does not affect runtime behavior of JPype, so it can be a bit more sloppy in some edge cases.

For the generics, the types should inherit from typing.Generic (directly or indirectly), and use typing.TypeVar arguments. You can check how this is done in the stubs e.g. for collections in java.util.
The type info is available in the reflection API via getParameterizedType() in cases where it is not erased. Then there are a few different cases to handle ... see https://gist.github.com/michi42/2110a8615a0e917a13ec6748c6168735#file-stubgenj-py-L308 for my attempt to deal with that (which may also not be correct in some edge cases)

[michi42]

A little update - together with @pelson we've further improved the stubgenj and turned it into a standalone installable package (for the time being): https://gitlab.cern.ch/scripting-tools/stubgenj

I'm also working on a little test suite for this.

[Christopher-Chianelli]

@michi42 How can I open a PR to stubgenj? I have a patch that add class Javadoc via from jpype._jclass import _jclassDoc. Method Javadoc from from jpype._jmethod import _jmethodGetDoc is not as useful (it only list overloads it seems).

[michi42]

@Christopher-Chianelli Good question - I did not consider this when I created the repository on the CERN GitLab. Indeed write access is restricted to CERN account holders.

To get this out as quickly as possible, could you just send me a patch for now?
In parallel I will discuss with some colleagues if we can move it to a public GitHub repo.

Also, I just pushed the latest version to pypi for your convenience: https://pypi.org/project/stubgenj/0.2.2/

[Christopher-Chianelli]

@michi42 patch is attached
add-javadoc.txt

[michi42]

@Christopher-Chianelli Thanks a lot for the patch. It looks good to me in general, but I think the specification do not allow adding Javadoc for empty class stubs (at least pycharm reports it as error) - therefore I have removed it.

I have released https://pypi.org/project/stubgenj/0.2.3/ with your changes.

[Christopher-Chianelli]

@michi42 I managed to find a way get method javadoc in the majority of cases. Sometimes method javadoc is missing, which appears to be the case when JPype cannot find the Javadoc for a class (for me, it happens when an interface extends another interface). (Basically, when _jclassDoc is called, it populates __javadoc__ on the class, which holds the javadoc for individual methods, fields and constructors in a dictionary. For overloaded methods, they are merged, but the first line is the "signature" of the overload, which can be used to put it in the corresponding overload method.) Patch attache
javadoc-method-patch.txt
d.

[michi42]

@Christopher-Chianelli Thanks for the patch, I tried to tune it a bit - basically in the end I call into org.jpype.javadoc.JavadocExtractor to avoid relying on the side effect of _jclassDoc.
Like this we can also avoid the default generated docstrings of JPype if no JavaDoc is available - when type stubs are available, it does not provide much extra information I think. If you think it's useful, I can make it optional...

Also I had to change your RegExps a little as it was matching too broadly, which lead to the wrong JavaDoc being attached to overloads.
It's not perfect, in particular it won't catch cases with the same number but different types of arguments for the overloads. Still much better than nothing.

Here is the MR, I will let my colleagues have a look ...
https://gitlab.cern.ch/scripting-tools/stubgenj/-/merge_requests/6
I think I will still implement an option to turn it on/off as it slows down the process a bit.

[michi42]

@Christopher-Chianelli I have just released the updated version as 0.2.4: https://pypi.org/project/stubgenj/0.2.4/

Thanks again for the patches!

[Christopher-Chianelli]

@michi42 FYI JPype1 1.2.1(which stubgenj depends on (and since ~= is used, pip will not allow a higher minor version (i.e. JPype1 1.3.0)) does not compile on Python 3.10 on Linux. Which in turns means it can't be listed as a build-system.requires. I just tested using a hack that only install JPype1>=1.3.0 and then does a pip install --no-deps stubgenj in the middle of the build, which works (i.e. stubgenj works with 1.3.0). Can you consider either updating the JPype version or changing it to range (i.e. '>= 1.2.1', or '>=0.2.1,<2.0.0', etc)?

[michi42]

Done: https://pypi.org/project/stubgenj/0.2.5/ - now asking for JPype1>=1.2.1,<2.0.0

This update also includes a fix for a misbehavior that directories in Javadoc JARs on the classpath were seen as "java packages", even if they contained no classes or subpackages in the real class JARs.
Such directories could have funny names (like "class-use" in Guava) that do not make valid python identifiers, and hence, break stubs.

[ldeluigi]

News? Can't this be merged into JPype directly?

[MrFizzyBubbs]

@michi42 I have been using stubgenj and it has worked great so far. Thank you for providing it! The only issue I've encountered is that it isn't generating stubs for abstract classes such as this I attempted to post an issue on gitlab but am unable to do so from a guest account. I apologize if this is the wrong avenue.

[michi42]

@MrFizzyBubbs apologies for the delay, I was quite busy with other work. Just had a look, it appears the problem comes from the fact that JPype's JPackage does not list this KoLCharacter class when using dir() - which is how stubgenj enumerates the classes in a package.

Looking a bit further, it appears the isPublic check of JPypePackage returns false for net.sourceforge.kolmafia.KoLCharacter even though it is public. Perhaps @Thrameos can have a look...

[jstasiak]

Hey all, thank you for all your work on this project and the problem at hand.

I needed this done so I generated the stubs with stubgenj and the OpenJDK I have installed (25), put that in a repository (https://github.com/jstasiak/jpype-java-stubs) and published two distributions from that:

• https://pypi.org/project/java-stubs/ (no automated string Java -> Python conversion)
• https://pypi.org/project/java-stubs-converted-strings/ (automated string conversion enabled)

There's probably a bunch of improvements but it works for me right now.

Note: stubgenj creates jpype-stubs but I remove that altogether and make my distributions depend on JPype instead. Trying to use jpype-stubs caused trouble for me type-checking-wise but I accept I might have dome something incorrectly.

If anyone wants to collaborate on this or if there's some will to merge this into JPype do let me know.