from java.util import * not working in 2.7.2 jdk 14

[lsgroup]

Hi - for some reason, import * does not work for java.util package. It works for other packages.

after import * on java.util, instantiating a class gets NameError. See below:

Jython 2.7.2 (v2.7.2:925a3cc3b49d, Mar 21 2020, 10:03:58)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java14.0.2
Type "help", "copyright", "credits" or "license" for more information.

from java.util import *
d = Date()
Traceback (most recent call last):
File "", line 1, in
NameError: name 'Date' is not defined

Anyone else have this issue?

[lsgroup]

also, this error does not occur when using jdk 1.8.

[lsgroup]

after calling from java import * and then calling the object and from import java.util import *, it works.

Can this be resolved by patching JavaImportHelper.java?

Jython 2.7.2 (v2.7.2:925a3cc3b49d, Mar 21 2020, 10:03:58)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java14.0.2
Type "help", "copyright", "credits" or "license" for more information.

from java import *
d = util.Date()
from java.util import *
d = Date()
h = Hashtable()
Traceback (most recent call last):
File "", line 1, in
NameError: name 'Hashtable' is not defined

[jeff5]

Import is a mess. I wrote these notes trying to understand it.

Can this be resolved by patching JavaImportHelper.java?

Could be. There is a test for star import. I wonder if we skip it on 9 or why it doesn't catch the problem?

also, this error does not occur when using jdk 1.8.

In that case, I think we have the introduction of Java modules at Java 9 to blame. I recall it was necessary to enumerate by hand what would get indexed, and so it gets distinct treatment. Ah yes, here is the last place I worked on that, which might be worth a look:

jython/src/org/python/core/packagecache/SysPackageManager.java

Line 159 in 0a58cc2

private void findAllPackages(Properties registry) {

You can see where I've worked on this problem, but evidently not nailed it in : ffffe7b . Also, my commit titles need to be slightly shorter :)

[lsgroup]

Hi Jeff - yeah, I saw the import_star_from _java.py test. It fails, so I figured you guys knew about it!

$ jython dist/Lib/test/import_star_from_java.py
org.python.package WARNING failed to create cache dir '/usr/local/jython2.7.2/Lib/test/.jython_cache/packages'
Traceback (most recent call last):
File "/home/mattma/Work/jython272latest/jython-master/dist/Lib/test/import_star_from_java.py", line 4, in
p = Pattern.compile("foo")
NameError: name 'Pattern' is not defined

Blame it on jigsaw!

Will fiddle around with SysPackageManager.java.

Thanks!

[lsgroup]

(Edited by Jeff to make the code display.)

OK, mystery solved!
class: org.python.core.packagecache.CachedJarsPackageManager
method: private void addPackages(Map<String, String> packageToClasses, String jarfile)
the lines:

            int idx = classes.indexOf('@');
            if (idx >= 0 && Options.respectJavaAccessibility) {
                classes = classes.substring(0, idx);
            }

inside the for loop essentially wipes out the class list whenever

python.security.respectJavaAccessibility = true

in the jython registry.

The class list returned from jrt:/java.base looks like this:

addPackages(jrt:/java.base) ===> java.util : @AbstractCollection,AbstractList,...

The "@" character precedes all the other classes - so the substring method return an empty string.

The jython ships with this set to true by default and changing this setting to false crashes jython.

Commenting out the 4 lines above solves the problem and doesn't seem to produce any errors. But using my hack in production makes me nervous.

Do you think commenting out those lines will be safe? What are your thoughts on how this should be handled?

Let me know if I can help.

woohoo!

>>> from java.util import *
>>> h = HashMap()
>>> h.put("one", 1)
>>> print h
{u'one': 1}

FYI - jython crashes with this error when respectJavaAccessibility=false

Exception in thread "main" Traceback (most recent call last):
  File "/usr/local/jython2.7.2/Lib/site.py", line 68, in <module>
    import os
  File "/usr/local/jython2.7.2/Lib/os.py", line 45, in <module>
    from posix import *
java.lang.reflect.InaccessibleObjectException: Unable to make private java.lang.constant.ConstantDesc java.lang.constant.DynamicConstantDesc.tryCanonicalize() accessible: module java.base does not "opens java.lang.constant" to unnamed module @bae7dc0
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:349)

[lsgroup]

I noticed today that this is not an issue in Java 11. So sometime between Java 11 and 14, a bunch of modules got marked as protected.

[jeff5]

I think something has changed to be more strict about setAccessible and its relatives, such that our disrespect is now punished.

We should not depend on non-public "API" but in a few places we do. I'm not surprised it is the posix module giving us this problem. (It may even be in jnr.posix, not our code at all.)

Exploring the original complaint:

PS gh-105> jython
Jython 2.7.2 (v2.7.2:925a3cc3b49d, Mar 21 2020, 10:03:58)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java11.0.9
Type "help", "copyright", "credits" or "license" for more information.
>>> len(globals())
4
>>> from java.util import *
>>> len(globals())
118
>>> exit()
PS gh-105> Set-EnvJava 15
PS gh-105> jython
Jython 2.7.2 (v2.7.2:925a3cc3b49d, Mar 21 2020, 10:03:58)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java15.0.1
Type "help", "copyright", "credits" or "license" for more information.
>>> len(globals())
4
>>> from java.util import *
>>> len(globals())
18

I think you have found that the import (or the caching) gets quietly abandoned part way through, leaving us an incomplete cache for packages where it happens.

[lsgroup]

OK, did some more poking around... The code segment that looks for the "@" and excludes everything after it, is no longer correct with jigsaw becoming enabled by default.

In Java 9 and apparently at least through 11, jigsaw was disabled for compatibility. But going forward, that "@" character is always going to be in the beginning of the class list because the core jdk se is a module and the @ denotes module.

One way to solve this problem without code change would be to set:

python.security.respectJavaAccessibility = false

in the jython registry and then run the jvm with an add-opens parameter:

--add-opens java.base/java.lang.constant=ALL-UNNAMED

this will solve the problem without code change.

But this does not resolve the issue of using the @ character to cache java.base module packages. We could remove the @ substring lines of code, but how can we test to make sure this doesn't break something else?

Another idea - has anyone tried modularizing jython with module-info.java? Theoretically, we could specify all the java.base packages in module-info.java.

see console output below...

$ java --add-opens java.base/java.lang.constant=ALL-UNNAMED -jar /usr/jython/jython.jar
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.python.core.PyJavaType (file:/usr/local/jython2.7.2/jython.jar) to method java.lang.Object.finalize()
WARNING: Please consider reporting this to the maintainers of org.python.core.PyJavaType
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Jython 2.7.2 (v2.7.2:925a3cc3b49d, Mar 21 2020, 10:03:58)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java14.0.2
Type "help", "copyright", "credits" or "license" for more information.

from java.util import *
h = HashMap()
print h
{}

[lsgroup]

one more thing... using add-opens method creates a illegal reflective access operations that will be denied in JDK 17. Just to import java.util.HashMap requires this to avoid all the illegal operations:

$ java --add-opens java.base/java.lang.constant=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/java.nio.charset=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.util.function=ALL-UNNAMED --add-opens java.base/java.security=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED -jar /usr/jython/jython.jar

so setting python.security.respectJavaAccessibility = false is probably not the way to go.

[jwgish]

import of * from java.lang is also not working in JDK17

[jeff5]

I can reproduce all this with my installed 2.7.2, but not with the development tip 2.7.3a1, even after deleting and re-creating the cache with the later Java.

PS Jython-2> del -Recurse .jython_cache
PS Jython-2> dist\bin\jython
Jython 2.7.3a1-DEV (heads/master:1a784f52e, Jun 13 2022, 07:40:34)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java17.0.2
Type "help", "copyright", "credits" or "license" for more information.
>>> len(globals())
4
>>> from java.util import *
>>> len(globals())
120
>>> from java.lang import *
>>> len(globals())
233
>>> Date()
Mon Jun 13 09:48:17 BST 2022
>>> Math
<type 'java.lang.Math'>

This is a Good Thing, and it's tempting to close this immediately, but I'm not conscious what change fixed it.

The code @lsgroup identifies is the same. However, I think the problem was never there exactly, but in the position of the @ that separates private from public API class names, which is to say the problem may be in forming the cache not using it.

There are not many files in org.python.core.packagecache, where the problem surely lies. I'll look for changes since tag v2.7.2 before dismissing.

[jeff5]

There are no changes at all within org.python.core.packagecache since v2.7.2.

I find that the problem was fixed by upgrading ASM to 9.3 in 7c11f58 . If I revert just that commit on the development tip, and make a clean build, the problem is back.

PS Jython-2> dist\bin\jython
Jython 2.7.3a1-DEV (heads/master-dirty:1a784f52e, Jun 13 2022, 11:13:07)
[Java HotSpot(TM) 64-Bit Server VM (Oracle Corporation)] on java17.0.2
Type "help", "copyright", "credits" or "license" for more information.
>>> len(globals())
4
>>> from java.util import *
>>> len(globals())
19
>>> Date()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'Date' is not defined

The content for the cache string is collected here:

jython/src/org/python/core/packagecache/CachedJarsPackageManager.java

Lines 600 to 608 in 8c3734d

	// Put the class on the accessible or inaccessible list
	try (InputStream c = Files.newInputStream(file, StandardOpenOption.READ)) {
	int access = checkAccess(c);
	if ((access != -1) && !filterByAccess(fileName, access)) {
	classes.accessible.add(className);
	} else {
	classes.inaccessible.add(className);
	}
	}

We use ASM in checkAccess to read the access mask from the class and a failure gets treated silently as private.