Analyze future impact of Java SecurityManager removal on tests #743

scottmarlow · 2021-09-15T23:19:16Z

As per jakartaee/platform#406, analyze our options for running TCK tests against a future Java SE version that does not contain the SecurityManager classes.

One output from this issue should be analysis of how we could:

Remove SecurityManager references from TCK tests without losing non-SecurityManager aspects of those tests.
Or make the SecurityManager tests optional but still require any non-SecurityManager aspects of tests to be run.
Do not change any SecurityManager tests and instead allow test failures caused by SecurityManager classes being removed/stubbed out from future Java SE version to be ignored (e.g. certification compatibility request can contain test failures caused by JDK changes to SecurityManager).

For reference, see https://openjdk.java.net/jeps/411

scottmarlow · 2022-03-08T23:19:00Z

Update: I will soon attach new test logs from running the EE 9.1 Platform TCK test failures with an older WildFly that is purely Jakarta EE 9.1 based without the Security Manager. Thanks @brideck for noticing!

Attached are 103 EE 9.1 Platform TCK test failures when running WildFly (with some EE 10 changes) without the Security Manager:

15 Connector failures connector_logs.zip

Source link: https://github.com/eclipse-ee4j/jakartaee-tck/tree/master/src/com/sun/ts/tests/connector/permissiondd

18 servlet failures servlet_logs.zip

Source links:
https://github.com/eclipse-ee4j/jakartaee-tck/tree/master/src/com/sun/ts/tests/servlet/ee/platform/cdi/filter
https://github.com/eclipse-ee4j/jakartaee-tck/tree/master/src/com/sun/ts/tests/servlet/ee/platform/cdi/listener
https://github.com/eclipse-ee4j/jakartaee-tck/tree/master/src/com/sun/ts/tests/servlet/ee/platform/cdi/servlet
https://github.com/eclipse-ee4j/jakartaee-tck/tree/master/src/com/sun/ts/tests/servlet/ee/spec/security/permissiondd

12 ejb30/sec failures ejb30_sec_logs.zip

Source link: https://github.com/eclipse-ee4j/jakartaee-tck/tree/master/src/com/sun/ts/tests/ejb30/sec/permsxml

58 securityapi failures securityapi_logs.zip

arjantijms · 2022-03-09T17:18:38Z

Thanks, much appreciated!

brideck · 2022-03-12T18:44:54Z

Running the EE 9.1 Platform TCK with Open Liberty and Java 18, we immediately hit a problem with the JavaTest harness that prevents any tests from running:

[javatest.batch] java.lang.UnsupportedOperationException: The Security Manager is deprecated and will be removed in a future release
[javatest.batch] 	at java.base/java.lang.System.setSecurityManager(System.java:416)
[javatest.batch] 	at com.sun.javatest.JavaTestSecurityManager.install(JavaTestSecurityManager.java:84)
[javatest.batch] 	at com.sun.javatest.tool.Main.run(Main.java:291)
[javatest.batch] 	at com.sun.javatest.tool.Main.main0(Main.java:150)
[javatest.batch] 	at com.sun.javatest.tool.Main.main(Main.java:130)

There is a property (javatest.security.noSecurityManager=true) that skips the call to setSecurityManager, but there is not currently a hook in the TCK automation to set it. I've hacked in the change for now to get past it and am re-running the TCK. I will share the results when it is complete.

LanceAndersen · 2022-03-12T19:12:44Z

What version of javatest are you using? You probably need a more recent build of javatest Also, you depending on the test(s), you may need to include: -Djava.security.manager=allow

On Mar 12, 2022, at 1:45 PM, Brian Decker ***@***.***> wrote: Running the EE 9.1 Platform TCK with Open Liberty and Java 18, we immediately hit a problem with the JavaTest harness that prevents any tests from running: [javatest.batch] java.lang.UnsupportedOperationException: The Security Manager is deprecated and will be removed in a future release [javatest.batch] at java.base/java.lang.System.setSecurityManager(System.java:416) [javatest.batch] at com.sun.javatest.JavaTestSecurityManager.install(JavaTestSecurityManager.java:84) [javatest.batch] at com.sun.javatest.tool.Main.run(Main.java:291) [javatest.batch] at com.sun.javatest.tool.Main.main0(Main.java:150) [javatest.batch] at com.sun.javatest.tool.Main.main(Main.java:130) There is a property (javatest.security.noSecurityManager=true) that skips the call to setSecurityManager, but there is not currently a hook in the TCK automation to set it. I've hacked in the change for now to get past it and am re-running the TCK. I will share the results when it is complete. — Reply to this email directly, view it on GitHub <#743 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABQPFO66ZSOP47X72INWFZLU7TQ3DANCNFSM5EDPYKGQ>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.

My best, Lance -- Lance Andersen USTA EMA President and CEO PTR Professional 5A USPTA Elite Professional TIA Global Cardio Tennis-Master Trainer USRSA Mobile: 978 857-0446 luckydogtennis.com -- luckydogtennis.com/TennisBlog -- cardiotennis.com

brideck · 2022-03-12T19:23:33Z

What version of javatest are you using? You probably need a more recent build of javatest

I'm not sure that a newer version of JavaTest would matter, unless they've changed the default behavior to not set a security manager. The property we need is there already, it's just that the invocation in the TCK's Ant scripts doesn't use it at present.

Also, you depending on the test(s), you may need to include: -Djava.security.manager=allow

I'm not trying to get all of the tests to pass with Java 18. Similar to what Scott did above, I'm trying to see what fails, so that we can be prepared for when there is a Java level that no longer lets you set this property to allow at all.

scottmarlow · 2022-03-14T13:27:57Z

My test results above are from running with Java 17.

Good idea to look at Java 18. I wonder which JavaTest version is needed to support Java 18 and possibly up to Java 21.

brideck · 2022-03-14T14:25:36Z

Good idea to look at Java 18. I wonder which JavaTest version is needed to support Java 18 and possibly up to Java 21.

We shouldn't need a new version of JavaTest. I'll create a PR to show the kind of change in the TCK scripts that would be needed for this.

brideck · 2022-03-14T14:44:08Z

I've run tests with Open Liberty three ways now: (1) Java 17 with -Djava.security.manager=disallow, (2) Java 17 with the EE security permissions setup in our OL automation disabled, and (3) Java 18. In each run, I got the same set of 33 failures -- a subset of what Scott reported upthread.

connector/permissiondd - 6 failures (testValidateLocalPermsInvalidName & testValidateMissingPermFails)
ejb30/sec/permsxml - 12 failures
securityapi/idstore/idstorepermission - 1 failure
servlet/ee/spec/security/permissiondd - 14 failures

The only surprising part to me (without having looked more closely yet) is that I would have expected more of the connector/permissiondd tests to fail. I'll look into that some more.

scottmarlow · 2022-03-16T12:53:00Z

From my updated test run with EE 9.1 (using the WildFly Preview 26.x branch which is actually EE 9.1):

connector has 15 failure(s)
ejb30_sec has 12 failure(s)
securityapi has 1 failure(s)
servlet has 14 failure(s)

I'll attach test jtr + logs soon.

scottmarlow · 2022-03-22T14:24:03Z

Running the EE 9.1 Platform TCK with Open Liberty and Java 18, we immediately hit a problem with the JavaTest harness that prevents any tests from running:
[javatest.batch] java.lang.UnsupportedOperationException: The Security Manager is deprecated and will be removed in a future release
[javatest.batch] 	at java.base/java.lang.System.setSecurityManager(System.java:416)
[javatest.batch] 	at com.sun.javatest.JavaTestSecurityManager.install(JavaTestSecurityManager.java:84)
[javatest.batch] 	at com.sun.javatest.tool.Main.run(Main.java:291)
[javatest.batch] 	at com.sun.javatest.tool.Main.main0(Main.java:150)
[javatest.batch] 	at com.sun.javatest.tool.Main.main(Main.java:130)
There is a property (javatest.security.noSecurityManager=true) that skips the call to setSecurityManager, but there is not currently a hook in the TCK automation to set it. I've hacked in the change for now to get past it and am re-running the TCK. I will share the results when it is complete.

#904 could benefit from the jakartaee-tck/pull/894 as well I think.

brideck · 2022-03-22T15:24:02Z

#904 could benefit from the jakartaee-tck/pull/894 as well I think.

Agreed. Shall I turn 894 into a real PR and have folks review it? The new version of JavaTest would also resolve this, but I'm not sure when it might be getting out of Beta releases.

scottmarlow · 2022-03-22T15:29:57Z

#904 could benefit from the jakartaee-tck/pull/894 as well I think.

Agreed. Shall I turn 894 into a real PR and have folks review it?

Yes, please do turn i894 into a real PR.

The new version of JavaTest would also resolve this, but I'm not sure when it might be getting out of Beta releases.

…m-tck#894 change to address jakartaee/platform-tck#743 which didn't get released with jakarta-xml-ws-tck-4.0.0.zip Signed-off-by: Scott Marlow <smarlow@redhat.com>

…m-tck#894 change to address jakartaee/platform-tck#743 which didn't get released with jakarta-xml-ws-tck-4.0.0.zip (#708) Signed-off-by: Scott Marlow <smarlow@redhat.com>

scottmarlow added the 10.0 Issues related to the Jakarta EE 10 Platform TCK release label Sep 15, 2021

scottmarlow added this to Unassigned in Platform TCK 10 Sep 15, 2021

scottmarlow changed the title ~~Remove use of Java SecurityManager in tests~~ Analyze future impact of Java SecurityManager removal on tests Oct 6, 2021

scottmarlow removed this from Unassigned in Platform TCK 10 Jan 12, 2022

brideck mentioned this issue Mar 14, 2022

Set javatest.security.noSecurityManager=true if Java 18+ #894

Merged

scottmarlow removed the 10.0 Issues related to the Jakarta EE 10 Platform TCK release label Jun 7, 2022

scottmarlow mentioned this issue Mar 14, 2024

Update jaxws TCK version to 4.0.1 to prepare for staging jakarta-xml-ws-tck-4.0.1.zip to see if https://github.com/jakartaee/platform-tck/pull/894 change is included #1253

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Analyze future impact of Java SecurityManager removal on tests #743

Analyze future impact of Java SecurityManager removal on tests #743

scottmarlow commented Sep 15, 2021 •

edited

scottmarlow commented Mar 8, 2022 •

edited

arjantijms commented Mar 9, 2022

brideck commented Mar 12, 2022

LanceAndersen commented Mar 12, 2022 via email

brideck commented Mar 12, 2022 •

edited

scottmarlow commented Mar 14, 2022 •

edited

brideck commented Mar 14, 2022

brideck commented Mar 14, 2022

scottmarlow commented Mar 16, 2022

scottmarlow commented Mar 22, 2022

brideck commented Mar 22, 2022

scottmarlow commented Mar 22, 2022

Analyze future impact of Java SecurityManager removal on tests #743

Analyze future impact of Java SecurityManager removal on tests #743

Comments

scottmarlow commented Sep 15, 2021 • edited

scottmarlow commented Mar 8, 2022 • edited

arjantijms commented Mar 9, 2022

brideck commented Mar 12, 2022

LanceAndersen commented Mar 12, 2022 via email

brideck commented Mar 12, 2022 • edited

scottmarlow commented Mar 14, 2022 • edited

brideck commented Mar 14, 2022

brideck commented Mar 14, 2022

scottmarlow commented Mar 16, 2022

scottmarlow commented Mar 22, 2022

brideck commented Mar 22, 2022

scottmarlow commented Mar 22, 2022

scottmarlow commented Sep 15, 2021 •

edited

scottmarlow commented Mar 8, 2022 •

edited

brideck commented Mar 12, 2022 •

edited

scottmarlow commented Mar 14, 2022 •

edited