review how easy it is to run manual tests

[pjfanning]

• sbt test works fine on my laptop
• the tests in this repo are simpler than the ones in https://github.com/apache/incubator-pekko
• there could be some flaky tests that might cause issues when other people review this module (during a release, for instance)

[samueleresca]

I have tried to run sbt test for this module multiple times and they always succeeded. I have tried on the following configurations:

• sbt 1.9.1 and Ubuntu Java 11.0.19 on Ubuntu 20.04.6 LTS
• sbt 1.9.2 and Microsoft Java 11.0.14.1 on Ubuntu 20.04.6 LTS (Newly created devbox)

Is there anything I can help with for this issue?

[pjfanning]

Thanks @samueleresca - we can leave this open to see if anyone else has time to try out a build but it's good to get independent runs to see if the build is stable.

[lomigmegard]

I have tried on my machine sbt test with the current HEAD b33f85110c75130661156bf6a4099342c5febcdf, with the following setup:

• sbt 1.9.2
• OpenJDK Runtime Environment Temurin-20.0.1+9 (build 20.0.1+9)
• Apple M1 with Ventura 13.4.1 (c)

I got a few failed tests:

[info] *** 12 TESTS FAILED ***
[error] Failed tests:
[error] 	org.apache.pekko.http.impl.engine.http2.H2SpecIntegrationSpec

[...]

[info] *** 3 TESTS FAILED ***
[error] Failed tests:
[error] 	org.apache.pekko.http.scaladsl.ClientServerHttp2EnabledSpec
[error] 	org.apache.pekko.http.scaladsl.ClientServerSpec

A couple examples of such failed tests:

[info] - should report failure if bind fails *** FAILED *** (29 milliseconds)
[info]   org.apache.pekko.stream.impl.io.ConnectionSourceStage$$anon$1$$anon$2: Bind failed because of java.net.BindException: [localhost/127.0.0.1:1025] Address already in use
[info]   Cause: java.net.BindException: [localhost/127.0.0.1:1025] Address already in use

[info] - should complete a request/response when the request side immediately closes the connection after sending the request *** FAILED *** (7 seconds, 68 milliseconds)
[info]   java.lang.AssertionError: assertion failed: expected no StreamSupervisor children, but got [Actor[pekko://PekkoSpecWithMaterializer_/system/Materializers/StreamSupervisor-73/flow-124-0-headSink#-84814694]]

[info] H2Spec
--> [H2Spec must pass rule: 3.5. HTTP/2 Connection Preface] Start of log messages of test that [Failed(org.scalatest.exceptions.TestFailedException: 139 was not equal to 0)]
| [DEBUG] [07/22/2023 12:26:16.074] [pool-1-thread-1-ScalaTest-running-H2SpecIntegrationSpec] [WithLogCapturing(pekko://PekkoSpec_)] Logging started for test [H2Spec must pass rule: 3.5. HTTP/2 Connection Preface]
| [DEBUG] [07/22/2023 12:26:16.075] [pool-1-thread-1-ScalaTest-running-H2SpecIntegrationSpec] [H2SpecIntegrationSpec(pekko://PekkoSpec_)] Executing h2spec: List(/Users/lomig/programmation/github/incubator-pekko-http/http2-tests/target/h2spec_darwin_amd64/h2spec, -k, -t, -p, 55864, -j, target/test-reports/h2spec-junit-3.5.xml, -s, 3.5)
| [DEBUG] [07/22/2023 12:26:16.659] [pool-1-thread-1-ScalaTest-running-H2SpecIntegrationSpec] [WithLogCapturing(pekko://PekkoSpec_)] Logging finished for test [H2Spec must pass rule: 3.5. HTTP/2 Connection Preface]
<-- [H2Spec must pass rule: 3.5. HTTP/2 Connection Preface] End of log messages of test that [Failed(org.scalatest.exceptions.TestFailedException: 139 was not equal to 0)]
[info] - must pass rule: 3.5. HTTP/2 Connection Preface *** FAILED *** (591 milliseconds)
[info]   139 was not equal to 0 (H2SpecIntegrationSpec.scala:194)

--> [H2Spec must pass rule: 6.7. PING] Start of log messages of test that [Failed(org.scalatest.exceptions.TestFailedException: 139 was not equal to 0)]
| [DEBUG] [07/22/2023 12:26:16.825] [pool-1-thread-1-ScalaTest-running-H2SpecIntegrationSpec] [WithLogCapturing(pekko://PekkoSpec_)] Logging started for test [H2Spec must pass rule: 6.7. PING]
| [DEBUG] [07/22/2023 12:26:16.825] [pool-1-thread-1-ScalaTest-running-H2SpecIntegrationSpec] [H2SpecIntegrationSpec(pekko://PekkoSpec_)] Executing h2spec: List(/Users/lomig/programmation/github/incubator-pekko-http/http2-tests/target/h2spec_darwin_amd64/h2spec, -k, -t, -p, 55864, -j, target/test-reports/h2spec-junit-6.7.xml, -s, 6.7)
| [DEBUG] [07/22/2023 12:26:16.839] [pool-1-thread-1-ScalaTest-running-H2SpecIntegrationSpec] [WithLogCapturing(pekko://PekkoSpec_)] Logging finished for test [H2Spec must pass rule: 6.7. PING]
<-- [H2Spec must pass rule: 6.7. PING] End of log messages of test that [Failed(org.scalatest.exceptions.TestFailedException: 139 was not equal to 0)]
[info] - must pass rule: 6.7. PING *** FAILED *** (16 milliseconds)
[info]   139 was not equal to 0 (H2SpecIntegrationSpec.scala:194)

And more of those must pass rule: [...]

[mdedetrich]

If these tests are flaky then I think what I wrote in apache/pekko-connectors#198 (comment) can apply here.

[lomigmegard]

The tests are not flaky, I tried 4 times.

However I checked and the port 1025 was indeed already bound on my machine because it's the default one for Proton Mail Bridge (cmd: sudo lsof -nP -i4TCP:1025 | grep LISTEN). Killing that process fixed the failed tests in ClientServerHttp2EnabledSpec.

The tests consistently fail in org.apache.pekko.http.scaladsl.ClientServerSpec (1) and org.apache.pekko.http.impl.engine.http2.H2SpecIntegrationSpec (12).

[jrudolph]

Weird that it expects port 1025 to be available, maybe something to fix... Not sure if someone already tested if the build runs on JDK 20. Maybe you can try with JDK 8 or 11 and see if that works? I general the HTTP tests should be pretty stable, much more so than the pekko core ones.

[lomigmegard]

With OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10) the tests don't finish and blow up with a java.lang.OutOfMemoryError: Java heap space.

I increased the heap to 4GB

sbt -v --java-home /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home -J-Xmx4g

More errors now:

[info] *** 6 SUITES ABORTED ***
[error] Error during tests:
[error] 	org.apache.pekko.http.impl.engine.http2.TelemetrySpiPlaintextSpec
[error] 	org.apache.pekko.http.impl.engine.http2.H2SpecIntegrationSpec
[error] 	org.apache.pekko.http.impl.engine.http2.Http2PersistentClientPlaintextSpec
[error] 	org.apache.pekko.http.impl.engine.http2.Http2PersistentClientTlsSpec
[error] 	org.apache.pekko.http.impl.engine.http2.TelemetrySpiCypherSpec
[error] 	org.apache.pekko.http.impl.engine.http2.Http2ClientServerSpec

A few of those NoClassDefFoundError:

[info] org.apache.pekko.http.scaladsl.GracefulTerminationSpec *** ABORTED *** (17 milliseconds)
[info]   java.lang.NoClassDefFoundError: Could not initialize class org.apache.pekko.http.impl.util.ExampleHttpContexts$

And a new one:

[info] Support for telemetry
[info] org.apache.pekko.http.impl.engine.http2.TelemetrySpiPlaintextSpec *** ABORTED *** (603 milliseconds)
[info]   java.lang.ExceptionInInitializerError:
[info]   at org.apache.pekko.http.impl.engine.http2.TelemetrySpiSpec.bindAndConnect(TelemetrySpiSpec.scala:105)
[...]
[info]   Cause: java.io.IOException: parseAlgParameters failed: ObjectIdentifier() -- data isn't an object ID (tag = 48)
[info]   at sun.security.pkcs12.PKCS12KeyStore.parseAlgParameters(PKCS12KeyStore.java:819)

EDIT: Looking deeper those errors all stem from line 44 of ExampleHttpContexts. Somehow my JDK 8 doesn't like the example keystore in keys/server.p12.

[lomigmegard]

Using JDK 11 OpenJDK Runtime Environment Homebrew (build 11.0.20+0)

I get again the same failures as with JDK 20:

[info] *** 12 TESTS FAILED ***
[error] Failed tests:
[info] Passed: Total 2, Failed 0, Errors 0, Passed 2
[error] 	org.apache.pekko.http.impl.engine.http2.H2SpecIntegrationSpec

[info] - must pass rule: 8.2. Server Push *** FAILED *** (18 milliseconds)
[info]   139 was not equal to 0 (H2SpecIntegrationSpec.scala:194)

[pjfanning]

I have no issues running these tests on my old Mac with Java 8 or Java 11. We haven't made much attempt to test Java 20 yet.

@lomigmegard The tests that fail for you rely on opening ports. Is there something on your machine blocking these tests from opening ports?

[lomigmegard]

Just to be sure I disabled LittleSnitch and macOS firewall, still the same. The binding of the HTTP/2 server appears be working, a random port is taken.

Let me know if you have other ideas I can try, but we can also ignore my problems as it seems I am the only one have them. Note the files target/test-reports/h2spec-junit-$sectionNr.xml don't exist, combined with the error code 139 – usually a Seg Fault – might indicate some problem with the h2spec binary.

Ah and indeed, running it manually directly seg fault:

./http2-tests/target/h2spec_darwin_amd64/h2spec --dryrun              
[1]    3295 segmentation fault   --dryrun

Looking at the build definition, we are at version 1.5.0 which is from 2016. Latest version is 2.6.0, replacing doesn't work as the download URL is then incorrect. I will open an issue to track this, and let you prioritize it (edit: #262)

[jrudolph]

Can you check again after the h2spec fix, @lomigmegard ?

[pjfanning]

@jrudolph the h2spec releases don't include one for Mac Silicon. https://github.com/summerwind/h2spec/releases

We might need eventually to support building h2spec from source or allowing users to provide their own h2spec runtime.

[jrudolph]

Ah, forgot that even the newest version does not support Arm yet.

[pjfanning]

Java 20 build is now part of nightly build. The one thing that I know of that doesn't work with Java 20 is building the Paradox HTML (doc build). This is because of Java reflection usage in the Paradox code and we will require to set --add-opens settings in our sbt build to allow this to run.

lightbend/paradox#491 (comment) might also work but we appear to already use paradox-java 1.4.1.

[error] Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make protected final java.lang.Class java.lang.ClassLoader.findLoadedClass(java.lang.String) accessible: module java.base does not "opens java.lang" to unnamed module @76951454
[error] 	at java.base/java.lang.reflect.AccessibleObject.throwInaccessibleObjectException(AccessibleObject.java:387)
[error] 	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:363)
[error] 	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:311)
[error] 	at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:201)
[error] 	at java.base/java.lang.reflect.Method.setAccessible(Method.java:195)
[error] 	at org.parboiled.transform.AsmUtils.findLoadedClass(AsmUtils.java:210)
[error] 	at org.parboiled.transform.ParserTransformer.transformParser(ParserTransformer.java:35)
[error] 	at org.parboiled.Parboiled.createParser(Parboiled.java:54)
[error] 	at com.lightbend.paradox.markdown.Reader.<init>(Reader.scala:40)
[error] 	at com.lightbend.paradox.sbt.ParadoxPlugin$.$anonfun$baseParadoxSettings$2(ParadoxPlugin.scala:83)

[jrudolph]

No, we are forcing older paradox versions in our plugins.sbt to stay compatible with Java 8. There's currently no version of paradox that works for both Java 8 and all later versions. After 1.0, we should drop Java 8 support (or at least build with Java 11 by default), so that we can update to a more recent paradox/parboiled version.

[lomigmegard]

Surprisingly the new version of h2spec is working fine on my Apple Silicon, all the tests are green 🎉 Rosetta is probably running automatically on it (why not on the previous version is not clear to me).
@pjfanning the note added in #264 could be removed or tweaked to say it works transparently if Rosetta is installed (which most developer machine would have already).