RFC - Smaller bundled JRE

[SomeTroglodyte]

Before creating

• This is NOT a gameplay feature from Civ VI, BNW, or outside - see Roadmap
• This is NOT a gameplay feature from Vanilla Civ V or from G&K - If so, it should be a comment in Missing features from Civ V - G&K #4697

Problem Description

Thinking about those 90MB download zip...

Related Issue Links

No response

Desired Solution

I tried, out of a hunch, to ask the jdk to build a jre adapted to Unciv. The installed size shrinks by ~70MB, the ZIP size by ~9MB.

I asked jdeps which modules are required: jdk\bin\jdeps.exe --print-module-deps --ignore-missing-deps --recursive --multi-release 11 Unciv.jar (supplying an unpacked jar to --class-path and/or --module-path did not change the result).
Then I asked jlink to make a runtime: Delete jre from extracted zip, leave exe and jar in place, then: jdk\bin\jlink.exe --add-modules java.base,java.desktop,java.instrument,java.logging,java.management,java.prefs,jdk.unsupported,jdk.crypto.ec --strip-debug --no-man-pages --no-header-files --compress=2 --output .\jre (jdk.crypto.ec was missed by jdeps, missing it blocks any https)

Now jre/lib/modules is 20M instead of 90M, and zipping it back up gives 79.6MiB.

Alternative Approaches

Not investigated whether a similar effect can be achieved using our current packer.

Additional Context

There's not much savings to be had elsewhere - translations is 15M uncompressed, kotlin reflection is >8M of class files, looking into why com.unciv is taking so much space just leads to tons of successively smaller files...

[yairm210]

Sounds good!!!

[SomeTroglodyte]

I was tending to "not worth the extra hassle". This came from that "can no longer start the gradle 8 builds" issue that doesn't answer our questions, and thus awakened curiosity ("were we not much much leaner not too long ago, in the 10M range???").

What prepares our current jre? Anuken packr? If it has options to control jlink... If not, this would mean one packr run discarding everything packr produces except the exe, then running jlink separately? Also, the module list needs manual maintenance if any major library is added or changed - jdeps autodetection, as shown, fails with a few modules that aren't loaded the normal way. (superficial duckduck skimming suggests the ssl stuff is loaded via reflection)
... also, we're currently pulling the "jre" assets which do not contain these tools, only the "jdk" ones do.

Wait - packr refers to jlink in their readme - will maybe read later.

[yairm210]

No, LibGDX packr run by cli command

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 15 days.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[touhidurrr]

Building JREs from JDK on demand? Cool idea.

[touhidurrr]

I did not even know something like this was even possible.
One problem though, how do you plan to build the windows version?
Solution can be something like this:

1. split packr-build task into 3 tasks, build-linux-jre, build-windows-jre, packr-build where packr-build depends on [build-linux-jre, build-windows-jre]
2. run build-windows-jre on windows-latest obviously.
3. Instead of using Adoptium API, use actions/setup-java to setup latest temurin JDKs
4. Now run your commands to build JREs
5. Use JREs built to pack everything with packr

[touhidurrr]

Found related discussions in:
https://adoptium.net/blog/2021/10/jlink-to-produce-own-runtime/

packr support seems sus, the docs are discussed here:
https://github.com/libgdx/packr/blob/master/README.md#minimization
Their example: https://github.com/libgdx/packr/blob/master/TestAppJreDist/testAppJreDist.gradle.kts

This looks a little too complex to me, jlink solution feels better.

jlink: https://docs.oracle.com/en/java/javase/11/tools/jlink.html

[touhidurrr]

Anyways, I can try to write actions code for this issue with jlink with the approach in #11091 (comment) if you guys are willing to get onboard.

[SomeTroglodyte]

My brain says: Uuuuugghhhh... There's areas where it feels like a Neanderthal seeing the Monolith - so I agree where I understand, and shy away cowardly where learning is required. I fear to get this automated requires a little trial and error, which I didn't want to back then - I know a little more about actions now, such as how you can get the "release patch" uncivbot to run on your own fork - so actual testing before submitting the script to the boss fork is possible... But add some laziness, and I'm happy if someone else does all that.

[touhidurrr]

ok, I tried the tutorial here: https://adoptium.net/blog/2021/10/jlink-to-produce-own-runtime/

Here is the result:

312M    jdk-11.0.23+9      // unpacked official temurin 11 jdk
125M    jdk-11.0.23+9-jre  // unpacked official temurin 11 jre
187M    jdk.tar.gz         // official temurin 11 jdk
42M     jre.tar.gz         // official temurin 11 jre
53M     minimized-jre      // the minimized jre that i generated
53M     Unciv.jar          // latest Unciv jar

So, 125M to 53M is like ~57.6% save. And yes, I tried running Unciv, it works.

[touhidurrr]

Ok, there one problem though. Maybe this is not as easy as it looks. I tried running Unciv with minimized-jre and network calls seems to be not working. Works fine with jdk shipped jre.

~$ ~/test$ ./minimized-jre/bin/java -jar Unciv.jar
sh: 1: xdg-mime: not found  // ignore this, common across all jres
2024-06-15T10:42:25.721715Z [threadpool-daemon-1] [Github] [ERROR] Exception during GitHub download | javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
        at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
        at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
        at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
        at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Unknown Source)
        at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
        at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
        at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
        at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
        at com.unciv.logic.github.Github$tryGetGithubReposWithTopic$inputStream$1.invoke(Github.kt:209)
        at com.unciv.logic.github.Github$tryGetGithubReposWithTopic$inputStream$1.invoke(Github.kt:208)
        at com.unciv.logic.github.Github.download(Github.kt:52)
        at com.unciv.logic.github.Github.tryGetGithubReposWithTopic(Github.kt:208)
        at com.unciv.logic.github.Github.tryGetGithubReposWithTopic$default(Github.kt:200)
        at com.unciv.ui.screens.modmanager.ModManagementScreen$tryDownloadPage$1.invokeSuspend(ModManagementScreen.kt:288)
        at com.unciv.ui.screens.modmanager.ModManagementScreen$tryDownloadPage$1.invoke(ModManagementScreen.kt)
        at com.unciv.ui.screens.modmanager.ModManagementScreen$tryDownloadPage$1.invoke(ModManagementScreen.kt)
        at com.unciv.utils.ConcurrencyKt$launchCrashHandling$1.invokeSuspend(Concurrency.kt:89)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
        at com.unciv.utils.Dispatchers$CrashHandlingDispatcher$dispatch$1.invoke(Concurrency.kt:190)
        at com.unciv.utils.Dispatchers$CrashHandlingDispatcher$dispatch$1.invoke(Concurrency.kt:190)
        at com.unciv.ui.crashhandling.CrashHandlingExtensionsKt$wrapCrashHandling$1.invoke(CrashHandlingExtensions.kt:17)
        at com.unciv.ui.crashhandling.CrashHandlingExtensionsKt$wrapCrashHandlingUnit$1.invoke(CrashHandlingExtensions.kt:33)
        at com.unciv.ui.crashhandling.CrashHandlingExtensionsKt$wrapCrashHandlingUnit$1.invoke(CrashHandlingExtensions.kt:33)
        at com.unciv.utils.Dispatchers$CrashHandlingDispatcher.dispatch$lambda$0(Concurrency.kt:190)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

2024-06-15T10:42:25.945385Z [threadpool-daemon-1] [Github] [ERROR] Exception during GitHub download | javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
        at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
        at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
        at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
        at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Unknown Source)
        at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
        at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
        at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
        at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(Unknown Source)
        at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
        at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
        at com.unciv.logic.github.Github$tryGetGithubReposWithTopic$inputStream$1.invoke(Github.kt:209)
        at com.unciv.logic.github.Github$tryGetGithubReposWithTopic$inputStream$1.invoke(Github.kt:208)
        at com.unciv.logic.github.Github.download(Github.kt:52)
        at com.unciv.logic.github.Github.tryGetGithubReposWithTopic(Github.kt:208)
        at com.unciv.logic.github.Github.tryGetGithubReposWithTopic$default(Github.kt:200)
        at com.unciv.ui.screens.modmanager.ModManagementScreen$tryDownloadPage$1.invokeSuspend(ModManagementScreen.kt:288)
        at com.unciv.ui.screens.modmanager.ModManagementScreen$tryDownloadPage$1.invoke(ModManagementScreen.kt)
        at com.unciv.ui.screens.modmanager.ModManagementScreen$tryDownloadPage$1.invoke(ModManagementScreen.kt)
        at com.unciv.utils.ConcurrencyKt$launchCrashHandling$1.invokeSuspend(Concurrency.kt:89)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
        at com.unciv.utils.Dispatchers$CrashHandlingDispatcher$dispatch$1.invoke(Concurrency.kt:190)
        at com.unciv.utils.Dispatchers$CrashHandlingDispatcher$dispatch$1.invoke(Concurrency.kt:190)
        at com.unciv.ui.crashhandling.CrashHandlingExtensionsKt$wrapCrashHandling$1.invoke(CrashHandlingExtensions.kt:17)
        at com.unciv.ui.crashhandling.CrashHandlingExtensionsKt$wrapCrashHandlingUnit$1.invoke(CrashHandlingExtensions.kt:33)
        at com.unciv.ui.crashhandling.CrashHandlingExtensionsKt$wrapCrashHandlingUnit$1.invoke(CrashHandlingExtensions.kt:33)
        at com.unciv.utils.Dispatchers$CrashHandlingDispatcher.dispatch$lambda$0(Concurrency.kt:190)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

2024-06-15T10:42:25.946604Z [threadpool-daemon-1] [ModManagementScreen$tryDownloadPage] [ERROR] Could not download mod list | java.lang.NullPointerException
        at com.unciv.ui.screens.modmanager.ModManagementScreen$tryDownloadPage$1.invokeSuspend(ModManagementScreen.kt:288)
        at com.unciv.ui.screens.modmanager.ModManagementScreen$tryDownloadPage$1.invoke(ModManagementScreen.kt)
        at com.unciv.ui.screens.modmanager.ModManagementScreen$tryDownloadPage$1.invoke(ModManagementScreen.kt)
        at com.unciv.utils.ConcurrencyKt$launchCrashHandling$1.invokeSuspend(Concurrency.kt:89)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
        at com.unciv.utils.Dispatchers$CrashHandlingDispatcher$dispatch$1.invoke(Concurrency.kt:190)
        at com.unciv.utils.Dispatchers$CrashHandlingDispatcher$dispatch$1.invoke(Concurrency.kt:190)
        at com.unciv.ui.crashhandling.CrashHandlingExtensionsKt$wrapCrashHandling$1.invoke(CrashHandlingExtensions.kt:17)
        at com.unciv.ui.crashhandling.CrashHandlingExtensionsKt$wrapCrashHandlingUnit$1.invoke(CrashHandlingExtensions.kt:33)
        at com.unciv.ui.crashhandling.CrashHandlingExtensionsKt$wrapCrashHandlingUnit$1.invoke(CrashHandlingExtensions.kt:33)
        at com.unciv.utils.Dispatchers$CrashHandlingDispatcher.dispatch$lambda$0(Concurrency.kt:190)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)

[touhidurrr]

Anyways, here are the steps that I used so that you guys can try debugging.

wget https://api.adoptium.net/v3/binary/latest/11/ga/linux/x64/jdk/hotspot/normal/eclipse -O jdk.tar.gz
tar -xzvf jdk.tar.gz
rm jdk.tar.gz
mv jdk-11* jdk-11
./jdk-11/bin/jdeps -s Unciv.jar > deps.txt
UNCIV_MODULES=$(cat deps.txt | grep -v 'not found' | cut -d ' ' -f 3 | tr '\n' ',' | sed 's/,$//')
./jdk-11/bin/jlink --add-modules $UNCIV_MODULES --strip-debug --no-man-pages --no-header-files --compress=2 --output minimized-jre
wget https://github.com/yairm210/Unciv/releases/download/4.11.19-patch1/Unciv.jar
./minimized-jre/bin/java -jar Unciv.jar

[touhidurrr]

I think this will take time to figure out. The best for now maybe just to merge #11751 for now as we try to figure this out.
I don't think can fix this dependency issue alone or this will get done by the next version or anytime soon.
We need to fix this issue in such a way so that we don't need to touch it in future. As for now, this is not working as intended:

./jdk-11/bin/jdeps -s Unciv.jar > deps.txt
UNCIV_MODULES=$(cat deps.txt | grep -v 'not found' | cut -d ' ' -f 3 | tr '\n' ',' | sed 's/,$//')

[SomeTroglodyte]

network calls seems to be not working

This one I did cover up there... Look for "(jdk.crypto.ec was missed by jdeps, missing it blocks any https)" in the first post. A search turns up some explanations hinting why, and hinting this is pretty common IIRC.

[touhidurrr]

This one I did cover up there... Look for "(jdk.crypto.ec was missed by jdeps, missing it blocks any https)" in the first post. A search turns up some explanations hinting why, and hinting this is pretty common IIRC.

So, just adding jdk.crypto.ec manually will solve this issue?

[SomeTroglodyte]

Yes, but 4 eyes see more than 2, so don't buy that bag untested, or don't let me prejudice you on which Unciv features to thoroughly test for missing modules from jlink minification. 'Later, I need to do some RL.

[touhidurrr]

don't buy that bag untested

That is the concern I have also. How to know if the JRE we have shipped will work? We can do trial and error by actually shipping it first then debugging but that would be hilarious (maybe something like putting the cart before the horse also).
About workarounds for this concern, I can't think of any.

[touhidurrr]

Ok found a solution that looks like it might resolve any such unforseen dependency missing issues. It is here:
https://stackoverflow.com/a/62359298
But we need the maven commands gradle counterpart.