[K/N] Expose program name in runtime #5281

vonox7 · 2024-03-21T15:39:55Z

*^[KTI-54606](https://youtrack.jetbrains.com/issue/KT-54606) Fixed*

SvyatoslavScherbina · 2024-04-26T14:53:10Z

kotlin-native/backend.native/tests/build.gradle

@@ -380,6 +380,10 @@ standaloneTest("throw_from_except_constr") {
    source = "runtime/exceptions/throw_from_except_constr.kt"
 }

+standaloneTest("program_name") {
+    source = "runtime/program_name/runtime_program_name.kt"
+}


This file represents the legacy test infrastructure that we are actively trying to get rid of.

Please create a proper test class in
https://github.com/JetBrains/kotlin/tree/master/native/native.tests/tests/org/jetbrains/kotlin/konan/test/blackbox instead.

I ported the testcase to the new infrastructure.

SvyatoslavScherbina · 2024-04-26T14:58:47Z

kotlin-native/runtime/src/launcher/cpp/launcher.cpp

@@ -30,6 +30,8 @@ using namespace kotlin;
 //--- Setup args --------------------------------------------------------------//

 OBJ_GETTER(setupArgs, int argc, const char** argv) {
+  kotlin::programName = argv[0];


Is argv[0] guaranteed to live long enough? What will happen if some code accesses kotlin::programName after the main function finishes?
Is argv guaranteed to always have at least one element?

programName will now be set to null before argv leaves scope. I could not find a definite answer if it would be guaranteed to live long enough, so I opted for the safe choice.

Theoretically it is guaranteed by the POSIX standard. However if it has 0 arguments, the code 1 line below would have crashed. I also added a testcase for argc=0, and added support for it via std::max(0, ...). So now kotlin executables don't crash on startup when they are launched in a non posix compatible way.

kotlin-native/runtime/src/main/kotlin/kotlin/native/Platform.kt

SvyatoslavScherbina · 2024-04-26T15:02:50Z

kotlin-native/backend.native/tests/runtime/program_name/runtime_program_name.kt

+    val programFileName = Platform.programName.substringAfterLast("/").substringBeforeLast(".")
+
+    assertEquals("program_name", programFileName)
+}


More tests are necessary. For example, the original issue mentions a nice use case:

Build multi-call binaries. For example, busybox is a single binary that contains many tools, and decides which tool to run based on how the symbolic link is called: https://busybox.net/downloads/BusyBox.html#usage (that's how it can be so small)

So checking a case with symbolic link would also be useful.

I added test for that by calling execv() in the C code. This allowed to test even more use-cases (no programName at all), while still covering the use-case of renaming/linking a binary.

# Conflicts: # kotlin-native/backend.native/tests/build.gradle

SvyatoslavScherbina · 2024-05-13T10:22:24Z

native/native.tests/build.gradle.kts

@@ -51,6 +51,7 @@ val stdlibK2Test = nativeTest("stdlibK2Test", "stdlib & frontend-fir")
 val kotlinTestLibraryTest = nativeTest("kotlinTestLibraryTest", "kotlin-test & !frontend-fir")
 val kotlinTestLibraryK2Test = nativeTest("kotlinTestLibraryK2Test", "kotlin-test & frontend-fir")
 val partialLinkageTest = nativeTest("partialLinkageTest", "partial-linkage")
+val programNameTest = nativeTest("programNameTest", "program-name")


This seems a bit redudant. I mean, running these tests separately shouldn't happen frequently, so adding a Gradle task is overkill.

Ok, I've removed it.

SvyatoslavScherbina · 2024-05-13T12:38:02Z

native/native.tests/testData/programName/kotlinPrintEntryPoint.kt

+
+fun main(args: Array<String>) {
+    // Remove path and extension (.kexe or .exe)
+    val programFileName = Platform.programName?.substringAfterLast("/")?.substringBeforeLast(".")


This test fails on Windows, because it uses \ instead of /.

Good catch, thank you. The sanitization is now platform agnostic.

SvyatoslavScherbina · 2024-05-13T13:07:19Z

native/native.tests/tests/org/jetbrains/kotlin/konan/test/blackbox/ProgramNameTest.kt

+            val result = runProcess(cExecutable.absolutePath, binaryName, *args) {
+                timeout = 60.seconds
+            }
+            assertEquals("calling exec...\n$expected", result.stdout)


I suspect this code should be fixed to handle \r\n on Windows.

Good catch, thank you again! The assertion is now platform agnostic due to additional sanitization.

SvyatoslavScherbina · 2024-05-13T13:07:42Z

native/native.tests/tests/org/jetbrains/kotlin/konan/test/blackbox/ProgramNameTest.kt

+
+        fun validate(expected: String, vararg args: String) {
+            val binaryName = kotlinCompilation.resultingArtifact.executableFile.path
+            val result = runProcess(cExecutable.absolutePath, binaryName, *args) {


This fails if the test target is different from the host. You can reproduce this by running this test with -Pkotlin.internal.native.test.target=<target> Gradle property.

Please use a proper executor through

kotlin/native/native.tests/tests/org/jetbrains/kotlin/konan/test/blackbox/support/settings/SettingsExecutor.kt

Line 18 in 8c69667

val Settings.executor: Executor

Thank you for this insight. I changed it accordingly.

The test fails on linuxArm64 target with:

java.lang.AssertionError: Expected <calling exec... programName: app args:>, actual <calling exec... exec failed>. at kotlin.test.DefaultAsserter.fail(DefaultAsserter.kt:16) at kotlin.test.Asserter$DefaultImpls.assertTrue(Assertions.kt:694) at kotlin.test.DefaultAsserter.assertTrue(DefaultAsserter.kt:11) at kotlin.test.Asserter$DefaultImpls.assertEquals(Assertions.kt:713) at kotlin.test.DefaultAsserter.assertEquals(DefaultAsserter.kt:11) at kotlin.test.AssertionsKt__AssertionsKt.assertEquals(Assertions.kt:63) at kotlin.test.AssertionsKt.assertEquals(Unknown Source) at kotlin.test.AssertionsKt__AssertionsKt.assertEquals$default(Assertions.kt:62) at kotlin.test.AssertionsKt.assertEquals$default(Unknown Source) at org.jetbrains.kotlin.konan.test.blackbox.ProgramNameTest.programNameTest$validate(ProgramNameTest.kt:49) at org.jetbrains.kotlin.konan.test.blackbox.ProgramNameTest.programNameTest(ProgramNameTest.kt:54) ...

To reproduce, you can run the test with -Pkotlin.internal.native.test.target=linuxArm64 Gradle property on a Linux/x86_64 machine. I don't have access to such a machine at the moment. Could you please check the test, if possible?

Our test infrastructure runs tests on linuxArm64 through qemu, maybe the problem is somehow related to that.

I could reproduce the problem by running the test via ./gradlew :native:native.tests:test --tests "org.jetbrains.kotlin.konan.test.blackbox.ProgramNameTest" -Pkotlin.internal.native.test.target=linux_arm64 on an x86 linux machine. However you are right, the current test infrastructure with qemu user mode emulation seems to not handle the exec syscall in the way we need it. I also tried upgrading qemu to version 9.0.0, but this also didn't resolve the issue. If I understand the following qemu-comment correctly, qemu can't exec into a new process which runs again on qemu, but always does a syscall to the host system. This also explains why exec fails with error code 8 (Exec format error), as qemu would instruct the x86 host system to run the arm64 binary:

at the point of execve the process leaves QEMU's control
(qemu syscall.c)

I see now some ways to approach the problem:
A) Disable the testcase in case of testRunSettings.executor is EmulatorExecutable. Note that this testcase already tests the handling of the "default" program name from the exec syscall the JVM does for us. So we already know that reading a value from argv[0] works. Then this PR could probably be merged when no new points come up.
B) Create a FullHostSystemEmulatorExecutable, which uses qemu system emulation instead of qemu user space emulation. I'm not sure if this wouldn't be way out of scope for this PR.

Do you have any other ideas? If not, are you fine with approach A?

--
I also managed to run the EmulatorExecutable with the following command, which just does qemu -> main.cexe -> qemu -> app.kexe: /.../qemu-aarch64 -L /.../sysroot /.../main.cexe /.../qemu-aarch64 qemu-aarch64 -L /.../sysroot /.../app.kexe (minimal refactoring needed in EmulatorExecutable.kt). However, also in this case we can not control the exec syscall that actually starts app.kexe, as this is part of the qemu user mode emulation infrastrucutre. There qemu sets then manually argv[0]=app when launching app.kexe.

SvyatoslavScherbina · 2024-05-13T13:08:48Z

native/native.tests/tests/org/jetbrains/kotlin/konan/test/blackbox/ProgramNameTest.kt

+        // No program name - this would not be POSIX compliant, see https://pubs.opengroup.org/onlinepubs/9699919799/functions/exec.html:
+        // "[...] requires a Strictly Conforming POSIX Application to pass at least one argument to the exec function"
+        // However, we should not crash the Kotlin runtime because of this.
+        validate("programName: null\nargs:")


On Linux, this fails for me with

java.lang.AssertionError: Expected <calling exec... programName: null args:>, actual <calling exec... programName: args:>.

Please take a look.

This is very interesting. Linux behaves here differently as windows & macOS. I fixed the problem, and added more explanation in the code. I used the following C standard PDF: https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf.

Please tell me if you would want to let windows+macOS behave like linux. The way I wrote it now feels more natural to me, however I fully understand that all 3 options have its benefits and drawbacks:

A) The way I wrote it now: all 3 platforms behave the same; no program name leads to programName=null, and empty programName leads to programName="".
B) all 3 platforms behave the same; however both no program name and empty program Name lead to programName="". (the linux way of thinking)
C) Fully native behaviour: no programName will be programName="" on linux and programName=null on macOS/windows; and empty programName will be everywhere programName="".

…ek/program-name

SvyatoslavScherbina · 2024-05-28T15:03:24Z

kotlin-native/runtime/src/main/cpp/Runtime.h

@@ -8,6 +8,7 @@

 #include "Porting.h"
 #include "Memory.h"
+#include "KString.h"


Seems unused.

SvyatoslavScherbina · 2024-05-28T15:12:11Z

native/native.tests/testData/programName/main.c

+
+    // Kotlin executable name is in argv[1]
+    // Forward argv[2..n] to kotlin executable as arguments (the first one should be the programName according to posix)
+    execv(argv[1], &(argv[2]));


Btw, execv requires that the list of arguments must be terminated by a NULL pointer. Is it always guaranteed here?

argv is an array of size argc + 1 and the last member is always NULL: https://en.cppreference.com/w/c/language/main_function

SvyatoslavScherbina · 2024-05-28T16:35:42Z

native/native.tests/tests/org/jetbrains/kotlin/konan/test/blackbox/ProgramNameTest.kt

+
+        fun validate(expected: String, vararg args: String) {
+            val binaryName = kotlinCompilation.resultingArtifact.executableFile.path
+            val result = runProcess(cExecutable.absolutePath, binaryName, *args) {


The test fails on linuxArm64 target with:

java.lang.AssertionError: Expected <calling exec... programName: app args:>, actual <calling exec... exec failed>. at kotlin.test.DefaultAsserter.fail(DefaultAsserter.kt:16) at kotlin.test.Asserter$DefaultImpls.assertTrue(Assertions.kt:694) at kotlin.test.DefaultAsserter.assertTrue(DefaultAsserter.kt:11) at kotlin.test.Asserter$DefaultImpls.assertEquals(Assertions.kt:713) at kotlin.test.DefaultAsserter.assertEquals(DefaultAsserter.kt:11) at kotlin.test.AssertionsKt__AssertionsKt.assertEquals(Assertions.kt:63) at kotlin.test.AssertionsKt.assertEquals(Unknown Source) at kotlin.test.AssertionsKt__AssertionsKt.assertEquals$default(Assertions.kt:62) at kotlin.test.AssertionsKt.assertEquals$default(Unknown Source) at org.jetbrains.kotlin.konan.test.blackbox.ProgramNameTest.programNameTest$validate(ProgramNameTest.kt:49) at org.jetbrains.kotlin.konan.test.blackbox.ProgramNameTest.programNameTest(ProgramNameTest.kt:54) ...

To reproduce, you can run the test with -Pkotlin.internal.native.test.target=linuxArm64 Gradle property on a Linux/x86_64 machine. I don't have access to such a machine at the moment. Could you please check the test, if possible?

Our test infrastructure runs tests on linuxArm64 through qemu, maybe the problem is somehow related to that.

SvyatoslavScherbina · 2024-05-28T16:38:49Z

kotlin-native/runtime/src/main/kotlin/kotlin/native/Platform.kt

+     * [null] if the Kotlin code was compiled to a native library and the executable is not a Kotlin program.
+     */
+    public val programName: String?
+        get() = Platform_getProgramName()


@qurbonzoda could you please review the stdlib change?

projedi · 2024-05-30T11:26:13Z

kotlin-native/runtime/src/launcher/cpp/launcher.cpp

@@ -30,8 +30,12 @@ using namespace kotlin;
 //--- Setup args --------------------------------------------------------------//

 OBJ_GETTER(setupArgs, int argc, const char** argv) {
+  if (argc > 0 && argv[0][0] != '\0') {
+    kotlin::programName = argv[0];


How about doing kotlin::programName = strndup(argv[0], 4096)? No need to free the result, and we don't have to worry about lifetimes. 4096 is just some random not-too-small not-too-big number to protect us from argv[0] being too large. Technically, the OS has some limits already, but might as well protect ourselves.

Ok, I applied your idea.

projedi · 2024-05-30T11:39:27Z

kotlin-native/runtime/src/launcher/cpp/launcher.cpp

@@ -30,8 +30,12 @@ using namespace kotlin;
 //--- Setup args --------------------------------------------------------------//

 OBJ_GETTER(setupArgs, int argc, const char** argv) {
+  if (argc > 0 && argv[0][0] != '\0') {


Need a comment explaining the purpose of argv[0][0] != '\0' check.

You're right, I also commented it here and not only indirectly in the testcase.

projedi · 2024-05-30T11:41:54Z

native/native.tests/testData/CExport/InterfaceV1/programName/main.c

+
+int main() {
+    programName();
+    fflush(NULL);


Let's add an explicit return 0;?

projedi · 2024-05-30T11:42:08Z

native/native.tests/testData/programName/main.c

+    execv(argv[1], &(argv[2]));
+
+    printf("exec failed\n");
+    return 0;


Let's return non-zero exit code?

Good point. I also now print the errno for easier debugging in case the exec syscall fails.

This reverts commit fd2ebbf.

[K/N] Expose program name in runtime

1ff362a

*^[KTI-54606](https://youtrack.jetbrains.com/issue/KT-54606) Fixed*

ivakub added the Native label Apr 8, 2024

SvyatoslavScherbina requested changes Apr 26, 2024

View reviewed changes

vonox7 added 5 commits April 29, 2024 14:57

Merge branch 'refs/heads/master' into slawicek/program-name

05d57ee

# Conflicts: # kotlin-native/backend.native/tests/build.gradle

Move programName testcase to new test infrastructure

f73045e

Add testcase for calling Platform.getProgramName from within a library

fb67427

Execute Kotlin executable via execv to pass custom programNames

2703f15

Support argc=0

3b66b6b

vonox7 requested a review from SvyatoslavScherbina April 30, 2024 14:05

Merge branch 'JetBrains:master' into slawicek/program-name

285dd67

SvyatoslavScherbina requested changes May 13, 2024

View reviewed changes

vonox7 added 6 commits May 15, 2024 10:45

Use proper executor

69e660f

Merge remote-tracking branch 'vali/slawicek/program-name' into slawic…

a69b027

…ek/program-name

Remove unneeded gradle task

6d07cf2

Ignore different windows path & line-endings

2fcf1f0

Let linux behave the same as windows/macos

3c933a7

Add empty program name validation

802c8e7

vonox7 requested a review from SvyatoslavScherbina May 26, 2024 19:03

Fix testcase on windows

64ffcfc

SvyatoslavScherbina reviewed May 28, 2024

View reviewed changes

SvyatoslavScherbina requested a review from projedi May 28, 2024 16:39

projedi requested changes May 30, 2024

View reviewed changes

vonox7 added 8 commits June 3, 2024 18:36

Add comment for argv[0] == "" check

2d44f5f

Add explicit return

06b0d67

Set return value to 1 in case of program fail

27d7688

Remove unneeded include

fba8ce4

Copy programName so no need to reset it

0aa4600

Merge branch 'JetBrains:master' into slawicek/program-name

7fb104f

Print errno for easier exec debugging

7868f85

Merge branch 'JetBrains:master' into slawicek/program-name

6323590

vonox7 added 3 commits June 4, 2024 20:30

Add missing import for linux

e39351d

Try to update qemu

fd2ebbf

Revert "Try to update qemu"

d09c9f0

This reverts commit fd2ebbf.

vonox7 requested review from SvyatoslavScherbina and projedi June 6, 2024 12:22

vonox7 added 2 commits June 8, 2024 11:24

Remove include

841f6e0

Add include back

ee0ed60

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[K/N] Expose program name in runtime #5281

[K/N] Expose program name in runtime #5281

vonox7 commented Mar 21, 2024

SvyatoslavScherbina Apr 26, 2024

vonox7 Apr 30, 2024

SvyatoslavScherbina Apr 26, 2024

vonox7 Apr 30, 2024 •

edited

SvyatoslavScherbina Apr 26, 2024

vonox7 Apr 30, 2024

SvyatoslavScherbina May 13, 2024

vonox7 May 26, 2024

SvyatoslavScherbina May 13, 2024

vonox7 May 26, 2024

SvyatoslavScherbina May 13, 2024

vonox7 May 26, 2024

SvyatoslavScherbina May 13, 2024

vonox7 May 26, 2024

SvyatoslavScherbina May 28, 2024

vonox7 Jun 6, 2024

SvyatoslavScherbina May 13, 2024

vonox7 May 26, 2024

SvyatoslavScherbina May 28, 2024

vonox7 Jun 6, 2024

SvyatoslavScherbina May 28, 2024

projedi May 30, 2024

SvyatoslavScherbina May 28, 2024

SvyatoslavScherbina May 28, 2024

projedi May 30, 2024

vonox7 Jun 6, 2024 •

edited

projedi May 30, 2024

vonox7 Jun 6, 2024

projedi May 30, 2024

vonox7 Jun 6, 2024

projedi May 30, 2024

vonox7 Jun 6, 2024 •

edited

[K/N] Expose program name in runtime #5281

Are you sure you want to change the base?

[K/N] Expose program name in runtime #5281

Conversation

vonox7 commented Mar 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vonox7 Apr 30, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vonox7 Jun 6, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vonox7 Jun 6, 2024 • edited

Choose a reason for hiding this comment

vonox7 Apr 30, 2024 •

edited

vonox7 Jun 6, 2024 •

edited

vonox7 Jun 6, 2024 •

edited