Investigate / add Kotlin native support

[bitspittle]

Context: https://www.reddit.com/r/Kotlin/comments/q17zjq/introducing_konsole_a_kotlinidiomatic_library_for/hfggbsv/

Currently, this library is JVM only. Adding native support would allow users to use this library in a Kotlin native context, which would:

• result in smaller file sizes / let people run on systems that don't have Java installed(?)
• have better performance(?)
• make this library available to Kotlin native projects

Current challenges:

• I'm currently leaning on the JVM library Jline for system terminal support, although there is almost certainly a C library that we can hook into instead
• I'm using Swing to create a virtual terminal, but we can exclude Virtual Terminal support for native targets.
• I've never written a Kotlin Native app before. I may be relying on JVM-only features without realizing it, like my threading logic or who knows what else?

This bug isn't a priority for me at the moment because it's not clear to me how many people actually want this support, but hey, if you do, please indicate so by adding a thumbs up reaction to it (and consider leaving a comment on why you don't want to just use the JVM, I'd be curious!)

[jamesward]

I'd love this for a project I'm working on, but I realize the Jline replacement piece would be tough. Maybe I can find some cycles to help. :)

[bitspittle]

If you want to explore this any time, let me know and I'll give you all the cycles I can! Happy to meet first at some point and tell you all known pitfalls.

…

On Mon, Jul 18, 2022, 4:24 PM James Ward ***@***.***> wrote: I'd love this for a project I'm working on, but I realize the Jline replacement piece would be tough. Maybe I can find some cycles to help. :) — Reply to this email directly, view it on GitHub <#63 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKNONASRH6MG4RRLVA3PV4LVUXRSHANCNFSM5FMJECLA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[rohanprabhu]

We are working on building a CLI application and this library looks like something that would be perfect for our use case. The only issue is that our CLI has to have native targets, and hence kotlin-native support is imperative. Talking about the JLine functionality, is there an interface (or interfaces) that abstracts out the JLine functionality - such that a native implementation of the same would suffice for a kotlin-native target? This is something I'd be very interested to pick up depending on what complexity we are looking at.

[bitspittle]

Hey! Thanks for your interest in Kotter!

I'd be happy to set up some time to meet, to share with you over video chat the current situation and perhaps accelerate any quick experiments you want to try. I'll send a message to your github account with details after writing this.

To answer your question, Kotter has a Terminal interface which is pretty minimal, and it sounds like what you're looking for. If you can implement it effectively for native, then you won't need JLine. Check out TestTerminal for a in-memory terminal I created for tests, and the SystemTerminal implementation which is the only part of the code that references JLine.

Note that there will still be some difficulties ahead, particularly threading. I am not too familiar with how threading works in Kotlin/native, because as far as I can tell the docs kind of suck. Note here how they bounce you to the "modern memory management" docs but those don't talk about threading at all.

There are a few places where I kick off threads (usually via coroutines, but in one case from an executor).

• kotter/kotter/src/main/kotlin/com/varabyte/kotter/runtime/Section.kt

  Line 184 in bf919e8

  return CoroutineScope(session.executor.asCoroutineDispatcher()).launch {

  • Renders happen sequentially on a dedicated render thread, managed by this single threaded executor. Read more in the README.

• kotter/kotter/src/main/kotlin/com/varabyte/kotter/foundation/input/InputSupport.kt

  Line 84 in bf919e8

  CoroutineScope(Dispatchers.IO).launch {

  • This extra thread was required because console ESC handling is awful. You can't tell the difference between the user pressing ESC or the system sending you an encoding command that starts with ESC. I couldn't find a good way to handle this except by waiting after the first ESC key came in to see if anything else followed. (Once you are aware of this hack, you can actually discover your own favorite terminal is probably doing something like this too)

• kotter/kotter/src/main/kotlin/com/varabyte/kotter/foundation/input/InputSupport.kt

  Line 115 in bf919e8

  .shareIn(CoroutineScope(Dispatchers.IO), SharingStarted.Lazily)

  • The Terminal interface exposes a Flow of raw ANSI bytes that are fired when keys are pressed. Those need to be translated into the Keys instances exposed by Kotter. This thread handles the flow collection and mapping.

• kotter/kotter/src/main/kotlin/com/varabyte/kotter/foundation/timer/TimerSupport.kt

  Line 90 in bf919e8

  private val job = CoroutineScope(Dispatchers.IO).launch {

  • Timers run on their own long-lived thread.

You'll want to make sure you understand all those cases. Getting this to work for native may be a trivial case of adding actual / expect or it may be a show stopper due to my incompatible design, all based on how native threads work. I'm simply not sure, and that's the first place I would experiment with.

Beyond that, you should clone the project and do a search for import java under kotter/kotter/src/main. That should show you some of the trouble you might be getting into :) I should have probably used kotlin Duration instead of java Duration, but it was totally experimental when I started writing Kotter, and now the API exposes java Durations in a bunch of places.

Otherwise, in theory you just need to move the following packages from kotter/kotter/src/main to commonMain:

com.varabyte
   kotter
      foundation.*
      runtime.*
   kotterx.*

and then move com.varabyte.kotter.terminal implementations to jvmMain. (And you'd create your new native terminal implementation in the native main section that you create).

At this point, brace yourself for a bunch of compile errors :)

Finally, here are some alternate solutions to make sure you've considered other options:

• Write your CLI using the JVM but call out to a native binary for some performance critical component. This might work well combined with graalvm!
  • Honestly, I'd try this first...
• There is a project called Mordant you should check out, in case it satisfies your use case as well. It has multiplatform support today.

[bitspittle]

I heard that Mosaic went multiplatform recently. If I can see how they did it and it doesn't seem that hard, I'll give a shot porting Kotter to multiplatform.

At one point, I did some experiments with GraalVM which seemed promising at first, and I really wanted to make that my recommended solution, but soon I got a class not found exception from the kotlin standard library (kotlin.random.Random wasn't found??!), and I didn't enjoy the runaround to try to figure out how to fix it. I found the docs for GraalVM occasionally as inscrutable as I sometimes find Gradle docs to be.

I also found a bug reported against the GraalVM Gradle plugin for someone who had a similar exception to what I have, including they did some leg work where they reported the exact line that needed to get fixed, and after that -- nothing. No response or change from the GraalVM team.

---

Anyway, since even my initial impressions with GraalVM were frustration, I think multiplatform (if I can get it to work) will be a much easier thing for me to recommend to kotter users.

[bitspittle]

First hurdle - Kotter makes use of reentrant locks. It seems like this is not officially supported in Kotlin/Native. There's atomicfu which provides reentrant locks but says that it's a heavily experimental feature and that library authors should not use them.

I may be able to reimplement what I need using expect / actual but this could get ugly...

[bitspittle]

Initial prototype success:

Final debug exe size: 7.6MB, release exe size: 1.4MB. That's not bad....

Still, best to temper any excitement with the following points:

1. I am implementing my own ReentrantLock and ReentrantReadWriteLock classes at this point. This scares the hell out of me. I am NOT a concurrency expert. It's possible my code will break on a really hard to reproduce edge cases later, or it will work just fine but might have relatively poor performance characteristics.

2. I am not supporting reading yet. Printing text is easy. Reading characters in a platform independent way may or may not be easy. I need to put the terminal into raw mode (so keys aren't repeated when the user types them) and I have no idea how to do that in Kotlin/Native at this point.

3. I have not found out how to handle CTRL-C yet. There's a Kotlin native method for registering a callback but it complains because I'm passing in a lambda to it, and now amount of inlining has made things happy. I'm not sure what the proper solution is yet or if there even is one.

4. I have not tested Windows yet.

5. I have not tested most of the samples yet.

6. While many user programs might not be affected, there are a handful of APIs now that are not currently backwards compatible. I might be able to smooth these over but I'm not sure.

Got farther than I expected to today, but I'm still not sure yet I'm sold on the extra maintenance burden / complexity this change might bring. If I could get GraalVM working consistently instead, that would be so tempting...

[bitspittle]

In case anyone is still around who is interested in helping me test this in the next few days, please ping me or give a thumbs up to this comment. It would really help the library to get other eyes on this besides mine.

---

Lots of improvements today. Kotter native is on track.

kotter-native-input.mp4

1. I am feeling a little better about my reentrant lock code, which I may still come to regret someday. But at least it seems like it's holding up well to a barrage of tests.

2. Reading user input is now supported 🎉 See video above, which runs the examples/input example using native.

3. CTRL-C is now handled.

4. Windows still not tested. Will test in the next few days.

5. Most samples still not tested yet. Will run a bunch of them in the next few days.

6. I'm more and more OK with breaking backwards compatibility, as the migration steps are trivial. (In most cases, users will just convert lines line Duration.ofMillis(100) to 100.milliseconds, or reimport extension methods that now live in a new place).

   • The next version of Kotter will be 1.1.0, which is technically breaking semver rules (since it's not totally backwards compatible), but I'm not comfortable calling this Kotter 2.0 at this point.

---

In summary, still to do...

• Verify lots of examples work when compiled to native (excluding a few that lean on JVM dependencies)
• Verify Windows is working
• Figure out how to publish a Kotlin native library (probably using GitHub CI to create artifacts for each target, and then pull those down and push them to maven?)
• (Maybe) Add native targets to my testing library, truthish, and port tests over to native

[bitspittle]

Kotlin/Native looking solid on Linux at this point.

Main thing left at this point is figuring out Windows (which it seems will need its own Terminal implementation, and then publishing artifacts on Maven.

It's my top priority for the next few days, hopefully no major snags.

[jamesward]

Loving this progress! Thank you @bitspittle :)

[bitspittle]

Thank you @jamesward!

It's a testament to Kotlin/Native how smoothly this has been going. I was sure I was going to run into those memory freeze exceptions I heard so much about, or other problems that would crop up since I didn't really consider Kotlin/Native when I first built Kotter. But the Kotlin standard library is pretty impressive at this point. And it wasn't long before I had some samples running locally in Kotlin/Native.

My only real regret with Kotter was using Java durations instead of Kotlin durations. I believe Kotlin durations were still experimental when I started writing Kotter, but still -- I should have been more confident that at that point it was probably safe to use them.

The documentation on JB's side for Kotlin/Native is a little meh (JB has a style where they dump a bunch of information on you all at once, much of it intimidating and unrelated, where what I think most people are looking for is a step-by-step book or codelab that introduces concepts in layers).

But once I got going, it was pretty neat to write C-style code with the magic of memScoped blocks. The team did some fancy stuff with Kotlin with their native APIs, which is crazy seeing how few people probably will ever truly use or appreciate them.

Anyway, let's see how Windows goes today...

[bitspittle]

Easy part is done (the text example just prints, it's totally not interactive):

Tomorrow will be focused on getting interactive mode / input working. Hopefully I can find a good article about modifying the terminal from C in Windows.

[bitspittle]

😌

kotter-native-input-win.mp4

Aiming to drop 1.1.0-rc1 on Friday. We'll see how that goes.

P.S. Dealing with the Windows API was a PITA :) It didn't help that I was working on a slow ass virtual machine. Anyway, hopefully I can put this dark memory behind me...

[bitspittle]

Good news -- I have a multiplatform project with a simpler setup than Kotter (https://github.com/varabyte/truthish for the curious) which I used as a test run for figuring out publishing Kotlin / Native artifacts on maven central. Took a bit longer than I expected to set up, but as of this morning, it's working.

It shouldn't be too complicated to carry that logic over to Kotter.

At this point, Kotlin / Native support for Kotter is close. New ETA: should be up within a few days, maybe even tomorrow if things go smoothly.

The scariest thing that could hit me, I think, is a subtle race condition showing up in tests.

The final checklist:

• Add one final feature I want to get in for 1.1.0
• Run tests aggressively (all tests are currently passing for JVM / Linux, but check other platforms and also run a bunch of times to ensure no race conditions). Depending on how this goes, this could push back my ETA.
• Add migration docs for 1.0.x to 1.1.0
• Add GitHub CI support for publishing artifacts to maven
• Add a README section in the Gradle area demonstrating how to target native platforms
• Add a README section for how to distribute a Kotter application
• Publish artifacts to maven

[bitspittle]

Alright folks! If anyone is eager to try Kotter+Multiplatform out right now, you can grab some dev snapshots I just published.

🕵️ With multiplatform increasing the surface of things that can go wrong, I can really use as many eyes as I can get on this! Especially if you use Windows, Mac Intel, and/or Mac M1. Non US-keyboards on Windows a plus. THANK YOU! 🕵️

First, you'll need to declare the following repository:

repositories {
    maven("https://s01.oss.sonatype.org/content/repositories/snapshots/")
}

Then, declare your dependency:

JVM project

dependencies {
   // IMPORTANT! This *used* to be "com.varabyte.kotter:kotter:..."
   implementation("com.varabyte.kotter:kotter-jvm:1.1.0-SNAPSHOT")
}

Multiplatform project

kotlin {
   // declare targets, e.g. linuxX64(), mingwX64()...
   sourceSets {
      val commonMain by getting {
         dependencies {
            implementation("com.varabyte.kotter:kotter:1.1.0-SNAPSHOT")
         }
      }
   }
}

If you've never set up a Kotlin/Native project before, please see the official docs.

You can also confer with the Kotter native example.

Migration from Kotter 1.0.x

• Java Duration -> Kotlin Duration
  • You will probably have a few of these, but fixing them is straightforward.
  • e.g. Duration.ofMillis(250) -> 250.milliseconds
  • e.g. Duration.ofSeconds(5) -> 5.seconds
  • e.g. duration.toMillis() -> duration.inWholeMilliseconds
  • See also: https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.time/-duration/
• java.net.URI -> com.varabyte.kotter.platform.net.Uri
  • You probably aren't using this.
• java.util.concurrent.locks.ReentrantReadWriteLock.write -> com.varabyte.kotter.platform.concurrent.locks.ReentrantReadWriteLock.write
  • You probably aren't using this.

---

I still have a fair bit of busy work to do, so a stable release may take a few more days. But, at this point, the snapshots should be a really solid representation of the final release. I'm expecting mostly docs and bug fixes from here on out.

Questions? Feedback? If I'm awake, you can usually catch me on my Discord server. My timezone is US Pacific Time (UTC-8)

[bitspittle]

Kotlin 1.1.0-rc1 has been released!

Release notes here

I'm really hoping I can turn this as is into 1.1.0 in a week or two. However, I'm nervous because I'm just one guy. Despite all my pounding on it, it's easy to feel paranoid about missing something.

If anyone in here wouldn't mind spending a few minutes even just trying to set up a quick dummy project, any confirmation that this is working for others would help me immensely.

Thank you all very much for your interest in Kotter. Getting Kotter+Native to work was an interesting journey. I hope the feature ultimately helps even one person get a Kotter app out when they might not have otherwise.

[bitspittle]

Has anyone had a chance to try out Kotter 1.1.0-rc1?

I've been using it myself non-stop and haven't had any issues, so I may just put out 1.1.0 in the next day or two.

However, if anyone seeing this message could commit giving it a try, I'd be happy to wait a few more days, to make sure I don't send out a 1.1.0 that's fundamentally busted in some way I'm just not running into in my own environment.

[jamesward]

This is awesome! I'm giving it a try and hitting a strange runtime exception. Code:

fun main() = session {
    val rpc = BarsRPC(Config.barsUrl)

    var loaded by liveVarOf(false)

    var bars by liveVarOf<List<Bar>>(emptyList())

    run {
        section {
            textLine("Connecting to: ${Config.barsUrl}")

            if (loaded) {
                textLine("loaded")
                if (bars.isEmpty()) {
                    textLine("No Bars")
                } else {
                    textLine("Bars:")
                    bars.forEach {
                        textLine("  ${it.name}")
                    }
                }
            }
        }.runUntilKeyPressed(Keys.ESC) {
            bars = rpc.fetchBars()
            loaded = true
        }
    }
}

Getting:

Connecting to: https://kotlinbars.jamesward.com/api/bars
Uncaught Kotlin exception: kotlin.ArithmeticException
    at 0   tui.kexe                            0x441a66           kfun:kotlin.Throwable#<init>(){} + 86 
    at 1   tui.kexe                            0x43af7c           kfun:kotlin.Exception#<init>(){} + 76 
    at 2   tui.kexe                            0x43b16c           kfun:kotlin.RuntimeException#<init>(){} + 76 
    at 3   tui.kexe                            0x43bd2c           kfun:kotlin.ArithmeticException#<init>(){} + 76 
    at 4   tui.kexe                            0x46e134           ThrowArithmeticException + 116 
    at 5   tui.kexe                            0x8e4ca0           kfun:com.varabyte.kotter.runtime.internal.text#numLines__at__kotlin.collections.List<com.varabyte.kotter.runtime.internal.TerminalCommand>(kotlin.Int){}kotlin.Int + 640 
    at 6   tui.kexe                            0x8c90f7           kfun:com.varabyte.kotter.runtime.Section.$renderOnceAsync$lambda$3COROUTINE$17.invokeSuspend#internal + 2471 
    at 7   tui.kexe                            0x4457e4           kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} + 916 
    at 8   tui.kexe                            0x7009e3           kfun:kotlinx.coroutines.DispatchedTask#run(){} + 3571 
    at 9   tui.kexe                            0x72a9e3           kfun:kotlinx.coroutines.MultiWorkerDispatcher.$workerRunLoop$lambda$1COROUTINE$225.invokeSuspend#internal + 1027 
    at 10  tui.kexe                            0x4457e4           kfun:kotlin.coroutines.native.internal.BaseContinuationImpl#resumeWith(kotlin.Result<kotlin.Any?>){} + 916 
    at 11  tui.kexe                            0x7009e3           kfun:kotlinx.coroutines.DispatchedTask#run(){} + 3571 
    at 12  tui.kexe                            0x687ab1           kfun:kotlinx.coroutines.EventLoopImplBase#processNextEvent(){}kotlin.Long + 1569 
    at 13  tui.kexe                            0x725244           kfun:kotlinx.coroutines.BlockingCoroutine.joinBlocking#internal + 660 
    at 14  tui.kexe                            0x72490d           kfun:kotlinx.coroutines#runBlocking(kotlin.coroutines.CoroutineContext;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>){0§<kotlin.Any?>}0:0 + 2125 
    at 15  tui.kexe                            0x724b66           kfun:kotlinx.coroutines#runBlocking$default(kotlin.coroutines.CoroutineContext?;kotlin.coroutines.SuspendFunction1<kotlinx.coroutines.CoroutineScope,0:0>;kotlin.Int){0§<kotlin.Any?>}0:0 + 358 
    at 16  tui.kexe                            0x72a09a           kfun:kotlinx.coroutines.MultiWorkerDispatcher.workerRunLoop#internal + 186 
    at 17  tui.kexe                            0x72a49e           kfun:kotlinx.coroutines.MultiWorkerDispatcher.<init>$lambda$0#internal + 62 
    at 18  tui.kexe                            0x72b262           kfun:kotlinx.coroutines.MultiWorkerDispatcher.$<init>$lambda$0$FUNCTION_REFERENCE$72.invoke#internal + 66 
    at 19  tui.kexe                            0x72b342           kfun:kotlinx.coroutines.MultiWorkerDispatcher.$<init>$lambda$0$FUNCTION_REFERENCE$72.$<bridge-UNN>invoke(){}#internal + 66 
    at 20  tui.kexe                            0x454b8a           WorkerLaunchpad + 170 
    at 21  tui.kexe                            0x5cbe30           _ZN6Worker19processQueueElementEb + 4624 
    at 22  tui.kexe                            0x5cab5e           _ZN12_GLOBAL__N_113workerRoutineEPv + 142 
    at 23  libc.so.6                           0x7f467f294fd3     0x0 + 139940757852115 
    at 24  libc.so.6                           0x7f467f31566b     0x0 + 139940758378091 

Any ideas?

Note that I'll add input prompt into this loop.

[bitspittle]

@jamesward Moved this specific issue to #98. Let's move the discussion over there and squash this!

[bitspittle]

OK folks, Kotter 1.1.0 is out. It's always a little terrifying releasing something to mavenCentral.

If you try this and find any problems with Kotlin/Native, please open a new issue. Thanks!!