[Idea] Reworking architecture of fxlauncher

[mainrs]

As discussed in #15, reworking the architecture of fxlauncher would allow to bring new features to it such as:

1. Enabling updates for the launcher itself
2. Version control of the application and rollback function
3. Downloading the appropriate JRE (if not present on the system)

All of those changes would result in a small native executable (exe, app, etc.) that would download all needed dependencies on the first launch. The native stub wouldn't be bigger than 1 MB for sure, meaning that the launcher would function as a 'web installer'.

[edvin]

I agree, the next step would definitely be to investigate native stubs. I tried creating a small Swift based executable for MacOS X. It actually weighed in at about 5MB for a plain hello world app, and the Objective C version was actually about the same size. For Windows I think it would be a lot slimmer.

[mainrs]

Wow, 5 MB for a Hello World app is huge. What is causing that much disk space? Looks quite inefficient to me (in a language design aspect).

I agree, the next step would definitely be to investigate native stubs.

Looks like POSIX filesystem allows the replacement of files at runtime, meaning that the swift stub could replace itself before doing other things.
The Windows stub would need to get updated by the jar launcher, because Windows' NTFS filesystem doesn't allow any operations on an open file by another process (which means that replacing it at runtime is impossible).
Looks like we either

• Do it in an unified way and update the native stubs through the jar launcher (by just downloading the native stub as an dependency) and implement certain "Strategies" that are used to replace the right files at the right locations or
• Let the native stub update itself on POSIX systems and use the jar launcher to update the native stub only on Windows systems.

[miho]

The size comes from static compilation to make the binary independent from additional dynamic library. I would recommend golang, however. Swift is a macOS only solution that does not really come with any benefits. Go binaries also tend to be a little bit smaller. My go Java launcher is 2-4MB in size. It works on all operating systems, since go can cross-compile. That is, the Windows and macOS versions can be created on a linux machine.

[mainrs]

Go binaries also tend to be a little bit smaller. My go Java launcher is 2-4MB in size. It works on all operating systems, since go can cross-compile. That is, the Windows and macOS versions can be created on a linux machine.

Sounds nice to be honest. I would consider this as an option. Bundling it as an OS X application bundle shouldn't be hard too, since it is basically just a folder. Would like to hear your opinion on this @edvin

[edvin]

I think the choice of programming language for the native stubs ultimately doesn't matter. The reason I mentioned Switft is that it has become the new standard for macOS, and the alternative Objective-C actually creates even larger binaries.

[miho]

I think the choice of programming language for the native stubs ultimately doesn't matter.

Yes, absolutely true. This is not a Go vs. Swift argument. Language features are irrelevant here.

The point is that Go creates static executables for every OS. I think, it is really overkill that we need Windows, Linux and OS X/macOS just for compiling a small launcher. Oh, and for at least Windows and Linux you need x86 and x64 versions which gives us a total of 5 launchers.

[mainrs]

I didn't know that Go can be compiled into several binaries for different platforms. Sounds good to me. I would stick with it. Shouldn't be a problem to write a launcher that can download some stuff.

[edvin]

Absolutely. Does Go provide either access to the underlying UI toolkit or a UI toolkit of it's own? The native launcher would need to provide a minimal progress UI while it downloads the JRE and a the fxlauncher.jar, then the Java code can take over with a JavaFX UI from there.

[mainrs]

Does Go provide either access to the underlying UI toolkit or a UI toolkit of it's own?

https://github.com/google/gxui Looks good :) Alternatively, there is another library for this. It is cross-platform too as far as I could see.

[edvin]

"Unfortunately due to a shortage of hours in a day, GXUI is no longer maintained."

If it's not part of Go, I guess the executable will be bigger. If it gets big, maybe the core + javafx modules of Java9 compiled via LLVM or something is just as good a bet?

[edvin]

Or maybe Avian is an even better alternative? :)

[edvin]

Avian can utilize SWT, so it would create native looking binaries for all platforms. The launcher would probably just be about 1MB in size. Sounds almost too good to be true :)

[mainrs]

Ye, so it would mean that we could write the "native stub" in pure Java by utilizing the small JVM provided by Avian. It would then download a normal Java distribution and install it. Am I right?

[edvin]

Yes, that's right. Of course, since we can already create native installers bundled with the JRE, the only thing this buys us is a smaller executable, which isn't all that important these days.

[mainrs]

That's true but I still like the original idea we had. Sure, bundling the JRE with the launcher would be, in the end, probably easier.
For people that already have java installed, bundling a native JRE is unnecessary.

[miho]

I really like Go for small native stuff without UI. But UI is not one of Go's strengths. If the UI only has to be minimal then https://github.com/andlabs/ui might work. For a JDK9 approach that uses JavaFX for the UI we would end up with a binary that is about 40-50MB. That isn't bad either if you need a UI.

[mainrs]

The problem with Avian is that it does not include UI code. So it is quite inconvenient to use it without a method to show feedback to the user. I took a look at the library you mentioned @miho. We would just need a ProgressBar, Buttons (for acceptin the JRE license) and that's it. Maybe some Labels, but those are included too. I think that the library should be enough for our purpose.

[edvin]

As I mentioned, Avian can encapsulate SWT, which is the UI toolkit used in Eclipse. It gives access to the native UI elements on each platform, and an executable with UI is only about 1MB :)

[mainrs]

As I mentioned, Avian can encapsulate SWT, which is the UI toolkit used in Eclipse. It gives access to the native UI elements on each platform, and an executable with UI is only about 1MB :)

I am really sorry. My brain just forgot about it D:
Would prefer Avian over Go to be honest. Probably because I can write Java better than Go or C/++

[edvin]

Haha :) It would be interesting to try to create a small app that basically downloads a file with a progress indicator and make a binary for each platform to see if it works adequately. I'm however tied up with TornadoFX and some stuff at work for a couple of weeks, but unless any of you guys want to give it a shot, I'll try it out after that, OK?

[mainrs]

Sure, can try it out. The only thing that looks like pain in **** is the part where you embed your jar into avian. I can't find any maven plugin or gradle plugin that does it automatically. But for a test something like that shouldn't be needed. Will write something later on :)

[edvin]

Sweet. I guess you can just use a shell script or a batch file depending on your platform for now, and we'll figure out a more streamlined way to handle this if this turns out to be a viable path.

[mainrs]

Here you go. It will probably only compile on OS X and Linux without any tweaks. If we decide to use Avian, we definitely need good deploy scripts from the beginning on 😄
Here is a short video of the application in action, quite simple.
The executable is around 2 MB in size. I like it so far :)
BTW: The python code is really ugly as hell. Just used python because I never used bash before.

Looks like the app freezes for some reason when I run it outside my IDE (the native stub). It does one iteration of copying and then the output stops on the terminal. No idea why this happens. I even compile it against the boot.jar. Looks like it has to do with the I/O code.
It is definitely an Avian bug. There http/https support is bare bones and not really working.
I also tried to use the openjdk distribution but it didn't work out that well. I would probably stick with the golfing solution. Seems easier then using Avian.

Tried Go for the first time. Looks quite promising: https://github.com/SirWindfield/go-launcher

[mainrs]

Just a small question: Is a UI really needed for the native stub? It's only purpose it to download the jar launcher and start it. That will just take some seconds to do. I personally think that a progress bar in a terminal would be enough for this. The task does not need to be done every launch and just takes under 10 seconds. Just my opinion on it.
Something along the lines of this:
37158 / 100000 [================>_______________________________] 37.16% 1m11s

[edvin]

Well, you need to get a hold of a JRE, and if the laucher can't find a satisfactory one locally (based on requirements from the app manifest) it would have to download one. We could of course still do that with just a shell window though, but I think such an executable might always open a shell/dos window when it starts, and that would look ugly/not very desktop integrated. What do you guys think?

[mainrs]

That is true. If you print something, a shell will pop up. The problem is that there won't be a solution that is cross-platform and easy to maintain/build. The library I found with GO works good but compiling the launcher would mean that we need to compile them on each platform by its own (since cross compiling won't work on it).
A terminal isn't the prettiest solution possible, that's for sure. But I think that it is the sweet spot between compatibility and maintainability. It would be possible to create the binaries using cross compiling, meaning that it doesn't depend on which platform we develop.
Writing own native stubs (in c or swift) would mean a lot more maintenance. It would also open up the problem of specialization. Not everybody has access to OS X and therefore can't help out.

Avian looked good but the protocol support isn't. It does only support http request. So links with https and ftp won't work.

That's my opinion on it. I did some more research and came to this final conclusion:

• Use GO to create native stubs (terminal)
• Use Lua to create them. Lua provides bindings for all major UI frameworks out there. This would mean that we need a different executable/project for each platform/arch.
• Write them on ourselves.

I agree with you. It would not really look desktop integrated. But (as far as I can tell after researching
a bit), there is no easy and clean way to create native stubs for each platform in an easy to maintain way without sacrificing the UI part.

[edvin]

Let's look at an alternative approach. If we make native stubs for each platform with the tools best suited for each platform, and then simply augment these native stubs for each application by injecting the right icon and app manifest url into them.. Then users don't need to build anything, just use a Gradle or Maven plugin to augment the installers that gets put in the build folder. I wonder if that's possible...

[edvin]

Another approach would be to offer a build server infrastructure to build everything in the cloud.

[mainrs]

Another approach would be to offer a build server infrastructure to build everything in the cloud.

Wouldn't that cost something each month? I don't think there are services that offer something like that for free. At least I don't know any service.

Then users don't need to build anything, just use a Gradle or Maven plugin to augment the installers that gets put in the build folder. I wonder if that's possible..

You mean we write the 'core' of them and distribute them. Gradle or Maven will then download them and inject the needed information into the executables later on while the application is building. Am I right? I am no expert at native applications/code but theoretically something like this should be possible. Do you know launch4j? It's an application that creates an exe wrapper around a jar. It basically does the same thing. It has a core exe that executes some code and calls the jar afterwards. So somehow the needed information (VM arguments, name of the jar, additional arguments) need to be injected too. It is probably a exe that is designed to get the needed information and create another exe itself (like a build job). Not sure here though.

[edvin]

Yeah, that's exactly what I mean. I happen to own a hosting company, so I could set up a free service which probably would be more than enough for the foreseeable future. If the build server/config was open source, anybody could set up their own, or build locally if they have the right tools installed.

However, if we can somehow augment the native stubs, the only thing we need to change is the icon and the app manifest. The app manifest would contain all the other information.

I guess launch4j already solved this for Windows :)

[mainrs]

Ye, sadly, launch4j or any other solution (like install4j) cost so much money. It's ridiculous.
We should investigate into augmenting files within native stubs. I found a link to a tool for windows: http://www.heaventools.com/rtconsole-embed-manifest.htm

Seems that it is definitely possible to do somehow.

[edvin]

Great to know :) I will ask some guys at work about this as well.

[edvin]

I talked to a guy at work, and he had some interesting insight.

Let's talk about Mac first.

Let's say we write the launcher as a first class Mac App in Swift. The result of building this is essentially a directory structure of files. Here it would be easy to swap out the icon and include an app.manifest before re-sealing the dmg. The swift app knows how to look for a satisfactory JRE or download it, update the application artifacts mentioned in the manifest and hand over control to the JavaFX app via a normal java launch.

For Windows we can do something similar, but

[edvin]

I talked to a guy at work, and he had some interesting insight.

For Mac: Let's say we write the launcher as a first class Mac App in Swift. The result of building this app is essentially a directory structure of files, ca 5MB. Here it would be easy to swap out the icon and include an app.manifest before re-sealing the dmg. The swift app would know how to look for a satisfactory JRE or download it, update the application artifacts mentioned in the manifest and hand over control to the JavaFX app via a normal java launch.

For Windows we can do something similar (possibly just C++ against the Win32 API to create a really small binary), but package it inside an installer that can also include icon + app manifest. There are some open source windows installers, I think javapackager depends on one of them.

[mainrs]

So instead of writing simple binaries we create installers for each major platform. That would allow us to bundle more than just a simple native stub. Sounds good to me 😄
What about Linux/FreeBSD? Is there some similar solution that we can use?

[edvin]

I'm sure there are similar solutions for Linux as well :)

[mainrs]

I'm sure there are similar solutions for Linux as well :)

What about .rpm files? As far as I know, those are files like the .dmg on OS X.

RPM Package Manager (RPM) (originally Red Hat Package Manager; now a recursive acronym) is a package management system. The name RPM refers to the following: the .rpm file format, files in the .rpm file format, software packaged in such files, and the package manager program itself.

[edvin]

We would probably need to build rpm and deb (Debian/Ubuntu), but I think that's pretty easy to do.

[mainrs]

I can probably implement the C++ app for Windows.
The next question would be on how we manage the files that need to be downloaded. I had an idea but would like to hear your opinion on it.
So basically, each developer bundles its own manifest that points to a URL that holds the app.xm file you already used but for our purpose it will have a small modification.

<application uri="https://sirwindfield.github.io/taskflow/application-support/"
  version="1.5">

  <configuration>
    <cache>cache</cache>
    <launch>de.zerotask.framework.Test</launch>
    <java-version>8u92-b14</java-version>

    <!-- this is only needed if the client should rollback to another version -->
    <rollback>1.4</rollback>
  </configuration>

  <file name="guice.jar" size="123456" checksum="sha256"/>
  <file name="guice.jar" size="123456" checksum="sha256"/>
  <file name="guice.jar" size="123456" checksum="sha256"/>
  <file name="guice.jar" size="123456" checksum="sha256"/>
  <file name="guice.jar" size="123456" checksum="sha256"/>
  <file name="guice.jar" size="123456" checksum="sha256"/>
  <file name="guice.jar" size="123456" checksum="sha256"/>
  <file name="guice.jar" size="123456" checksum="sha256"/>

  <!-- all files marked as launcher will get downloaded by the native stub. If anything changes, it gets re-downloaded -->
  <launcher-file name="launcher.jar" size="45675" checksum="sha256"/>
  <launcher-file name="resources.jar" sze="54304" checksum="sha256"/>
  <launcher-file name="signed.cer" size="465" checksum="sha256"/>

</application>

We would just reuse the already existing app manifest and extend it with a new entry called launcher-file. Those files will only get downloaded by the native stub and will be ignored by the java launcher. The checksum can be used replaced with Adler32 to be honest.

It doesn't make a lot of sense to implement a test application before we discuss the infrastructure of the update process. We need to consider rollback functionality and decide on how we want to structure the repository (or even depend on an Interface so people can implement there own Strategies).
My idea was to have the base url + version + file url. Meaning that guice would resolve to https://sirwindfield.github.io/taskflow/application-support/v1.4/guice.jar because of the rollback. If the rollback gets disabled, it will be https://sirwindfield.github.io/taskflow/application-support/v1.5/guice.jar.
The launcher should then be able to calculate the differences and only download the files that changed from v1.4 to v1.5, meaning that any file that is not in the v1.4 folder on the hard drive will get added from the next available version, here v1.5. This would mean that the user doesn't need to download each file, only the ones that changed. This could be accomplished by calculating the differences at runtime or by calculating them at build time using maven or gradle. Those tools could then create a patch information file containing each changed file. Thus, the launcher would only need to look for that file for all the needed informations.

[mainrs]

Mind if I ask a small question? If we chose the path to write those native stubs ourselves to get the UI going, how would we manage the build process for users? MSI files can only be created on Windows (at least all tools and libraries I found are written for Windows, even the Java dependency WIX). Is it this were you wanted to use build servers?

[edvin]

Build servers are one option for people that don't have access to a Windows environment, but it would be so nice if we could just augment an existing install or assemble the Windows binary on other platforms. We should investigate this further as well.

As for your previous post, that's exactly how I'm envisioning this. The native stub is an optional extra wrapper that can help with installing, downloading the JRE and keeping the installer up to date. If you skip this and distribute the jar, you get the app updates via fxlauncher.jar, but you basically require the user to have a JRE already, and the launcher can not be updated.

If the native stub/installer only takes care of updating the launcher and JRE, we don't have any overlap in functionality between the two installers.

For this native step to be viable however, we have to provide something more than what can already be achieved with fxlauncher in combination with javapackager. A bit smaller download size isn't really revolutionary, but if we can circumvent the requirement of having to build the installers on 3 separate operating systems then we're actually adding some value.

If we can't make that happen I'm not sure how much extra value we would add with this.

[edvin]

Btw, I think it's possible to build windows binaries from linux (and possibly osx) using c++ and mingw32.

[mainrs]

If we can't make that happen I'm not sure how much extra value we would add with this

If we can't, javapackager would be better off by simply including the JRE with it.
I am not that familiar with native code and compiling it. I started to code on windows and had to use mingw32, which was so much pain that I later on gave up on it. That's the time were I switched to Java for development. I can imagine that it is somehow possible. but developers need to make sure that there environment is properly set up to ensure all things are working.

I know that javapackager is using WIX but WIX isn't available on any other platform that windows. I am not sure if javapackager can even build msi files on other OS than windows.

[mainrs]

Btw, what is your opinion concerning Java 9 and its modules? Wouldn't it be a good idea to consider those so we would have an update system capable of updating Java 9 and Java 8 applications?

[edvin]

I'm very interested in Java 9 because of Project Jigsaw. I've played with it a little, and I love the fact that you can bundle a cutdown version of the JRE with only the needed modules. That alone will actually reduce the download size so considerably that it might not even be worth it to create a native launcher :)

I haven't followed it's progress closely the last year, but I think it's feature complete already right? It might be a good idea to understand the module system completely so we can cater for it in a future proof way.

[mainrs]

It is feature complete. There are some test JDK out from oracle. We would probably just need the JavaFX and HTTP module, along with the dependencies. That could reduce the size immensely.

[edvin]

It would be interesting to see how small core + javafx + http (if that's a module?) would be. I looks like my schedule will clear up a bit towards next week so hopefully I can start playing with these ideas a little :)

[mainrs]

Looking forward to that :)
Just a small question along that side: Would it theoretically be possible to provide a bar bones JRE with the launcher that can download the other dependencies needed (modules) like the application dependencies? Would be really nice :D

[mainrs]

If you got Java 9 and any IDE working together, please let me know. I couldn't manage to get something done using either Netbeans, Eclipse or IDEA.

[edvin]

Hmm.. Maybe it's too early yet? I have lots of other features for TornadoFX in my pipe line right now, so I'll finish those and then have a look at Java 9.

[mainrs]

Ye, I am fine with that :) May I ask you something? Is developing JavaFX applications with TornadoFX much easier/faster?

[edvin]

TornadoFX removes almost all of the boiler plate from writing JavaFX applications, and it has lots of abstractions that helps you do intricate stuff in a single line of code. It also has type safe stylesheets and even type safe inline styles. Combined with hot view reloading (and hot CSS reloading) the development cycle is not even comparable :) Have a look at the examples on the project page and join us on slack to get some more insight. There are also some YouTube videos of the framework in action.

https://kotlinlang.slack.com/messages/tornadofx/

[edvin]

Guys, we have lots of good ideas now, and I even have some more, but I'm too busy with TornadoFX and KDBC to start a reboot of this right now, so I'll close the issue. FXLauncher works good like it is, but I'll revisit this issue when it's time to start on FXLauncher 2.0 :) Thanks for all your input.