Updating a jar while the application is still running

[thokuest]

I just changed our application delivery so that getdown is responsible to install the application and keep it up-to-date. Now, I have the following situation:

1. I run the application via getdown. It opens and everything is fine.
2. While the application is running, I open another instance of the application. getdown is now running an update because I changed a jar file (by intention).
3. Back in the previously opened application I issue a command that results in a ClassNotFoundException.

As far as I know this is the expected result if a jar file is being replaced while an application is running. What can be done in order to prevent such situations? I'm on Windows 8.1 if that helps.

I'm still new to getdown, so bear with me if that's a configuration issue or this question has already been asked.

Regards,
-- Thomas

[thokuest]

The more I dive into the details, the more I think it's actually a "bug" in getdown. An updater should either ensure that the app is shut down before an update is applied or it should apply a technique that supports multiple open instances of the application.

What do you think?

[samskivert]

Getdown was never designed for this use case. The "ping a URL to see if there's an update" mode was always intended only for development use and never for production apps. The "versioned" update mode requires that the APP check whether there's an update, and when there is, the APP should write a version.txt file telling Getdown that the new version is available, and then the app should shut itself down and invoke Getdown, Getdown would then update the app to the specified version and launch it.

There is no simple cross platform way that Getdown can coordinate with the app to ensure that it's not already running, so Getdown, in production mode, never updates the app unless the app says that it needs to be updated.

All of this confusion and overwriting files and corruption and blah blah is the result of people using the "development" mode of Getdown for production. It's easier to set up because you don't have to write code in your app to track new versions, but it isn't robust and cannot easily be made robust without coordination from the app. That's why we have the versioned mode in the first place.

[thokuest]

Indeed, I wasn't aware that this mode isn't intended for production use. My bad! I re-read the Design Goals and, keeping Getdown's background in mind, it totally makes sense what you write. So, thanks for the clarification.

-- Thomas

[thokuest]

It's actually not hard to integrate version tracking in an existing app which will then instruct Getdown to run an update. The hardest part is to coordinate a shutdown of all application instances so that an update of the code resources can take place. Lots of our customers are using Citrix, so we have a lot of users running our application from the same host (and directory). I think you can imagine how hard it is in this scenario to run an update.

I experimented a bit and modified Getdown so that it starts the application from a "cache" directory instead of the installation directory. The cache is structured like so:

+---cache
|   +---11
|   |       114bb82aae54994171682133f2bb93aa.jar
|   |       114bb82aae54994171682133f2bb93aa.jar.lastAccessed
|   |       
|   +---76
|   |       766c146b9f2f414a44384506daea64d9.jar
|   |       766c146b9f2f414a44384506daea64d9.jar.lastAccessed
|   |       768a6f66e943575558150093f057c35a.jar
|   |       768a6f66e943575558150093f057c35a.jar.lastAccessed
|   \---f3
|           f36f5fc57600e37610a76147caeaefac.jar
|           f36f5fc57600e37610a76147caeaefac.jar.lastAccessed

So, whenever an update is available new files are installed into the cache and new application instances are started from there. No existing jars are actually replaced. The usage of the jar files is tracked by the .lastAccessed file which should enable Getdown to run a garbage collection periodically, for example weekly, monthly, what ever. Using the cache, the "ping a URL to see if there's an update" mode is working (at least) for me.

As far as I remember, Java Web Start uses a similar mechanism. Microsoft's ClickOnce, to a certain extent, caches application versions as well.

Although I know that this cannot be incorporated into Getdown easily without breaking current functionality (I think of unpacked resources), I want to share my idea in the hope it helps others.

-- Thomas

[samskivert]

On Thu, Sep 8, 2016 at 11:51 AM, Thomas Küstermann <notifications@github.com

wrote:

I experimented a bit and modified Getdown so that it starts the
application from a "cache" directory instead of the installation directory.
The cache is structured like so:

...

Although I know that this cannot be incorporated into Getdown easily
without breaking current functionality (I think of unpacked resources), I
want to share my idea in the hope it helps others.

This is nice and it would be great to avoid problems with updates failing
due to the application still running while updates take place. However,
it's not feasible for the use case we originally designed Getdown for,
which was the deployment and installation of very large MMO games. Our
installations are regularly 500MB-1GB or larger, and even making a copy of
the entire game as a part of applying an otherwise potentially very small
patch, let alone keeping many copies of the game around simultaneously,
would be problematic even in this world of very large hard-drives.

However, it would not be unreasonable to allow an app to opt-into this more
robust behavior. There are many non-game users of Getdown that could have
isolated caches for each update and could benefit from not worry about apps
running while updates took place. We could probably also use this mechanism
to make the non-versioned update process more robust, which would be nice
since many people are content to use that mechanism rather than deal with
the extra work of generating patches for explicit versions.

Indeed, with the benefit of 12 years of hindsight, Getdown could do with a
few design changes. :)

Assuming your modifications simply copy the code + resources into a cache
directory prior to launch (reusing an existing copy if it already exists),
it sounds like that's something that could easily be enabled or disabled
via a getdown.txt configuration option. If you want to add such an option
and submit a pull request, I'd be happy to make the feature available to
everyone (with the caveat that it is not compatible with certain features
like resource unpacking).

-- mdb@samskivert.com

[thokuest]

Sounds great! Before I submit a PR, let's talk about what is to be implemented exactly, so everyone has a clear understanding.

What I have in mind is to populate a cache for all classpath resources (code & resource jars) after Getdown has downloaded all bits and before the application is launched. It will be ensured that each file is contained in the cache only once. The cache won't contain resource jars which are not part of the classpath as well as unpacked resources. Although it's possible to have different "working folders" per application instance, I consider this way too error prone and complex to be implemented in a timely manner. Let's start simple and evolve if necessary.

The (code) cache should only be used if it is activated by a specific configuration property in the getdown.txt file, let's say useCodeCache.

useCodeCache = true

The cache contains the jar files along with .lastAccessed files tracking their last usage. Those tracking files will help a garbage collector to clean up unused files. A cached files retention period is to be configured in getdown.txt as well, say cacheRetentionPeriod. I'm undecided how to configure the value, I'm thinking of defining the number of millis/seconds, or an expression like 7 days.

What do you think, is this sufficient? Please feel free to add more requirements.

[samskivert]

That sounds pretty good to me. I assume the current working directory will be the same (the main application directory, not the cache directory) and only the classpath will be adjusted to reference the jars from the cache directory. This would ensure that apps will otherwise mostly work the same, just without the jar file stomping during update.

I think for cacheRetentionPeriod you could probably just use an integer number of days and maybe call it cacheRetentionDays to reflect the units. I can't imagine needing to prune things more frequently than that given that a new cache directory is only created when the app actually changes, and one would have to do an awful lot of updates on the same day to cause trouble with that minimum granularity.