New release package architecture proposal

[caesay]

Prelude

The structure of our full release and delta packages is, by design, similar enough to Squirrel packages that users of Squirrel can drop in Velopack packages and have them update seamlessly.

Making this change to our package architecture needs to be done very carefully, and may need to initially be an opt-in feature. Later down the line, (eg. v2) we can make this new format on-by-default. This means users who are migrating to Velopack from Squirrel post v2 will be able to out out and generate legacy packages for the purposes of migrating to Velopack, and then use new package formats from then on. We can also limit RELEASES generation to legacy mode.

Proposal

The new proposed internal format is:

YourAppId-1.0.1-win-full.vpkg
├── splashimage
├── manifest.json
└── application.tar

The tar command ships by default on all our target operating systems. By using tar to store application files, we get automatic internal symlink handling (see #52), as well as determinism out of the box. Also, it simplifies our delta process - because we can now just perform a zstd patch (already implemented) of the tar bundle instead of our (now complex) delta create/reconstruction algorithms.

The resulting package could then compressed using zip/deflate as we currently do. Alternatively, for a possibly faster algorithm, we could compress the internal .tar with Zstandard, and then zip the package using STORE instead of DEFLATE. It's important we can get at that manifest quickly and easily, so compressing the entire package with zstd is not viable.

Benefits

• Smaller deltas (larger dictionary)
• Much more resilient deltas, because both tar and zstd are established and stable
• Much faster delta reconstruction during updates since it only requires us to patch one tar file
• Remove complex zip handling logic related to symlinks and determinism
• Remove complex delta handling in C#, and avoid the need to implement it for Rust
• Deprecate the XML nuspec, so can avoid including that (large) parser in our Rust binaries

[Daniel-Svensson]

Do you envision that the application/tar file could be reused between different packIds/channels or would it be one copy per channel?

Background:
We are currently using clickonce where we deploy the same files in ~5 internal + 5 public environments (which might be mapped to channels). (Each environment in turns have a number of tenant specific manifest files, over 100 for "production", in order to allow side by side install and allow the customer get their name in the start menu)

Releases are generally rolled out/promoted gradually. And all files are "identical" except the "manifest files" which contains the application name/identity similar to pack.

I think we might be able to switch to velopack and change to deploying one fat "fullversion" +delta per environment/channel but it would be nice (not required) if the application package could be reused.

[caesay]

The above proposal does not make what you suggest possible because the manifest and application.tar are still combined as a single file inside a STORE'd zip container. For sharing application.tar to work, the manifest and application files would need to be shipped separately.

It's an interesting idea, but I don't know how feasible it is given the current Velopack design goals. The "perfect" implementation of updates in my opinion would deduplicate packages on a per-file basis, and during updates UpdateManager would just ask the server for exactly the files it needs to perform the update. If it's not done this way, even a single small file being changed (eg. company logo or name in your case) would render the application.tar totally un-reusable.

A lot of people today are publishing packages by hand to storage services that would make this kind of structure a nightmare, because you lose the "one file one update" principle.

I would be happy to consider more concrete proposals of how we could make files reusable between channels without losing the publishing ease for self-hosters / those using cloud storage.