feat: add support for delta package uploads using go-octodiff

[borland]

Overview

Adds support for delta compression to the CLI.

Precursors:

• Creation of the go-octodiff library, ported from our existing C# OctoDiff
• Integration of go-octodiff into the go-octopusdeploy SDK client library for Octopus Deploy

Delta compression is off by default. You can enable it with the --use-delta-compression flag, which has aliases of --delta and -d. For example

octopus package upload --package SomePackage.2.0.0.zip --use-delta-compression

Delta compression works by only transmitting the difference between two files - for example if the Octopus Server already holds version 1.0.0 of a nuget or zip package, and you are uploading 1.2.0 - if the zip contains a lot of the same files, then OctoDiff can avoid having to re-send these over a slow network link.

It is off by default because it is not a good fit for:

• Fast network links, e.g. physical servers connected locally, or cloud servers. The additional Disk I/O and CPU processing required to compute the delta between the old and new files can exceed the cost of simply sending the file
• Packages where there are no files in common between the old and new versions.
• tar.gz files. Zip files compress each contained file individually, so the same file will compress to the same output, enabling OctoDiff to detect this. Tar.Gz files compress all files together in a single unit; the compressed form of a file can change depending on what other files are also in the archive.
• Zip files that only contain other compressed files (e.g. you put a .msi in a zip). This produces the same outcome as tar.gz files.

If you are not sure whether it will help you, please experiment using the CLI using the files and network conditions that you have. The CLI will output messages such as these:

Delta upload for package MyPackage.2.0.0.nupkg. Delta size was 54.7% of full file, saving 200 MiB
Full upload for package MyPackage.2.0.0.nupkg. Delta size was 99% of full file (too large)

You can use an external utility such as time on unix or Measure-Command in powershell to determine whether delta compression will speed things up for you.

Quality

During the creation of go-octodiff, I added a significant number of tests to C# OctoDiff, to capture its behaviour at appropriate levels of detail. These tests were used to scaffold the go implementation, ensuring it produces bit-for-bit identical output across many test cases.

Additionally, a lot of manual testing has been done, over a variety of real-world inputs. I ran the C# and Go implementations side by side and verified they produced bit-for-bit identical output in all cases.

Beta Disclaimer

I have confidence that the implementation in the CLI is of high quality, however, the introduction of any complex feature brings risk, and there is always the slim chance that a bug exists which has not been caught during the development process. As such, please consider this feature in beta until the end of 2024.

Performance

note: Octodiff comprises 3 operations:

• Creating a signature file for a prior version of a package
• Using the signature to build a delta against a new version of the package
• Applying the delta to reconstruct the new version

Of these, Creating the signature and applying the Delta are done by the Octopus Server and are not relevant to the CLI.
Time taken to build delta's for different file sizes on the same PC (Windows 11)

Scenario	C#	go
33 MB nuget package	0.43s	0.31s
108 MB powershell zip	2.3s	2.2s
2.1 GB docker image	25.6s	21.1s

[benPearce1]

Would like to see a callout that delta-compression is not production ready

[eero-dev]

Any chances that this will be implemented? Currently blocking us from switching from the .NET CLI

[borland]

Hi @eero-dev

I've had a brief chat about this internally, and while I can't promise any dates, I am planning to pick this back up again next week if I am able to.

If possible, would you consider creating an issue here requesting octodiff support, and add whatever information you feel comfortable sharing about your use-cases? (e.g. which kinds of files you upload, how big, etc)? This will help us justify the feature going forward, and help me target some tests to better help you. If you can't do this that's OK, I'll still try and move the feature forward if I can.

Thanks, Orion

[borland]

Hi @eero-dev
End of year has been busy so I haven't been able to look at this. In the absence of any feedback I'll keep it at a lower priority and hope to pick up in early 2024

[eero-dev]

Hi @eero-dev End of year has been busy so I haven't been able to look at this. In the absence of any feedback I'll keep it at a lower priority and hope to pick up in early 2024

We are using the CLI action in github to push packages to octopus whenever a new release is created, however the upload takes 10 minutes compared to 1 minute before with .NET CLI
The files total around 300MB, multiple NET component projects in a monorepo

Made issue here: #303

[benPearce1]

Successfully tested this locally with a large file

first package version

❯ octopus package upload -p pkg1.1.0.0.zip --use-delta-compression true
Uploaded package pkg1.1.0.0.zip
    Full upload for package pkg1.1.0.0.zip. No previous versions available
    Timing: Signature 57.88ms, Upload 823.01ms

second package with ~100 kb difference

❯ octopus package upload -p pkg1.2.0.0.zip --use-delta-compression true
Uploaded package pkg1.2.0.0.zip
    Delta upload for package pkg1.2.0.0.zip.
    Delta size was 0.0% of full file, saving 252737468 bytes
    Timing: Signature 669.13ms, Build Delta 1.44s, Upload 493.55ms

[eero-dev]

Any updates on this? github is deprecating node16 actions and our version will stop working and CI time will increase like crazy 😔

https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/

[borland]

Any updates on this? github is deprecating node16 actions and our version will stop working and CI time will increase like crazy

Hi @eero-dev

It's not clear what you're referring to with the node16 action?

I'm guessing you might be using our other OctopusDeploy/install-octopus-cli-action to install the legacy CLI that supports OctoDiff?

That action has been updated to support node 20, but if you're pinning an old version of it in your yaml file you may not be aware of this. Your GitHub action yaml file should be referencing v3 of the action, e.g.

uses: OctopusDeploy/install-octopus-cli-action@v3

Unfortunately I can't provide any further update on adding OctoDiff support to the Go CLI at this time

[eero-dev]

Any updates on this? github is deprecating node16 actions and our version will stop working and CI time will increase like crazy

Hi @eero-dev

It's not clear what you're referring to with the node16 action?

I'm guessing you might be using our other OctopusDeploy/install-octopus-cli-action to install the legacy CLI that supports OctoDiff?

That action has been updated to support node 20, but if you're pinning an old version of it in your yaml file you may not be aware of this. Your GitHub action yaml file should be referencing v3 of the action, e.g.

uses: OctopusDeploy/install-octopus-cli-action@v3

Unfortunately I can't provide any further update on adding OctoDiff support to the Go CLI at this time

Any updates on this? github is deprecating node16 actions and our version will stop working and CI time will increase like crazy

Hi @eero-dev

It's not clear what you're referring to with the node16 action?

I'm guessing you might be using our other OctopusDeploy/install-octopus-cli-action to install the legacy CLI that supports OctoDiff?

That action has been updated to support node 20, but if you're pinning an old version of it in your yaml file you may not be aware of this. Your GitHub action yaml file should be referencing v3 of the action, e.g.

uses: OctopusDeploy/install-octopus-cli-action@v3

Unfortunately I can't provide any further update on adding OctoDiff support to the Go CLI at this time

Yes we are unable to switch over to a newer version as it would install the Go CLI (as stated here https://github.com/OctopusDeploy/install-octopus-cli-action/blob/main/migration-guide.md) and the upload times without octodiff are way too high.

[borland]

@eero-dev ah, my apologies. I missed the fact that v3 installed the new CLI. I’ll ask internally if there are any other options

[danzhik]

Hello everyone. We have just recently updated from old OctoCLI to a new one, and after finally resolving all migration issues we almost come to a conclusion to revert back to Octo CLI. The reason is that a part of our CI/CD pipeline is creating a really weighty (hunderds of MBs) NuGet in GH actions, which we push to our Octopus server instance, and now with the new CLI, because of delta upload functionality being unavailable, uploading of this NuGet to Octopus takes about 10min vs previous 10 seconds in Octo CLI. Any update on when we could expect the parity on this one? Thank you!

[borland]

Hi @eero-dev @danzhik

I've just released version 2.4.0 of the CLI, which adds delta compression. It is off by default, but you can enable it if you specify the --use-delta-compression flag according to the description on this PR.

I hope you find it useful!

[danzhik]

Great news @borland, thanks a lot! I hope to test it soon, but it seems like releases.json file still has v2.2.0 stated as the latest available, thus it's not possible to install the new version of cli using a GH action☹️

[borland]

Thanks @danzhik. I will look into releases.json sometime this week, hopefully tomorrow

[borland]

Hi @danzhik. With an assist from @IsaacCalligeros95, the releases.json should now contain the 2.4.0 version, enabling you to pull it via install-octopus-cli-action. Let me know if there are any further issues, thanks!

[danzhik]

Hi @borland, that's wonderful, thanks a lot! I've been really looking forward to it since v2.3.0. After a sample GH actions run, I'm happy to report that feature works as intended, huge thanks for a prompt turnover on this one! Not sure if it matches the performance of the old one for 100%, but even if not, the discrepancy is really minor and in our case can be neglected. Again, thanks for the efforts!