Support .zip and .tar.gz package formats #2183
Description
A goal for Octopus in 2016 is this idea of Octopus Everywhere. Any application you might want to deploy, any target environment - Octopus should have a great (if opinionated) way to do it. PowerShell in deployments should happen less and less.
If we want to achieve this, something we need to address is packaging apps - NuGet just isn't going to work. We've written previously about the need for a packaging format designed specifically for applications, and it's time to take a leap.
Background
Historically Octopus has required NuGet. This was a pragmatic choice made because of the ecosystem around NuGet (and Octopus being .NET focussed). NuGet provided us with:
- A package format (.nupkg)
- A repository protocol (NuGet feeds)
The NuGet package format isn't particularly valuable; the real value is the repository protocol. Tools like TeamCity, VS Online, MyGet, Artifactory, Nexus and other repositories supporting the NuGet protocol, so it makes sense for Octopus to simply work with them.
However, NuGet also provides some challenges:
- NuGet packages are based on ZIP files, but actually use the Open Packaging Convention format. You can't simply create a .zip and rename it - files have to be referenced in metadata in the package, and there are rules around file naming (URI escaping). It's therefore difficult to create them without specialised tools and hooking them into build processes. Nearly any platform/build system can make a .zip, but not many are used to creating OPC package.
- NuGet isn't designed for applications. Normal NuGet packages follow certain conventions (libraries must go in
lib, other files incontent, etc.) which don't apply to packaging applications. Application packages also should be self contained, and don't use dependencies.
So we first must teach people about creating .nupkgs, and then we have to tell them to ignore lots of parts about what .nupkgs are about. And while the NuGet team and Microsoft won't say "don't use NuPkgs for this", it's not a use case they are particularly interested in either.
Decisions
With ASP.NET 5 switching from MSBuild-based builds to command-line tools, the ways that we plugged in to produce NuGet packages need to be revised. Octopus is also starting to be used to deploy more than just .NET applications. We have a few options:
- We could persist with NuGet, and try and make other platforms use it, building and maintaining OctoPack equivalents
- We could support dozens of different platform-specific packaging and repository formats (NPM, Maven, PIP, Gems, etc.)
- We could support something simple that everyone can understand (.zip and .tar.gz), but keep .nupkg for people who really, really want to use an external package repository.
Proposal
Since Octopus server comes with a built-in package repository, we'll extend this to support:
- .nupkg as we already do
- .zip files
- .tar.gz files
From a metadata point of view, there's only two pieces of metadata we need from every package:
- A name/ID ("MyApp")
- A version ("1.0.0")
So the convention would be to simply name the archive:
<id>.<version>.<ext>
Where:
id = letters, numbers, dashes, underscores, dots
version = any valid [Semver 2](http://semver.org) string
ext = one of the supported extensions (zip, tar.gz)
Since the ID can include dots, the version is assumed to start the first time we encounter a segment with just a number. E.g.,
foo.bar.baz.11.2.3.tar.gz
The version starts at 11 because it is the first numerical segment.
If we ever need to supply further metadata in the future, we could do this based on the contents of the package - either with an Octopus metadata file, or some conventions (e.g., release notes might be in a file called CHANGELOG).
Benefits
Builds from languages other than .NET would be greatly simplified. When building a Node.js app, for example, your Gulp/Grunt tasks would just have to:
- Take all the files you want and zip them
- Push to Octopus:
PUT /package/foo.1.0.0.zip
If we ever added npm as a package repository provider, this would also work transparently since NPM packages are just .tar.gz.
Changes we'd need to make
- Octopus built-in package repository would need to accept and index these package types
- Let's add a simpler HTTP POST API to push new packages
- Calamari would need to know how to extract them
I'd envision that the packages would flow through Octopus/Calamari as-is; we wouldn't try and convert them to/from other formats.
Here's a stab at the regex:
var regex = new Regex(@"
^(?<id>[\w\.]*?)\.
(?<semver>
(?<major>0|[1-9][0-9]*)\.
(?<minor>0|[1-9][0-9]*)
(\.(?<patch>0|[1-9][0-9]*))?
(?<pre>-[\da-z\-]+(?:\.[\da-z-]+)*)?
(?<meta>\+[\da-z\-]+(?:\.[\da-z\-]+)*)?
)
(?<ext>\.tar\.gz|\.tgz|\.zip|\.nupkg)
$",
RegexOptions.IgnoreCase | RegexOptions.Compiled | RegexOptions.Singleline | RegexOptions.IgnorePatternWhitespace);