Support .zip and .tar.gz package formats #2183

Closed
PaulStovell opened this Issue Nov 25, 2015 · 9 comments

Comments

Projects
None yet
5 participants
@PaulStovell
Member

PaulStovell commented Nov 25, 2015

A goal for Octopus in 2016 is this idea of Octopus Everywhere. Any application you might want to deploy, any target environment - Octopus should have a great (if opinionated) way to do it. PowerShell in deployments should happen less and less.

If we want to achieve this, something we need to address is packaging apps - NuGet just isn't going to work. We've written previously about the need for a packaging format designed specifically for applications, and it's time to take a leap.

Background

Historically Octopus has required NuGet. This was a pragmatic choice made because of the ecosystem around NuGet (and Octopus being .NET focussed). NuGet provided us with:

  • A package format (.nupkg)
  • A repository protocol (NuGet feeds)

The NuGet package format isn't particularly valuable; the real value is the repository protocol. Tools like TeamCity, VS Online, MyGet, Artifactory, Nexus and other repositories supporting the NuGet protocol, so it makes sense for Octopus to simply work with them.

However, NuGet also provides some challenges:

  • NuGet packages are based on ZIP files, but actually use the Open Packaging Convention format. You can't simply create a .zip and rename it - files have to be referenced in metadata in the package, and there are rules around file naming (URI escaping). It's therefore difficult to create them without specialised tools and hooking them into build processes. Nearly any platform/build system can make a .zip, but not many are used to creating OPC package.
  • NuGet isn't designed for applications. Normal NuGet packages follow certain conventions (libraries must go in lib, other files in content, etc.) which don't apply to packaging applications. Application packages also should be self contained, and don't use dependencies.

So we first must teach people about creating .nupkgs, and then we have to tell them to ignore lots of parts about what .nupkgs are about. And while the NuGet team and Microsoft won't say "don't use NuPkgs for this", it's not a use case they are particularly interested in either.

Decisions

With ASP.NET 5 switching from MSBuild-based builds to command-line tools, the ways that we plugged in to produce NuGet packages need to be revised. Octopus is also starting to be used to deploy more than just .NET applications. We have a few options:

  1. We could persist with NuGet, and try and make other platforms use it, building and maintaining OctoPack equivalents
  2. We could support dozens of different platform-specific packaging and repository formats (NPM, Maven, PIP, Gems, etc.)
  3. We could support something simple that everyone can understand (.zip and .tar.gz), but keep .nupkg for people who really, really want to use an external package repository.

Proposal

Since Octopus server comes with a built-in package repository, we'll extend this to support:

  • .nupkg as we already do
  • .zip files
  • .tar.gz files

From a metadata point of view, there's only two pieces of metadata we need from every package:

  • A name/ID ("MyApp")
  • A version ("1.0.0")

So the convention would be to simply name the archive:

<id>.<version>.<ext>

Where:

id = letters, numbers, dashes, underscores, dots
version = any valid [Semver 2](http://semver.org) string
ext = one of the supported extensions (zip, tar.gz)

Since the ID can include dots, the version is assumed to start the first time we encounter a segment with just a number. E.g.,

foo.bar.baz.11.2.3.tar.gz

The version starts at 11 because it is the first numerical segment.

If we ever need to supply further metadata in the future, we could do this based on the contents of the package - either with an Octopus metadata file, or some conventions (e.g., release notes might be in a file called CHANGELOG).

Benefits

Builds from languages other than .NET would be greatly simplified. When building a Node.js app, for example, your Gulp/Grunt tasks would just have to:

  • Take all the files you want and zip them
  • Push to Octopus: PUT /package/foo.1.0.0.zip

If we ever added npm as a package repository provider, this would also work transparently since NPM packages are just .tar.gz.

Changes we'd need to make

  • Octopus built-in package repository would need to accept and index these package types
  • Let's add a simpler HTTP POST API to push new packages
  • Calamari would need to know how to extract them

I'd envision that the packages would flow through Octopus/Calamari as-is; we wouldn't try and convert them to/from other formats.

Here's a stab at the regex:

var regex = new Regex(@"
    ^(?<id>[\w\.]*?)\.
    (?<semver>
        (?<major>0|[1-9][0-9]*)\.
        (?<minor>0|[1-9][0-9]*)
        (\.(?<patch>0|[1-9][0-9]*))?
        (?<pre>-[\da-z\-]+(?:\.[\da-z-]+)*)?
        (?<meta>\+[\da-z\-]+(?:\.[\da-z\-]+)*)?
    )
    (?<ext>\.tar\.gz|\.tgz|\.zip|\.nupkg)
    $",
    RegexOptions.IgnoreCase | RegexOptions.Compiled | RegexOptions.Singleline | RegexOptions.IgnorePatternWhitespace);
@zentron

This comment has been minimized.

Show comment
Hide comment
@zentron

zentron Nov 26, 2015

A great extension to this would be to allow users to use existing conventions like package.json to specify some of other metadata required. This can be pulled out of the package on demand.

zentron commented Nov 26, 2015

A great extension to this would be to allow users to use existing conventions like package.json to specify some of other metadata required. This can be pulled out of the package on demand.

@vanessalove

This comment has been minimized.

Show comment
Hide comment
@vanessalove

vanessalove Nov 26, 2015

Contributor

To consider: extending our SemVer use.

Contributor

vanessalove commented Nov 26, 2015

To consider: extending our SemVer use.

@michaelnoonan

This comment has been minimized.

Show comment
Hide comment
@michaelnoonan

michaelnoonan Nov 26, 2015

Contributor

Brilliant concept from my point of view. I also agree that during this effort we should try to support SemVer 2 where possible - primarily escaping the 20 character limit imposed on the SpecialVersion string by NuGet.

Contributor

michaelnoonan commented Nov 26, 2015

Brilliant concept from my point of view. I also agree that during this effort we should try to support SemVer 2 where possible - primarily escaping the 20 character limit imposed on the SpecialVersion string by NuGet.

@michaelnoonan

This comment has been minimized.

Show comment
Hide comment
@michaelnoonan

michaelnoonan Nov 26, 2015

Contributor

I also agree that we avoid transformation into some intermediary/common package format. Allowing packages to pass-through as-is will be less complicated, and we can cater for that at the point of extraction. I think people will encounter less surprises using this approach.

Contributor

michaelnoonan commented Nov 26, 2015

I also agree that we avoid transformation into some intermediary/common package format. Allowing packages to pass-through as-is will be less complicated, and we can cater for that at the point of extraction. I think people will encounter less surprises using this approach.

@zentron zentron self-assigned this Nov 26, 2015

@zentron

This comment has been minimized.

Show comment
Hide comment
@zentron

zentron Nov 26, 2015

Might need to consider\test what this means for OctoDiff

zentron commented Nov 26, 2015

Might need to consider\test what this means for OctoDiff

@zentron

This comment has been minimized.

Show comment
Hide comment

@zentron zentron referenced this issue in OctopusDeploy/Calamari Dec 2, 2015

Closed

Feature nonnupkg #74

@michaelnoonan michaelnoonan added this to the 3.3.0 milestone Dec 21, 2015

@zentron

This comment has been minimized.

Show comment
Hide comment
@zentron

zentron Jan 4, 2016

Octopus Server
  • Support .tar.gz, .zip, etc packages in package store to be loaded and removed on start-up
  • Support uploading non-NuGet packages via API
  • Support uploading non-NuGet packages via Client
  • Update front end UI to remove explicit references to NuGet packages and instead just mention Packages
Calamari
  • Calamari can aquire non-NuGet packages (from local repo only. Currently no support for non NuGet repositories)
  • Calamari can unpack and deploy non-NuGet packages
Node
  • Produce npm octpus pack library for users
  • Produce gulp octpus pack library wrapper
  • Produce grunt octpus pack library wrapper
  • Put into publicly accessible repository
Tests
  • E2E Upload non NuGet package
  • E2E Deploy non NuGet package
  • E2E Package Libary updated on startup from filesystem
  • Calamari test date preservation during extraction (Once new version of SharpCompress is available this should work... Not a blocking task for this feature)
Documentation

zentron commented Jan 4, 2016

Octopus Server
  • Support .tar.gz, .zip, etc packages in package store to be loaded and removed on start-up
  • Support uploading non-NuGet packages via API
  • Support uploading non-NuGet packages via Client
  • Update front end UI to remove explicit references to NuGet packages and instead just mention Packages
Calamari
  • Calamari can aquire non-NuGet packages (from local repo only. Currently no support for non NuGet repositories)
  • Calamari can unpack and deploy non-NuGet packages
Node
  • Produce npm octpus pack library for users
  • Produce gulp octpus pack library wrapper
  • Produce grunt octpus pack library wrapper
  • Put into publicly accessible repository
Tests
  • E2E Upload non NuGet package
  • E2E Deploy non NuGet package
  • E2E Package Libary updated on startup from filesystem
  • Calamari test date preservation during extraction (Once new version of SharpCompress is available this should work... Not a blocking task for this feature)
Documentation
@klh

This comment has been minimized.

Show comment
Hide comment
@klh

klh Jan 26, 2017

while doing this remember to add support for specifying compression levels, with uncompressed / stream being specifically interesting to speed up file deploys / packaging when having large files / many files

klh commented Jan 26, 2017

while doing this remember to add support for specifying compression levels, with uncompressed / stream being specifically interesting to speed up file deploys / packaging when having large files / many files

@zentron

This comment has been minimized.

Show comment
Hide comment
@zentron

zentron Jan 27, 2017

@klh this ticket has already been closed for some time as support for zips & tars has been in Octopus Deploy since 3.3.
If you are experiencing any issues with your deployments that include zip files we would encourage you to contact our support team with details so that we can provide assistance and, where necessary, triage for action.

zentron commented Jan 27, 2017

@klh this ticket has already been closed for some time as support for zips & tars has been in Octopus Deploy since 3.3.
If you are experiencing any issues with your deployments that include zip files we would encourage you to contact our support team with details so that we can provide assistance and, where necessary, triage for action.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment