Permalink
Fetching contributors…
Cannot retrieve contributors at this time
52 lines (39 sloc) 3.52 KB

Automatic Release Creation for External Feeds

To support Automatic Release Creation for external feeds Octopus needs to know about new package versions that are pushed. While some feeds may support webhooks, such support is going to be varied in extent or not available at all. Since Octopus already has access to query the feeds, the simplest approach is to have Octopus query them periodically to determine package changes.

The question is then, what are the circumstances under which Octopus can say, "a new package is available to be turned into a release"? The answer is simple in the case of the Built-In feed since this notification follows a push-model. By definition since the feed is internal to Octopus, when a new package arrives then it is not one that Octopus could have known about and so must be new and considered for deployment.

With external packages however we can't know when a new package is pushed. All that we can do is get the list of packages currently available and compare with the packages that we knew were available at a previous point in time. The packages can be considered to be "new" if after the second request, packages appear in the list that were not present in the initial list. This feature requires a minimum of 2 requests to make such a determination. It is not simple enough to use 1 request and compare with releases already created since it may be the case that a release was created for that package but was subsequently removed. While a similar situation may occur with polling, at least the window for such an event will be constrained by the polling frequency.

Steps

Feed can be configured to be polled package X seconds.

Feed Settings

Every 20 seconds:

  • Check all projects for packages currently in use (un-frozen deployments)
  • Group by feed
  • Filter those with sync disabled
  • Pass the package list to the feed-specific sync process (some feeds might be optimal for multiple packages).

For each pacakage

  • Update with the current list of versions.
  • Some feeds may need to pull all versions, others may be able to optimise (e.g. page until same packages returned).

On server startup this process needs to take place once before polling starts to record the "initial" set of versions available to be compared at the laster poll. We dont make any claims to be able to created releases for packages pushed while server was offline. Given this property it may be simpler to store in memory rather than DB.

Back of envelope usage

  • Assuming on average a version is XX.XX.XXX = 9 chars. 16 bytes a char = approx 144 bytes per version.
  • Assuming 10 package builds per work day = 2600 versions per year = approx 0.3MB per year
  • Assuming 5 different Projects thats 1.5MB per year.

Scenarios

  • User creates release for 1.0.0

    1. v1.0.1 pushed: ARC Triggers
    2. v0.5.0 Pushed: ARC Triggers??
  • User has two channels with the following rules

    • [1.0, 2.0) - Master
    • [2.0,] - Dev
    1. v2.1 pushed: ARC Triggers Dev
    2. v1.5 pushed: ARC Triggers DEV
    3. v0.1 pushed: No Trigger
  • User has two channels with the following rules

    • null - Master
    • ^alpha.*$ - Hotfix
    1. v2.1 pushed: ARC Triggers Master
    2. v2.1-alpha.2 pushed: ARC Triggers Hotfix & Master??

Not just us

If you need a further vote of confidence, AWS Code pipeline appears to also operate on a polling mechanism. AWS Code Pipeline