A secure, modern artefact storage service for teams.
Airfreight is an artefact storage service. It sits in front of a blobstore (S3, Google Cloud Storage, etc.) and provides a abstraction that is useful for an organization with teams that produce and consume artefacts for and from one another.
Crates are the core component of Airfreight. A crate is a collection of related files that are stored together.
A warehouse is a collection of crates. All of the crates in a warehouse are assumed to be different versions of the same thing. Crates in a warehouse have no defined order.
Crates can be tagged with arbitrary values. This could be used for marking when a particular resource has passed a particular point in your CI system. Crates may also be given a human readable version number (e.g. v1.3.2).
When making requests to list all of the items in warehouse the user can specify how they would like the results filtered and ordered. This could be simply in the chronological order in which they were entered, the semantic order of the versions in the warehouse, or even a set of crates which have been marked with a particular tag.
A shipment is a set of crates from across many warehouses that can be grouped together in order to be distributed together. Shipments are the main distribution format between teams. They can persist the filtering and ordering state from above into persistent streams of versions.
teams & users
Teams can own warehouses and publish shipments.
Users can belong to multiple teams and have various levels of permission within those teams.
While the following aren't concrete design decisions in and of themselves, they should help guide the service in a healthy direction.
The entire system should be as append-only as possible. For example, an artefact has already been designated as v1.5 of a product then it should not be possible to change that. Having more than one artefact claim to be version 1.5 would be a nightmare.
The service should never lose data. This principle will mainly be pushed down on to the backing blobstore but the rest of the system should strive to help with this goal. This may be by performing soft deletions or making the system easy to backup and restore. The idempotency principle above complements this by ensuring that we do not lose data through unexpected mutation.
I think that each project should (if time permits) try a piece of technology that the authors don't have experience with and improve the experience with a piece of technology that the authors are familiar with.
To this end, I'd like to try out using Google Cloud (GCP) for as much of this project as possible. I'm very familiar with how AWS operates and I'm interested about what we can learn from GCP. I'd also like to improve the experience of developing a Go application that talks to the Cloud Foundry UAA. Many internal services resort to HTTPS and basic authentication because it's simple and quick. I'd like to make integrating with the UAA as simple and quick while taking advantage of the improved security.