gonda uses the following components:
- A stateless http web server, which serves user requests for down- and uploads.
- A SQL database storing
Package,PackageVersionandChanneldata, as well as user information, and data about the state of the system. - A file storage backend, typically cloud blob storage, to hold the package files.
- A job queue and one or more workers that process queued jobs.
gonda treats packages and channels as independent. Packages and PackageVersions exist on their own, and may or may not be associated to any Channel. What is available in a given channel is defined through ChannelBindings, which conntect PackageVersions to Channel. This scheme has the advantage that we can easily change the content of a Channel by adjusting the bindings, without actually moving files around. It also means that one package can be available through multiple channels, but we still only need to keep one physical copy of the file.
New package versions are created by users through the upload of a conda package file (.conda or .bz2). Package versions can only be created in this way for packages that have already been registered in the server.
File uploads require postprocessing for parsing and validation of the uploaded file. Therefore, the upload process is split into a short synchronous portion, and a subsequent asynchronous processing step. The folowing diagram illustrates the process.
- The user sends a mime post request including the form file to be uploaded to the server.
- The server performs synchronous actions:
- saves the file to
_upload/{temporary filename}in the storage backend - creates a new
Uploadrecord with statuspendingin the DB. - Queues a job to process the
Uploadrecord. - Returns an URL to the user that can be polled for the status of the
Upload.
- saves the file to
- Asynchronous processing:
- the temporary file is extracted and its meta data is read
- It is asserted that the
Packagefor this file exists - It is asserted that the
PackageVersionfor this file does not exist yet
TBD