Mini-server for Third-Party Library Repositories

[radeusgd]

Summary

While we will be using some cloud infrastructure for hosting our library repositories, we want also a way for users to create their own third-party repositories.

For now this can be a very simple webserver which just serves files from a directory.

This task does not include the service responsible for uploading - this will be a separate task.

Value

• It will be easier for clients to set-up custom/private library repositories.
• We will have a simple tool for testing the repository-related logic locally, without having to rely on the cloud infrastructure that may be developed later on.

Specification

• Clarify and document the directory structure of the library repository and editions storage.
  • Clarify what metadata is stored and where.
  • It should indicate whether the library has been yanked.
  • Clarify the archive format.
  • Clarify the manifest format (which will list available archives and maybe dependencies).
  • Clarify if edition files should be stored compressed (which may be useful as they will be relatively big and will be downloaded very often but should compress quite well).
    • Check if we can just have them compressed in transport and if we need any additional logic to support such a solution.
• Implement the serializers for the manifests with Circe.
• Implement a simple CLI which will launch a webserver and host the repository.
  • It should be lightweight.
  • It should allow to set basic configuration like the location of the repository or the port to listen on.
  • For now it will only support HTTP as the primary role of this server is debugging. Moreover HTTPS can always be implemented by listening on a local network and a proxy. Proper HTTPS support may be done later.
  • Initially it should probably be in Node.JS and be just a simple server with no special logic.

Acceptance Criteria & Test Cases

• The above specification has been completed.
• The directory structure design has been accepted.

[radeusgd]

@MichaelMauderer @iamrecursion I'd like to discuss whether we should HTTPS or HTTP for the custom repository be default.

Pros for HTTP:

• It is easier to set-up.
• If I'm not mistaken, if a company wants to expose the repository outside, they can host the service in an internal network and set-up a HTTPS proxy for the outside world.
  Cons for HTTP:
• It is insecure so it is not the best default to choose.

[radeusgd]

We also need to decide how these custom repositories shall be used in projects.

Should the edition configuration (#1769) include any external repositories that should be used? If yes, what should be the precedence in case that a single library is available in multiple repositories?

Or alternatively maybe instead of specifying a repository for the whole edition, maybe specific library entries in the custom edition could include a full-repository URL instead of just a name indicating that they should be sourced from another repo? With that it would be great to also support URLs on the local filesystem - that could simplify development.

Or maybe we should support both?

[MichaelMauderer]

We also need to decide how these custom repositories shall be used in projects.

Should the edition configuration (#1766) include any external repositories that should be used? If yes, what should be the precedence in case that a single library is available in multiple repositories?

Or alternatively maybe instead of specifying a repository for the whole edition, maybe specific library entries in the custom edition could include a full-repository URL instead of just a name indicating that they should be sourced from another repo? With that it would be great to also support URLs on the local filesystem - that could simplify development.

Or maybe we should support both?

I believe when talking with @wdanilo we were thinking of having an additional repositories configuration in the project. This would indicate repositories to use, and the order in there would define precedence for package resolution. So if you import foo edition X it will check "is foo edition X in repo 1, is foo edition X in repo 2, ...".

[radeusgd]

What do you mean by import foo edition X? My understanding is that we configure the edition per project, not per import.

And since the edition is the only mapping of libraries to versions and our editions will only list what is available in our repository, just providing custom repositories isn't going to be enough - it will also require specifying the custom provided libraries. In my opinion it would just make most sense to specify a custom edition and such an edition could specify these additional repositories in its configuration. I don't see how it could work to configure repositories in the project configuration itself.

[MichaelMauderer]

Hm. I was thinking that the custom repositories also has a configuration for editions. So, it contains the "edition" file and that again contains this repositories list of libraries for this edition.

If a project looks for a library it will already need to know which edition the project is in, so before we know the version of the library we look for, we look for "lib foo / edition bar". Which exact version that is, will be resolved by the repositories' edition file.

I realize that this means that a repository could "patch" some libraries by overriding another repositories libraries, but this could actually be a useful feature.

[radeusgd]

After discussion we settled for:

• at the top include a mapping alias -> repository URL,
• each pacakge refers from which repository (alias) it should be get,
• the main repository is not treated differently, it is just defined in the main edition like others.