Registry Draft v2

[f-f]

This pull request introduces a major revision of the "new PureScript Registry draft" that we currently have in the README of the master branch.
I'll call that "Draft v1", while this should be considered the "Draft v2".
Once this work graduates from the "draft" status (eventually after more revisions of this draft), it will be called "Registry v1" - this is important to note because we'll tie the version number to the breaking changes to the Dhall types of the Registry, currently in the v1 folder.

How to review this Pull Request

Before I get into the explanation of what has changed and how, I'd like to detail how I'd like the lifetime of this pull request to look like, and what I'd suggest the review workflow to be:

• since the changeset is admittedly giant, I don't expect anyone to fully look at it, and "reviewing it" in the traditional GitHub meaning is highly discouraged since that'd be a painful experience (of course you can still go ahead if you'd like that).
  Since 90% of the work here consists of rewriting the spec in the README, I'd strongly suggest just having a read at the new draft
• because we are still in draft stage, it's least confusing for everyone if this work gets merged as soon as possible in master, as I'd find highly undesirable to be stuck with an outdated "draft v1" in there and the updated work only shown in this PR.
  However I believe it's useful to have this PR open so we have time to catch obvious oversights - so I'll wait for 3 days, and then merge to master
• I'd like to encourage questions, requests for clarifications, etc in GitHub reviews, but we'll move all of these discussions over to Issues once this is merged (i.e. we should not hold the merge on anything short of a fundamental design flaw)

Notable changes

I'll assume here that you're familiar with the current design, and I'll summarize here a list of things that have changed since then:

• most importantly, the package Manifest storage format changes from Dhall to JSON
• however, we'll still conform the JSON Manifests to Dhall types to ensure they follow the Schema we specify here
• a contract of forward-compatibility has been defined for future versions of the Manifest schema
• it will be required for the package Manifest to be versioned in the repository sources (but we'll generate those for all the Bower packages, so the transition will be smooth)
• we propose the Manifest file to be called purs.json
• we won't store the package Manifests in this repo, and the single source of truth for them will be the package tarballs. However, we'll store a copy of that in a registry-index repository, for easier bulk-access.
• we introduce a Metadata file for every package, to be stored in this repo and to serve as permanent storage for data needed for registry operations. This includes storing the SHA256 of the package tarballs.
• there's no focus anymore on storing the package tarballs as artifacts in GitHub Releases, and instead we'll start off with a S3-like storage maintained by the Package Team
• and at the same time the contract for adding new mirrors of Registry storage has been clarified, both for package managers and for anyone that would like to mirror the registry
• source locations now support a "subdirectory" component, freeing packages from having to be in the root of the repository, and allowing multiple packages from the same repository
• the role of Trustees has been clarified
• an attempt to support nativeDependencies has been made, and subsequently abandoned. See Backend specific dependencies #20 for context, and this commit for the (removed) code that was implementing this.
• the work on the "CI infrastructure" that is supposed to power all the Registry operations has been started. It was initially thought as a single Haskell executable, but for ease of operation and maintenance this has changed into a modular PureScript codebase, to be stored in this repo. You can see it here
• the "Registry API" for package publishers has been specified, and it will consist of "JSON over GitHub issues". The possible interaction with package managers has been specified too.
• the "implementation plan" has been expanded and detailed, see next section for more

Some of these changes have been discussed in issues in this repo, so I am not listing all the reasons for all the changes above (after all it's supposed to be a summary), so feel free to ask for more context and I'll do my best to link to relevant places and otherwise detail that.

Where do we go from here

You can find more details for the implementation plan in the relevant section of the draft.

Quoting from "what is happening now":

1. we're figuring out the last details of the package Manifest, which is the big blocker for proceeding further, since it will be baked into all the tarballs uploaded to the storage.
2. setting up a storage backend maintained by the Packaging Team
3. writing up the CI code to import the Bower packages as described above

(1) and (3) are mostly this PR, (2) is happening but it's blocked by (1).

To summarize: we need to commit to the first version of the Manifest Schema before we move forward with the rest of this.

[hdgarrood]

This is starting to look really nice ❤️

[hdgarrood]

I don't think there's a need to refer to the semver package on npm, is there? The spec tells you how to sort versions unambiguously in all cases except for when the versions have the same major, minor, patch, and prerelease data, but their build metadata differs. However, I think this case should not come up, as the registry should not accept a package submission whose version is the same as an already-uploaded version in all respects except for the build metadata.

[f-f]

The keyword here is "when in doubt". The bottom line here is that we go by the standard, but in practice our CI will use the NPM package (which is the main implementation of the standard anyways), so I thought it was useful to refer to that here so that in case of inconsistencies it would be easy to verify if the problem is in the standard, in this implementation or in other implementations.

[thomashoneyman]

Is it possible to "deprecate" a field by replacing it with a new one, with tools using the new one but falling back to the old one?

[f-f]

That could theoretically work, but the reason why we have to leave old fields in is so that old clients will be able to read them (as they cannot rely on new fields). In this sense we'll never be able to stop writing data to the old fields, so I guess the answer to your question is "not really". Makes sense?

[thomashoneyman]

Is this part just enforced by CI? Or is there any reason to make TargetType either a) Lib (with an enforced "src/**/*.purs" source) or b) some other target?

[f-f]

I envisioned this as being just a CI check, but we could do something fancier as well here.
Right now the Dhall types are fairly basic, because there are some constraints on their values, which need to:

1. have a decent JSON representation
2. make it easy to guarantee forwards-compatibility

[thomashoneyman]

On the whole 👍!

[f-f]

@hdgarrood @thomashoneyman I merged this and opened issues for the review comments where we figured out action points or where I felt there's the need for more discussion, feel free to open more issues if you feel like I didn't address some concern or if I missed some other actionable item.

Thanks everyone for the thoughtful reviews! 🙂