Document registries and packaging process

website/.vitepress/config.mjs

@@ -68,7 +68,12 @@ const sidebar = {
     },
     {
       text: "Registries",
-      items: [p("Package tarball", "/docs/registries/package-tarball")],
+      items: [
+        p("Overview", "/docs/registries/overview"),
+        p("Packaging", "/docs/registries/packaging"),
+        p("Package tarball", "/docs/registries/package-tarball"),
+        p("Custom registry", "/docs/registries/custom-registry"),
+      ],
     },
     {
       text: "Appendices",

website/docs/registries/custom-registry.md

@@ -0,0 +1,69 @@
+# Hosting a registry
+
+Although official registry is in development, we want you to have an option to host your own. You just need to implement an interface imposed by Scarb.
+
+## Basic package registry
+
+The most basic registry needs to serve three types of files over HTTP: index file, package index info and package archives. That means a simple file server can be used as a package registry.
+
+### Registry index file
+
+A file that contains all the info about various endpoints of the registry that Scarb needs to be able to operate over.
+
+Index file has a defined structure:
+
+```json
+{
+  "version": 1,
+  "api": "https://your.domain.com/api/v1",
+  "dl": "https://your.domain.com/api/v1/dl/{package}/{version}",
+  "index": "https://your.domain.com/api/v1/index/{prefix}/{package}.json"
+}
+```
+
+- `version` - value reserved for versioning the registry interface.
+- `api` - a URL of your registry API. Currently isn't used at all, will be used for things like uploading or yanking a package.
+- `dl` - a download template URL. You can use `{package}` and `{version}` to construct a template that Scarb will populate with the name and version of the package that it needs to download. In case of a simple server it could look like `https://your.registry.com/{package}-{version}.tar.zst`. The request to that URL must return a package `.tar.zst` archive created by `scarb package` command.
+- `index` - a URL template that functions like the one in `dl` but it points to JSON files with index info about specific packages. It takes `{package}` parameter which is a package name but also a `{prefix}` value.
+
+  This prefix is useful when you want to organize a file structure into a deeper but narrower one. That's the structure `scarb publish` creates when you use it with local registry. Basically for a package that has name of 4 characters or longer (e.g foobar) the prefix will be `/fo/ob/`, for 3 characters `/3/f` and for 2 and 1 characters just `/2` and `/1` respectively.
+
+### Package index file
+
+When Scarb needs to figure out what version of package it needs to fetch in order to resolve dependency requirements it needs to know what versions are available. That's what package index files are used for. They contain information about package versions present in the registry together with data about what dependencies they have and checksums needed to verify if package haven't been tampered with.
+
+A structure of an example `foo` package index file looks like this:
+
+```json
+[
+  {
+    "v": "0.1.0",
+    "deps": [],
+    "cksum": "sha256:6607a3b860f35f55738360ff55917642282d772423e8120a013b479ddb9e3f89"
+  },
+  {
+    "v": "0.1.1",
+    "deps": [
+      {
+        "name": "bar",
+        "req": "^0.1.3"
+      }
+    ],
+    "cksum": "sha256:5917642282d772423e8120a013b4796607a3b860f35f55738360ff5ddb9e3f89"
+  }
+]
+```
+
+As you can see it is a JSON array with each entry being a version of the `foo` package available in the registry. An entry consist of `v` key that describes the version, `deps` array describing each version dependency requirements and `cksum` which is an `sha256` hash used to verify integrity.
+
+### Package archive
+
+Last type of files that needs to be served are the package archives. These are the output of `scarb package` command as described in [Packaging](/docs/registries/packaging) section.
+
+## Using custom registry
+
+To use a custom registry to download specific dependency you need to add `registry` key to the entry. It needs to point to a URL that returns the registry index file.
+
+```toml
+foo = { version = "0.1.3", registry = "https://custom.registry/index" }
+```

website/docs/registries/overview.md

@@ -0,0 +1,7 @@
+# Overview
+
+Scarb is not only a Cairo build toolchain but also a package manager, so it needs support for package storage. Currently, Git is the default for hosting community packages, but their visibility and ease of discovery are greatly limited. That's what registries are for, to have a concentration of packages in one place that are easily discoverable.
+
+Although the official Cairo package registry is in active development and not ready for use yet, you can [host your own](/docs/registries/custom-registry) registry or search for and [use](/docs/registries/custom-registry#using-custom-registry) a community-hosted one.
+
+If you are interested in learning about the packaging and publishing process, you can read about it [here](/docs/registries/packaging).

website/docs/registries/package-tarball.md

@@ -48,6 +48,26 @@ The normalization process consists of the following:
 
 The original `Scarb.toml` file from the package source directory, without any processing.
 
+### `README` and `LICENSE`
+
+If these files are present in the project root, or paths to them are provided in the manifest, they are copied to the archive as `README.md` and `LICENSE` respectively.
+
+### `VCS.json`
+
+File containing info about version control system used in your package.\
+Currently only git is supported.
+
+```json
+{
+  "git": {
+    "sha1": "a928d5ba03fc09d3316b39f04f30ee135df0c606"
+  },
+  "path_in_vcs": ""
+}
+```
+
+It contains information about the hash of the commit that the package was created on, together with `path_in_vcs` which describes package relative position to the git working directory root. It will be an empty string if it is the same as package root.
+
 ## Package source
 
 By default, only the `src` directory from package source is included in the tarball.

website/docs/registries/packaging.md

@@ -0,0 +1,22 @@
+# Packaging your project
+
+When you want to share your package over package registry, it needs to be turned into an archive first. The archive will then be uploaded to the registry and downloaded by other users.
+
+Use `scarb package` command to create an archive of your package. You can read about package compression algorithm and contents in [Package tarball](/docs/registries/package-tarball) section.
+Basically when you run the command Scarb looks for the source code of your package together with metadata files such as manifest file and put them in an archive in `target/package` directory.
+
+If you are in a git repository, Scarb will first check if the repo state is clean and error out in case of any changes present in a git working directory. To ignore this check you can use the `--allow-dirty` flag.
+
+Next step is the package verification. After creating an initial archive, Scarb will attempt to unpack it and compile to check for any corruptions in the packaging process. If you want to speed up the package process you can disable this step using the `--no-verify` flag.
+
+After successfully passing the whole process, the `{name}-{version}.tar.zst` archive waits in the `target/package` directory for being uploaded, where both `name` and `version` correspond to the values in `Scarb.toml`.
+
+## Publishing the package
+
+To upload your package, you can use the `scarb publish` command. Publishing your package over HTTP is not yet supported, therefore, the only way to upload the package is to use local registry. The command takes `--index` argument that you can use to pass local directory path where you want to store the packages.
+
+```shell
+scarb publish --index file:///Users/foo/bar
+```
+
+This is only useful when you are [hosting your own registry](/docs/registries/custom-registry).