From de137a580394e2aecc8d5b704091e27c08f481a0 Mon Sep 17 00:00:00 2001 From: Mattt Date: Thu, 28 Jan 2021 15:40:45 -0800 Subject: [PATCH 1/7] Revise SE-0292 --- proposals/0292-package-registry-service.md | 1068 ++++++++++++++------ 1 file changed, 743 insertions(+), 325 deletions(-) diff --git a/proposals/0292-package-registry-service.md b/proposals/0292-package-registry-service.md index 0ccb6d7d87..bc5ffdb860 100644 --- a/proposals/0292-package-registry-service.md +++ b/proposals/0292-package-registry-service.md @@ -5,21 +5,26 @@ [Whitney Imura](https://github.com/whitneyimura), [Mattt Zmuda](https://github.com/mattt) * Review Manager: [Tom Doron](https://github.com/tomerd) -* Status: **Returned for revision** +* Status: **Awaiting review** * Implementation: [apple/swift-package-manager#3023](https://github.com/apple/swift-package-manager/pull/3023) * Review: [Review](https://forums.swift.org/t/se-0292-package-registry-service/) +* Previous Revision: + [1](https://github.com/apple/swift-evolution/blob/b48527526b5748a60b0b23846d5880e9cc2c4711/proposals/0292-package-registry-service.md) ## Introduction Swift Package Manager downloads dependencies using Git. Our proposal defines a standard web service interface -that it can also use to download dependencies from package registries. +that it can also use to download dependencies from a package registry. + +Swift-evolution thread: +[Swift Package Registry Service](https://forums.swift.org/t/swift-package-registry-service/37219) ## Motivation -A package dependency is specified by a URL for its source repository. -When a project is built for the first time, -Swift Package Manager clones the Git repository for each dependency +A package dependency is currently specified by a URL to its source repository. +When Swift Package Manager builds a project for the first time, +it clones the Git repository for each dependency and attempts to resolve the version requirements from the available tags. Although Git is a capable version-control system, @@ -28,7 +33,7 @@ it's not well-suited to this kind of workflow for the following reasons: * **Reproducibility**: A version tag in the Git repository for a dependency can be reassigned to another commit at any time. - This can cause the same source code to produce different build results + This can cause a project to produce different build results depending on when it was built. * **Availability**: The Git repository for a dependency can be moved or deleted, @@ -38,77 +43,223 @@ it's not well-suited to this kind of workflow for the following reasons: downloads all versions of a package when only one is used at a time. * **Speed**: Cloning a Git repository for a dependency can be slow - for repositories with large histories. + if it has a large history. + Also, cloning a Git repository is expensive for both the server and client, + and may be significantly slower than downloading the same content + using HTTP through a [content delivery network (CDN)][CDN]. -Many language ecosystems have a package registry, including +Many language ecosystems have a *package registry*, including [RubyGems] for Ruby, [PyPI] for Python, [npm] for JavaScript, and [crates.io] for Rust. In fact, -many Swift developers develop apps today using +many Swift developers build apps today using [CocoaPods] and its index of libraries. -A package registry can offer faster and more reliable dependency resolution +A package registry for Swift Package Manager +could offer faster and more reliable dependency resolution than downloading dependencies using Git. -It can also support other useful functionality, -such as: - -* **Advisories**: - Vulnerabilities can be communicated directly to package consumers - in a timely manner. -* **Discoverability** - Package maintainers can annotate their releases with project metadata, - including its authors, license, and other information. -* **Search** - A registry can provide a standard interface for searching available packages, - or provide the information necessary for others to create a search index. -* **Flexibility** - Swift Package Manager requires an external dependency to be - hosted in a Git repository with a package manifest located in its root, - which may be a barrier to adoption for some projects. - A package registry imposes no requirements on - version control software or project structure. +It could also support other useful functionality, +including package search, security audits, and local offline caches. ## Proposed solution This proposal defines a standard interface for package registry services -and describes how Swift Package Manager can integrate with them +and describes how Swift Package Manager integrates with them to download dependencies. -The goal of this proposal is to make dependency resolution -more available and reproducible. -We believe our proposed solution -can meet or exceed the current performance of dependency resolution -and will allow for new functionality to be built in the future. +A user may [configure](#registry-configuration-subcommands) +a package registry for their project +by specifying a URL to a [conforming web service](#package-registry-service). +When a registry is configured, +Swift Package Manager resolves external dependencies +in the project's package manifest (`Package.swift`) file +that are [declared](#new-packagedescription-apis) +with a [scoped package identifier](#package-identity) in the form +`@scope/PackageName`. +These package identifiers resolve potential +[module name collisions](#module-name-collision-resolution) +across build targets. + +For each external dependency declared in the package manifest, +Swift Package Manager first sends a +`GET` request to `/{scope}/{name}` +to fetch a list of available releases +from the configured registry. +If a release is found that satisfies the declared version requirement +(for example, `.upToNextMinor(from: "1.1.0")`), +Swift Package Manager sends a +`GET` request to `/{scope}/{name}/{version}/Package.swift` +to fetch the manifest for that release. +This process continues with the package manifests of each dependency, +each of their respective dependencies, +and so on. +Once the dependency graph is [resolved](#dependency-graph-resolution), +Swift Package Manager downloads the +[source archive](#archive-source-subcommand) for each dependency +by sending a `GET` request to `/{scope}/{name}/{version}.zip`. ## Detailed design ### Package registry service -A package registry service implements REST API endpoints +A package registry service implements the following REST API endpoints for listing releases for a package, fetching information about a release, -downloading the source archive for a release, -and publishing a new release of a package. +and downloading the source archive for a release: -| Method | Path | Description | -| ------ | ---------------------------------------------------- | ------------------------------------------------ | -| `GET` | `/{package}` | List package releases | -| `GET` | `/{package}/{version}` | Fetch metadata for a package release | -| `GET` | `/{package}/{version}/Package.swift{?swift-version}` | Fetch manifest for a package release | -| `GET` | `/{package}/{version}.zip` | Download source archive for a package release | -| `PUT` | `/{package}/{version}{?commit,branch,tag,path,url}` | Publish a package release or update its metadata | +| Method | Path | Description | +| ------ | --------------------------------------------------------- | ----------------------------------------------- | +| `GET` | `/{scope}/{name}` | List package releases | +| `GET` | `/{scope}/{name}/{version}` | Fetch metadata for a package release | +| `GET` | `/{scope}/{name}/{version}/Package.swift{?swift-version}` | Fetch manifest for a package release | +| `GET` | `/{scope}/{name}/{version}.zip` | Download source archive for a package release | +| `GET` | `/identifiers{?url}` | Lookup package identifiers registered for a URL | A formal specification for the package registry interface is provided alongside this proposal. In addition, -an OpenAPI (v3) document and a reference implementation +an OpenAPI (v3) document +and a reference implementation written in Swift are provided for the convenience of developers interested in building their own package registry. ### Changes to Swift Package Manager +#### Package identity + +Currently, the identity of a package is computed from +the last path component of its effective URL +(which can be changed with dependency mirroring). +However, this approach can lead to a conflation of +distinct packages with similar names +and the duplication of the same package under different names. + +We propose using a scoped identifier +in the form `@scope/PackageName` +to identify package dependencies. + +A *scope* provides a namespace for related packages within a package registry. +A package scope consists of +an at-sign (`@`) followed by alphanumeric characters and hyphens. +Hyphens may not occur at the beginning or end, +nor consecutively within a scope. +The maximum length of a package name is 40 characters. +A valid package scope matches the following regular expression pattern: + +```regexp +\A@[a-zA-Z\d](?:[a-zA-Z\d]|-(?=[a-zA-Z\d])){0,39}\z +``` + +A package's *name* is specified by the `name` provided in its manifest. + +The maximum length of a package scope is 128 characters. +A valid package name matches the following regular expression pattern: + + +```regexp +\A\p{XID_Start}\p{XID_Continue}{0,127}\z +``` + +> For more information, +> see [Unicode Identifier and Pattern Syntax][UAX31]. + +Package scopes are case-insensitive +(for example, `mona` ≍ `MONA`). +Package names are +case-insensitive, +diacritic-insensitive +(for example, `Å` ≍ `A`), and +width-insensitive +(for example, `Ａ` ≍ `A`). +Package names are compared using +[Normalization Form Compatible Composition (NFKC)][UAX15]. + +#### New `PackageDescription` API + +The `Package.Dependency` type adds the following static method: + +```swift +extension Package.Dependency { + /// Adds a dependency on a package with the specified identifier + /// that uses the provided version requirement. + public static func package( + id: String, + _ requirement: Package.Dependency.VersionBasedRequirement + ) -> Package.Dependency +} +``` + +These methods may be called in the `dependencies` field of a package manifest +to declare one or more dependencies by their respective package identifier. + +```swift +dependencies: [ + .package(id: "@mona/LinkedList", .upToNextMinor(from: "1.1.0")), + .package(id: "@mona/RegEx", .exact("2.0.0")) +] +``` + +A package dependency declared with an identifier using this method +may only specify a version-based requirement. +`Package.Dependency.VersionBasedRequirement` is a new type +that provides the same interface as `Package.Dependency.Requirement` +for version-based requirements, +but excluding branch-based and commit-based requirements. + +#### Module name collision resolution + +Consider a dependency graph that includes both +a package declared with the identifier `@mona/LinkedList` and +an equivalent package declared with the URL `https://github.com/mona/LinkedList`. + +When Swift Package Manager fetches a list of releases for the identified package +(`GET /@mona/LinkedList`), +the response includes a `Link` header field +with URLs to that project's source repository +that are known to the registry. + +```http +Link: ; rel="canonical", + ; rel="alternate" +``` + +Swift Package Manager uses this information +to reconcile the URL-based dependency declaration with +the package identifier `@mona/LinkedList`. +Link relation URLs may also be normalized to mitigate insignificant variations. +For example, +a package with an ["scp-style" URL][scp-url] like +`git@github.com:mona/LinkedList.git` +is determined to be equivalent to a URL with an explicit scheme like +`ssh:///git@github.com/mona/LinkedList`. +Swift Package Manager may additionally consult the registry +to associate a URL-based package declaration with a package identifier +by sending a `GET /identifiers{?url}` request with that package's URL. + +A package identifier serves as the package name +in target-based dependency declarations — +that is, the `package` parameter in `.product(name:package)` method calls. + +```diff + targets: [ + .target(name: "MyLibrary", + dependencies: [ + .product(name: "LinkedList", +- package: "LinkedList") ++ package: "@mona/LinkedList") + ] + ] +``` + +Any path-based dependency declaration +or URL-based declaration without an associated package identifier +will continue to synthesize its identity from +the last path component of its location. + +#### Dependency graph resolution + In its `PackageGraph` module, Swift Package Manager defines the `PackageContainer` protocol as the top-level unit of package resolution. Conforming types are responsible for @@ -122,73 +273,27 @@ There are currently two concrete implementations of `PackageContainer`: This proposal adds a new `RegistryPackageContainer` type that adopts `PackageContainer` and performs equivalent operations with HTTP requests to a registry service. - -This proposal also adds a new `CompoundRepositoryProvider` type -that conforms to `RepositoryPackageContainerProvider` -and attempts to use the package registry interface when available -for qualifying remote packages. +These client-server interactions are facilitated by +a new `RegistryManager` type. The following table lists the tasks performed by Swift Package Manager during dependency resolution alongside the Git operations used and their corresponding package registry API calls. -| Task | Git operation | Registry request | -| ------------------------------------- | --------------------------- | ---------------------------------------- | -| Fetch the contents of a package | `git clone && git checkout` | `GET /{package}/{version}.zip` | -| List the available tags for a package | `git tag` | `GET /{package}` | -| Fetch a package manifest | `git clone` | `GET /{package}/{version}/Package.swift` | - -Initially, -Swift Package Manager will use a package registry to resolve dependencies -only when the user passes the `--enable-package-registries` command-line flag. -This may change in a future release. - -```terminal -$ swift build --enable-package-registries -``` - -When package registries are enabled, -Swift Package Manager will first attempt to use package registry API calls -to resolve qualifying dependencies, -falling back to Git operations if those API calls fail. -A dependency qualifies for resolution through a package registry -if it satisfies all of the following criteria: - -* The package has a url with an `https` scheme. -* The last path component of the package url has no file extension. -* The dependency specifies an exact version or range of versions. - -For example, -here are a list of dependencies that do and do not qualify: - -```swift -// ✅ These dependencies qualify for resolution with package registry -.package(url: "https://github.com/mona/LinkedList", from: "1.1.0") -.package(url: "https://github.com/mona/LinkedList", .exact("1.1.0")) -.package(url: "https://github.com/mona/LinkedList", .upToNextMajor(from: "1.1.0")) -.package(url: "https://github.com/mona/LinkedList", .upToNextMinor(from: "1.1.0")) - -// ❌ These dependencies can only be resolved using Git -.package(url: "git@github.com:mona/LinkedList.git", from: "1.1.0") // No https scheme -.package(url: "https://github.com/mona/LinkedList.git", from: "1.1.0") // .git file extension -.package(url: "https://github.com/mona/LinkedList", .branch("master")) // No version -.package(url: "https://github.com/mona/LinkedList", .revision("d6ca4e56219a8a5f0237d6dcdd8b975ec7e24c89")) // No version -.package(path: "../LinkedList") // No https scheme or version -``` +| Task | Git operation | Registry request | +| ------------------------------------- | --------------------------- | ---------------------------------------------------- | +| Fetch the contents of a package | `git clone && git checkout` | `GET /{scope}/{name}/{version}.zip` | +| List the available tags for a package | `git tag` | `GET /{scope}/{name}` | +| Fetch a package manifest | `git clone` | `GET /{scope}/{name}/{version}/Package.swift` | Package registries support [version-specific _manifest_ selection][version-specific-manifest-selection] by providing a list of versioned manifest files for a package (for example, `Package@swift-5.3.swift`) -in its response to `GET /{package}/{version}/Package.swift`. -However, package registries won't support -[version-specific _tag_ selection][version-specific-tag-selection], -and instead rely on [Semantic Versioning][SemVer] -to accomodate different versions of Swift -(for example, -by using major release versions -or build metadata like `1.0.0+swift-5_3`). +in its response to `GET /{scope}/{name}/{version}/Package.swift`. +However, package registries don't support +[version-specific _tag_ selection][version-specific-tag-selection]. ### Changes to `Package.resolved` @@ -196,7 +301,7 @@ Swift package registry releases are archived as Zip files. When an external package dependency is downloaded through a registry, Swift Package Manager compares the integrity checksum provided by the server -against any existing checksum for that release in `Package.resolved` +against any existing checksum for that release in the `Package.resolved` file as well as the integrity checksum reported by the `compute-checksum` subcommand: ```terminal @@ -212,8 +317,7 @@ it's saved to `Package.resolved`. "object": { "pins": [ { - "package": "LinkedList", - "url": "https://github.com/mona/LinkedList", + "package": "@mona/LinkedList", "state": { "checksum": "ed008d5af44c1d0ea0e3668033cae9b695235f18b1a99240b7cf0f3d9559a30d", "version": "1.2.0" @@ -225,34 +329,36 @@ it's saved to `Package.resolved`. } ``` -If the checksum reported by the server is different from the existing checksum -(or the checksum of the downloaded artifact is different from either of them), -that's an indication that a package's contents may have changed at some point. +Suppose the checksum reported by the server +is different from the existing checksum +(or the checksum of the downloaded artifact is different from either of them). +In that case, +a package's contents may have changed at some point. Swift Package Manager will refuse to download dependencies if there's a mismatch in integrity checksums. ```terminal $ swift build -error: checksum of downloaded source archive of dependency 'LinkedList' (c2b934fe66e55747d912f1cfd03150883c4f037370c40ca2ad4203805db79457) does not match checksum specified by the manifest (ed008d5af44c1d0ea0e3668033cae9b695235f18b1a99240b7cf0f3d9559a30d) +error: checksum of downloaded source archive of dependency '@mona/LinkedList' (c2b934fe66e55747d912f1cfd03150883c4f037370c40ca2ad4203805db79457) does not match checksum specified by the manifest (ed008d5af44c1d0ea0e3668033cae9b695235f18b1a99240b7cf0f3d9559a30d) ``` Once the correct checksum is determined, the user can update `Package.resolved` with the correct value and try again. -### Archive subcommand +### Archive-source subcommand An anecdotal look at other package managers suggests that a checksum mismatch is more likely to be a disagreement in how to create the archive and/or calculate the checksum than, say, a forged or corrupted package. -This proposal adds a new `swift package archive` subcommand +This proposal adds a new `swift package archive-source` subcommand to provide a standard way to create source archives for package releases. ```manpage SYNOPSIS - swift package archive [--output=] + swift package archive-source [--output=] OPTIONS -o , --output= @@ -260,7 +366,8 @@ OPTIONS If unspecified, the package is written to `\(PackageName).zip`. ``` -Run the `swift package archive` subcommand in the root directory of a package +Run the `swift package archive-source` subcommand +in the root directory of a package to generate a source archive for the current working tree. For example: @@ -280,24 +387,24 @@ import PackageDescription let package = Package( name: "LinkedList", -$ swift package archive +$ swift package archive-source Created LinkedList.zip ``` By default, -the filename of the generated archive is +generated archive's filename is the name of the package with a `.zip` extension (for example, "LinkedList.zip"). -This can be configured with the `--output` option: +You can override this behavior with the `--output` option: ```terminal $ git checkout 1.2.0 -$ swift package archive --output="LinkedList-1.2.0.zip" +$ swift package archive-source --output="LinkedList-1.2.0.zip" # Created LinkedList-1.2.0.zip ``` -The `archive` subcommand has the equivalent behavior of [`git-archive(1)`] -using the `zip` format at its default compression level. +The `archive-source` subcommand has the equivalent behavior of +[`git-archive(1)`] using the `zip` format at its default compression level. Therefore, the following command produces equivalent output to the previous example: @@ -305,13 +412,227 @@ equivalent output to the previous example: $ git archive --format zip --output LinkedList-1.2.0.zip 1.2.0 ``` -If desired, this behavior may be changed in future tool versions. +If desired, this behavior could be changed in future tool versions. -> **Note:** +> **Note**: > `git-archive` ignores files with the `export-ignore` Git attribute. > By default, this ignores hidden files and directories, > including`.git` and `.build`. +### Registry configuration subcommands + +This proposal adds a new `swift package-registry` subcommand +for managing the registry used for all packages +and/or packages in a particular scope. + +Custom registries can serve a variety of purposes: + +- **Private dependencies**: + Users may configure a custom registry for a particular scope + to incorporate private packages with those fetched from a public registry. +- **Geographic colocation**: + Developers working under adverse networking conditions can + host a mirror of official package sources on a nearby network. +- **Policy enforcement**: + A corporate network can enforce quality or licensing standards, + so that only approved packages are available through a custom registry. +- **Auditing**: + A custom registry may analyze or meter access to packages + for the purposes of ranking popularity or charging licensing fees. + +#### Setting a custom registry + +```manpage +SYNOPSIS + swift package-registry set [options] +OPTIONS: + --global Apply settings to all projects for this user + --scope Associate the registry with a given scope +``` + +Running the `package-registry set` subcommand +in the root directory of a package +creates or updates the `.swiftpm/config/registries.json` file +with a new top-level `registries` key +that's associated with an object containing the specified registry URLs. + +For example, +a build server that doesn't allow external network connections +may configure a registry URL to resolve dependencies +using an internal registry service. + +```terminal +$ swift package-registry set https://internal.example.com/ +$ cat .swiftpm/config/registries.json +``` + +```json +{ + "registries": { + "default": { + "url": "https://internal.example.com" + } + }, + "version": 1 +} + +``` + +If no registry is configured, +Swift Package Manager commands like +`swift package resolve` and `swift package update` +fail with an error. + +```terminal +$ swift package resolve +error: cannot resolve dependency '@mona/LinkedList' without a configured registry +``` + +#### Associating a registry with a scope + +The user can associate a package scope with a custom registry +by passing the `--scope` option. + +For example, +a user might resolve all packages with the package scope `@example` +(such as `@example/PriorityQueue`) +to a private registry. + +```terminal +$ swift package-registry set https://internal.example.com/ --scope @example +$ cat .swiftpm/config/registries.json +``` + +```json +{ + "registries": { + "@example": { + "url": "https://internal.example.com" + } + }, + "version": 1 +} + +``` + +When a custom registry is associated with a package scope, +package dependencies with that scope are resolved through the provided URL. +A custom registry may be associated with one or more scopes, +but a scope may be associated with only a single registry at a time. +Scoped custom registries override any unscoped custom registry. + +#### Unsetting a custom registry + +This proposal also adds a new `swift package-registry unset` subcommand +to complement the `package-registry set` subcommand. + +```manpage +SYNOPSIS + swift package-registry unset [options] +OPTIONS: + --global Apply settings to all projects for this user + --scope Removes the registry's association to a given scope +``` + +Running the `package-registry unset` subcommand +in the root directory of a package +updates the `.swiftpm/config` file +to remove the `default` entry in the top-level `registries` key, if present. +If a `--scope` option is passed, +only the entry for the specified scope is removed, if present. + +#### Global registry configuration + +The user can pass the `--global` option to the `set` or `unset` subcommands +to update the user-level configuration file located at +`~/.swiftpm/config/registries.json`. + +Any default or scoped registries configured locally in a project directory +override any values configured globally for the user. +For example, +consider the following global and local registry configuration files: + +```jsonc +// Global configuration (~/.swiftpm/config/registries.json) +{ + "registries": { + "default": { + "url": "https://global.example.com" + }, + "@foo": { + "url": "https://global.example.com" + }, + }, + "version": 1 +} + +// Local configuration (.swiftpm/config/registries.json) +{ + "registries": { + "@foo": { + "url": "https://local.example.com" + } + }, + "version": 1 +} + +``` + +Running the `swift package resolve` command with these configuration files +resolves packages with the `@foo` scope +using the registry located at "https://local.example.com", +and all other packages +using the registry located at "https://global.example.com". + +In summary, +the behavior of `swift package resolve` and related commands +depends on the following factors, +in descending order of precedence: + +* The package manifest in the current directory (`./Package.swift`) +* Any existing lock file (`./Package.resolved`) +* Any local configuration (`./.swiftpm/config`) +* Any global configuration file (`~/.swiftpm/config/registries.json`) + +### Changes to config subcommand + +#### Set-mirror option for package identifiers + +A user can currently specify an alternate location for a package +by setting a [dependency mirror][SE-0219] for that package's URL. + +```terminal +$ swift package config set-mirror \ + --original-url https:///github.com/mona/linkedlist \ + --mirror-url https:///github.com/octocorp/swiftlinkedlist +``` + +This proposal updates the `swift package config set-mirror` subcommand +to accept a `--package-identifier` option in place of an `--original-url`. +Running this subcommand with a `--package-identifier` option +creates or updates the `.swiftpm/config` file, +modifying the array associated with the top-level `object` key +to add a new entry or update an existing entry +for the specified package identifier, +that assigns its alternate location. + +```json +{ + "object": [ + { + "mirror": "https://github.com/OctoCorp/SwiftLinkedList.git", + "original": "@mona/LinkedList" + } + ], + "version": 1 +} + +``` + +When a mirror URL is set for a package identifier, +Swift Package Manager resolves any dependencies with that identifier +through Git using the provided URL. + ## Security Adding external dependencies to a project @@ -320,6 +641,46 @@ However, much of the associated risk can be mitigated, and a package registry can offer stronger guarantees for safety and security compared to downloading dependencies using Git. +Core security measures, +such as the use of HTTPS and integrity checksums, +are required by the registry service specification. +Additional decisions about security +are delegated to the registries themselves. +For example, +registries are encouraged to adopt a +scoped, revocable authorization framework like [OAuth 2.0][RFC 6749], +but this isn't a strict requirement. +Package maintainers and consumers should +consider a registry's security posture alongside its other features +when deciding where to host and fetch packages. + +Our proposal's package identity scheme is designed to prevent or mitigate +vulnerabilities common to packaging systems and networked applications: + +- Package scopes are marked by an at-sign (`@`) prefix + and restricted to a limited set of characters, + preventing [homograph attacks]. + For example, + "А" (U+0410 CYRILLIC CAPITAL LETTER A) is an invalid scope character + and cannot be confused for "A" (U+0041 LATIN CAPITAL LETTER A). +- Package scopes disallow leading, trailing, or consecutive hyphens (`-`), + and disallows underscores (`_`) entirely, + which mitigates look-alike package scopes + (for example, "@llvm--swift" and "@llvm_swift" are both invalid + and cannot be confused for "@llvm-swift"). +- Package scopes disallow dots (`.`), + which prevents potential confusion with domain variants of scopes + (for example, "@apple.com" is invalid + and cannot be confused for "@apple"). +- Packages are registered within a scope, + which mitigates [typosquatting]. + Package registries may further restrict the assignment of new scopes + that are intentionally misleading + (for example, "@G00gle", which looks like "@Google"). +- Package names disallow punctuation and whitespace characters used in + [cross-site scripting][xss] and + [CRLF injection][http header injection] attacks. + To better understand the security implications of this proposal — and Swift dependency management more broadly — we employ the @@ -341,11 +702,10 @@ goes a long way to mitigate the overall risk. Swift Package Manager could further mitigate this risk by taking the following measures: -* Enforcing HTTPS for all dependency URLs -* Resolving dependency URLs using DNS over HTTPS (DoH) -* Requiring dependency URLs with Internationalized Domain Names (IDNs) +* Enforcing HTTPS for all URLs +* Resolving URLs using DNS over HTTPS (DoH) +* Requiring URLs with Internationalized Domain Names (IDNs) to be represented as Punycode -* Normalizing package names to use an ASCII-compatible subset of characters ### Tampering @@ -369,9 +729,11 @@ and provide a valid checksum for a malicious package. `Package.resolved` provides a [Trust on first use (TOFU)][TOFU] security model that can offer strong guarantees about the integrity of dependencies over time. -A registry can further improve on this model -by implementing a [transparent log] or some comparable, -tamper-proof system for associating artifacts with valid checksums. +A registry can further improve on this model by implementing a +[transparent log], +[checksum database], +or another comparable, tamper-proof system +for authenticating package contents. ### Repudiation @@ -384,41 +746,75 @@ and individual commits may be cryptographically signed by authors. Unless you can establish a direct connection between an artifact and a commit in a source tree, there's no way to determine the provenance of that artifact. -However, -a [transparent log] of checksums or the use of digital signatures -can provide similar non-repudiation guarantees. -### Information disclosure +Source archives generated by [`git-archive(1)`] +include the checksum of the `HEAD` commit as a comment. +If the history of a project is available +and the commit used to generate the source archive is signed with [GPG], +the cryptographic signature may be used to verify the authenticity. -An attacker could scrape public code repositories -for `Package.swift` files that use hardcoded credentials in dependency URLs, -and attempt to reuse those credentials to impersonate the user. +```terminal +$ git rev-parse HEAD +b7c37c81f164e5dce0f64e3d75c79a48fb1fe00b3 -```swift -dependencies: [ - .package(name: "TopSecret", - url: "https://:x-oauth-basic@github.com/mona/TopSecret", - checksum: "2c4a4ce92225fb766447c1757abb916e13f68eba0459f1287ee62e4941d89bbf") -] +$ swift package archive-source -o LinkedList-1.2.0.zip +Generated LinkedList-1.2.0.zip + +$ zipnote LinkedList-1.2.0.zip | grep "@ (zip file comment below this line)" -A 1 | tail -n 1 +b7c37c81f164e5dce0f64e3d75c79a48fb1fe00b3 + +$ git verify-commit b7c37c81f164e5dce0f64e3d75c79a48fb1fe00b3 +gpg: Signature made Tue Dec 16 00:00:00 2020 PST +gpg: using RSA key BFAA7114B920808AA4365C203C5C1CF +gpg: Good signature from "Mona Lisa Octocat " [ultimate] ``` -This kind of attack can be mitigated on an individual basis -by using an unauthenticated URL and setting a mirror. +Otherwise, +a checksum database and the use of digital signatures +can both provide similar non-repudiation guarantees. + +### Information disclosure + +A user may inadvertently reveal the existence of a private registry +or expose hardcoded credentials +by checking in their project's `.swiftpm/config` file. + +An attacker could scrape public code repositories for `.swiftpm/config` files +and attempt to reuse those credentials to impersonate the user. + +```json +{ + "registries": { + "default": { + "url": "https://:@swift.pkg.github.com//" + } + }, + "version": 1 +} -```terminal -$ swift package config set-mirror \ - --original-url https://github.com/mona/TopSecret \ - --mirror-url https://:x-oauth-basic@github.com/mona/TopSecret ``` +This kind of attack can be mitigated on an individual basis +by adding `.swiftpm/config` to a project's `.gitignore` file. The risk could be mitigated for all users -if Swift Package Manager forbids the use of hardcoded credentials -in `Package.swift` files. +if Swift Package Manager included a `.gitignore` file +in its new project template. +Code hosting providers can also help minimize this risk +by [detecting secrets][secret scanning] +that are committed to public repositories. + +Credentials may also be unintentionally disclosed +by Swift Package Manager or other tools in logging statements. +Care should be taken to redact the user info component of URLs +when displaying feedback to the user +(for example, +the URL `https://:@swift.pkg.github.com` +is logged as `https://***@swift.pkg.github.com`). ### Denial of service An attacker could scrape public code repositories -for `Package.swift` files that declare dependencies +for `.swiftpm/config` files that declare one or more custom registries and launch a denial-of-service attack in an attempt to reduce the availability of those resources. @@ -426,103 +822,80 @@ The likelihood of this attack is generally low but could be used in a targeted way against resources known to be important or expensive to distribute. -This threat can be mitigated by obfuscating dependency URLs, -such that they can't be pattern matched from source code. - -```swift -func rot13(_ string: String) -> String { - String(string.unicodeScalars.map { unicodeScalar in - var value = unicodeScalar.value - switch unicodeScalar { - case "A"..."M", "a"..."m": value += 13 - case "N"..."Z", "n"..."z": value -= 13 - default: break - } - - return Character(Unicode.Scalar(value)!) - }) -} - -dependencies: [ - .archive(name: "TopSecret", - url: rot13("uggcf://tvguho.pbz/zban/GbcFrperg"), - // ^ "https://github.com/mona/TopSecret" - checksum: "2c4a4ce92225fb766447c1757abb916e13f68eba0459f1287ee62e4941d89bbf") -] -``` - -> **Important**: -> Never store credentials in code — -> _even if they're obfuscated_. - ### Escalation of privilege Even authentic packages from trusted creators can contain malicious code. Code analysis tools can help to some degree, as can system permissions and other OS-level security features. -But developers are ultimately the ones responsible -for the code they ship to users. +However, developers are ultimately responsible for the code they ship to users. ## Impact on existing packages Current packages won't be affected by this change, -as they'll continue to be able to download dependencies directly through Git. - -HTTP content negotiation can be used to migrate existing package dependencies -to take advantage of Swift package registries -without changing their specification. +as they'll continue to download dependencies directly through Git. -For example, -consider the following `Package.swift` manifest for a package -that includes `LinkedList` as a dependency: +## Alternatives considered -```swift -// swift-tools-version:5.2 -import PackageDescription +### Use of alternative naming schemes -let package = Package( - name: "Example", - dependencies: [ - .package(url: "https://github.com/mona/LinkedList", from: "1.1.0"), - ], - targets: [ - .target(name: "Example", dependencies: ["LinkedList"]) - ] -) -``` +Some package systems, +including [RubyGems], [PyPI], and [CocoaPods] +identify packages with bare names in a flat namespace +(for example, `rails`, `pandas`, or `Alamofire`). +Other systems, +including [Maven], +use [reverse domain name notation] to identify software components +(for example, `com.squareup.okhttp3`). -Currently, -Swift package manager uses the provided URL -to request the Git repository for the LinkedList dependency. -A future version of Swift Package Manager -could specify `application/vnd.swift.registry.v1+json` in its `Accept` header -(or set a versioned `User-Agent` header) -to opt-in to package registry APIs when available. - -## Alternatives considered +We considered these and other schemes for identifying packages, +but they were rejected in favor of a scoped package identity +similar to the one used by [npm]. ### Use of `tar` or other archive formats Swift Package Manager currently uses Zip archives for binary dependencies, which is reason enough to use it again here. +Zip files are also a convenient format for package registries, +because they support the access of individual files within an archive. +This allows a registry to satisfy +the package manifest endpoint +(`GET /{scope}/{name}/{version}/Package.swift`) +without storing anything separately from the archive used for the +package archive endpoint +(`GET /{scope}/{name}/{version}.zip`). + We briefly considered `tar` as an archive format -but concluded that its behavior of preserving symlinks and executable bits +but concluded that its behavior of preserving symbolic links and executable bits served no useful purpose in the context of package management, and instead raised concerns about portability and security. -> As an aside, -> Zip files are also a convenient format for package registries, -> because they support the access of individual files within an archive. -> This allows a registry to satisfy -> the package manifest endpoint (`GET /{package}/{version}/Package.swift`) -> without storing anything separately from the archive used for the -> package archive endpoint (`GET /{package}/{version}.zip`). +### Inclusion of alternative source locations in package releases payload + +To maintain compatibility with existing, URL-based dependency declarations +Swift Package Manager needs to reconcile source locations +with their respective identifiers. +For example, +the declarations +`.package(url: "https://github.com/mona/LinkedList", .exact("1.1.0"))` and +`.package(id: "@mona/LinkedList", .exact("1.1.0"))`, +must be deemed equivalent +to resolve a dependency graph that contains both of them. + +We considered including alternative source locations in the response body, +but rejected that in favor of using link relations. + +[Web linking][RFC 8288] provides a standard way to +describe the relationships between resources. +Standard `canonical` and `alternative` [IANA link relations] +convey precise semantics for +the relationship between a package and its source repositories +that are broadly useful beyond any individual client. -### Addition of an `unarchive` subcommand +### Addition of an `unarchive-source` subcommand -This proposal adds an `archive` subcommand +This proposal adds an `archive-source` subcommand as a standard way for developers and registries to create source archives for packages. Having a canonical tool for creating source archives @@ -530,14 +903,14 @@ avoids any confusion when attempting to verify the integrity of Zip files sent from a registry with the source code for that package. -We considered including a complementary `unarchive` subcommand +We considered including a complementary `unarchive-source` subcommand but ultimately decided against it, -reason being that unarchiving a Zip archive +the reason being that unarchiving a Zip archive is unambiguous and well-supported on most platforms. ### Use of digital signatures -[SE-0272] includes discussion about +[SE-0272] includes a discussion about the use of digital signatures for binary dependencies, concluding that they were unsuitable because of complexity around transitive dependencies. @@ -552,71 +925,90 @@ For the reasons outlined in the preceding Security section, we believe that digital signatures may offer additional guarantees of authenticity and non-repudiation beyond what's possible with checksums alone. -### Addition of a `publish` subcommand - -Most package managers — -including the ones described in the introduction to this proposal — -follows what we describe as a "push" model. -When a package owner releases a new version of their software, -a client runs a command locally and pushes the results to a server. - -However, -the "push" model reflects a tradition of software deployment -that predates modern source code management and build automation. -As package maintainers can attest, -this approach often involves a lot of manual effort -and trial-and-error guesswork. -It also lacks strong guarantees about reproducibility and software traceability. - -Taking inspiration from current best-practices like -continuous integration (CI) and continuous delivery (CD), -this proposal instead follows what we describe a "pull" model. -When a package owner releases a new version of their software, -their sole responsibility is to notify the package registry. -The server does all the work of downloading the source code -and packaging it up for distribution. - -We considered but rejected the idea of a `publish` subcommand -for a few different reasons. -For one, we worried that the existence of `swift package archive` -would cause confusion to anyone more familiar with -another "push"-style package ecosystem. -The specification's open-ended policy for a registry's authentication model -also proved to be a complicating factor. -But the deciding factor was that we saw `publish` as unnecessary; -we imagine package publication to be the -final outcome of a successful CI /CD pipeline to be run automatically, -rather than a command to be run manually. - ## Future directions Defining a standard interface for package registries lays the groundwork for several useful features. -### Package removal +### Package dependency URL normalization + +As described in ["Module name collision resolution"](#module-name-collision-resolution) +Swift Package Manager cannot build a project +if two or more packages in the project +are located by URLs with the same (case-insensitive) last path component. +Swift Package Manager may improve support URL-based dependencies +by normalizing package URLs to mitigate insignificant variations. +For example, +a package with an ["scp-style" URL][scp-url] like +`git@github.com:mona/LinkedList.git` +may be determined to be equivalent to a package with an HTTPS scheme like +`https:///github.com/mona/LinkedList`. + +### Local offline cache + +Swift Package Manager could implement an [offline cache] +that would allow it to work without network access. +While this is technically possible today, +a package registry makes for a simpler and more secure implementation +than would otherwise be possible with Git repositories alone. + +### Package publishing + +A package registry is responsible for determining +which package releases are made available to a consumer. +This proposal sets no policies for how +package releases are published to a registry. +Nor does it specify how package scopes are registered or verified. + +Many package managers — +including the ones mentioned above — +and artifact repository services, such as +[Docker Hub], +[JFrog Artifactory], +and [AWS CodeArtifact] +follow what we describe as a *"push"* model of publication: +When a package owner wants to releases a new version of their software, +they produce a build locally and push the resulting artifact to a server. +This model has the benefit of operational simplicity and flexibility. +For example, +maintainers have an opportunity to digitally sign artifacts +before uploading them to the server. + +Alternatively, +a system might incorporate build automation techniques like +continuous integration (CI) and continuous delivery (CD) +into what we describe as a *"pull"* model: +When a package owner wants to release a new version of their software, +their sole responsibility is to notify the package registry; +the server does all the work of downloading the source code +and packaging it up for distribution. +This model can provide strong guarantees about +reproducibility, quality assurance, and software traceability. + +We intend to work with industry stakeholders +to develop standards for publishing Swift packages +in an optional extension to the registry specification. -The proposed specification defines an endpoint for publishing package releases, -but not for removing them. +### Package removal There are several reasons why a package release may be removed, including: -* The package maintainer publishes a release by mistake. -* A security vulnerability is found in a release. -* The registry is compelled by law enforcement to remove a release. +* The package maintainer publishing a release by mistake +* A security researcher disclosing a vulnerability for a release +* The registry being compelled by law enforcement to remove a release However, removing a package release has the potential to break any packages that depend on it. -Many package management systems have their own processes for -how removal works (or whether it's supported in the first place). It's unclear whether or to what extent such policies should be informed by registry specification itself. For now, a registry is free to exercise its own discretion about how to respond to out-of-band removal requests. -We look forward to discussing this further -and updating the specification once we arrive at a consensus -about the correct behavior. + +We plan to consider these questions +as part of the future, optional extension to the specification +described in the previous section. ### Binary framework distribution @@ -625,7 +1017,7 @@ to support distributing packages as [XCFramework] bundles. ```http GET /github.com/mona/LinkedList/1.1.1.xcframework HTTP/1.1 -Host: packages.example.com +Host: packages.github.com Accept: application/vnd.swift.registry.v1+xcframework ``` @@ -640,22 +1032,44 @@ let package = Package( targets: [ .binaryTarget( name: "LinkedList", - url: "https://packages.example.com/github.com/mona/LinkedList/1.1.1.xcframework", + url: "https://packages.github.com/github.com/mona/LinkedList/1.1.1.xcframework", checksum: "ed04a550c2c7537f2a02ab44dd329f9e74f9f4d3e773eb883132e0aa51438b37" ), ] ) ``` -### Offline cache +### Updates to package editor commands -Swift Package Manager could implement an [offline cache] -that would allow it to work without network access. -While this is technically possible today, -a package registry makes for a simpler and more secure implementation -than would otherwise be possible with Git repositories alone. +[Package editor commands][SE-0301] +could be extended to add dependencies using scoped identifiers +in addition to URLs. + +```terminal +$ swift package add-dependency @mona/LinkedList +# Installed LinkedList 1.2.0 +``` + +```diff ++ .package(id: "@mona/LinkedList", .exact("1.2.0")) +``` + +### Package manifest dependency migration + +Swift Package Manager could add tooling +to help package maintainers adopt registry-supported identifiers +in their projects. + +```terminal +$ swift package-registry migrate +``` + +```diff +- .package(url: "https://github.com/mona/LinkedList", .exact("1.2.0")) ++ .package(id: "@mona/LinkedList", .exact("1.2.0")) +``` -### Security auditing +### Security audits The response for listing package releases could be updated to include information about security advisories. @@ -666,7 +1080,7 @@ information about security advisories. "advisories": [{ "cve": "CVE-20XX-12345", "cwe": "CWE-400", - "package_name": "github.com/mona/LinkedList", + "package_name": "@mona/LinkedList", "vulnerable_versions": "<=1.0.0", "patched_versions": ">1.0.0", "severity": "moderate", @@ -682,19 +1096,19 @@ or as part of a new `swift package audit` subcommand. ```terminal $ swift package audit -┌───────────────┬──────────────────────────────────────────────────────────────┐ -│ High │ Regular Expression Denial of Service │ -├───────────────┼──────────────────────────────────────────────────────────────┤ -│ Package │ RegEx │ -├───────────────┼──────────────────────────────────────────────────────────────┤ -│ Dependency of │ PatternMatcher │ -├───────────────┼──────────────────────────────────────────────────────────────┤ -│ Path │ SomePackage > PatternMatcher > RegEx │ -├───────────────┼──────────────────────────────────────────────────────────────┤ -│ More info │ https://example.com/advisories/526 │ -└───────────────┴──────────────────────────────────────────────────────────────┘ - -Found 3 vulnerability (1 low, 1 moderate, 1 high) in 12 scanned packages. +┌───────────────┬────────────────────────────────────────────────┐ +│ High │ Regular Expression Denial of Service │ +├───────────────┼────────────────────────────────────────────────┤ +│ Package │ @mona/RegEx │ +├───────────────┼────────────────────────────────────────────────┤ +│ Dependency of │ PatternMatcher │ +├───────────────┼────────────────────────────────────────────────┤ +│ Path │ SomePackage > PatternMatcher > RegEx │ +├───────────────┼────────────────────────────────────────────────┤ +│ More info │ https://example.com/advisories/526 │ +└───────────────┴────────────────────────────────────────────────┘ + +Found 3 vulnerabilities (1 low, 1 moderate, 1 high) in 8 scanned packages. Run `swift package audit fix` to fix 3 of them. ``` @@ -713,20 +1127,25 @@ LinkedList (github.com/mona/LinkedList) - One thing links to another. RegEx (github.com/mona/RegEx) - Expressions on the reg. ``` -### Package installation from the command-line - -Swift Package Manager could be extended with an `install` subcommand -that adds a dependency by its URL. - -```terminal -$ swift package install github.com/mona/LinkedList -# Installed LinkedList 1.2.0 -``` - -This functionality could be implemented separately from this proposal -but is included here as a complement to the search subcommand described above. - +[AWS CodeArtifact]: https://aws.amazon.com/codeartifact/ [BCP 13]: https://tools.ietf.org/html/rfc6838 "Media Type Specifications and Registration Procedures" +[CDN]: https://en.wikipedia.org/wiki/Content_delivery_network "Content delivery network" +[checksum database]: https://sum.golang.org "Go Module Mirror, Index, and Checksum Database" +[CocoaPods]: https://cocoapods.org "A dependency manager for Swift and Objective-C Cocoa projects" +[crates.io]: https://crates.io "crates.io: The Rust community’s crate registry" +[Docker Hub]: https://hub.docker.com +[GPG]: https://gnupg.org +[homograph attacks]: https://en.wikipedia.org/wiki/IDN_homograph_attack +[http header injection]: https://en.wikipedia.org/wiki/HTTP_header_injection +[IANA link relations]: https://www.iana.org/assignments/link-relations/link-relations.xhtml "IANA Link Relation Types" +[ICANN]: https://www.icann.org +[JFrog Artifactory]: https://jfrog.com/artifactory/ +[JSON-LD]: https://w3c.github.io/json-ld-syntax/ "JSON-LD 1.1: A JSON-based Serialization for Linked Data" +[Maven]: https://maven.apache.org +[npm]: https://www.npmjs.com "The npm Registry" +[offline cache]: https://yarnpkg.com/features/offline-cache "Offline Cache | Yarn - Package Manager" +[PyPI]: https://pypi.org "PyPI: The Python Package Index" +[reverse domain name notation]: https://en.wikipedia.org/wiki/Reverse_domain_name_notation [RFC 2119]: https://tools.ietf.org/html/rfc2119 "Key words for use in RFCs to Indicate Requirement Levels" [RFC 3230]: https://tools.ietf.org/html/rfc5843 "Instance Digests in HTTP" [RFC 3492]: https://tools.ietf.org/html/rfc3492 "Punycode: A Bootstring encoding of Unicode for Internationalized Domain Names in Applications (IDNA)" @@ -744,27 +1163,26 @@ but is included here as a complement to the search subcommand described above. [RFC 7807]: https://tools.ietf.org/html/rfc7807 "Problem Details for HTTP APIs" [RFC 8288]: https://tools.ietf.org/html/rfc8288 "Web Linking" [RFC 8446]: https://tools.ietf.org/html/rfc8446 "The Transport Layer Security (TLS) Protocol Version 1.3" -[TR36]: http://www.unicode.org/reports/tr36/ "Unicode Technical Report #36: Unicode Security Considerations" -[IANA Link Relations]: https://www.iana.org/assignments/link-relations/link-relations.xhtml -[JSON-LD]: https://w3c.github.io/json-ld-syntax/ "JSON-LD 1.1: A JSON-based Serialization for Linked Data" -[SemVer]: https://semver.org/ "Semantic Versioning" +[RubyGems]: https://rubygems.org "RubyGems: The Ruby community’s gem hosting service" [Schema.org]: https://schema.org/ +[scp-url]: https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols#_the_ssh_protocol +[SE-0219]: https://github.com/apple/swift-evolution/blob/master/proposals/0219-package-manager-dependency-mirroring.md "Package Manager Dependency Mirroring" +[SE-0272]: https://github.com/apple/swift-evolution/blob/master/proposals/0272-swiftpm-binary-dependencies.md "Package Manager Binary Dependencies" +[SE-0301]: https://github.com/apple/swift-evolution/blob/main/proposals/0301-package-editing-commands.md "Package Editor Commands" +[secret scanning]: https://docs.github.com/en/github/administering-a-repository/about-secret-scanning +[SemVer]: https://semver.org/ "Semantic Versioning" [SoftwareSourceCode]: https://schema.org/SoftwareSourceCode -[DUST]: https://doi.org/10.1145/1462148.1462151 "Bar-Yossef, Ziv, et al. Do Not Crawl in the DUST: Different URLs with Similar Text. Association for Computing Machinery, 17 Jan. 2009. January 2009" - -[GitHub / Swift Package Management Service]: https://forums.swift.org/t/github-swift-package-management-service/30406 - -[RubyGems]: https://rubygems.org "RubyGems: The Ruby community’s gem hosting service" -[PyPI]: https://pypi.org "PyPI: The Python Package Index" -[npm]: https://www.npmjs.com "The npm Registry" -[crates.io]: https://crates.io "crates.io: The Rust community’s crate registry" -[CocoaPods]: https://cocoapods.org "A dependency manager for Swift and Objective-C Cocoa projects" +[STRIDE]: https://en.wikipedia.org/wiki/STRIDE_(security) "STRIDE (security)" [thundering herd effect]: https://en.wikipedia.org/wiki/Thundering_herd_problem "Thundering herd problem" -[offline cache]: https://yarnpkg.com/features/offline-cache "Offline Cache | Yarn - Package Manager" -[XCFramework]: https://developer.apple.com/videos/play/wwdc2019/416/ "WWDC 2019 Session 416: Binary Frameworks in Swift" -[SE-0272]: https://github.com/apple/swift-evolution/blob/master/proposals/0272-swiftpm-binary-dependencies.md "Package Manager Binary Dependencies" -[transparent log]: https://research.swtch.com/tlog [TOFU]: https://en.wikipedia.org/wiki/Trust_on_first_use "Trust on First Use" -[version-specific-tag-selection]: https://github.com/apple/swift-package-manager/blob/master/Documentation/Usage.md#version-specific-tag-selection "Swift Package Manager - Version-specific Tag Selection" +[transparent log]: https://research.swtch.com/tlog +[typosquatting]: https://en.wikipedia.org/wiki/Typosquatting +[UAX15]: http://www.unicode.org/reports/tr15/ "Unicode Technical Report #15: Unicode Normalization Forms" +[UAX18]: http://www.unicode.org/reports/tr18/ "Unicode Technical Report #18: Unicode Regular Expressions" +[UAX31]: http://www.unicode.org/reports/tr31/ "Unicode Technical Report #31: Unicode Identifier and Pattern Syntax" +[UAX36]: http://www.unicode.org/reports/tr36/ "Unicode Technical Report #36: Unicode Security Considerations" +[UTI]: https://en.wikipedia.org/wiki/Uniform_Type_Identifier [version-specific-manifest-selection]: https://github.com/apple/swift-package-manager/blob/master/Documentation/Usage.md#version-specific-manifest-selection "Swift Package Manager - Version-specific Manifest Selection" -[STRIDE]: https://en.wikipedia.org/wiki/STRIDE_(security) "STRIDE (security)" +[version-specific-tag-selection]: https://github.com/apple/swift-package-manager/blob/master/Documentation/Usage.md#version-specific-tag-selection "Swift Package Manager - Version-specific Tag Selection" +[XCFramework]: https://developer.apple.com/videos/play/wwdc2019/416/ "WWDC 2019 Session 416: Binary Frameworks in Swift" +[xss]: https://en.wikipedia.org/wiki/Cross-site_scripting From 719fb169ecd799dab9bc48206887952155af013b Mon Sep 17 00:00:00 2001 From: Mattt Date: Wed, 24 Mar 2021 09:42:38 -0700 Subject: [PATCH 2/7] Change package identity format to scope.package-name This change was made at the request of the Swift core team ahead of second review --- proposals/0292-package-registry-service.md | 76 +++++++++++----------- 1 file changed, 37 insertions(+), 39 deletions(-) diff --git a/proposals/0292-package-registry-service.md b/proposals/0292-package-registry-service.md index bc5ffdb860..8c3742a3e1 100644 --- a/proposals/0292-package-registry-service.md +++ b/proposals/0292-package-registry-service.md @@ -77,7 +77,7 @@ Swift Package Manager resolves external dependencies in the project's package manifest (`Package.swift`) file that are [declared](#new-packagedescription-apis) with a [scoped package identifier](#package-identity) in the form -`@scope/PackageName`. +`scope.package-name`. These package identifiers resolve potential [module name collisions](#module-name-collision-resolution) across build targets. @@ -137,27 +137,26 @@ distinct packages with similar names and the duplication of the same package under different names. We propose using a scoped identifier -in the form `@scope/PackageName` +in the form `scope.package-name` to identify package dependencies. A *scope* provides a namespace for related packages within a package registry. A package scope consists of -an at-sign (`@`) followed by alphanumeric characters and hyphens. +alphanumeric characters and hyphens. Hyphens may not occur at the beginning or end, nor consecutively within a scope. -The maximum length of a package name is 40 characters. +The maximum length of a package name is 39 characters. A valid package scope matches the following regular expression pattern: ```regexp -\A@[a-zA-Z\d](?:[a-zA-Z\d]|-(?=[a-zA-Z\d])){0,39}\z +\A[a-zA-Z\d](?:[a-zA-Z\d]|-(?=[a-zA-Z\d])){0,39}\z ``` A package's *name* is specified by the `name` provided in its manifest. -The maximum length of a package scope is 128 characters. +The maximum length of a package name is 128 characters. A valid package name matches the following regular expression pattern: - ```regexp \A\p{XID_Start}\p{XID_Continue}{0,127}\z ``` @@ -196,8 +195,8 @@ to declare one or more dependencies by their respective package identifier. ```swift dependencies: [ - .package(id: "@mona/LinkedList", .upToNextMinor(from: "1.1.0")), - .package(id: "@mona/RegEx", .exact("2.0.0")) + .package(id: "mona.LinkedList", .upToNextMinor(from: "1.1.0")), + .package(id: "mona.RegEx", .exact("2.0.0")) ] ``` @@ -211,11 +210,11 @@ but excluding branch-based and commit-based requirements. #### Module name collision resolution Consider a dependency graph that includes both -a package declared with the identifier `@mona/LinkedList` and +a package declared with the identifier `mona.LinkedList` and an equivalent package declared with the URL `https://github.com/mona/LinkedList`. When Swift Package Manager fetches a list of releases for the identified package -(`GET /@mona/LinkedList`), +(`GET /mona/LinkedList`), the response includes a `Link` header field with URLs to that project's source repository that are known to the registry. @@ -227,7 +226,7 @@ Link: ; rel="canonical", Swift Package Manager uses this information to reconcile the URL-based dependency declaration with -the package identifier `@mona/LinkedList`. +the package identifier `mona.LinkedList`. Link relation URLs may also be normalized to mitigate insignificant variations. For example, a package with an ["scp-style" URL][scp-url] like @@ -248,7 +247,7 @@ that is, the `package` parameter in `.product(name:package)` method calls. dependencies: [ .product(name: "LinkedList", - package: "LinkedList") -+ package: "@mona/LinkedList") ++ package: "mona.LinkedList") ] ] ``` @@ -317,7 +316,7 @@ it's saved to `Package.resolved`. "object": { "pins": [ { - "package": "@mona/LinkedList", + "package": "mona.LinkedList", "state": { "checksum": "ed008d5af44c1d0ea0e3668033cae9b695235f18b1a99240b7cf0f3d9559a30d", "version": "1.2.0" @@ -339,7 +338,7 @@ if there's a mismatch in integrity checksums. ```terminal $ swift build -error: checksum of downloaded source archive of dependency '@mona/LinkedList' (c2b934fe66e55747d912f1cfd03150883c4f037370c40ca2ad4203805db79457) does not match checksum specified by the manifest (ed008d5af44c1d0ea0e3668033cae9b695235f18b1a99240b7cf0f3d9559a30d) +error: checksum of downloaded source archive of dependency 'mona.LinkedList' (c2b934fe66e55747d912f1cfd03150883c4f037370c40ca2ad4203805db79457) does not match checksum specified by the manifest (ed008d5af44c1d0ea0e3668033cae9b695235f18b1a99240b7cf0f3d9559a30d) ``` Once the correct checksum is determined, @@ -485,7 +484,7 @@ fail with an error. ```terminal $ swift package resolve -error: cannot resolve dependency '@mona/LinkedList' without a configured registry +error: cannot resolve dependency 'mona.LinkedList' without a configured registry ``` #### Associating a registry with a scope @@ -494,19 +493,19 @@ The user can associate a package scope with a custom registry by passing the `--scope` option. For example, -a user might resolve all packages with the package scope `@example` -(such as `@example/PriorityQueue`) +a user might resolve all packages with the package scope `example` +(such as `example.PriorityQueue`) to a private registry. ```terminal -$ swift package-registry set https://internal.example.com/ --scope @example +$ swift package-registry set https://internal.example.com/ --scope example $ cat .swiftpm/config/registries.json ``` ```json { "registries": { - "@example": { + "example": { "url": "https://internal.example.com" } }, @@ -559,7 +558,7 @@ consider the following global and local registry configuration files: "default": { "url": "https://global.example.com" }, - "@foo": { + "foo": { "url": "https://global.example.com" }, }, @@ -569,7 +568,7 @@ consider the following global and local registry configuration files: // Local configuration (.swiftpm/config/registries.json) { "registries": { - "@foo": { + "foo": { "url": "https://local.example.com" } }, @@ -579,7 +578,7 @@ consider the following global and local registry configuration files: ``` Running the `swift package resolve` command with these configuration files -resolves packages with the `@foo` scope +resolves packages with the `foo` scope using the registry located at "https://local.example.com", and all other packages using the registry located at "https://global.example.com". @@ -621,7 +620,7 @@ that assigns its alternate location. "object": [ { "mirror": "https://github.com/OctoCorp/SwiftLinkedList.git", - "original": "@mona/LinkedList" + "original": "mona.LinkedList" } ], "version": 1 @@ -657,8 +656,7 @@ when deciding where to host and fetch packages. Our proposal's package identity scheme is designed to prevent or mitigate vulnerabilities common to packaging systems and networked applications: -- Package scopes are marked by an at-sign (`@`) prefix - and restricted to a limited set of characters, +- Package scopes are restricted to a limited set of characters, preventing [homograph attacks]. For example, "А" (U+0410 CYRILLIC CAPITAL LETTER A) is an invalid scope character @@ -666,17 +664,17 @@ vulnerabilities common to packaging systems and networked applications: - Package scopes disallow leading, trailing, or consecutive hyphens (`-`), and disallows underscores (`_`) entirely, which mitigates look-alike package scopes - (for example, "@llvm--swift" and "@llvm_swift" are both invalid - and cannot be confused for "@llvm-swift"). + (for example, "llvm--swift" and "llvm_swift" are both invalid + and cannot be confused for "llvm-swift"). - Package scopes disallow dots (`.`), which prevents potential confusion with domain variants of scopes - (for example, "@apple.com" is invalid - and cannot be confused for "@apple"). + (for example, "apple.com" is invalid + and cannot be confused for "apple"). - Packages are registered within a scope, which mitigates [typosquatting]. Package registries may further restrict the assignment of new scopes that are intentionally misleading - (for example, "@G00gle", which looks like "@Google"). + (for example, "G00gle", which looks like "Google"). - Package names disallow punctuation and whitespace characters used in [cross-site scripting][xss] and [CRLF injection][http header injection] attacks. @@ -849,8 +847,8 @@ use [reverse domain name notation] to identify software components (for example, `com.squareup.okhttp3`). We considered these and other schemes for identifying packages, -but they were rejected in favor of a scoped package identity -similar to the one used by [npm]. +but they were rejected in favor of the scoped package identity +described in this proposal. ### Use of `tar` or other archive formats @@ -879,7 +877,7 @@ with their respective identifiers. For example, the declarations `.package(url: "https://github.com/mona/LinkedList", .exact("1.1.0"))` and -`.package(id: "@mona/LinkedList", .exact("1.1.0"))`, +`.package(id: "mona.LinkedList", .exact("1.1.0"))`, must be deemed equivalent to resolve a dependency graph that contains both of them. @@ -1046,12 +1044,12 @@ could be extended to add dependencies using scoped identifiers in addition to URLs. ```terminal -$ swift package add-dependency @mona/LinkedList +$ swift package add-dependency mona.LinkedList # Installed LinkedList 1.2.0 ``` ```diff -+ .package(id: "@mona/LinkedList", .exact("1.2.0")) ++ .package(id: "mona.LinkedList", .exact("1.2.0")) ``` ### Package manifest dependency migration @@ -1066,7 +1064,7 @@ $ swift package-registry migrate ```diff - .package(url: "https://github.com/mona/LinkedList", .exact("1.2.0")) -+ .package(id: "@mona/LinkedList", .exact("1.2.0")) ++ .package(id: "mona.LinkedList", .exact("1.2.0")) ``` ### Security audits @@ -1080,7 +1078,7 @@ information about security advisories. "advisories": [{ "cve": "CVE-20XX-12345", "cwe": "CWE-400", - "package_name": "@mona/LinkedList", + "package": "mona.LinkedList", "vulnerable_versions": "<=1.0.0", "patched_versions": ">1.0.0", "severity": "moderate", @@ -1099,7 +1097,7 @@ $ swift package audit ┌───────────────┬────────────────────────────────────────────────┐ │ High │ Regular Expression Denial of Service │ ├───────────────┼────────────────────────────────────────────────┤ -│ Package │ @mona/RegEx │ +│ Package │ mona.RegEx │ ├───────────────┼────────────────────────────────────────────────┤ │ Dependency of │ PatternMatcher │ ├───────────────┼────────────────────────────────────────────────┤ From 9503099028a823b0bc00aee1a3b7bf2e06f165e1 Mon Sep 17 00:00:00 2001 From: Mattt Date: Wed, 24 Mar 2021 16:26:24 -0700 Subject: [PATCH 3/7] Mention SE-0305 and possibility of distributing artifact archives --- proposals/0292-package-registry-service.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/proposals/0292-package-registry-service.md b/proposals/0292-package-registry-service.md index 8c3742a3e1..93313058d7 100644 --- a/proposals/0292-package-registry-service.md +++ b/proposals/0292-package-registry-service.md @@ -1010,8 +1010,8 @@ described in the previous section. ### Binary framework distribution -The package registry specification could be amended -to support distributing packages as [XCFramework] bundles. +The registry specification could be amended to support the distribution of +[XCFramework] bundles or [artifact archives][SE-0305]. ```http GET /github.com/mona/LinkedList/1.1.1.xcframework HTTP/1.1 @@ -1167,6 +1167,7 @@ RegEx (github.com/mona/RegEx) - Expressions on the reg. [SE-0219]: https://github.com/apple/swift-evolution/blob/master/proposals/0219-package-manager-dependency-mirroring.md "Package Manager Dependency Mirroring" [SE-0272]: https://github.com/apple/swift-evolution/blob/master/proposals/0272-swiftpm-binary-dependencies.md "Package Manager Binary Dependencies" [SE-0301]: https://github.com/apple/swift-evolution/blob/main/proposals/0301-package-editing-commands.md "Package Editor Commands" +[SE-0305]: https://github.com/apple/swift-evolution/blob/main/proposals/0305-swiftpm-binary-target-improvements.md "Package Manager Binary Target Improvements" [secret scanning]: https://docs.github.com/en/github/administering-a-repository/about-secret-scanning [SemVer]: https://semver.org/ "Semantic Versioning" [SoftwareSourceCode]: https://schema.org/SoftwareSourceCode From c9022045683aca6bcd73fc7db4db874c9f56d214 Mon Sep 17 00:00:00 2001 From: Mattt Date: Wed, 24 Mar 2021 16:28:04 -0700 Subject: [PATCH 4/7] Rename section 'Module name collision resolution' to 'Package name collision resolution' --- proposals/0292-package-registry-service.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/0292-package-registry-service.md b/proposals/0292-package-registry-service.md index 93313058d7..68ae04e038 100644 --- a/proposals/0292-package-registry-service.md +++ b/proposals/0292-package-registry-service.md @@ -207,7 +207,7 @@ that provides the same interface as `Package.Dependency.Requirement` for version-based requirements, but excluding branch-based and commit-based requirements. -#### Module name collision resolution +#### Package name collision resolution Consider a dependency graph that includes both a package declared with the identifier `mona.LinkedList` and @@ -930,7 +930,7 @@ lays the groundwork for several useful features. ### Package dependency URL normalization -As described in ["Module name collision resolution"](#module-name-collision-resolution) +As described in ["Package name collision resolution"](#package-name-collision-resolution) Swift Package Manager cannot build a project if two or more packages in the project are located by URLs with the same (case-insensitive) last path component. From 9158f9fdc6552ada119ef6ec5c60f827fa34b387 Mon Sep 17 00:00:00 2001 From: Mattt Date: Thu, 25 Mar 2021 11:12:37 -0700 Subject: [PATCH 5/7] .swiftpm/config is now a directory This change was made at the request of the Swift core team ahead of second review --- proposals/0292-package-registry-service.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/proposals/0292-package-registry-service.md b/proposals/0292-package-registry-service.md index 68ae04e038..3f804e8eb8 100644 --- a/proposals/0292-package-registry-service.md +++ b/proposals/0292-package-registry-service.md @@ -535,7 +535,7 @@ OPTIONS: Running the `package-registry unset` subcommand in the root directory of a package -updates the `.swiftpm/config` file +updates the `.swiftpm/config/registries.json` file to remove the `default` entry in the top-level `registries` key, if present. If a `--scope` option is passed, only the entry for the specified scope is removed, if present. @@ -590,7 +590,7 @@ in descending order of precedence: * The package manifest in the current directory (`./Package.swift`) * Any existing lock file (`./Package.resolved`) -* Any local configuration (`./.swiftpm/config`) +* Any local configuration (`./.swiftpm/config/registries.json`) * Any global configuration file (`~/.swiftpm/config/registries.json`) ### Changes to config subcommand @@ -609,7 +609,7 @@ $ swift package config set-mirror \ This proposal updates the `swift package config set-mirror` subcommand to accept a `--package-identifier` option in place of an `--original-url`. Running this subcommand with a `--package-identifier` option -creates or updates the `.swiftpm/config` file, +creates or updates the `.swiftpm/config/mirrors.json` file, modifying the array associated with the top-level `object` key to add a new entry or update an existing entry for the specified package identifier, @@ -775,7 +775,7 @@ can both provide similar non-repudiation guarantees. A user may inadvertently reveal the existence of a private registry or expose hardcoded credentials -by checking in their project's `.swiftpm/config` file. +by checking in their project's `.swiftpm/config` directory. An attacker could scrape public code repositories for `.swiftpm/config` files and attempt to reuse those credentials to impersonate the user. @@ -812,7 +812,8 @@ is logged as `https://***@swift.pkg.github.com`). ### Denial of service An attacker could scrape public code repositories -for `.swiftpm/config` files that declare one or more custom registries +for `.swiftpm/config/registries.json` files +that declare one or more custom registries and launch a denial-of-service attack in an attempt to reduce the availability of those resources. From 0193e4920bf33182c1c24160d98b98311472dbb1 Mon Sep 17 00:00:00 2001 From: Mattt Date: Thu, 25 Mar 2021 11:24:12 -0700 Subject: [PATCH 6/7] Add HTTP client implementation details under 'Dependency graph resolution' This change was made at the request of the Swift core team ahead of second review --- proposals/0292-package-registry-service.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/proposals/0292-package-registry-service.md b/proposals/0292-package-registry-service.md index 3f804e8eb8..490f57591e 100644 --- a/proposals/0292-package-registry-service.md +++ b/proposals/0292-package-registry-service.md @@ -274,14 +274,18 @@ that adopts `PackageContainer` and performs equivalent operations with HTTP requests to a registry service. These client-server interactions are facilitated by a new `RegistryManager` type. +When requesting resources from a registry, +Swift Package Manager will employ techniques like +exponential backoff, circuit breakers, and client-side validation +to safeguard against adverse network conditions and malicious server responses. The following table lists the tasks performed by Swift Package Manager during dependency resolution alongside the Git operations used and their corresponding package registry API calls. -| Task | Git operation | Registry request | -| ------------------------------------- | --------------------------- | ---------------------------------------------------- | +| Task | Git operation | Registry request | +| ------------------------------------- | --------------------------- | --------------------------------------------- | | Fetch the contents of a package | `git clone && git checkout` | `GET /{scope}/{name}/{version}.zip` | | List the available tags for a package | `git tag` | `GET /{scope}/{name}` | | Fetch a package manifest | `git clone` | `GET /{scope}/{name}/{version}/Package.swift` | From 3349caaa629e2556acb681185a9f4e978a98b7d9 Mon Sep 17 00:00:00 2001 From: Mattt Date: Thu, 25 Mar 2021 11:43:24 -0700 Subject: [PATCH 7/7] A package's *name* uniquely identifies a package in a scope This change was made at the request of the Swift core team ahead of second review --- proposals/0292-package-registry-service.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/proposals/0292-package-registry-service.md b/proposals/0292-package-registry-service.md index 490f57591e..99185ac74d 100644 --- a/proposals/0292-package-registry-service.md +++ b/proposals/0292-package-registry-service.md @@ -152,8 +152,7 @@ A valid package scope matches the following regular expression pattern: \A[a-zA-Z\d](?:[a-zA-Z\d]|-(?=[a-zA-Z\d])){0,39}\z ``` -A package's *name* is specified by the `name` provided in its manifest. - +A package's *name* uniquely identifies a package in a scope. The maximum length of a package name is 128 characters. A valid package name matches the following regular expression pattern: