diff --git a/src/SUMMARY.md b/src/SUMMARY.md index dc7828d..6697802 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -8,6 +8,11 @@ - [Strategical decision making in a DependencyProvider](./pubgrub_crate/strategy.md) - [Solution and error reporting](./pubgrub_crate/solution.md) - [Writing your own error reporting logic](./pubgrub_crate/custom_report.md) +- [Advanced usage and limitations](./limitations/intro.md) + - [Optional dependencies](./limitations/optional_deps.md) + - [Allowing multiple versions of a package](./limitations/multiple_versions.md) + - [Versions in a continuous space](./limitations/continuous_versions.md) + - [Pre-release versions](./limitations/prerelease_versions.md) - [Internals of the PubGrub algorithm](./internals/intro.md) - [Overview of the algorithm](./internals/overview.md) - [Terms](./internals/terms.md) diff --git a/src/limitations/continuous_versions.md b/src/limitations/continuous_versions.md new file mode 100644 index 0000000..dacba76 --- /dev/null +++ b/src/limitations/continuous_versions.md @@ -0,0 +1,82 @@ +# Versions in a continuous space + +The current design of pubgrub exposes a `Version` trait demanding two properties, (1) that there exists a lowest version, and (2) that versions live in a discrete space where the successor of each version is known. +So versions are basically isomorph with N^n, where N is the set of natural numbers. + +## The successor design + +There is a good reason why we started with the successor design for the `Version` trait. +When building knowledge about the dependency system, pubgrub needs to compare sets of versions, and to perform common set operations such as intersection, union, inclusion, comparison for equality etc. +In particular, given two sets of versions S1 and S2, it needs to be able to answer if S1 is a subset of S2 (S1 ⊂ S2). +And we know that S1 ⊂ S2 if and only if S1 ∩ S2 == S1. +So checking for subsets can be done by checking for the equality between two sets of versions. +Therefore, **sets of versions need to have unique canonical representations to be comparable**. + +We have the interesting property that we require `Version` to have a total order. +As a consequence, the most adequate way to represent sets of versions with a total order, is to use a sequence of non intersecting segments, such as `[0, 3] ∪ ]5, 9[ ∪ [42, +∞[`. + +> Notation: we note segments with close or open brackets depending on if the value at the frontier is included or excluded of the interval. +> It is also common to use a parenthesis for open brackets. +> So `[0, 14[` is equivalent to `[0, 14)` in that other notation. + +The previous set is for example composed of three segments, +- the closed segment [0, 3] containing versions 0, 1, 2 and 3, +- the open segment ]5, 9[ containing versions 6, 7 and 8, +- the semi-open segment [42, +∞[ containing all numbers above 42. + +For the initial design, we did not want to have to deal with close or open brackets on both interval bounds. +Since we have a lowest version, the left bracket of segments must be closed to be able to contain that lowest version. +And since `Version` does not impose any upper bound, we need to use open brackets on the right side of segments. +So our previous set thus becomes: `[0, ?[ ∪ [?, 9[ ∪ [42, +∞[`. +But the question now is what do we use in place of the 3 in the first segment and in place of the 5 in the second segment. +This is the reason why we require the `bump()` method on the `Version` trait. +If we know the next version, we can replace 3 by bump(3) == 4, and 5 by bump(5) == 6. +We finally get the following representation `[0, 4[ ∪ [6, 9[ ∪ [42, +∞[`. +And so the `Range` type is defined as follows. + +```rust +pub struct Range { + segments: Vec>, +} +type Interval = (V, Option); +// set = [0, 4[ ∪ [6, 9[ ∪ [42, +∞[ +let set = vec![(0, Some(4)), (6, Some(9)), (42, None)]; +``` + +## The bounded interval design + +We may want however to have versions live in a continuous space. +For example, if we want to use fractions, we can always build a new fraction between two others. +As such it is impossible to define the successor of a fraction version. + +We are currently investigating the use of bounded intervals to enable continuous spaces for versions. +If it happens, this will only be in the next major release of pubgrub, probably 0.3. +The current experiments look like follows. + +```rust +/// New trait for versions. +/// Bound is core::ops::Bound. +pub trait Version: Clone + Ord + Debug + Display { + /// Returns the minimum version. + fn minimum() -> Bound; + /// Returns the maximum version. + fn maximum() -> Bound; +} + +/// An interval is a bounded domain containing all values +/// between its starting and ending bounds. +/// RangeBounds is core::ops::RangeBounds. +pub trait Interval: RangeBounds + Debug + Clone + Eq + PartialEq { + /// Create an interval from its starting and ending bounds. + /// It's the caller responsability to order them correctly. + fn new(start_bound: Bound, end_bound: Bound) -> Self; +} + +/// The new Range type is composed of bounded intervals. +pub struct Range { + segments: Vec, +} +``` + +It is certain though that the flexibility of enabling usage of continuous spaces will come at a performance price. +We just have to evaluate how much it costs and if it is worth sharing a single implementation, or having both a discrete and a continuous implementation. diff --git a/src/limitations/intro.md b/src/limitations/intro.md new file mode 100644 index 0000000..7879932 --- /dev/null +++ b/src/limitations/intro.md @@ -0,0 +1,23 @@ +# Advanced usage and limitations + +By design, the current implementation of PubGrub is rather well suited to handle a dependency system with the following constraints: + +1. Packages are uniquely identified. +2. Versions are in a discrete set, with a total order. +3. The successor of a given version is always uniquely defined. +4. Dependencies of a package version are fixed. +5. Exactly one version must be selected per package depended on. + +The fact that packages are uniquely identified (1) is perhaps the only constraint that makes sense for all common dependency systems. +But for the rest of the constraints, they are all inadequate for some common real-world dependency systems. +For example, it's possible to have dependency systems where order is not required for versions (2). +In such systems, dependencies must be specified with exact sets of compatible versions, and bounded ranges make no sense. +Being able to uniquely define the successor of any version (3) is also a constraint that is not a natural fit if versions have a system of pre-releases. +Indeed, what is the successor of `2.0.0-alpha`? +We can't tell if that is `2.0.0` or `2.0.0-beta` or `2.0.0-whatever`. +Having fixed dependencies (4) is also not followed in programming languages allowing optional dependencies. +In Rust packages, optional dependencies are called "features" for example. +Finally, restricting solutions to only one version per package (5) is also too constraining for dependency systems allowing breaking changes. +In cases where packages A and B both depend on different ranges of package C, we sometimes want to be able to have a solution where two versions of C are present, and let the compiler decide if their usages of C in the code are compatible. + +In the following subsections, we try to show how we can circumvent those limitations with clever usage of dependency providers. diff --git a/src/limitations/multiple_versions.md b/src/limitations/multiple_versions.md new file mode 100644 index 0000000..d49d4ac --- /dev/null +++ b/src/limitations/multiple_versions.md @@ -0,0 +1,253 @@ +# Allowing multiple versions of a package + +One of the main goals of PubGrub is to pick one version per package depended on, under the assumption that at most one version per package is allowed. +Enforcing this assumption has two advantages. +First, it prevents data and functions interactions of the same library at different, potentially incompatible versions. +Second, it reduces the code size and therefore also the disk usage and the compilation times. + +However, under some circonstances, we may relax the "single version allowed" constraint. +Imagine that our package "a" depends on "b" and "c", and "b" depends on "d" @ 1, and "c" depends on "d" @ 2, with versions 1 and 2 of "d" incompatible with each other. +With the default rules of PubGrub, that system has no solution and we cannot build "a". +Yet, if our usages of "b" and "c" are independant with regard to "d", we could in theory build "b" with "d" @ 1 and build "c" with "d" @ 2. + +Such situation is sufficiently common that most package managers allow multiple versions of a package. +In Rust for example, multiple versions are allowed as long as they cross a major frontier, so 0.7 and 0.8 or 2.0 and 3.0. +So the question now is can we circumvent this fundamental restriction of PubGrub? +The short answer is yes, with buckets and proxies. + +## Buckets + +We saw that implementing optional dependencies required the creation of on-demand feature packages, which are virtual packages created for each optional feature. +In order to allow for multiple versions of the same package to coexist, we are also going to play smart with packages. +Indeed, if we want two versions to coexist, there is only one possibility, which is that they come from different packages. +And so we introduce the concept of buckets. +A package bucket is a set of versions of the same package that cannot coexist in the solution of a dependency system, basically just like before. +So for Rust crates, we can define one bucket per major frontier, such as one bucket for 0.1, one for 0.2, ..., one for 1.0, one for 2.0, etc. +As such, versions 0.7.0 and 0.7.3 would be in the same bucket, but 0.7.0 and 0.8.0 would be in different buckets and could coexist. + +Are buckets enought? Not quite, since once we introduce buckets, we also need a way to depend on multiple buckets alternatively. +Indeed, most packages should have dependencies on a single bucket, because it doesn't make sense to depend on potentially incompatible versions at the same time. +But rarely, dependencies are written such that they can depend on multiple buckets, such as if one write `v >= 2.0`. +Then, any version from the 2.0 bucket would satisfy it, as well as any version from the 3.0 or any other later bucket. +Thus, we cannot write "a depends on b in bucket 2.0". +So how do we write dependencies of "a"? +That's where we introduce the concept of proxies. + +## Proxies + +A proxy package is an intermediate on-demand package, placed just between one package and one of its dependencies. +So if we need to express that package "a" has a dependency on package "b" for different buckets, we create the intermediate proxy package "a->b". +Then we can say instead that package "a" depends on any version of the proxy package "a->b". +And then, we create one proxy version for each bucket. +So if there exists the following five buckets for "b", 0.1, 0.2, 1.0, 2.0, 3.0, we create five corresponding versions for the proxy package "a->b". +And since our package "a" depends on any version of the proxy package "a->b", it will be satisfied as soon as any bucket of "b" has a version picked. + +## Example + +We will consider versions in the form `major.minor` with a major component starting at 1, and a minor component starting at 0. +The smallest version is 1.0, and each major component represents a bucket. + +> Note that we could start versions at 0.0, but since different dependency system tends to interpret versions before 1.0 differently, we will simply avoid that problem by saying versions start at 1.0. + +For convenience, we will use a string notation for proxies and buckets. +Buckets will be indicated by a "#", so "a#1" is the 1.0 bucket of package "a", and "a#2" is the 2.0 bucket of package "a". +And we will use "@" to denote exact versions, so "a" @ 1.0 means package "a" at version 1.0. +Proxies will be represented by the arrow "->" as previously mentioned. +Since a proxy is tied to a specific version of the initial package, we also use the "@" symbol in the name of the proxy package. +For example, if "a" @ 1.4 depends on "b", we create the proxy package "a#1@1.4->b". +It's a bit of a mouthful, but once you learn how to read it, it makes sense. + +> Note that we might be tempted to just remove the version part of the proxy, so "a#1->b" instead of "a#1@1.4->b". +> However, we might have "a" @ 1.4 depending on "b" in range `v >= 2.2` and have "a" @ 1.5 depending on "b" in range `v >= 2.6`. +> Both dependencies would map to the same "a#1->b" proxy package, but we would not know what to put in the dependency of "a#1->b" to the "b#2" bucket. +> Should it be "2.2 <= v < 3.0" as in "a" @ 1.4, or should it be "2.6 <= v < 3.0" as in "a" @ 1.5? +> That is why each proxy package is tied to an exact version of the source package. + +Let's consider the following example, with "a" being our root package. +- "a" @ 1.4 depends on "b" in range `1.1 <= v < 2.9` +- "b" @ 1.3 depends on "c" at version `1.1` +- "b" @ 2.7 depends on "d" at version `3.1` + +Using buckets and proxies, we can rewrite this dependency system as follows. +- "a#1" @ 1.4 depends on "a#1@1.4->b" at any version (we use the proxy). +- "a#1@1.4->b" proxy exists in two versions, one per bucket of "b". +- "a#1@1.4->b" @ 1.0 depends on "b#1" in range `1.1 <= v < 2.0` (the first bucket intersected with the dependency range). +- "a#1@1.4->b" @ 2.0 depends on "b#2" in range `2.0 <= v < 2.9` (the second bucket intersected with the dependency range). +- "b#1" @ 1.3 depends on "c#1" at version `1.1`. +- "b#2" @ 2.7 depends on "d#3" at version `3.1`. + +There are potentially two solutions to that system. +The one with the newest versions is the following. +- "a#1" @ 1.4 +- "a#1@1.4->b" @ 2.0 +- "b#2" @ 2.7 +- "d#3" @ 3.1 + +Finally, if we want to express the solution in terms of the original packages, we just have to remove the proxy packages from the solution. + +## Example implementation + +A complete example implementation of this extension allowing multiple versions is available in the [`allow-multiple-versions` crate of the `advanced_dependency_providers` repository][multiple-versions-crate]. +In that example, packages are of the type `String` and versions of the type `SemanticVersion` defined in pubgrub, which does not account for pre-releases, just the (Major, Minor, Patch) format of versions. + +[multiple-versions-crate]: https://github.com/pubgrub-rs/advanced_dependency_providers/tree/main/allow-multiple-versions + +### Defining an index of packages + +Inside the `index.rs` module, we define a very basic `Index`, holding all packages known, as well as a helper function `add_deps` easing the writing of tests. + +```rust +/// Each package is identified by its name. +pub type PackageName = String; +/// Alias for dependencies. +pub type Deps = Map>; + +/// Global registry of known packages. +pub struct Index { + /// Specify dependencies of each package version. + pub packages: Map>, +} + +// Initialize an empty index. +let mut index = Index::new(); +// Add package "a" to the index at version 1.0.0 with no dependency. +index.add_deps::("a", (1, 0, 0), &[]); +// Add package "a" to the index at version 2.0.0 with a dependency to "b" at versions >= 1.0.0. +index.add_deps("a", (2, 0, 0), &[("b", (1, 0, 0)..)]); +``` + +### Implementing a dependency provider for the index + +Since our `Index` is ready, we now have to implement the `DependencyProvider` trait for it. +As explained previously, we'll need to differenciate packages representing buckets and proxies, so we define the following new `Package` type. + +```rust +/// A package is either a bucket, or a proxy between a source and a target package. +pub enum Package { + /// "a#1" + Bucket(Bucket), + /// source -> target + Proxy { + source: (Bucket, SemVer), + target: String, + }, +} + +/// A bucket corresponds to a given package, and match versions +/// in a range identified by their major component. +pub struct Bucket { + pub name: String, // package name + pub bucket: u32, // 1 maps to the range 1.0.0 <= v < 2.0.0 +} +``` + +In order to implement the first required method, `choose_package_version`, we simply reuse the `choose_package_with_fewest_versions` helper function provided by pubgrub. +That one requires a list of available versions for each package, so we have to create that list. +As explained previously, listing the existing (virtual) versions depend on if the package is a bucket or a proxy. +For a bucket package, we simply need to retrieve the original versions and filter out those outside of the bucket. + +```rust +match package { + Package::Bucket(p) => { + let bucket_range = Range::between((p.bucket, 0, 0), (p.bucket + 1, 0, 0)); + self.available_versions(&p.name) + .filter(|v| bucket_range.contains(*v)) + } + ... +``` + +If the package is a proxy however, there is one version per bucket in the target of the proxy. + +```rust +match package { + Package::Proxy { target, .. } => { + bucket_versions(self.available_versions(&target)) + } + ... +} + +/// Take a list of versions, and output a list of the corresponding bucket versions. +/// So [1.1, 1.2, 2.3] -> [1.0, 2.0] +fn bucket_versions( + versions: impl Iterator +) -> impl Iterator { ... } +``` + +Additionally, we can filter out buckets that are outside of the dependency range in the original dependency leading to that proxy package. +Otherwise it will add wastefull computation to the solver, but we'll leave that out of this walkthrough. + +The `get_dependencies` method is slightly hairier to implement, so instead of all the code, we will just show the structure of the function in the happy path, with its comments. + +```rust +fn get_dependencies( + &self, + package: &Package, + version: &SemVer, +) -> Result, ...> { + let all_versions = self.packages.get(package.pkg_name()); + ... + match package { + Package::Bucket(pkg) => { + // If this is a bucket, we convert each original dependency into + // either a dependency to a bucket package if the range is fully contained within one bucket, + // or a dependency to a proxy package at any version otherwise. + let deps = all_versions.get(version); + ... + let pkg_deps = deps.iter().map(|(name, range)| { + if let Some(bucket) = single_bucket_spanned(range) { + ... + (Package::Bucket(bucket_dep), range.clone()) + } else { + ... + (proxy, Range::any()) + } + }) + .collect(); + Ok(Dependencies::Known(pkg_deps)) + } + Package::Proxy { source, target } => { + // If this is a proxy package, it depends on a single bucket package, the target, + // at a range of versions corresponding to the bucket range of the version asked, + // intersected with the original dependency range. + let deps = all_versions.get(&source.1); + ... + let mut bucket_dep = Map::default(); + bucket_dep.insert( + Package::Bucket(Bucket { + name: target.clone(), + bucket: target_bucket, + }), + bucket_range.intersection(target_range), + ); + Ok(Dependencies::Known(bucket_dep)) + } + } +} + +/// If the range is fully contained within one bucket, +/// this returns that bucket identifier, otherwise, it returns None. +fn single_bucket_spanned(range: &Range) -> Option { ... } +``` + +That's all! +The implementation also contains tests, with helper functions to build them. +Here is the test corresponding to the example we presented above. + +```rust +#[test] +/// Example in guide. +fn success_when_simple_version() { + let mut index = Index::new(); + index.add_deps("a", (1, 4, 0), &[("b", (1, 1, 0)..(2, 9, 0))]); + index.add_deps("b", (1, 3, 0), &[("c", (1, 1, 0)..(1, 1, 1))]); + index.add_deps("b", (2, 7, 0), &[("d", (3, 1, 0)..(3, 1, 1))]); + index.add_deps::("c", (1, 1, 0), &[]); + index.add_deps::("d", (3, 1, 0), &[]); + assert_map_eq( + &resolve(&index, "a#1", (1, 4, 0)).unwrap(), + &select(&[("a#1", (1, 4, 0)), ("b#2", (2, 7, 0)), ("d#3", (3, 1, 0))]), + ); +} +``` + +Implementing a dependency provider allowing both optional dependencies and multiple versions per package is left to the reader as an exercise (I've always wanted to say that). diff --git a/src/limitations/optional_deps.md b/src/limitations/optional_deps.md new file mode 100644 index 0000000..dff7303 --- /dev/null +++ b/src/limitations/optional_deps.md @@ -0,0 +1,220 @@ +# Optional dependencies + +Sometimes, we want the ability to express that some functionalities are optional. +Since those functionalities may depend on other packages, we also want the ability to mark those dependencies optional. + +Let's imagine a simple scenario in which we are developing package "A" and depending on package "B", itself depending on "C". +Now "B" just added new functionalities, very convenient but also very heavy. +For architectural reasons, they decided to release them as optional features of the same package instead of in a new package. +To enable those features, one has to activate the "heavy" option of package "B", bringing a new dependency to package "H". +We will mark optional features in dependencies with a slash separator "/". +So we have now "A" depends on "B/heavy" instead of previously "A" depends on "B". +And the complete dependency resolution is thus now ["A", "B/heavy", "C", "H"]. + +But what happens if our package "A" start depending on another package "D", which depends on "B" without the "heavy" option? +We would now have ["A", "B", "B/heavy", "C", "D", "H"]. +Is this problematic? +Can we conciliate the fact that both "B" and "B/heavy" are in the dependencies? + +## Strictly additive optional dependencies + +The most logical solution to this situation is to require that optional features and dependencies are strictly additive. +Meaning "B/heavy" is entirely compatible with "B" and only brings new functions and dependencies. +"B/heavy" cannot change dependencies of "B" only adding new ones. +Once this hypothesis is valid for our dependency system, we can model "B/heavy" as a different package entirely, depending on both "B" and "H", leading to the solution ["A", "B", "B/heavy", "C", "D", "H"]. +Whatever new optional features that get added to "B" can similarly be modeled by a new package "B/new-feat", also depending on "B" and on its own new dependencies. +When dependency resolution ends, we can gather all features of "B" that were added to the solution and compile "B" with those. + +## Dealing with versions + +In the above example, we eluded versions and only talked about packages. +Adding versions to the mix actually does not change anything, and solves the optional dependencies problem very elegantly. +The key point is that an optional feature package, such as "B/heavy", would depend on its base package, "B", exactly at the same version. +So if the "heavy" option of package "B" at version v = 3 brings a dependency to "H" at v >= 2, then we can model dependencies of "B/heavy" at v = 3 by ["B" @ v = 3, "H" @ v >= 2]. + +## Example implementation + +A complete example implementation is available in the [`optional-deps` crate of the `advanced_dependency_providers` repository][optional-deps-crate]. +Let's give an explanation of that implementation. +For the sake of simplicity, we will consider packages of type `String` and versions of type `NumberVersion`, which is just a newtype around `u32` implementing the `Version` trait. + +### Defining an index of packages + +We define an `Index`, storing all dependencies (`Deps`) of every package version in a double map, first indexed by package, then by version. + +```rust +// Use NumberVersion, which are simple u32 for the versions. +use pubgrub::version::NumberVersion as Version; +/// Each package is identified by its name. +pub type PackageName = String; + +/// Global registry of known packages. +pub struct Index { + /// Specify dependencies of each package version. + pub packages: Map>, +} +``` + +Dependencies listed in the `Index` include both mandatory and optional dependencies. +Optional dependencies are identified, grouped, and gated by an option called a "feature". + +```rust +pub type Feature = String; + +pub struct Deps { + /// The regular, mandatory dependencies. + pub mandatory: Map, + /// The optional, feature-gated dependencies. + pub optional: Map>, +} +``` + +Finally, each dependency is specified with a version range, and with a set of activated features. + +```rust +pub struct Dep { + /// The range dependended upon. + pub range: Range, + /// The activated features for that dependency. + pub features: Set, +} +``` + +For convenience, we added the `add_deps` and `add_feature` functions to help building an index in the tests. + +```rust +let mut index = Index::new(); +// Add package "a" to the index at version 0 with no dependency. +index.add_deps("a", 0, &[]); +// Add package "a" at version 1 with a dependency to "b" at version "v >= 1". +index.add_deps("a", 1, &[("b", 1.., &[])]); +// Add package "c" at version 1 with a dependency to "d" at version "v < 4" with the feature "feat". +index.add_deps("c", 1, &[("d", ..4, &["feat"])]); +// Add feature "feat" to package "d" at version 1 with the optional dependency to "f" at version "v >= 1". +// If "d" at version 1 does not exist yet in the index, this also adds it with no mandatory dependency, +// as if we had called index.add_deps("d", 1, &[]). +index.add_feature("d", 1, "feat", &[("f", 1.., &[])]); +``` + +### Implementing a dependency provider for the index + +Now that our `Index` is ready, let's implement the `DependencyProvider` trait on it. +As we explained before, we'll need to differenciate optional features from base packages, so we define a new `Package` type. + +```rust +/// A package is either a base package like "a", +/// or a feature package, corresponding to a feature associated to a base package. +pub enum Package { + Base(String), + Feature { base: String, feature: String }, +} +``` + +Let's implement the first function required by a dependency provider, `choose_package_version`. +For that we defined the `base_pkg()` method on a `Package` that returns the string of the base package. +And we defined the `available_versions()` method on an `Index` to list existing versions of a given package. +Then we simply called the `choose_package_with_fewest_versions` helper function provided by pubgrub. + +```rust +fn choose_package_version, U: Borrow>>( + &self, + potential_packages: impl Iterator, +) -> Result<(T, Option), Box> { + Ok(pubgrub::solver::choose_package_with_fewest_versions( + |p| self.available_versions(p.base_pkg()).cloned(), + potential_packages, + )) +} +``` + +This was very straightforward. +Implementing the second function, `get_dependencies`, requires just a bit more work but is also quite easy. +The exact complete version is available in the code, but again, for the sake of simplicity and readability, let's just deal with the happy path. + +```rust +fn get_dependencies( + &self, + package: &Package, + version: &Version, +) -> Result, ...> { + // Retrieve the dependencies in the index. + let deps = + self.packages + .get(package.base_pkg()).unwrap() + .get(version).unwrap(); + + match package { + // If we asked for a base package, we simply return the mandatory dependencies. + Package::Base(_) => Ok(Dependencies::Known(from_deps(&deps.mandatory))), + // Otherwise, we concatenate the feature deps with a dependency to the base package. + Package::Feature { base, feature } => { + let feature_deps = deps.optional.get(feature).unwrap(); + let mut all_deps = from_deps(feature_deps); + all_deps.insert( + Package::Base(base.to_string()), + Range::exact(version.clone()), + ); + Ok(Dependencies::Known(all_deps)) + }, + } +} +``` + +Quite easy right? +The helper function `from_deps` is where each dependency is transformed from its `String` form into its `Package` form. +In pseudo-code (not compiling, but you get how it works) it looks like follows. + +```rust +/// Helper function to convert Index deps into what is expected by the dependency provider. +fn from_deps(deps: &Map) -> DependencyConstraints { + deps.iter().map(create_feature_deps) + .flatten() + .collect() +} + +fn create_feature_deps( + (base_pkg, dep): (&String, &Dep) +) -> impl Iterator)> { + if dep.features.is_empty() { + // If the dependency has no optional feature activated, + // we simply add a dependency to the base package. + std::iter::once((Package::Base(base_pkg), dep.range)) + } else { + // Otherwise, we instead add one dependency per activated feature, + // but not to the base package itself (we could but no need). + dep.features.iter().map(|feat| { + (Package::Feature { base: base_pkg, feature: feat }, dep.range) + }) + } +} +``` + +We now have implemented the `DependencyProvider` trait. +The only thing left is testing it with example dependencies. +For that, we setup few helper functions and we wrote some tests, that you can run with a call to `cargo test --lib`. +Below is one of those tests. + +```rust +#[test] +fn success_when_multiple_features() { + let mut index = Index::new(); + index.add_deps("a", 0, &[("b", .., &["feat1", "feat2"])]); + index.add_feature("b", 0, "feat1", &[("f1", .., &[])]); + index.add_feature("b", 0, "feat2", &[("f2", .., &[])]); + index.add_deps::("f1", 0, &[]); + index.add_deps::("f2", 0, &[]); + assert_map_eq( + &resolve(&index, "a", 0).unwrap(), + &select(&[ + ("a", 0), + ("b", 0), + ("b/feat1", 0), + ("b/feat2", 0), + ("f1", 0), + ("f2", 0), + ]), + ); +} +``` + +[optional-deps-crate]: https://github.com/pubgrub-rs/advanced_dependency_providers/tree/main/optional-deps diff --git a/src/limitations/prerelease_versions.md b/src/limitations/prerelease_versions.md new file mode 100644 index 0000000..b8d20fb --- /dev/null +++ b/src/limitations/prerelease_versions.md @@ -0,0 +1,77 @@ +# Pre-release versions + +Pre-releasing is a very common pattern in the world of versioning. +It is however one of the worst to take into account in a dependency system, and I highly recommend that if you can avoid introducing pre-releases in your package manager, you should. +In the context of pubgrub, pre-releases break two fondamental properties of the solver. + +1. Pre-releases act similar to continuous spaces. +2. Pre-releases break the mathematical properties of subsets in a space with total order. + +(1) Indeed, it is hard to answer what version comes after "1-alpha0". +Is it "1-alpha1", "1-beta0", "2"? +In practice, we could say that the version that comes after "1-alpha0" is "1-alpha0?" where the "?" character is chosen to be the lowest character in the lexicographic order, but we clearly are on a stretch here and it certainly isn't natural. + +(2) Pre-releases are often semantically linked to version constraints written by humans, interpreted differently depending on context. +For example, "2.0.0-beta" is meant to exist previous to version "2.0.0". +Yet, it is not supposed to be contained in the set described by `1.0.0 <= v < 2.0.0`, and only within sets where one of the bounds contains a pre-release marker such as `2.0.0-alpha <= v < 2.0.0`. +This poses a problem to the dependency solver because of backtracking. +Indeed, the PubGrub algorithm relies on knowledge accumulated all along the propagation of the solver front. +And this knowledge is composed of facts, that are thus never removed even when backtracking happens. +Those facts are called incompatibilities and more info about those is available in the "Internals" section of the guide. +The problem is that if a fact recalls that there is no version within the `1.0.0 <= v < 2.0.0` range, backtracking to a situation where we ask for a version within `2.0.0-alpha <= v < 2.0.0` will return nothing even without checking if a pre-release exists in that range. +And this is one of the fundamental mechanisms of the algorithm, so we should not try to alter it. + +Point (2) is probably the reason why some pubgrub implementations have issues dealing with pre-releases when backtracking, as can be seen in [an issue of the dart implementation][dart-prerelease-issue]. + +[dart-prerelease-issue]: https://github.com/dart-lang/pub/pull/3038 + +## Playing again with packages? + +In the light of the "bucket" and "proxies" scheme we introduced in the section about allowing multiple versions per package, I'm wondering if we could do something similar for pre-releases. +Normal versions and pre-release versions would be split into two subsets, each attached to a different bucket. +In order to make this work, we would need a way to express negative dependencies. +For example, we would want to say: "a" depends on "b" within the (2.0, 3.0) range and is incompatible with any pre-release version of "b". +The tool to express such dependencies is already available in the form of `Term` which can be a `Positive` range or a `Negative` one. +We would have to adjust the API for the `get_dependencies` method to return terms instead of a ranges. +This may have consequences on other parts of the algorithm and should be thoroughly tested. + +One issue is that the proxy and bucket scheme would allow for having both a normal and a pre-release version of the same package in dependencies. +We do not want that, so instead of proxy packages, we might have "frontend" packages. +The difference being that a proxy links a source to a target, while a frontend does not care about the source, only the target. +As such, only one frontend version can be selected, thus followed by either a normal version or a pre-release version but not both. + +Another issue would be that the proxy and bucket scheme breaks strategies depending on ordering of versions. +Since we have two proxy versions, one targetting the normal bucket, and one targetting the pre-release bucket, a strategy aiming at the newest versions will lean towards normal or pre-release depending if the newest proxy version is the one for the normal or pre-release bucket. +Mitigating this issue seems complicated, but hopefully, we are also exploring alternative API changes that could enable pre-releases. + +## Multi-dimensional ranges + +We are currently exploring new APIs where `Range` is transformed into a trait, instead of a predefined struct with a single sequence of non-intersecting intervals. +For now, the new trait is called `RangeSet` and could be implemented on structs with multiple dimensions for ranges. + +```rust +pub struct DoubleRange { + normal_range: Range, + prerelease_range: Range, +} +``` + +With multi-dimensional ranges we could match the semantics of version constraints in ways that do not introduce alterations of the core of the algorithm. +For example, the constraint `2.0.0-alpha <= v < 2.0.0` could be matched to: + +```rust +DoubleRange { + normal_range: Range::none, + prerelease_range: Range::between("2.0.0-alpha", "2.0.0"), +} +``` + +And the constraint `2.0.0-alpha <= v < 2.1.0` would have the same `prerelease_range` but would have `2.0.0 <= v < 2.1.0` for the normal range. +Those constraints could also be intrepreted differently since not all pre-release systems work the same. +But the important property is that this enable a separation of the dimensions that do not behave consistently with regard to the mathematical properties of the sets manipulated. + +All this is under ongoing experimentations, to try reaching a sweet spot API-wise and performance-wise. +If you are eager to experiment with all the extensions and limitations mentionned in this section of the guide for your dependency provider, don't hesitate to reach out to us in our [zulip stream][zulip] or in [GitHub issues][issues] to let us know how it went! + +[zulip]: https://rust-lang.zulipchat.com/#narrow/stream/260232-t-cargo.2FPubGrub +[issues]: https://github.com/pubgrub-rs/pubgrub/issues