Proposal for `resolution: external` as support for mono-repositories

[sigurdm]

NOTE: This proposal has been partially succeeded by a proposal with a proper workspace file. See https://docs.google.com/document/d/1UEEAGdWIgVf0X7o8WPQCPmL4LSzaGCfJTM0pURoDLfE/edit?resourcekey=0-c5CMaOoc_pg3ZwJKMAM0og . We probably want to support resolution: external as well as resolution: workspace for cases such as the dart sdk where the package-config is not created by pub.

This issue suggests a simple mechanism in the pub client for better supporting a shared resolution between the packages of a mono-repo. Namely adding

resolution: external

to a pubspec.yaml to indicate that pub should not treat it as a root package.

The main parts of this proposal does not contain much support in the pub client for how to manage multiple packages together in a single project. We instead suggest pushing more advanced support to external tools such as package:mono_repo or package:melos.

Motivation

Mono-repositories are repositories with tightly related packages that are often developed and released together.

The dart team is maintaining a number of mono-repos Eg:
dart-lang/test
google/protobuf.dart
dart-lang/ecosystem

Another special case of mono-repos is the dart SDK where a common resolution is used, but not maintained by pub, even though many of the subpackages have a pubspec.yaml, and some are published to pub.dev individually...

Externally we know of several customers using mono-repositories.

When developing in a mono-repo it is often desirable to use a shared version resolution for ensuring consistent behavior.

By having a shared resolution, tools such as the analyzer can also share context between analyzing each package, with big potential memory savings.

Some packages are in a mono-repo for organizational purposes, and might not want to share the same resolution (and would not benefit from this proposal). One such repo could be: flutter/packages.

Some tutorials are split in multiple "chapters" each with a single pubspec.yaml, and the analyzer has been known to choke on those because they each have a separate context. They could perhaps benefit from this proposal (if developing the pubspec.yaml is not part of the tutorial).

Existing support for mono-repositories

There exists tooling to support handling of mono-repos, such as mono_repo.dart and melos. These tools can help running the same command for all sub-projects
They, however, don't allow a shared resolution between the packages. These could be updated to allow for generating a shared root pubspec.yaml.

Proposed mechanism

A new section in pubspec.yaml would mark a pubspec that it should not be used for a "local resolution", but instead rely on the Dart VM's mechanism for searching parent directories for a .dart_tool/package_config.json containing a resolution to use.

name: test
environment:
  sdk: ^3.1.0
dependencies:
  ...
resolution: external

• Given this, dart pub get would not generate a dart_tool/package_config.json file in the same directory. Instead it could either (still open):

  • Report an error: "This package is not intended for direct resolution"
  • Look for a pubspec.yaml in parent directories and resolve that instead.
  • Do nothing.

• dart run and dart test needs to ignore pubspec.yamls with resolution: external when resolving packages.
  dart pub publish should however consider this pubspec.yaml when doing validations.
  We could create a lint discouraging from having dev_dependencies and dependency_overrides in a pubspec.yaml with resolution: external.

Creating a mono-repo

Now to create a mono-repo, one would create a "root" pubspec.yaml with dependency_overrides with path-dependencies on all the sub-projects.

name: my_mono_repo_root
publish_to: none
environment:
  sdk: ^3.0.0
dev_dependencies:
  build_runner: ^2.0.0
dependency_overrides:
  pkg1:
    path: pkgs/pkg1
  pkg2:
    path: pkgs/pkg2

All dev_dependencies should be collected in the root pubspec.yaml, and thus shared between the subprojects.

Running dart pub get in this root-directory will create a shared pubspec.lock (that can be checked into source control if desired) and a shared dart_tool/package_config.json for consumption by the Dart VM and analyzer.

• We now rely on the dart run resolution mechanism to look up through parent directories for finding the dart_tool/package_config.json file, and providing the package resolution. This should enable dart test to still work from the sub-project folder (we need to test that this works).

• One could think of mechanisms for "linking" the sub-projects into the root-project, such that they could be operated on as a whole (eg. when doing upgrades). We propose to (at least as a first step) leave such functionality out of the pub client, and for example

The Dart SDK

In the Dart SDK we can mark all the in-sdk-repo packages as resolution: external.

There will not (at least for now) be a root pubspec.yaml, but we will instead rely on the current alternative tooling to generate a shared dart_tool/package_config.json.

Open questions:

• Do we need a way to mark the sub-dependencies as special, such that Pub knows how to update them in sync? Or are dependency_overrides good enough?
  • We could add this later if there's a high demand
  • We could also autodetect overridden path-dependencies from the root-package and handle them special.
• We need to figure out how build_runner would work in such a setup. Can each sub-project contain its own build.yaml?
• How does this affect dart pub add? Can you add to a sub-package, and should that resolve together with other packages?
• Should dart pub add --dev find the root pubspec.yaml?
• How does this affect dart pub upgrade --major-versions? Is there a way to upgrade all the pubspec.yaml's of a project in sync? (Could perhaps be added later)

We might need a way to force resolution of the single package, e.g. for deployment of a sub-package. (Could we piggy back on dart pub get --enforce-lockfile - then we should have called it dart pub get --deploy 😐)

Potential downsides

• Relying on dependencies_overrides can cause some issues

  • Dependency constraints among sub-projects are not checked.
    When publishing you might have constraints not matching sibling dependencies

    This can to some extent be helped by doing a sub-package-only resolution in dart pub publish.

• Diamonds where you depend on other packages on pub.dev, and these packages also depends on another package in your mono-repository
  • These are rather rare - perhaps we don't need strong support for those.
• cycles with other packages on pub.dev
  • These are rather rare - perhaps we don't need to support those.
• Mono-repository root pubspec.yaml might be missing the dependency_override that should include a child.
  • Could we warn against this?

[jonasfj]

I think the core of this approach is to enable tooling like mono_repo and melos to manage mono-repositories with a single resolution better.

So if the root pubspec.yaml does have dependency_overrides for all child-packages, that seems like an issue for mono-repo tooling to deal with.

I suspect that when dart pub get encounters a pubspec.yaml with resolution: external then it should walk up until it finds a root pubspec.yaml that should be resolved, or finds a .dart_tool/package_config.json which provides a resolution.

Regarding, tools that modify pubspec.yaml, such as:

• dart pub upgrade --major-versions
• dart pub add
• dart pub add --dev

I don't know what we should do. If pub isn't doing the resolution. Then maybe we do not support these, how can we?

To be clear: the idea is to let mono-repo tooling disable pub get locally. Whether mono-repository tooling delegates resolution to a root pubspec.yaml, or does other magic to produce a resolution is left for the mono-repo tooling to decide.

[olexale]

I’m working in a mono-repo with more than 300 packages, so I’m very curious about this proposal.
I’m unsure if I fully understand it, so I prepared some questions/comments:

1. The name «external» makes a lot of sense from the tooling perspective, when you are thinking about individual packages, but in the context of the application development, an «internal» would make more sense, as for my mono-repo, these packages are internal, and external packages are packages from pub.dev and other sources.

2. Could you please elaborate more on the expected additional support from the mono_repo and melos?

3. What if I have multiple applications in the same monorepo? For example, I have the following structure:

apps
 - app1
 - app2
utilities
 - util1
 - util2

I.e. I have app1 that depends on util1 and util2, and app2 depends on both these utilities too. Will this scenario still be supported?

4. I see that is still an open question, but if code generation and testing behavior changes, i.e. it will not be possible to run tests and generate code per package, that would kill the idea of having a mono-repo for our project as we will not be able to optimize CI pipelines (we are running tests and other checks on modified packages only), and locally I’ll not be able to quickly run tests for a package I’m working in - tests for the whole app will be executed, which, for our project, will take a ridiculous amount of time if I’m working on a single feature.

5. The «dependency_overrides» has clear semantics, but adding internal (I mean «external») packages to the list mixes packages that are indeed overridden and the regular ones. If that change is going to happen, I think it will make more sense to have a separate section in the pubspec for it.

Unfortunately, I don’t think I understand the real benefit of this change. I can see how the proposed changes fix the analyzer performance, which is IMO the main pain for mono-repos now, but it seems like we are solving this problem of a tool at the expense of the developer’s experience. Even if the proposed functionality already existed, I would prefer keeping the «Only Analyze Projects With Open Files» flag on in my IDE (which fixes the analyzer performance, not without drawbacks ofc) and keep the project simpler.

[blaugold]

This proposal looks really promising.

Analyzer performance in large mono-repos is an issue I have heard of from multiple people, so reducing analyzer contexts would be a big win. Having to only resolve one package instead of 10s or 100s also has the potential to speed up CI jobs.

Potential issues

• It might not always be possible to find a shared resolution for all packages in a mono-repo. Resolving the dependencies of many packages with many more transitive dependencies together makes it that much more likely that there are conflicting dependency constraints. It's easy enough to opt-out individual packages out of shared resolution by removing resolution: external, though. Tools like package:mono_repo and package:melos may have to specifically support a mixed setup.
• Is depend_on_referenced_packages still going to work when dev_dependencies are specified in the shared package used for resolution?
• Is the flutter tool going to work without a dart_tool/package_config.json in the package root?

[sigurdm]

@olexale

Thanks for the feedback! Some initial responses:

1. The name «external» makes a lot of sense from the tooling perspective, when you are thinking about individual packages, but in the context of the application development, an «internal» would make more sense, as for my mono-repo, these packages are internal, and external packages are packages from pub.dev and other sources.

Interesting point. But I still think that even though the packages are internal, the resolution is external, and thus I think the term makes some sense. But we are not fixed on the naming if you have better suggestions.

2. Could you please elaborate more on the expected additional support from the mono_repo and melos?

Well I cannot promise what these packages will implement. But I could imagine that these packages could help generate the repo-wide pubspec.yaml. Perhaps they could also help making repo-wide upgrades (somehow).

3. What if I have multiple applications in the same monorepo?

Yeah, I think your example should be supported, as these will all be included via dependency overrides.

4. I see that is still an open question, but if code generation and testing behavior changes,

I don't think this is as much an open question, as tasks that will have to happen for this proposal to be useful.

5. The «dependency_overrides» has clear semantics, but adding internal (I mean «external») packages to the list mixes packages that are indeed overridden and the regular ones. If that change is going to happen, I think it will make more sense to have a separate section in the pubspec for it.

Again, it is not the packages that are external, but the resolution mechanism.

The idea is to keep the proposal minimal for now.
If we find that in practice a special way of declaring that a dependency is part of a mono-repo is needed it can be added later.

Unfortunately, I don’t think I understand the real benefit of this change. I can see how the proposed changes fix the analyzer performance, which is IMO the main pain for mono-repos now, but it seems like we are solving this problem of a tool at the expense of the developer’s experience. Even if the proposed functionality already existed, I would prefer keeping the «Only Analyze Projects With Open Files» flag on in my IDE (which fixes the analyzer performance, not without drawbacks ofc) and keep the project simpler.

I think the main point is ensuring that all the sub-packages in a mono-repo share a single resolution (a single pubspec.lock), and thus reduces the potential for the confusion that could occur if two sub-packages end up with different versions of a dependency - causing surprising behavior.

Also there is potentially some time to save by only running pub get once.

[sigurdm]

@blaugold

Also thanks for feedback!

It might not always be possible to find a shared resolution for all packages in a mono-repo.

IMO if a shared resolution cannot be found, it is an advantage to find out early and solve that issue somehow, rather than developing with different versions of dependencies in different sub-projects.

It's easy enough to opt-out individual packages out of shared resolution by removing resolution: external, though.

Good point - though it would perhaps be a confusing state of the mono-repo, but it shows there are ways out if really needed.

Tools like package:mono_repo and package:melos may have to specifically support a mixed setup.

yeah - they probably need to recognize this pattern. Really good point.

Is depend_on_referenced_packages still going to work when dev_dependencies are specified in the shared package used for resolution?

Good question! I guess that depends what pubspec that lint is going to look into - @pq do you have input here?

Is the flutter tool going to work without a dart_tool/package_config.json in the package root?

Another good question. Ideally it 'should just work' (from the definition of package config), but there will probably be stuff breaking.

@christopherfujino can you foresee stuff that would work poorly in this model?

[pq]

Is depend_on_referenced_packages still going to work when dev_dependencies are specified in the shared package used for resolution?

Good question! I guess that depends what pubspec that lint is going to look into - @pq do you have input here?

Good question. @jakemac53? 📟

[jakemac53]

Is depend_on_referenced_packages still going to work when dev_dependencies are specified in the shared package used for resolution?

No it would not work. Is there any particular reason the proposal here says not to correctly specify dev_dependencies per-pubspec? I would suggest just requiring them to be there for each package still.

If we don't have a standard that requires a root level pubspec, I don't see how this could be supported either. Unless we just wanted the lint to ignore entirely non-lib dirs in packages with resolution: external.

• We need to figure out how build_runner would work in such a setup. Can each sub-project contain its own build.yaml?

This should work fine as long as they still have valid pubspecs as well. This would need to include dev dependencies though, similarly to the depend_on_referenced_packages lint use case.

• How does this affect dart pub add? Can you add to a sub-package, and should that resolve together with other packages?

It should error imo. And pub get/upgrade should also error.

[jakemac53]

There is a potential separate issue here - there might be tools which just assume a file at exactly .dart_tool/package_config.json. These tools could certainly be updated but up until now it was a safe assumption to make. It is hard to know what would break without actually making the change, but I would expect some amount of pain in the transition.

[christopherfujino]

There is a potential separate issue here - there might be tools which just assume a file at exactly .dart_tool/package_config.json. These tools could certainly be updated but up until now it was a safe assumption to make. It is hard to know what would break without actually making the change, but I would expect some amount of pain in the transition.

Yeah, I'm pretty sure many things would break in the tool. Worse, it seems like it would be more difficult to reason about where to find this information? Obviously, this is something we could overcome, but I suspect it would be painful.

[christopherfujino]

What about if pub still wrote the package_config.json, but perhaps if its contents contained the absolute path to the root file?

[jakemac53]

Yeah, I'm pretty sure many things would break in the tool. Worse, it seems like it would be more difficult to reason about where to find this information? Obviously, this is something we could overcome, but I suspect it would be painful.

Fwiw, the behavior is fully specified https://github.com/dart-lang/language/blob/main/accepted/2.8/language-versioning/package-config-file-v2.md#finding-the-file. But, that doesn't mean all tools actually do that lookup logic given they don't really have to for most external use cases.

[jakemac53]

Also, reading the specification it actually does currently talk a lot about Pub specifically, and its current behavior of putting a file in each package root.

So we would want to probably treat this as a breaking change and go through the normal breaking change processes.