Request: common dependencies

[isacjunior]

In some cases, we have common dependencies around some packages. For now, we have two workarounds:

• Create a package common_dependencies and centralized the common dependencies here.
  [✗] The dependency is installed as a transitive dependency and we don't have IntelliSense.
  [✓] All common dependencies in a unique locale. This is easier to manipulate versions.
• Install repeatedly dependencies in each package.
  [✗] Is difficult to manipulate the version, needs to change in many locales.
  [✓] We have the dependency as direct and so we have IntelliSense

Any know the workaround to solve this?

Attempt

I tried to use the following below but it doesn't work.

melos version: 0.4.10+1

# melos.yaml
dependencies:
  http: ^0.13.1

Suggestion:

When the melos copy pubspec.yaml and pubspec.lock, we could insert these dependencies on the specific packages.

# melos.yaml
dependencies:
  http: ^0.13.1
    scope: "package_one,package_two"

dev_dependencies:
  build_runner: ^1.12.2
    ignore: "package_one"

[Ehesp]

Have you tried to add the dependencies to the root pubspec.yaml (not melos.yaml)?

Then add the root package, like so: https://github.com/invertase/melos/blob/master/melos.yaml#L4

[isacjunior]

Have you tried to add the dependencies to the root pubspec.yaml (not melos.yaml)?

Then add the root package, like so: https://github.com/invertase/melos/blob/master/melos.yaml#L4

This is the same approach that I mentioned above. The root project would a common_dependencie package.

The solution for now that I find is to use getx_cli to install the common packages. But would amazing if melos could have this feature.

common_dependencies:
  run: melos exec -c 1 --scope="package_one,package_two" -- exec "get install provider:5.0.0 http:0.13.1"

[SAGARSURI]

This would be great if we can define common dependencies in melos.yaml.

[SAGARSURI]

Hey @Salakar would it possible to add this feature in Melos?

[marctan]

is this already supported in melos? this is really useful especially with dart_code_metrics . To use the analyzer , https://dartcodemetrics.dev/docs/analyzer-plugin, we need to add dart_code_metrics: ^4.15.0 to dev_dependencies on each pubspec.yaml. if you have a lot of packages, then you need to add it one by one for it to work. I was hoping, there's a way in melos where we can add only in one place.

[blaugold]

This is not currently supported by Melos. But if you just want to add dart_code_metrics to all packages, you can use melos exec:

melos exec -- dart pub add --dev dart_code_metrics:^4.15.0

[BenVercammen]

I have a similar issue with build_runner: ^2.2.0 being resolved as 2.2.0 in one package and 2.2.1 in another... I was assuming that melos would pull in the same version for each package, but apparently not?

[blaugold]

Melos only overrides dependencies on packages in the same workspace to path dependencies.

But even if we had common dependencies as a feature, I don't think that resolving dependencies across all packages would be wise. It would make it a lot harder for pub to resolve all dependency constraints of direct and indirect dependencies.

@BenVercammen Could you explain what issue you're facing?

[BenVercammen]

@blaugold Not really an issue (for now), just the fact that 2 packages in the same workspace have the same dependency, with the same ^ version, but actually resolve to 2 different (minor/patch) versions. Was a bit tricky when debugging. I guess I'll either have to work with fixed versions across all packages or just learn to deal with it somehow... Unless there is already some best practice that I'm unaware of?

[blaugold]

Ah, ok. I guess one thing that could help is to regularly run pub upgrade in all packages. If the package is published to pub.dev it should not have pubspec.lock checked in to git, so it should be much easier to do that than for apps. If common dependencies can be resolved to the same highest version, they will be after running pub upgrade in each package.

[fncap]

Hi I just wrote a little dart script that updates common packages versions inside all pubspec.yaml under libs and apps workspace folders.
Common packages versions are sourced from a dedicated yaml file.
This script could be executed in the pre bootstrap lifecycle hook!

pubspec.yaml at workspace root

name: my_workspace

environment:
  sdk: '>=2.18.0 <3.0.0'

dependencies:
  yaml: any
  yaml_edit: ^2.1.0
dev_dependencies:
  melos: ^2.9.0

common_packages.yaml

dependencies:
  package_name_01: ^1.0.0
  package_name_02: ^1.2.0

dev_dependencies:
  package_name_03: ^2.3.2
  package_name_04: ^6.5.4

common_packages_injector.dart

import 'dart:async';
import 'dart:io' show Directory, File;

import 'package:path/path.dart' as p;
import 'package:yaml/yaml.dart';
import 'package:yaml_edit/yaml_edit.dart';

Future<List<Directory>> getSubfolder(Directory dir) {
  var folders = <Directory>[];
  var completer = Completer<List<Directory>>();
  var lister = dir.list(recursive: false);
  lister.listen((x) {
    if (x is! Directory) {
      return;
    }
    folders.add(x);
  },
      // should also register onError
      onDone: () => completer.complete(folders));
  return completer.future;
}

Future update(YamlMap commonDependencies, List<Directory> folders,
    String targetNode) async {
  for (var x in folders) {
    final pubspecPath = File(p.joinAll([x.path, 'pubspec.yaml']));
    final yamlStr = await pubspecPath.readAsString();
    final yaml = loadYaml(yamlStr) as YamlMap;
    final editable = YamlEditor(yamlStr);
    final libDependencies = (yaml.nodes[targetNode] as YamlMap);

    bool shouldUpdate = false;
    for (var dep in commonDependencies.entries) {
      if (!libDependencies.containsKey(dep.key)) {
        continue;
      }
      editable.update([targetNode, dep.key], dep.value);
      shouldUpdate = true;
    }

    if (shouldUpdate) {
      await pubspecPath.writeAsString(editable.toString());
    }
  }
}

Future<void> main() async {
  final commonPackagesYamlPath =
      p.joinAll([Directory.current.path, 'common_packages.yaml']);
  final commonPackagesYaml = File(commonPackagesYamlPath);
  final commonPackages =
      loadYaml(commonPackagesYaml.readAsStringSync()) as YamlMap;
  //
  final libsPath = p.joinAll([Directory.current.path, 'libs']);
  final libs = await getSubfolder(Directory(libsPath));

  final appsPath = p.joinAll([Directory.current.path, 'apps']);
  final apps = await getSubfolder(Directory(appsPath));

  final paths = [...libs, ...apps];

  for (var target in ['dependencies', 'dev_dependencies']) {
    await update(commonPackages.nodes[target] as YamlMap, paths, target);
  }
}

melos.yaml pre 3.0.0

# ...
scripts:
  bootstrap: dart run common_packages_injector.dart
# ...

melos.yaml 3.0.0 and above

# ...
command:
  bootstrap:
    hooks:
      pre: dart run common_packages_injector.dart
# ...

Enjoy!

[matthewfx]

@fncap That's what I needed! Thank you very much! :)

[feinstein]

How about when we don't need a common dependency? In my case half of my packages need the same dependency, and the others don't. So I wanted to use the CLI to update this dependency version only on the packages that use the dependency.

[fncap]

My solution use the common_packages.yaml as a reference and whenever it find one of the referenced package it replace its value only in the specific package!
So I thinks that is exactly what you need!
I'm using it to solve the same problem that you have!

[lohnn]

Since we started using custom_lint in our monorepo project at work, we've realised that we need to keep all our package dependencies in sync, or else it does not really want to function.
Adding functionality for a central common_packages.yaml that would update imports in all our packages and apps would be a real life saver.

I am more than willing to take a stab at implementing such functionality. Are there any maintainers that would mind?

[vincent-hoodoo]

How about when we don't need a common dependency? In my case half of my packages need the same dependency, and the others don't. So I wanted to use the CLI to update this dependency version only on the packages that use the dependency.

That's an edge case in my opinion. In that case you would just not use the common dependencies method, and simply specify the version each time.

[feinstein]

I don't believe this is an edge case, my app is fairly big, and modularizing it is crucial to maintainability. But some modules depend on some stuff and others don't. The whole idea of using melos, for us, is to avoid the big manual hassle to deal with so many dependencies spread around so many modules.

[vincent-hoodoo]

@feinstein
I'm not saying that having a way to specify common dependencies is an edge case at all, quite the opposite, I really care about this feature.

What I'm saying is that specifying different dependencies versions for separate local packages is, and that it can easily be achieved by using dart's standard of way of defining package dependencies.

Just to get an idea of your use case, can you detail:

• how many dependencies you have that are on different versions
• how many local packages you have that depend on these different versions?

[feinstein]

Oh, sorry, I misunderstood you. Actually I think I misunderstood the whole thing. I was afraid melos would start adding dependencies to all packages, even the ones that don't need them. But I understand that's not the case, since the tree shaker would remove this unused dependencies. My bad.

[lohnn]

@blaugold and @Ehesp, would you be fine with me implementing this?
My Idea is to do a solution similar to #94 (comment), but where the ´common_dependencies.yaml´ is inserted into the subpackages in bootstrap.

[blaugold]

@lohnn Would be great to get this feature implemented, so feel free to go a head.

👍 I think during the bootstrap command is a good time to sync the dependencies.

Since Melos 3 we require a pubspec.yaml at the root of the workspace. Instead of a separate common_dependencies.yaml file, we could take the dependencies from there and apply them as common dependencies. That's also how other monorepo tools like nx handle common dependencies. Keeping the common dependencies in a pubspec.yaml file has the advantage that we get code completion and a better dev experience. If we go with this approach, we need an option to toggle common dependencies.

[lohnn]

Great!
Sounds like a good idea. I guess it should then be added as an option to melos.yaml under `BootstrapCommandConfigs´?

Like this maybe?

name: my_app

packages:
  - packages/**
  - apps/**

command:
  bootstrap:
    shareDependencies: true <--
  version:
    branch: HEAD

scripts: ...

And I'm guessing we turn this off by default?

[blaugold]

I guess it should then be added as an option to melos.yaml under BootstrapCommandConfigs?

👍

And I'm guessing we turn this off by default?

I think that's the save choice.

[vincent-hoodoo]

Keeping the common dependencies in a pubspec.yaml file has the advantage that we get code completion and a better dev experience

@blaugold could you clarify what you mean by that?

The decision of sharing the dependencies listed in the root pubspec.yaml would come with its own issues:

1. Packages will be installed at the root when they don't need to be
   It could for example generate useless .flutter-plugins and .flutter-plugins-dependencies files at the root

2. Local packages (potentially used by a lib script in the root package) will have to apply their version to children packages, even when this is not the developer's intention.

3. The distinction between the root package's responsibilities and the children responsibilities will be blurred.
   A good way to understand what a package does (and doesn't do) is to look at its pubspec.yaml file. If common children dependencies are mixed with the root package's dependencies, it will be impossible to know what the root package responsibilities are.

There are other API designs that wouldn't suffer from these issues, while being more explicit and more extensible.
As others have suggested in this thread, I think we would be better off with new fields, added either to the melos.yaml file, or to a melos namespace in pubspec.yaml.

e.g. in melos.yaml

common_dependencies:
  my_package: 1.0.0
common_dev_dependencies:
  my_dev_package: 1.0.0

e.g. in pubspec.yaml

melos:
  common_dependencies:
    my_package: 1.0.0
  common_dev_dependencies:
    my_dev_package: 1.0.0

[feinstein]

Sorry for interrupting the discussion, but regardless of the final implementation, I think It would be useful if we could run melos outdated to get the latest versions of the common dependencies that satisfy all our packages transitive dependencies.

[lohnn]

I have created a draft PR where I went with common_dependencies.yaml, I get code completion (with my plugins for IntelliJ).
Created it as a draft to be able to start discussions regarding what type of solution we want to go with with file names and such, but also give possibility for you to also give feedback on the solution itself.

[godilite]

what exactly does bootstrap command do? if it doesnt do this?

[feinstein]

The flutter repo has a tool that tracks one file with common dependency versions, and updates all pubspec.yaml it finds with those versions, leaving a text saying the version was updated by a tool and should not be changed manually.

Melos could have the same functionality, looking in each pubspec.yaml if there's a reference to a dependency in a central pubspec.yaml, if positive change it's version to the one in the central file and append a comment saying it was automatically added.

Example here:

  async: 2.11.0 # THIS LINE IS AUTOGENERATED - TO UPDATE USE "flutter update-packages --force-upgrade"

[davidmigloz]

This has already been supported since v3.2.0: #526

[feinstein]

And how's that different from what has been asked in this issue?

[davidmigloz]

I guess this issue can now be closed

[spydon]

Closed since implemented in #526

[feinstein]

@spydon thar PR mentions the common_packages.yaml, I am trying to find this mechanism in the docs website, but there I can only see a dependencies section in bootstrap. Has the common_packages.yaml file being removed?

[vincent-hoodoo]

@feinstein if you read past the initial description of the PR, you'll get an answer to your question.
TL,DR: indeed there is no common_packages.yaml file.

[spydon]

@feinstein here are the docs for it :)
https://melos.invertase.dev/commands/bootstrap#shared-dependencies

[fncap]

Hi to all, it would be very useful if you can include dependency_overrides in sharing logic!
Thanks!

[spydon]

Hi to all, it would be very useful if you can include dependency_overrides in sharing logic!
Thanks!

The problem with shared dependency overrides is that there is no good way to remove them after they have been removed from the melos.yaml file, and since dependency overrides should be temporary that should be taken into consideration, that's why #520 and #594 haven't been merged.

There is a specific issue for this: #439