Find a way to coordinate / share / merge with mono_repo

[kevmoo]

Hey folks!

This is purely speculation, but it might be worth at least discussing.

I help maintain https://github.com/google/mono_repo.dart/ with a couple of folks from the Dart team.

There is a LOT of overlap here. Wondering if it'd make sense to at least chat.

[Salakar]

Hey @kevmoo thanks for reaching out 👋

There is a LOT of overlap here. Wondering if it'd make sense to at least chat.

Would love to chat and see how we can work together on our shared project goals of making monorepos more manageable in Dart.

In terms of overlap I don't actually think there is much overlap here code wise apart from our shared goals of the projects. Our docs are heavily in progress and at the moment so they do fail to shine a light on what Melos really does. I'll clarify with some high level points comparing Melos features with mono_repo;

(please correct me if I'm wrong on any of the mono_repo points)

---

Pub

mono_repo

Runs pub get or upgrade in each individual package in the repository.

melos

Pub files such .packages, .dart_tool & .flutter-plugins are generated for each package in melos via the bootstrap command, we don't run pub get in each package. This gives us many benefits;

• We can interlink all local packages in the repository by generating the pub files to point to a directory rather than 'remote'.
  • Scenario: In a repository, package A depends on package B. Both packages A & B exist in the monorepo. However if you pub get inside package A, pub will retrieve package B from the pub.dev registry as it's unaware of B existing locally. However with melos, it's aware that package B exists locally too so will generate the various pub files to point to a relative path in the repository locally.
    • If you wanted to use pub you could of course define a dependency override in the pubspec of package A that sets a path for package B but then you'd have to do this manually every time and then manually remove it again when you want to publish or even commit. This doesn't scale well and doesn't help with making a repository contributor friendly. FlutterFire for example has almost 50 packages with various interlinking levels (you can run melos list --graph to see it's local dependency graph).
      • This also gets phenomenally worse for example say if we introduce package C that package B depends on but package C then also depends on A.
• Interlinking highlights pub issues early, e.g. version constraints.
  • Scenario: Package A depends on a package C with a constraint of ^1.0.0. However package B depends on the same C package but with a higher constraint of ^2.0.0. If you pub get in both package A & B you wouldn't be made aware of this and eventually your packages get published to pub and a user that uses both A & B finds themselves with a versions constraint conflict and can't use the packages together anymore (same applies to external deps).
    • In this scenario melos would fail on the bootstrap and highlight the constraint issue. This happened frequently on FlutterFire before melos.
• Interlinking highlights dart/analyzer issues early.
  • Scenario: Package A relies on package B from the same mono repo. Package B gets a minor API change. Via pub get on package A the dart analyzer and IDEs report no issues with the package as it installed package B from the pub registry and not local (which hasn't been published yet). With melos, the dart analyzer / IDEs would highlight this issue immediately. The same applies for non-breaking deprecations, package A wouldn't show there was a deprecated API in use without being interlinked through melos.
• Interlinking allows working on new, not yet published packages easily, without defining dependency overrides.

---

CI

mono_repo

At the time we started building melos, mono_repo was for travis only. (I see there is some actions stuff now as well).

melos

Melos is CI agnostic, it provides all the tools/commands you'd ever need for any CI. e.g. exec, run, list, version, publish;

$ melos --help

Usage: melos <command> [arguments]

Global options:
-h, --help                                       Print this usage information.
    --verbose                                    Enable verbose logging.
    --no-private                                 Exclude private packages (`publish_to: none`). They are included by default.
    --[no-]published                             Filter packages where the current local package version exists on pub.dev. Or "-no-published" to filter packages that have not had their current version published yet.
    --scope=<glob>                               Include only packages with names matching the given glob. This option can be repeated.
    --ignore=<glob>                              Exclude packages with names matching the given glob. This option can be repeated.
    --since=<ref>                                Only include packages that have been changed since the specified `ref`, e.g. a commit sha or git tag.
    --dir-exists=<dirRelativeToPackageRoot>      Include only packages where a specific directory exists inside the package.
    --file-exists=<fileRelativeToPackageRoot>    Include only packages where a specific file exists in the package.

Available commands:
  bootstrap   Initialize the workspace, link local packages together and install remaining package dependencies.
  clean       Clean this workspace and all packages. This deletes the temporary pub & ide files such as ".packages" & ".flutter-plugins". Supports all package filtering options.
  exec        Execute an arbitrary command in each package. Supports all package filtering options.
  list        List local packages.
  publish     Publish any unpublished packages or package versions in your repository to pub.dev. Dry run is on by default.
  run         Run a script by name defined in the workspace melos.yaml config file.
  version     Automatically version and generate changelogs for all packages. Supports all Melos filtering flags.

Example:

Run dart analyze in all packages with a concurrency of 5 but fail fast (stop immediately if any package fails):

melos exec -c 5 --fail-fast -- \
  dart analyze .

You can also define scripts in the root melos.yaml for a repository, e.g.

name: FlutterFire

packages:
  - packages/**

scripts:
  analyze:
    run: |
      melos exec -c 5 --fail-fast -- \
        dart analyze .
    description: |
      Run `dart analyze` in all packages.
       - Note: you can also rely on your IDEs Dart Analysis / Issues window.

 # ...

Contributors or CI just needs to run melos run analyze with that defined. Or even just melos run to select from the available defined scripts and view their descriptions:

Note that the script definitions can be as basic as [name]: something_to_run or as advanced as giving it a description, pre-defining package filters (see Filters below) and prompting user to select a package to run the script against from a list of packages (filtered from pre-defined filters). See #34 for screenshot examples of this.

---

Filtering

All melos commands support some basic to very advanced package filtering flags;

--no-private
Exclude private packages (publish_to: none). They are included by default.

--[no-]published
Filter packages where the current local package version exists on pub.dev. Or "-no-published" to filter packages that have not had their current version published yet.

--scope=<glob>
Include only packages with names matching the given glob. This option can be repeated.
Example: --scope="*platform_interface*" - filter packages to platform interfaces ones only.

--ignore=<glob>
Exclude packages with names matching the given glob. This option can be repeated.
Example: --ignore="*example*" - ignore example apps.

--since=<ref>
Only include packages that have been changed since the specified ref, e.g. a commit sha or git tag.
Example: --since="dev" - filter packages that have had changes made since the dev branch.

--dir-exists=<dirRelativeToPackageRoot>
Include only packages where a specific directory exists inside the package. This option can be repeated.
Example: --dir-exists="macos" --dir-exists="test_driver" --scope="*example*" - filter example apps that have test_driver tests directory and also have a macos directory (signifying macos supported).

--file-exists=<fileRelativeToPackageRoot>
Include only packages where a specific file exists in the package. This option can be repeated.
Example: --file-exists="./test_driver/MELOS_PARENT_PACKAGE_NAME_e2e.dart" --scope="*example*" - filter example apps that have test_driver file named the same as the parent packages name, e.g. if firebase_auth/example then the MELOS_PARENT_PACKAGE_NAME placeholder gets automatically replaced to be firebase_auth. See #3 for details all the injectable environment variables melos automatically injects.

The best command to test out these global filter flags with is the list command:

$ melos list --help
List local packages.

Usage: melos list
-h, --help        Print this usage information.
-l, --long        Show extended information.
-a, --all         Show private packages that are hidden by default.
-p, --parsable    Show parsable output instead of columnified view.
    --json        Show information as a JSON array.
    --graph       Show dependency graph as a JSON-formatted adjacency list.

---

Automated versioning and release management.

Melos provides support for automatic semantic versioning (based on the angular conventional commits standard) of releases and changelog generation via the melos version & melos publish commands (which both also support the global filtering flags).

$ melos version --help
Automatically version and generate changelogs for all packages. Supports all Melos filtering flags.

Usage: melos version [arguments]
-h, --help                    Print this usage information.
-p, --prerelease              Version any packages with changes as a prerelease. Cannot be combined with graduate flag.
-g, --graduate                Graduate current prerelease versioned packages to stable versions, e.g. "0.10.0-dev.1" would become "0.10.0". Cannot be combined with prerelease flag.
-c, --[no-]changelog          Update CHANGELOG.md files.
                              (defaults to on)
-t, --[no-]git-tag-version    By default, melos version will commit changes to pubspec.yaml files and tag the release. Pass --no-git-tag-version to disable the behavior.
                              (defaults to on)
    --yes                     Skip the Y/n prompt at the beginning of the command.
    --preid                   When run with this option, melos version will increment prerelease versions using the specified prerelease identifier, e.g. using a "nullsafety" preid along with the --prerelease flag would result in a version in the format "1.0.0-nullsafety.0".
                              (defaults to "dev")

$ melos publish --help
Publish any unpublished packages or package versions in your repository to pub.dev. Dry run is on by default.

Usage: melos publish [arguments]
-h, --help               Print this usage information.
-n, --[no-]dry-run       Validate but do not publish the package.
                         (defaults to on)
-t, --git-tag-version    Add any missing git tags for release. Note tags are only created if --no-dry-run is also set.

This has allowed us to publish some ~50 FlutterFire packages in minutes with guaranteed versioning and changelog consistency. Previously this could take over a day and could still be wrong, e.g. package A has been updated therefore package B that depends on A should get a dependency bump and also be updated.

Additionally this removed so much headache for maintaining the FlutterFire project, previously users had to as part of their PRs also bump the package versions and add change log entries - this caused so many inconsistencies and blocked merges due to conflicts when a lot of PRs were working on the same packages. Now users just have to submit their proposed changes and thats it, no changelogs or version bumps. This was a huge bottleneck for contributing to FlutterFire because as soon as you merged a PR then all the other PRs for the same packages would then be out of date and have version & changelog conflicts - you could only have 1 PR ready at a time.

These commands both also support custom script hooks, e.g. define a custom version or postversion script in your melos.yaml and it will be automatically called as part of the versioning process - useful for generated files that need to be updated before publishing. As an example Melos itself uses this to generate it's version.g.dart file; https://github.com/invertase/melos/blob/master/melos.yaml#L39

---

IDE integration

Melos provides deep IDE integration support. E.g. for IntelliJ, the various IntelliJ config xml files & iml files are generated into the melos workspace for all the packages. See #9 for what this means. More advanced integration for Vscode is also in the works.

---

There's probably a lot of other things I've not covered but I don't want to overload this response more than I already have. For some examples of melos in the wild checkout the following;

• FlutterFire
  • Config: https://github.com/FirebaseExtended/flutterfire/blob/master/melos.yaml
  • CI Example:
    • https://github.com/FirebaseExtended/flutterfire/blob/master/.github/workflows/firebase_auth.yaml
    • https://github.com/FirebaseExtended/flutterfire/blob/master/.github/workflows/scripts/drive-example.sh
• Melos
  • Config: https://github.com/invertase/melos/blob/master/melos.yaml
  • CI Example: https://github.com/invertase/melos/blob/master/.github/workflows/validate.yaml
• Flutter Community - Plus Plugins
  • Config: https://github.com/fluttercommunity/plus_plugins/blob/main/melos.yaml
• AWS Amplify Flutter
  • Config: https://github.com/aws-amplify/amplify-flutter/blob/master/melos.yaml

[kevmoo]

🤣 Ha! Maybe we should just retire mono_repo. You guys are crankin!

I'll chat with the other owners. mono_repo is a bit of a special ❄️ at the moment. It does exactly what our team needs for the bits we need it for.

But I may start recommending folks come here – if that's okay with you.

We do have some nice logic for doing GitHub Actions now...but maybe you guys are already there and beyond.

[Ehesp]

@kevmoo thanks! Melos mainly started from working on FlutterFire where we needed to link the packages together and not worry about modifying every single pubspec.yaml file.

But I may start recommending folks come here – if that's okay with you.

Fine with us :)