Support extensions from globally activated pub packages

[kenzieschmoll]

An example of an extension that fits this use case would be an arbitrary development tool package (like an AI-assisted development experience extension or a tool that interacts with the Dart analysis server). These extensions may be runtime or static extensions.

It would be a bit awkward to require a user to add a dev_dependency to every Dart project where they want to use this DevTools extension, since tools like these are not specific to a Dart / Flutter project, but rather to the general Dart / Flutter development experience. The workflow for an end user would be:

1. dart pub global activate some_pkg
2. If some_pkg provides a DevTools extension, we should be able to load this in DevTools (and also embedded in the IDE).

@jonasfj @sigmundch is there a way to tell from the pub cache what the globally activated pub packages are?

[sigmundch]

@sigurdm 😄

[sigurdm]

We have dart pub global list, however I'm not sure it gives you enough to go on.

You probably want a way to find the .dart_tool/package_config.json for each of these to detect extensions - right?

@jonasfj should we extend the list command with more info? Any other ideas?

[jonasfj]

I'm not sure globally activated packages should be used for this.

Global package activation is a mechanism for installing a package onto your system.
I'm not sure we should use it to configure extensions for devtools.

That said, a somewhat ludicrous idea would be for devtools to discover system-level extensions by searching PATH.

Imagine that devtools discovers system-level extensions by:

• Listing all binaries from any folder in PATH.
• Filter for executables.
• Take all executables prefixed _dart_devtools_extensions_
• These executables are now considered extensions, if each extension is supposed to provide a Flutter Web application or something like that, you simply run _dart_devtools_extensions_myglobaldevtoolsextension --print-extension-location and have it print where the resources are located.

If you are authoring a global devtools extension, you don't have to ship it as a pub package to be globally activate. You can ship as a globally activated package, because you can use the executables section in pubspec.yaml to specify executables to be installed on PATH. Provided that the user added $PUB_CACHE/bin to $PATH.

I suppose it would also be possible for devtools to try and guess the location of the global PUB_CACHE, and then also search PUB_CACHE/bin for executables prefixed _dart_devtools_extensions_.

With this approach system-level devtools extensions could also be installed using system package manager like apt-get, or just extracted from a zip to /opt and manually added to PATH. Or it could be installed with a different package manager like homebrew, npm or pypi. Because devtools is just searching for executables on PATH prefixed by _dart_devtools_extensions_ (or whatever string you want).

Maybe, it's a bit crazy, but:

• dart global activate is for installing tools.
• I think searching PATH is the best way to discover tools.
• Listing all executables on PATH is probably fast.
• Executables like _dart_devtools_extension_my_extension_name is not going to pollute bash-autocompletion (a prefix like dart_... would probably be annoying).
• A globally activate pub package can find its own location in PUB_CACHE using something like Isolate.resolvePackageUriSync('package:my_own_package_name/my_own_package_name.dart') (and then do a relative path from there)

It might just work :D

[kenzieschmoll]

You probably want a way to find the .dart_tool/package_config.json for each of these to detect extensions - right?

@sigurdm package:extension_discovery's findExtensions method expects a package_config.json location so that it can:

1. Look for any pub package that contains an extension/devtools/ directory
2. Return a list of Extension objects that describe the extension and provide a location for the extension's assets.

For a globally activated package, we would really just need to look at its contents for the extension/devtools directory. Maybe package:extension_discovery could be extended to support this use case as well, if it is non-trivial to find a package_config.json file for globally activated packages.

Global package activation is a mechanism for installing a package onto your system. I'm not sure we should use it to configure extensions for devtools.

@jonasfj Since DevTools extensions are shipped as part of the pub ecosystem, I actually think this is a reason we need to support this use case. As a user, I would expect that if I install a package onto my system, that I can use anything that package provides, including executables, extensions, etc.

if each extension is supposed to provide a Flutter Web application or something like that, you simply run _dart_devtools_extensions_myglobaldevtoolsextension --print-extension-location and have it print where the resources are located.

While this could work, my concern is that this is one more thing that extension authors have to manually set up, and they have to know when they need this and when they do not. I don't expect extension authors to have as much familiarity with what it means to be a "static" extension vs. a "runtime" extension and "global" vs "non-global". We want to make it as simple as possible for authors to build a tool, and know that no matter how users install their package (as a global or as a project dependency), that their users will be able to discover the tool they have built and use it.

With this approach system-level devtools extensions could also be installed using system package manager like apt-get, or just extracted from a zip to /opt and manually added to PATH. Or it could be installed with a different package manager like homebrew, npm or pypi. Because devtools is just searching for executables on PATH prefixed by dart_devtools_extensions (or whatever string you want).

I don't think we want to diverge the extension distribution (and enablement) mechanism, since this will be less discoverable for users and will make the development process more difficult for extension authors. Having one way to ship and install an extension (via pub) is ideal. Shipping / installing with pub is straight forward and keeps DevTools extensions in alignment with the Dart ecosystem standards.

[kenzieschmoll]

Pinging this issue again since another use case has come up where we'd like to be able to load a DevTools extension for a globally activated package.

Rationale:

1. Requiring users to add a dependency to a Dart or Flutter project in order to use an extension is not always ideal. The project may be a pure Dart project, but the package containing the extension may have a Flutter dependency (in the case of standalone extensions, the extension requires a Flutter dep).
2. Adding the dependency for a devtools extension or the package containing a devtools extension may cause unnecessary version solve issues for the project that are unrelated to using the tool.

For these reasons and the ones I outlined above, extension detection for globally activated packages is something we'd like to support.

Could package:extension_discovery be extended to support this use case? @jonasfj @sigurdm Or the other option that Sigurd suggested using dart pub global list would work fine. Could we add dart pub global list --verbose or similar that includes the location of the package_config.json for globally activated packages, and then we could use package:extension_discovery as it is today to find the extensions for global packages.

[jonasfj]

In an ideal world:

(1) is something that should be solved by:

• Using a dev_dependency, or,
• Support publishing Flutter plugin + Dart standalone hybrid packages dart-lang/pub#3563

(2) should be addressed by allowing multiple versions of the same package in a resolution (probably by marking some dependencies as private_dependencies).

The motivation for global extensions should be that there may be extensions which:

• (i) Make sense for all projects on a particular developer machine, say an extension that makes working over a remote session (SSH) easier...
• (ii) Make sense for some users working on a particular project (though a pubspec_override.yaml or other configuration option to disable/enable an extension for a particular checkout might be better).

---

In practice, I think (2) private_dependencies (or something similar) allowing multiple versions is highly requested, but it's also a large project and it may be quite a while before we have consensus for a thing like that.

I suspect that (1) is something we do need to solve, we haven't found a good way yet. But I don't think that an optional Flutter SDK constraint is a controversial feature. And progress on native hooks will probably only make this more urgent.

---

This is my thoughts of the top of my head. Not sure what we should do. Or if there are other reasons to do global extensions. Or maybe, tracking devtools extensions in pubspec.lock is just overkill and we should allow some things to be chaotic.

[sigurdm]

Given that devtools extensions are "just" precompiled flutter(-web) binaries. There should (from the point of view of distributing the extension) not be the need for any other dependencies?
I am not sure pub.dev is the best place for distributing these binaries, but it seems to me one strategy is to publish packages that contain just the precompiled extension and none of the source code or dependencies used for compiling the extension.

With no extensions there is no potential for conflicts?