should extensions be easily installable via npm?

[boneskull]

I understand the reasons for providing a CLI to manage extensions (appium driver / appium plugin), but I am wondering if anyone would want to just use npm (or Yarn or pnpm or w/e) to install extensions?

Setting the implementation discussion or technical challenges aside for a moment, there are drawbacks to the current strategy:

1. A Node.js-based project will need to take extra steps to get Appium with extensions installed as a dependency (currently this happens via special properties in package.json or env, and only as a postinstall hook).
2. The special properties mentioned above don't seem to take versions into account. It should be possible to use semver to define what extensions are needed.
3. We cannot guarantee postinstall hooks will forever remain supported or enabled by default. This would mean extensions would need manual installation.
4. Does not take into account that other package managers may be used or required.
5. Extensions escape checks built with standard tooling, e.g., npm audit, dependabot, license compliance checks like FOSSA, etc.
6. IDEs may need special instruction to understand where these extensions live. e.g., if a plugin offers a new command, a project cannot offer "intellisense" about the new command unless that extension has been installed locally or has been otherwise configured manually.
7. A comfortable Node.js or JS developer will find the bespoke workflow annoying, since it's bespoke.

To be clear, I'm not proposing that we abandon e.g., appium driver install, because it's undoubtedly quite useful for many end-users. However, this isn't everyone, and I believe we could find an alternative means of extension management--one which would offer the usual benefits of ecosystem conventions to which Node.js/JS developers are accustomed.

Looking for feedback from @jlipps especially 😄

[jlipps]

I think there are lots of benefits to the extension CLI and we should keep it, which it sounds like you agree with. Beyond that I don't care. If there's a way to make it nicely interoperable with standard NPM tooling (via some discovery process we add to Appium as a fallback or additional search parameter for appium home), then that's cool too. I guess I would just be interested to consider a proposal sketch that you have in mind as the next step.

[boneskull]

Before I can do that, I think I need some more information about the current design goals:

• I'm unclear about the intent of extensions.yaml. It is not a manifest of what should be installed (like a package.json) because that information is stored elsewhere (NPM_CONFIG_DRIVERS, NPM_CONFIG_PLUGINS). Is it a description of what is installed, at least at runtime? Are missing extensions pruned from extensions.yaml? Why is it a thing?
• Is extensions.yaml YAML because it's intended to be human-readable & editable?
• No package.json or package-lock.json is created when installing into APPIUM_HOME. Rather than being treated like node_modules, it is itself a kind of node_modules. However, dependencies do not seem to be commingled; only the extension modules themselves are subdirectories of APPIUM_HOME, and not the dependencies of the extensions. While this certainly makes for a "cleaner" directory, was there any other intent here?
• Is there anything else about APPIUM_HOME that I should know about?
• Is --source=local mainly aimed at extension developers? Is it actually necessary? npm install /path/to/module (and thusly appium driver install /path/to/module) does basically the same thing, AFAIK.

[jlipps]

I'm unclear about the intent of extensions.yaml

Right now it's a way for Appium itself to know what is installed, and have easy access to metadata about what's installed, so it doesn't have to do weird searches of NPM directory structures to try to figure that out.

It is not a manifest of what should be installed

No, but one vision for your extension.yaml would be to move it to a new computer and then run some (not yet implemented) command like appium sync which would make your local appium home reflect what's in the extensions yaml. so yes, part of the idea was that this could be a definition sysadmins could use to ensure identical appium setups across systems.

The only direction I ever envisioned was YAML -> NPM, not the other way around (i.e., I never thought we should try to figure out what is actually installed and write that back to the YAML).

Is extensions.yaml YAML because it's intended to be human-readable & editable?

Yes, for the reason I just mentioned.

However, dependencies do not seem to be commingled; only the extension modules themselves are subdirectories of APPIUM_HOME, and not the dependencies of the extensions. While this certainly makes for a "cleaner" directory, was there any other intent here?

No, the only intent was to be absolutely sure that each extension has whatever dependencies it would have in isolation, without any range dependencies being pulled up or down by what some other extension more specifically requires. I think I actually implemented this as a result of running afoul of that situation in a concrete fashion. Hence the weird two level install to ensure isolation.

Is there anything else about APPIUM_HOME that I should know about?

If you have a package.json in some directory above APPIUM_HOME, it messes things up (NPM looks for the highest package.json in the directory tree to figure out where to install things). Or maybe it did but I prevented that now?

Is --source=local mainly aimed at extension developers? Is it actually necessary?

Extension developers yes, but more importantly anyone who needs a driver that's not available externally. Think privately distributed drivers. We use it at HeadSpin, for example, for drivers we've developed that aren't on NPM.

[boneskull]

@jlipps

Is --source=local mainly aimed at extension developers? Is it actually necessary?
Extension developers yes, but more importantly anyone who needs a driver that's not available externally. Think privately distributed drivers. We use it at HeadSpin, for example, for drivers we've developed that aren't on NPM.

what I was trying to get at was that, if appium driver install <driver> checked if driver was in the list of known drivers, and it was not, perhaps it's a filepath or other URL, and you could just pass that through to npm, and it'd do the same thing as --source=local.

[boneskull]

@jlipps

No, the only intent was to be absolutely sure that each extension has whatever dependencies it would have in isolation, without any range dependencies being pulled up or down by what some other extension more specifically requires. I think I actually implemented this as a result of running afoul of that situation in a concrete fashion. Hence the weird two level install to ensure isolation.

IIRC this can be accomplished with npm install --global-style <pkg> which does no deduping or commingling of dependencies.

If you have a package.json in some directory above APPIUM_HOME, it messes things up (NPM looks for the highest package.json in the directory tree to figure out where to install things). Or maybe it did but I prevented that now?

I'll test with --global-style and see what happens here.

You mentioned some time ago that you had clients that didn't want to deal with npm at all. I'm curious about the "why" behind that.

[jlipps]

oh it's just that they don't care about node or npm or know how to use them, they just use appium. for these people it is a stretch to ask them to npm install anything including appium. so once they have appium it's nice to feel that they're just working with "appium"

[boneskull]

I can confirm that --global-style avoids the "package.json in a parent dir" problem. likewise, once a package.json has been created in an otherwise empty APPIUM_HOME, npm install --prefix=. --no-package-lock --global-style will reinstall as expected.

we can create an $APPIUM_HOME/.npmrc which has the configuration we need, and admins can then use these two files to reproduce driver installations across machines. add a .gitignore w/ node_modules and you could just commit APPIUM_HOME to VCS.

[jlipps]

ok that's pretty cool. so maybe we can undo my extra nested directory hack. i didn't know about --global-style. and then we can write an appium driver reinstall command that would reinstall drivers based on .npmrc? i guess that doesn't work if we mix drivers and plugins.