From e63094a66d3b690561c129599861f221ffb7305c Mon Sep 17 00:00:00 2001 From: Darcy Clarke Date: Wed, 30 Mar 2022 20:35:30 +0000 Subject: [PATCH 1/4] RFC: Dependency Selector Syntax & `npm query` --- accepted/0000-dependency-selector-syntax.md | 421 ++++++++++++++++++++ 1 file changed, 421 insertions(+) create mode 100644 accepted/0000-dependency-selector-syntax.md diff --git a/accepted/0000-dependency-selector-syntax.md b/accepted/0000-dependency-selector-syntax.md new file mode 100644 index 000000000..85cedc583 --- /dev/null +++ b/accepted/0000-dependency-selector-syntax.md @@ -0,0 +1,421 @@ +# `npm query` Command & Dependency Selector Syntax + +## Summary + +Introduce a new `npm query` commmand which exposes a new dependency selector syntax (informed by & respecting many aspects of the [CSS Selectors 4 Spec](https://dev.w3.org/csswg/selectors4/#relational)). + +## Motivation + +- Standardize the shape of, & querying of, dependency graphs with a robust object model, metadata & selector syntax +- Leverage existing, known language syntax & operators from CSS to make disperate package information broadly accessible +- Unlock the ability to anwser complex, multi-faceted questions about dependencies, their relationships & associative metadata +- Consolidate redundant logic of similar query commands in `npm` (ex. `npm fund`, `npm ls`, `npm outdated`, `npm audit` ...) + +## Detailed Explanation + +This RFC's spec & implementation should closely mimic the capabilities of existing CSS Selector specifications. Notably, we'll introduce limited net-new classes, states & syntax to ensure the widest adoption & understanding of paradigms. When deviating, we'll explicitely state why & how. + +## Implementation + +- `Arborist`'s `Node` Class will have a new `.querySelectorAll()` method + - this method will return a filtered, flattened dependency Arborist `Node` list based on a valid query selector +- Introduce a new command, `npm query`, which will take a dependency selector & output a flattened dependency Node list (output is in `json` by default, but configurable) + +### Dependency Selector Syntax + +#### Overview: +- there is no "type" or "tag" selectors (ex. `div, h1, a`) as a dependency/target is the only type of `Node` that can be queried +- the term "dependencies" is in reference to any `Node` found in the `idealTree` returned by `Arborist` + +#### Selectors + +- `*` universal selector +- `#` dependency selector (equivalent to `[name="..."]`) +- `#@` (equivalent to `[name=]:semver()`) +- `,` selector list delimiter +- `.` class selector +- `:` pseudo class selector +- `>` direct decendent/child selector +- `~` sibling selector + +#### [Attribute Selectors](https://developer.mozilla.org/en-US/docs/Web/CSS/Attribute_selectors) +- `[]` attribute selector (ie. existence of attribute) +- `[attribute=value]` attribute value is equivalant... +- `[attribute~=value]` attribute value contains word... +- `[attribute*=value]` attribute value contains string... +- `[attribute|=value]` attribute value is equal to or starts with... +- `[attribute^=value]` attribute value begins with... +- `[attribute$=value]` attribute value ends with... + +#### Pseudo Selectors +- [`:not()`](https://developer.mozilla.org/en-US/docs/Web/CSS/:not) +- [`:has()`](https://developer.mozilla.org/en-US/docs/Web/CSS/:has) +- [`:where()`](https://developer.mozilla.org/en-US/docs/Web/CSS/:where) +- [`:is()`](https://developer.mozilla.org/en-US/docs/Web/CSS/:is) +- [`:root`](https://developer.mozilla.org/en-US/docs/Web/CSS/:root) matches the root node/dependency +- [`:scope`](https://developer.mozilla.org/en-US/docs/Web/CSS/:scope) matches node/dependency it was queried against +- [`:empty`](https://developer.mozilla.org/en-US/docs/Web/CSS/:empty) when a dependency has no dependencies +- [`:private`](https://docs.npmjs.com/cli/v8/configuring-npm/package-json#private) when a dependency is private +- `:link` when a dependency is linked +- `:deduped` when a dependency has been deduped +- `:override` when a dependency is an override +- `:extraneous` when a dependency could be but is not deduped +- `:outdated` when a dependency is not `latest` +- `:vulnerable` when a dependency has a `CVE` +- `:semver()` matching a valid [`node-semver`](https://github.com/npm/node-semver) spec +- `:path()` [glob](https://www.npmjs.com/package/glob) matching based on dependencies path relative to the project +- `:realpath()` [glob](https://www.npmjs.com/package/glob) matching based on dependencies realpath +- `:type()` [based on currently recognized types](https://github.com/npm/npm-package-arg#result-object) + +#### Generic Pseudo Selectors: + +A standardized pseudo selector pattern will be used for `Object`s, `Array`s or `Arrays` of `Object`s accessible via `Arborist`'s `Node.package` metadata. Pseudo classes generated in this way will allow for iterative attribute selection. + +`Array`s specifically use a special `value` keyword in place of a typical attribute name. `Arrays` also support exact `valu`e matching when a `String` is passed to the selector. See examples below: + +#### Example of an `Object`: +```css +/* return dependencies that have a `scripts.test` containing `"tap"` */ +*:scripts([test~=tap]) +``` + +#### Example of an `Array` Attribute Selection: +```css +/* return dependencies that have a keyword that begins with "react" */ +*:keywords([value^="react"]) +``` + +#### Example of an `Array` matching directly to a value: +```css +/* return dependencies that have the exact keyword "react" */ +/* this is equivalent to `*:keywords([value="react"])` */ +*:keywords("react") +``` + +#### Example of an `Array` of `Object`s: +```css +/* returns */ +*:contributors([email="ruyadorno@github.com"])` +``` + +### Classes + +Given that `Arborist` will control our understanding of the DOM (Dependency Object Model), claseses are predefined by their relationships to their ancestors & are specific to their dependency type. Dependencies may have, & are allowed to have, multiple classifications (ex. a `workspace` may also be a `dev` dependency). + +- `.prod` +- `.dev` +- `.optional` +- `.peer` +- `.bundled` +- `.workspace` + +### Attributes + +There are several known attributes that are normalized & queryable living in `Node.package` (aka. `package.json`). + +- `[name]` +- `[version]` +- `[description]` +- `[homepage]` +- `[bugs]` +- `[author]` +- `[license]` +- `[funding]` +- `[files]` +- `[main]` +- `[browser]` +- `[bin]` +- `[man]` +- `[directories]` +- `[repository]` +- `[scripts]` +- `[config]` +- `[workspaces]` +- `[dependencies]` +- `[devDependencies]` +- `[optionalDependencies]` +- `[bundledDependencies]` +- `[peerDependencies]` +- `[peerDependenciesMeta]` +- `[engines]` +- `[os]` +- `[cpu]` +- `[keywords]` + +### Command + +#### `npm query ""` (alias `q`) + +#### Options: + +- `query-output` + - Default: `json` + - Type: `json`, `list`, `explain`, `outdated`, `funding`, `audit`, `duplicates`, `file` + +#### Response Output + +- an array of dependency objects is returned which can contain multiple copies of the same package which may or may not have been linked or deduped + +```json +[ + { + "name": "", + "version": "", + "description": "", + "homepage": "", + "bugs": {}, + "author": {}, + "license": {}, + "funding": {}, + "files": [], + "main": "", + "browser": "", + "bin": {}, + "man": [], + "directories": {}, + "repository": {}, + "scripts": {}, + "config": {}, + "dependencies": {}, + "devDependencies": {}, + "optionalDependencies": {}, + "bundledDependencies": {}, + "peerDependencies": {}, + "peerDependenciesMeta": {}, + "engines": {}, + "os": [], + "cpu": [], + "workspaces": {}, + "keywords": [], + "ancenstry": "", + "path": "", + "realpath": "", + "parent": "", + "vulnerabilities": [], + "cwe": [] + }, + ... +``` + +#### Usage + +```bash +npm query ":root > .workspace > *" # get all workspace direct deps +``` + +### Extended Example Queries & Use Cases + +```stylus +// all deps +* + +// all direct deps +:root > * + +// direct production deps +:root > .prod + +// direct development deps +:root > .dev + +// any peer dep of a direct deps +:root > * > .peer + +// any workspace dep +.workspace + +// all workspaces that depend on another workspace +.workspace > .workspace + +// all workspaces that have peer deps +.workspace:has(*.peer) + +// any dep named "lodash" +// equivalent to [name="lodash"] +#lodash + +// any deps named "lodash" & within semver range ^"1.2.3" +#lodash@^1.2.3 +// equivalent to... +[name="lodash"]:semver(^1.2.3) + +// get the hoisted node for a given semver range +#lodash@^1.2.3:not(:deduped) + +// querying deps with a specific version +#lodash@2.1.5 +// equivalent to... +[name="lodash"][version="2.1.5"] + +// all deps living alongside vulnerable deps +*:vulnerable ~ *:not(:vulnerable) + +// has any deps +*:has(*) + +// has any vulnerable deps +*:has(*:vulnerable) + +// deps with no other deps (ie. "leaf" nodes) +*:empty + +// all vulnerable deps that aren't dev deps & that aren't vulnerable to CWE-1333 +*:vulnerable:not(.dev:cwe(1333)) + +// manually querying git dependencies +*[repository^="github:"], +*[repository^="git:"], +*[repository^="https://github.com"], +*[repository^="http://github.com"], +*[repository^="https://github.com"], +*[repository^="+git:..."] + +// querying for all git dependencies +*:type(git) + +// find all references to "install" scripts +*[scripts=install], +*[scripts=postinstall], +*[scripts=preinstall] + +// get production dependencies that aren't also dev deps +.prod:not(.dev) + +// get dependencies with specific licenses +*[license="MIT"], *[license="ISC"] + +// find all packages that have @ruyadorno as a contributor +*:contributors([email=ruyadorno@github.com]) +``` + +## Next Steps + +### Command Mapping to `query-output` + +Previous commands with similar behaivours will now be able to utilize `Aborist` `Node.querySelectorAll()` under-the-hood & will fast-follow the `npm query` implementation. + +#### `npm list` + +```bash +npm list # equivalent to... +npm query ":root > *" + +npm list --all # equivalent to... +npm query "*" --query-output list + +npm list # equivalent to... +npm query "#" --query-output list +``` + +#### `npm explain` + +```bash +npm explain # equivalent to... +npm query "#" --query-output explain +``` + +#### `npm outdated` + +```bash +npm outdated # equivalent to... +npm query ":root > *:outdated" --query-output outdated + +npm outdated --all # equivalent to... +npm query "*:outdated" --query-output outdated +``` + +#### `npm fund` + +```bash +npm fund # equivalent to... +npm query ":root > *[funding]" --query-output funding + +npm fund --all # equivalent to... +npm query "*[funding]" --query-output funding + +npm fund # equivalent to... +npm query "#" --query-output funding +``` + +#### `npm audit` + +```bash +npm audit # equivalent to... +npm query ":root > *:vulnerable" --query-output audit + +npm audit --all # equivalent to... +npm query "*:vulnerable" --query-output audit + +npm audit # equivalent to... +npm query "#" --query-output audit +``` + +#### `npm find-dupes` + +```bash +npm find-dupes # equivalent to... +npm query "*:deduped" --query-output duplicates +``` + +#### `npm view` + +```bash +npm view # equivalent to... +npm query ":root" --query-output view + +npm view # equivalent to... +npm query "#" --query-output view +``` + +## Commands _could_ read from `stdin` + +In a future RFC, & major version bump, `npm` could begin reading from `stdin` to chain commands together with a common understand of a dependency object. All of the below commands would add the ability to execute as if they were passed package specs (our current defulat representation of packages/dependencies). + +``` +audit, bugs, ci, config, deprecate, diff, dist-tag, docs, edit, exec, +explain, explore, find-dupes, fund, get, install, install-ci-test, +install-test, link, list, outdated, pack, pkg, prune, publish, rebuild, +repo, restart, run-script, set, set-script, shrinkwrap, star, stars, +start, stop, team, test, token, uninstall, unpublish, unstar, update, +version, view +``` + +### Example of piping from `npm query` to other commands + +```bash +# list workspaces w/ peer deps +npm query ".workspace:has(.peer)" | npm ls + +# list outdated direct dev deps +npm query ":root > .dev:outdated" | npm outdated + +# install the same dev deps across all workspaces +npm query ":root > .dev" | npm install --workspaces + +# show audit details for dependencies with a specific vulnerability/CWE +npm query "*:cwe(1333)" | npm audit + +# show audit details for vulnerable deps that aren't ReDoS dev deps +npm query "*:vulnerable:not(.dev:cwe(1333))" | npm audit +``` + +## Prior Art + +- [`HTML`](https://html.spec.whatwg.org/) & [DOM](https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model) Specifications +- [`CSS`](https://www.w3.org/Style/CSS/specs.en.html), [Selectors](https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Selectors) & [Pseudo Class](https://developer.mozilla.org/en-US/docs/Web/CSS/Pseudo-classes) Specifications +- AST Selector Libraries/Parsers + - [`estree`](https://github.com/estree/estree) + - [`abstract-syntax-tree`](https://www.npmjs.com/package/abstract-syntax-tree) + - [`postcss-selector-parser`](https://www.npmjs.com/package/postcss-selector-parser) / [API](https://github.com/postcss/postcss-selector-parser/blob/master/API.md) + - [`css-selector-parser`](https://www.npmjs.com/package/css-selector-parser) +- [pnpm's `--filter`](https://pnpm.io/filtering) Flag +- [Gzemnid](https://github.com/nodejs/Gzemnid) Package Querying + +## F.A.Q. +- Q. Is there any such thing as a bare specifier? + - A. No. Unlike CSS Selectors, there's no `"element"`, or `"dependency"` in this context, equivalent. The selector syntax presuposes all entities are packages. +- Q. Should this syntax cover **all** possible queries of a dependency graph? + - A. No. This spec is meant to provide a sufficently mature mechanism/building block for answering the majority of questions end-users have about their depndencies (re. 80/20 rule applies here) + +## Unresolved Questions & Bikeshedding +- N/A From abb2da7971c2a179421b500e754a514998847b8b Mon Sep 17 00:00:00 2001 From: Darcy Clarke Date: Wed, 30 Mar 2022 18:25:01 -0400 Subject: [PATCH 2/4] Update accepted/0000-dependency-selector-syntax.md Co-authored-by: Tierney Cyren --- accepted/0000-dependency-selector-syntax.md | 1 + 1 file changed, 1 insertion(+) diff --git a/accepted/0000-dependency-selector-syntax.md b/accepted/0000-dependency-selector-syntax.md index 85cedc583..1a7ef6b0d 100644 --- a/accepted/0000-dependency-selector-syntax.md +++ b/accepted/0000-dependency-selector-syntax.md @@ -24,6 +24,7 @@ This RFC's spec & implementation should closely mimic the capabilities of existi ### Dependency Selector Syntax #### Overview: + - there is no "type" or "tag" selectors (ex. `div, h1, a`) as a dependency/target is the only type of `Node` that can be queried - the term "dependencies" is in reference to any `Node` found in the `idealTree` returned by `Arborist` From 25ac373a0b94fd74b0dcfc8ce51630c69307eac5 Mon Sep 17 00:00:00 2001 From: Darcy Clarke Date: Wed, 6 Apr 2022 23:07:30 -0400 Subject: [PATCH 3/4] Update accepted/0000-dependency-selector-syntax.md Co-authored-by: Jordan Harband --- accepted/0000-dependency-selector-syntax.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/accepted/0000-dependency-selector-syntax.md b/accepted/0000-dependency-selector-syntax.md index 1a7ef6b0d..bf3510d14 100644 --- a/accepted/0000-dependency-selector-syntax.md +++ b/accepted/0000-dependency-selector-syntax.md @@ -195,7 +195,7 @@ There are several known attributes that are normalized & queryable living in `No "vulnerabilities": [], "cwe": [] }, - ... + … ``` #### Usage From 3d5b2130504139bdc8a3b599923aa07d2ff79c96 Mon Sep 17 00:00:00 2001 From: Darcy Clarke Date: Wed, 11 May 2022 20:55:24 +0000 Subject: [PATCH 4/4] fix: change arbitrary psuedo selector attribute selection to avoid conflicts w/ new :attr() --- accepted/0000-dependency-selector-syntax.md | 83 +++++++-------------- 1 file changed, 28 insertions(+), 55 deletions(-) diff --git a/accepted/0000-dependency-selector-syntax.md b/accepted/0000-dependency-selector-syntax.md index bf3510d14..c7f3ce891 100644 --- a/accepted/0000-dependency-selector-syntax.md +++ b/accepted/0000-dependency-selector-syntax.md @@ -7,8 +7,8 @@ Introduce a new `npm query` commmand which exposes a new dependency selector syn ## Motivation - Standardize the shape of, & querying of, dependency graphs with a robust object model, metadata & selector syntax -- Leverage existing, known language syntax & operators from CSS to make disperate package information broadly accessible -- Unlock the ability to anwser complex, multi-faceted questions about dependencies, their relationships & associative metadata +- Leverage existing, known language syntax & operators from CSS to make disparate package information broadly accessible +- Unlock the ability to answer complex, multi-faceted questions about dependencies, their relationships & associative metadata - Consolidate redundant logic of similar query commands in `npm` (ex. `npm fund`, `npm ls`, `npm outdated`, `npm audit` ...) ## Detailed Explanation @@ -39,15 +39,6 @@ This RFC's spec & implementation should closely mimic the capabilities of existi - `>` direct decendent/child selector - `~` sibling selector -#### [Attribute Selectors](https://developer.mozilla.org/en-US/docs/Web/CSS/Attribute_selectors) -- `[]` attribute selector (ie. existence of attribute) -- `[attribute=value]` attribute value is equivalant... -- `[attribute~=value]` attribute value contains word... -- `[attribute*=value]` attribute value contains string... -- `[attribute|=value]` attribute value is equal to or starts with... -- `[attribute^=value]` attribute value begins with... -- `[attribute$=value]` attribute value ends with... - #### Pseudo Selectors - [`:not()`](https://developer.mozilla.org/en-US/docs/Web/CSS/:not) - [`:has()`](https://developer.mozilla.org/en-US/docs/Web/CSS/:has) @@ -60,7 +51,9 @@ This RFC's spec & implementation should closely mimic the capabilities of existi - `:link` when a dependency is linked - `:deduped` when a dependency has been deduped - `:override` when a dependency is an override -- `:extraneous` when a dependency could be but is not deduped +- `:extraneous` when a dependency exists but is not defined as a dependency of any node +- `:invalid` when a dependency version is out of its ancestors specified range +- `:missing` when a dependency is not found on disk - `:outdated` when a dependency is not `latest` - `:vulnerable` when a dependency has a `CVE` - `:semver()` matching a valid [`node-semver`](https://github.com/npm/node-semver) spec @@ -68,40 +61,52 @@ This RFC's spec & implementation should closely mimic the capabilities of existi - `:realpath()` [glob](https://www.npmjs.com/package/glob) matching based on dependencies realpath - `:type()` [based on currently recognized types](https://github.com/npm/npm-package-arg#result-object) -#### Generic Pseudo Selectors: +#### [Attribute Selectors](https://developer.mozilla.org/en-US/docs/Web/CSS/Attribute_selectors) -A standardized pseudo selector pattern will be used for `Object`s, `Array`s or `Arrays` of `Object`s accessible via `Arborist`'s `Node.package` metadata. Pseudo classes generated in this way will allow for iterative attribute selection. +The attribute selector evaluates the key/value pairs in `package.json` if they are `String`s. -`Array`s specifically use a special `value` keyword in place of a typical attribute name. `Arrays` also support exact `valu`e matching when a `String` is passed to the selector. See examples below: +- `[]` attribute selector (ie. existence of attribute) +- `[attribute=value]` attribute value is equivalant... +- `[attribute~=value]` attribute value contains word... +- `[attribute*=value]` attribute value contains string... +- `[attribute|=value]` attribute value is equal to or starts with... +- `[attribute^=value]` attribute value begins with... +- `[attribute$=value]` attribute value ends with... + +#### `Array` & `Object` Attribute Selectors + +The generic `:attr()` pseudo selector standardizes a pattern which can be used for attribute selection of `Object`s, `Array`s or `Arrays` of `Object`s accessible via `Arborist`'s `Node.package` metadata. This allows for iterative attribute selection beyond top-level `String` evaluation. + +`Array`s specifically use a special `item` keyword in place of a typical attribute name. `Arrays` also support exact `valu`e matching when a `String` is passed to the selector. See examples below: #### Example of an `Object`: ```css /* return dependencies that have a `scripts.test` containing `"tap"` */ -*:scripts([test~=tap]) +*:attr(:scripts([test~=tap])) ``` #### Example of an `Array` Attribute Selection: ```css /* return dependencies that have a keyword that begins with "react" */ -*:keywords([value^="react"]) +*:attr(:keywords([item^="react"])) ``` #### Example of an `Array` matching directly to a value: ```css /* return dependencies that have the exact keyword "react" */ /* this is equivalent to `*:keywords([value="react"])` */ -*:keywords("react") +*:attr(:keywords("react")) ``` #### Example of an `Array` of `Object`s: ```css /* returns */ -*:contributors([email="ruyadorno@github.com"])` +*:attr(:contributors([email="ruyadorno@github.com"])) ``` ### Classes -Given that `Arborist` will control our understanding of the DOM (Dependency Object Model), claseses are predefined by their relationships to their ancestors & are specific to their dependency type. Dependencies may have, & are allowed to have, multiple classifications (ex. a `workspace` may also be a `dev` dependency). +Classes are defined by the package relationships to their ancestors (ie. the dependency types that are defined in `package.json`). This approach is user-centric as the ecosystem has been taught to think about dependencies in these classifications first-and-foremost. Dependencies are allowed to have multiple classifications (ex. a `workspace` may also be a `dev` dependency & may also be `bundled` - a selector for that type of dependency would look like: `*.workspace.dev.bundled`). - `.prod` - `.dev` @@ -110,39 +115,6 @@ Given that `Arborist` will control our understanding of the DOM (Dependency Obje - `.bundled` - `.workspace` -### Attributes - -There are several known attributes that are normalized & queryable living in `Node.package` (aka. `package.json`). - -- `[name]` -- `[version]` -- `[description]` -- `[homepage]` -- `[bugs]` -- `[author]` -- `[license]` -- `[funding]` -- `[files]` -- `[main]` -- `[browser]` -- `[bin]` -- `[man]` -- `[directories]` -- `[repository]` -- `[scripts]` -- `[config]` -- `[workspaces]` -- `[dependencies]` -- `[devDependencies]` -- `[optionalDependencies]` -- `[bundledDependencies]` -- `[peerDependencies]` -- `[peerDependenciesMeta]` -- `[engines]` -- `[os]` -- `[cpu]` -- `[keywords]` - ### Command #### `npm query ""` (alias `q`) @@ -153,7 +125,7 @@ There are several known attributes that are normalized & queryable living in `No - Default: `json` - Type: `json`, `list`, `explain`, `outdated`, `funding`, `audit`, `duplicates`, `file` -#### Response Output +#### Example Response Output - an array of dependency objects is returned which can contain multiple copies of the same package which may or may not have been linked or deduped @@ -194,6 +166,7 @@ There are several known attributes that are normalized & queryable living in `No "parent": "", "vulnerabilities": [], "cwe": [] + ... }, … ``` @@ -286,7 +259,7 @@ npm query ":root > .workspace > *" # get all workspace direct deps *[license="MIT"], *[license="ISC"] // find all packages that have @ruyadorno as a contributor -*:contributors([email=ruyadorno@github.com]) +*:attr(:contributors([email=ruyadorno@github.com])) ``` ## Next Steps