[rush] Initial SelectorExpressionParser implementation

[elliot-nelson]

Summary

• Introduce "Selector Expression JSON" syntax.
• Introduce "Selector Expression String" syntax.
• Implement a robust project selector at the API layer (instead of the CLI layer).

This PR is related to #3327 (but maybe doesn't close it).

Details

Introduction

Today, we select projects with a command like rush build --to rush-lib --from tag:App.

• This command uses two different selectors (selector scopes): the implicit scope name: and tag:.
• It uses two different parameters (which I like to refer to as "filters"): to and from.
• Finally, there's an implicit operator, logical OR (aka set union), which is what we always use in CLI today.

The idea behind selector expressions is to combine these selectors, filters, and operators in a more formal way that allows more complex selection sets.

Selector Expression JSON

The JSON for expressions come in small blocks, 3 flavors, one for each type above:

• { scope: 'git', value: 'origin/main' } is a selector, the equivalent of git:origin/main
• { filter: 'to', arg: <expr> } is a filter which operates on sub-expression, the equivalent of --to
• { op: 'and', args: [<expr>, <expr>] } is an operator, which takes one or more sub-expression operands (args)

We can now express a concept like "the set of all projects impacted by project XYZ and tagged sdk", like so:

{
  "op": "and"
  "args": [
    {
      "filter": "to",
      "arg": {"scope": "name", "value": "XYZ" }
    },
    { "scope": "tag", "value": "sdk" }
  ]
}

This JSON syntax could be generated on the fly and passed in via a file param, or it could even be used for more permanent arrangements, such as saved in Rush configuration files to filter certain future features by project in a configurable way.

Selector Expression Strings

For ad-hoc project selection on the command line, it would be convenient to express the JSON structure above in a standardized string format as well. This PR proposes a simple parser for such a string, with the following rules:

• The operators "and", "or", and "not" are expressed as space-separated words.
• Filter names (like "to" and "from") act just like "not" (i.e. like unary operators).
• Operator precedence AND > OR > NOT/FILTER > SELECTOR.
• Parentheses can be used anywhere to group expressions.
• For project names or selectors that could be misinterpreted (e.g. a project named "and" or other symbols in it), use square brackets to wrap the name - [and].

Thus, the JSON expression in the section above would be expressed as to XYZ and tag:sdk. On the command line:

rush build --select 'to XYZ and tag:sdk'

# All of the above are also totally valid and equivalent:
rush build --select 'to(XYZ) and (tag:sdk)'
rush build --select '(to XYZ) and (tag:sdk)'
rush build --select '((to XYZ) and tag:sdk)'

# The following is NOT equivalent (it is more like a --to CLI filter):
rush build --select 'to(XYZ and tag:sdk)'

API Layer

Because the new objects are built at API layer, they should be equally usable outside of the Rush CLI context, e.g.:

// a custom node utility
import { RushProjectSelector } from '@microsoft/rush-lib';

const projects = await new RushProjectSelector(rushConfiguration).selectExpressionString('....... some expression string .......');

for (const project of projects) {
  // custom processing
}

Future extensions

@dmichon-msft originally suggested that the concept of "selector expressions" could be a plugin for the selector types (essentially making it a complex selector type). I am proposing a different approach, which would be instead to make the concept of a selector expression a core feature of Rush, that is not extendable -- for example, there is no way to introduce new core keywords like "and".

However, future Rush plugin types could introduce new selector scopes (e.g. "tag"). Or, they could introduce new selector filters (e.g. "to"). The implementation proposed in this PR would support new scopes and filters, but not new operators or changes to the JSON syntax overall.

How it was tested

A few unit tests added for documentation.

Impacted documentation

• Document the new --select parameter
• Document the concept of selector expressions
• Document the concept of JSON selector expressions

[elliot-nelson]

@octogonz Hi there 👋 I've added enough now that you can actually use the proposed string formatting for selection expressions, e.g.:

rush list --select 'from tag:A and not tag:B'

Let me know what you think. There's still a little bit of cleanup to do -- in particular, maybe moving Terminal out of some of the SelectorParser classes -- that I'm postponing for now.

[elliot-nelson]

Note: this implementation was designed to be a ~minimal amount of code to get the job done, and not take on any new dependencies.

It might make sense, instead, to use an existing defined API for parsing -- something like https://www.npmjs.com/package/parsinator. An existing parsing utility library will likely be much better at providing detailed error messages, at the risk of containing more functionality than is necessary for these expressions.

[elliot-nelson]

Closing for now -- this pull request contains the string-based expression syntax which will require more design review, will revisit after the JSON expressions have merged.