feat(compartment-map): Decouple discovery and languages

[kriskowal]

Refs: #400 and #2294

Description

In this change, we carve and export -lite.js and -parsers.js out of import.js, archive.js, and import-archive.js, as well revealing mapNodeModules through @endo/compartment-mapper/node-modules.js revealing hidden flexibility already present in the Compartment Mapper. This allows the Compartment Mapper to mix and match “language behaviors” with different workflows (e.g., using import-parsers.js with archive-lite.js instead of archive-parsers.js generates archives with original sources instead of precompiled sources.) We also decouple the process of discovering the physical compartments such that node-modules.js can be substituted for alternate compartment exploration algorithms, e.g., combining a custom lockfile and package cache from a specific package management solution.

Security Considerations

Does not cross security considerations.

Scaling Considerations

Exporting more narrowly scoped modules allows consumers to be avoid entraining Babel in a greater variety of scenarios.

Documentation Considerations

Evolution of the reference documentation should suffice, though another pass at README.md to dig into these more surgical and composable APIs may be worthwhile.

Testing Considerations

The existing scenarios are extensive and fully cover the behaviors that we’ve factored out. I expect #2294 tests to cover the new scenarios.

Compatibility Considerations

This change takes great care to preserve the existing behavior of the composite import.js, archive.js, and import-archive.js interfaces, only revealing existing behaviors that were previously internal, allowing more expressible scenarios.

Upgrade Considerations

This change will not affect upgrade and paves a migration path forward for preserving backward-compatibility for bundleSource and importBundle as we draw closer to supporting XS native compartments.

[kriskowal]

Advise reviewing commits individually. I took some care to rename and edit in separate steps.

[boneskull]

if I'm being fussy, I'd want to see, for each of these new files, a module-level comment describing the module's use-case. and descriptions for each of the exported functions.

probably needs a NEWS.md update.

there's nothing here that I'd block about.

[boneskull]

(suggestion/observation/trolling) there's obviously a convention at play here, but I'd omit the .js from the keys

[kriskowal]

I very much want to. It’s not just convention, though. This makes the Compartment Mapper continue to work when used on some older foundations like node -r esm that do not read the "exports" directive, so we can’t use it for aliases, just to limit exposure. We have one lingering integration test that needs to be replaced before we can drop this pattern.

[boneskull]

Didn't realize that esm was a target

[boneskull]

Would it be reasonable to provide both extension-laden and extension-free exports?

[kriskowal]

I think that’s reasonable when after eliminate standardthings/esm from our toolchain. Until then, better to encourage usage patterns that are portable.

[boneskull]

(suggestion) IMO it isn't necessary to explain this, but nbd. I would probably even just zap the @typedef altogether and use typeof Compartment in the parseArchive signature's docstring

[kriskowal]

Using typedef Compartment doesn’t work for some reason, thus the comment. My preference as well.

[boneskull]

@kriskowal FWIW I cannot reproduce whatever the problem was that you ran into. This compiles cleanly for me:

/**
 * @typedef {object} Options
 * @property {string} [expectedSha512]
 * @property {HashFn} [computeSha512]
 * @property {Record<string, unknown>} [modules]
 * @property {ExitModuleImportHook} [importHook]
 * @property {typeof Compartment} [Compartment]
 * @property {ComputeSourceLocationHook} [computeSourceLocation]
 * @property {ComputeSourceMapLocationHook} [computeSourceMapLocation]
 * @property {ParserForLanguage} [parserForLanguage]
 */

[boneskull]

that said, it is a little weird that we are expecting the Compartment constructor here. there is only one Compartment constructor. is there a place where we're actually using a different one?

[kriskowal]

Yeah, there are Compartment compatible constructors created by wrapping the Compartment constructor. Notably, Compartment before lockdown is different than after. Every Compartment has its own instance of Compartment. They’re structurally compatible, though. Unclear to me why that would make a difference to types, but they do have different values.

[boneskull]

even though TS is structural, because Compartment is a function, it will only ever allow the global Compartment from whichever version of ses provides it. That may be fine, but also seems redundant. Though it may be useful if we're using some mocked-out Compartment in tests? even then it seems like it would be the same Compartment constructor

[kriskowal]

src/import-archive-lite.js:266:4 - error TS2502: 'Compartment' is referenced directly or indirectly in its own type annotation.

266  * @param {typeof Compartment} [options.Compartment]

[boneskull]

ah, I see that now. It's because of the DefaultCompartment. typeof DefaultCompartment works

[boneskull]

if I'm being fussy, I'd want to see, for each of these new files, a module-level comment describing the module's use-case. and descriptions for each of the exported functions.

probably needs a NEWS.md update.

there's nothing here that I'd block about.

[boneskull]

mo tags mo problems

[boneskull]

Didn't realize that esm was a target

[boneskull]