feat(bundle-source): Zip original sources with --no-transforms #2294

kriskowal · 2024-05-21T21:13:43Z

Description

This change adds a mode to the bundle-source command with the initial flag --no-transforms that generates “endo zip base64” style bundles without applying the module-to-program transform and SES shim censorship evasion transforms, such that the original files on disk appear in the zip file. This is a preparatory step, necessary for building test artifacts, in advance of full support for this bundle style.

Security Considerations

bundle-source is part of the Endo and Agoric toolkit and it, or its surrogate, participate in the toolchain for generating content that can be confined by Hardened JavaScript, but is not trusted by Hardened JavaScript at runtime. It does however currently run with all the authority of the developer in their development environment and its integrity must be carefully guarded.

Scaling Considerations

No improvements expected at this time, but in pursuit of #400, it may be possible to move the heavy and performance sensitive JavaScript transform components from bundle-source to import-bundle and only suffer the performance cost of these transforms on Node.js, where those costs are more readily born by some runtimes. Precompiled bundles may continue to be the preferred medium for deployment to the web, for example.

Documentation Considerations

We will need to advertise the --no-transforms flag eventually, since there will be a period where it is advisable if not necessary to generate contracts and caplets targeting the XS runtime.

Testing Considerations

I have included a test that verifies the API behavior and manually run the following to verify behavior for the CLI:

rm -rf bundles
yarn bundle-source --no-transforms --cache-json bundles demo/circular/a.js circular-a
rm -rf circular-a
mkdir -p circular-a
jq -r .endoZipBase64 bundles/bundle-circular-a.json | base64 -d > circular-a/circular-a.zip
(cd circular-a; unzip circular-a.zip)
jq . circular-a/compartment-map.json
# verifying the final module entires have parser: 'mjs'

Compatibility Considerations

This flag is opt-in and breaks no prior behaviors. This introduces a new entry to the build cache meta-data and may cause some bundles to be regenerated one extra time after upgrading.

Upgrade Considerations

This should not impact upgrade, though it participates in the greater #400 story which will require xsnap upgrades to come to bear.

kriskowal · 2024-05-21T21:20:56Z

Looking for review buddies for work pursuant to XS native compartments, #400. This change touches bundle-source code I’ve shared maintainership with @dckc and compartment-mapper which has benefited tremendously from contributions from @naugtur and @boneskull, built on substantial work by @michaelfig. I would like to page @gibson042 into Compartment Mapper since it touches upon work we propose to TC39.

boneskull

I had a vision... a prophecy. it was... merge conflicts

boneskull · 2024-05-21T21:59:51Z

packages/bundle-source/cache.js

@@ -251,11 +275,13 @@ export const makeBundleCache = (wr, cwd, readPowers, opts) => {
   * @param {string} rootPath
   * @param {string} [targetName]
   * @param {Logger} [log]
+   * @param {object} [options]


What is this type?

packages/bundle-source/src/main.js

packages/bundle-source/src/zip-base64.js

boneskull · 2024-05-21T22:05:55Z

packages/bundle-source/src/zip-base64.js

+  const {
+    dev = false,
+    cacheSourceMaps = false,
+    noTransforms = false,
+    commonDependencies,
+  } = options;


please add types for this function's parameters

packages/bundle-source/test/no-transforms.test.js

packages/compartment-mapper/src/archive-lite.js

packages/compartment-mapper/src/archive.js

boneskull · 2024-05-21T22:28:24Z

packages/compartment-mapper/src/import-parsers.js

+export const parserForLanguage = {
+  mjs: parserMjs,
+  cjs: parserCjs,
+  json: parserJson,
+  text: parserText,
+  bytes: parserBytes,


in case this informs anything you're doing here: I have a branch that has exposed user-defined languages and parsers, which seems like the least-awful way to get native modules running.

I’m not going to land this before Tuesday next week. I do not mind rebuilding this on top of your more rigorous treatment. I’m not jazzed about exposing parserForLanguage to the public API but it was sufficient and minimal for the bundleSource needs.

I'm not going to land my code before Tuesday next week, so it's kind of moot. I originally had a parserForLanguage option, but it turned out that I needed a new language, so it became an API that looks like:

parsers: [ { languages: ['native'], parser: someParserImpl, extensions: ['node'] } ]

I think I can make this work on top of your change without too much trouble. My intent is to make these -parsers.js modules continue to export parserForLanguage for internal use, but also languageForExtension and parsers like above that merges the two for public use. I can use the stub module to hide the implementation detail from the public.

For those tuning in, we landed @boneskull’s work with some modifications to compose well with this line of work, and I removed the overlapping commits from this PR in the rebase. I then reconstructed the middle commits of this PR as #2306, as that was common ground for our separate concerns, and then removed the overlapping commits from this PR. All that remains now is the final integration in bundle-source, with very little modification from the changes that have been already approved here. The only big difference is that I’ve additionally decoupled the function that constructs a preliminary compartment map through exploration of node_modules on a file system.

dckc · 2024-05-21T21:44:47Z

packages/compartment-mapper/src/import-lite.js

+ * @param {ArchiveOptions} [options]
+ * @returns {Promise<Application>}
+ */
+export const loadLocation = async (readPowers, moduleLocation, options) => {


I'm used to default args:

Suggested change

export const loadLocation = async (readPowers, moduleLocation, options) => {

export const loadLocation = async (readPowers, moduleLocation, options = {}) => {

dckc · 2024-05-30T16:38:07Z

packages/bundle-source/test/no-transforms.test.js

+  const moduleText = new TextDecoder().decode(moduleBytes);
+  const originalModuleText = await fs.promises.readFile(entryPath, 'utf-8');
+  // And, just to be sure, the text in the bundle matches the original text.
+  t.is(moduleText, originalModuleText);


Refs: #400 and #2294 ## Description In this change, we carve and export `-lite.js` and `-parsers.js` out of `import.js`, `archive.js`, and `import-archive.js`, as well revealing `mapNodeModules` through `@endo/compartment-mapper/node-modules.js` revealing hidden flexibility already present in the Compartment Mapper. This allows the Compartment Mapper to mix and match “language behaviors” with different workflows (e.g., using `import-parsers.js` with `archive-lite.js` instead of `archive-parsers.js` generates archives with original sources instead of precompiled sources.) We also decouple the process of discovering the physical compartments such that `node-modules.js` can be substituted for alternate compartment exploration algorithms, e.g., combining a custom lockfile and package cache from a specific package management solution. ### Security Considerations Does not cross security considerations. ### Scaling Considerations Exporting more narrowly scoped modules allows consumers to be avoid entraining Babel in a greater variety of scenarios. ### Documentation Considerations Evolution of the reference documentation should suffice, though another pass at README.md to dig into these more surgical and composable APIs may be worthwhile. ### Testing Considerations The existing scenarios are extensive and fully cover the behaviors that we’ve factored out. I expect #2294 tests to cover the new scenarios. ### Compatibility Considerations This change takes great care to preserve the existing behavior of the composite `import.js`, `archive.js`, and `import-archive.js` interfaces, only revealing existing behaviors that were previously internal, allowing more expressible scenarios. ### Upgrade Considerations This change will not affect upgrade and paves a migration path forward for preserving backward-compatibility for `bundleSource` and `importBundle` as we draw closer to supporting XS native compartments.

kriskowal force-pushed the kriskowal-zip-original-sources branch 2 times, most recently from 9f6119c to 61a24ef Compare May 21, 2024 21:16

kriskowal force-pushed the kriskowal-zip-original-sources branch 2 times, most recently from 80fb0d5 to bf2af00 Compare May 21, 2024 22:03

boneskull approved these changes May 21, 2024

View reviewed changes

kriskowal force-pushed the kriskowal-zip-original-sources branch from bf2af00 to 40ccdae Compare May 29, 2024 22:50

kriskowal mentioned this pull request May 29, 2024

feat(compartment-mapper): Custom parser support #2304

Merged

4 tasks

dckc approved these changes May 30, 2024

View reviewed changes

kriskowal mentioned this pull request Jun 3, 2024

feat(compartment-map): Decouple discovery and languages #2306

Merged

kriskowal force-pushed the kriskowal-zip-original-sources branch 2 times, most recently from 7b4bc52 to e807ddf Compare June 4, 2024 19:38

kriskowal added 3 commits June 4, 2024 13:07

feat(bundle-source): Zip original sources with --no-transforms

2af54c3

perf(bundle-source): Allow imports to run parallel

f3afe94

chore: Update yarn.lock

9d28fb8

kriskowal force-pushed the kriskowal-zip-original-sources branch from e807ddf to 9d28fb8 Compare June 4, 2024 20:08

kriskowal merged commit 0c0b93b into master Jun 4, 2024
17 checks passed

kriskowal deleted the kriskowal-zip-original-sources branch June 4, 2024 20:26

kriskowal mentioned this pull request Jun 4, 2024

docs(bundle-source): Document no transforms mode #2307

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(bundle-source): Zip original sources with --no-transforms #2294

feat(bundle-source): Zip original sources with --no-transforms #2294

kriskowal commented May 21, 2024 •

edited

kriskowal commented May 21, 2024

boneskull left a comment

boneskull May 21, 2024

boneskull May 21, 2024

boneskull May 21, 2024

kriskowal May 21, 2024

boneskull May 22, 2024

kriskowal May 29, 2024

kriskowal Jun 4, 2024

dckc May 21, 2024

dckc May 30, 2024

	export const loadLocation = async (readPowers, moduleLocation, options) => {
	export const loadLocation = async (readPowers, moduleLocation, options = {}) => {

feat(bundle-source): Zip original sources with --no-transforms #2294

feat(bundle-source): Zip original sources with --no-transforms #2294

Conversation

kriskowal commented May 21, 2024 • edited

Description

Security Considerations

Scaling Considerations

Documentation Considerations

Testing Considerations

Compatibility Considerations

Upgrade Considerations

kriskowal commented May 21, 2024

boneskull left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kriskowal commented May 21, 2024 •

edited