OverlayFS: readable FS might not have cache folder to lookup getEventsSince

[wherget]

🐛 bug report

Admittedly this is a bit of an advanced Parcel setup, I'm currently trying to use an OverlayFS of a MemoryFS and NodeFS as inputFs to Parcel. I'm doing this to generate ephemeral entry files in-memory for use in a storybook build.

After writing Snapshots on first shutdown (i.e. when .parcel-cache was empty), further attempts to run the same setup fail, since the snapshot cannot be read from MemoryFS.

🤔 Expected Behavior

This works independent of whether a snapshot exists in the parcel cache.

😯 Current Behavior

ERR! FSError: ENOENT: /PROJECT_DIR/.parcel-cache/846847a2e337eb8b.txt does not exist
ERR!     at $86a3ba4eb3a17970$export$3048eb7ec07c2c4e.readFileSync (/PROJECT_DIR/packages/core/fs/src/MemoryFS.js:204:13)
ERR!     at $86a3ba4eb3a17970$export$3048eb7ec07c2c4e.readFile (/PROJECT_DIR/packages/core/fs/src/MemoryFS.js:197:17)
ERR!     at $86a3ba4eb3a17970$export$3048eb7ec07c2c4e.getEventsSince (/PROJECT_DIR/packages/core/fs/src/MemoryFS.js:602:31)
ERR!     at $ff96e39fd0cf388b$export$5963299e2424ca1c.getEventsSince (/PROJECT_DIR/packages/core/fs/src/OverlayFS.js:177:46)
ERR!     at loadRequestGraph (/PROJECT_DIR/node_modules/@parcel/core/lib/RequestTracker.js:893:18)
ERR!     at Function.init (/PROJECT_DIR/node_modules/@parcel/core/lib/RequestTracker.js:854:17)
ERR!     at Parcel._init (/PROJECT_DIR/node_modules/@parcel/core/lib/Parcel.js:256:28)
ERR!     at Parcel.watch (/PROJECT_DIR/node_modules/@parcel/core/lib/Parcel.js:315:7)
ERR!     at Object.exports.start (/PROJECT_DIR/packages/storybook-builder-parcel/index.js:49:22)
ERR!     at async Promise.all (index 0)
ERR!  $86a3ba4eb3a17970$var$FSError: ENOENT: /PROJECT_DIR/.parcel-cache/846847a2e337eb8b.txt does not exist
ERR!     at $86a3ba4eb3a17970$export$3048eb7ec07c2c4e.readFileSync (/PROJECT_DIR/packages/core/fs/src/MemoryFS.js:204:13)
ERR!     at $86a3ba4eb3a17970$export$3048eb7ec07c2c4e.readFile (/PROJECT_DIR/packages/core/fs/src/MemoryFS.js:197:17)
ERR!     at $86a3ba4eb3a17970$export$3048eb7ec07c2c4e.getEventsSince (/PROJECT_DIR/packages/core/fs/src/MemoryFS.js:602:31)
ERR!     at $ff96e39fd0cf388b$export$5963299e2424ca1c.getEventsSince (/PROJECT_DIR/packages/core/fs/src/OverlayFS.js:177:46)
ERR!     at loadRequestGraph (/PROJECT_DIR/node_modules/@parcel/core/lib/RequestTracker.js:893:18)
ERR!     at Function.init (/PROJECT_DIR/node_modules/@parcel/core/lib/RequestTracker.js:854:17)
ERR!     at Parcel._init (/PROJECT_DIR/node_modules/@parcel/core/lib/Parcel.js:256:28)
ERR!     at Parcel.watch (/PROJECT_DIR/node_modules/@parcel/core/lib/Parcel.js:315:7)
ERR!     at Object.exports.start (/PROJECT_DIR/packages/storybook-builder-parcel/index.js:49:22)
ERR!     at async Promise.all (index 0) {
ERR!   code: 'ENOENT',
ERR!   path: '/PROJECT_DIR/.parcel-cache/846847a2e337eb8b.txt'
ERR! }

💁 Possible Solution

I can imagine multiple ways to handle this, some of which may not be sensible, but unfortunately I lack the insight into parcel to suggest which might be the best.

• OverlayFS could, similar to other places, ignore the exception on the readableFs and only report events from the writableFs.
  

  parcel/packages/core/fs/src/OverlayFS.js

  Lines 167 to 183 in 349a6ca

  	async getEventsSince(
  	dir: FilePath,
  	snapshot: FilePath,
  	opts: WatcherOptions,
  	): Promise<Array<Event>> {
  	let writableEvents = await this.writable.getEventsSince(
  	dir,
  	snapshot,
  	opts,
  	);
  	let readableEvents = await this.readable.getEventsSince(
  	dir,
  	snapshot,
  	opts,
  	);
  	return [...writableEvents, ...readableEvents];
  	}

• MemoryFS could first try to compute events by loading (the current behaviour), and failing that, construct appropriate (probably created or updated) events for all files.
• The snapshot file could be copied over to MemoryFS before running getEventsSince in order for regular Event lookup to proceed.

My current workaround is the first one, but that doesn't seem like the right thing to do for when e.g. the OverlayFS is longer-lived, say across multiple Parcel.run()s, where an actual snapshot file may exist in MemoryFS.

💻 Code Sample

let workers = createWorkerFarm();
let baseFs = new NodeFS();
let tempFs = new MemoryFS(workers);
let htmlPath = await generateHTML(tempFs);
await generateJS(tempFs);
let inputFs = new OverlayFS(baseFs, tempFs);
return new Parcel({
    entries: htmlPath,
    ...moreOptions,
    workerFarm: workers,
    inputFS: inputFs,
    outputFS: baseFs
}

🌍 Your Environment

Software	Version(s)
Parcel	2.8.3
Node	19.1.0
npm/Yarn	1.22.17
Operating System	macOS 10.15.7

[mischnic]

This is definitely a usecase we want to support (just like Storybook, so please do contribute if you get somewhere with this). But you're right, there is something off with the overlayfs/memoryfs and I'm not quite sure right now how it's supposed to work together with the cache

[mischnic]

Somewhat related PR: #8857

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs.