How are native addons going to interact with the VFS?

[RaisinTen]

• Is the VFS going to load shared objects in-memory? Is that even possible on all our supported platforms?
• Do we expect native addons to be able to access files inside the VFS? If so, how?

[jviotti]

Good questions!

Is the VFS going to load shared objects in-memory? Is that even possible on all our supported platforms?

Native add-ons are dynamically shared libraries. I believe you can load them from a random offset of a file on Linux, but they need to be separate files for macOS and Windows. @jesec can correct me if I'm wrong, but I believe PKG extracts them to a temporary directory and loads them from there.

Do we expect native addons to be able to access files inside the VFS? If so, how?

Hm, I don't think this will be possible at all with the current architecture. We are injecting the VFS logic into Node.js, so if you use any other mechanism of reading files, then they would by definition not work. @jesec does PKG deal with this in any way?

Worst case I guess we can expose the location of the VFS to the add-on and provide some native APIs to read files from there... Of course, native add-ons would have to use that and wouldn't inherit VFS capabilities out of the box.

[jviotti]

What if we unpack the SEA just once in some place like the equivalent of the app data directory on all platforms, either during the first run or during the installation phase if we consider that to be a thing?

Runtime I/O is definitely prone to these sort of things. Ideally, we would find a way to not extract anything.

I'm not sure I understand, you would need to have write access to place the binary in the filesystem in the first place, right?

@RaisinTen This is a very valid use case on enterprise settings. Usually, an IT admin with write access provisions the base system with the applications the employees would use, and then distribute the same thing as a read-only file-system for general consumption.

Fuse, but that's unlikely to be a reasonable option (non-portable, not-well-maintained on OSX / Windows, perf concerns).

The AppImage project (https://appimage.github.io) is a great example of how to do this well on Linux. AppImages "extract" the app at runtime by mounting it with fuse in a read-only manner. I don't think there are ways to accomplish something like this as well as in Linux in the other operating systems though, but sounds worth exploring.

@RaisinTen You can try opening a thread on the AppImage project and see if they have any ideas how would AppImages work i.e. on Windows?

[RaisinTen]

Ideally, we would find a way to not extract anything.

What if an SEA wants to run an executable binary? Is there a way to run that as a string without extracting it on the file system?

[jviotti]

I believe it can be done on Linux, but not sure about other OSes. We need to do some proper research... We can try to i.e. create a basic macOS terminal C/C++ program, inject another Mach-O executable on it using Postject, and attempt to execute it without extraction.

However, I don't think it will be possible to do for all operating systems. At least we can maybe keep extraction to a minimum?

[RaisinTen]

I believe it can be done on Linux, but not sure about other OSes.

Doing it on linux too seems to be complicated according to the discussion in https://stackoverflow.com/a/66053812.

At least we can maybe keep extraction to a minimum?

That seems to be the most viable option.

Another observation - even if we manage to run executables in memory / load shared objects in memory, those files might depend on files present in certain paths on the real file system. It won't be possible to configure those binaries to use the VFS C++ API if the user doesn't have access to the source code of those binaries.

[jviotti]

Hm, not sure what the best solution would be here then. The only model I know that solves this problem is the FUSE mounting approach followed by AppImages. FUSE on Linux is super fast, so you don't encounter all the issues @arcanis raised, but I don't know if it's possible in other OSes at all.

Can you research whether it's feasible to do FUSE in other systems, or at least use something that is relatively close? For example, DMGs on macOS come pretty close in that you don't "extract" the DMG when double-clicking, but only mount a "view" into the DMG without occupying extra space, but DMG mounting is pretty slow as far as I understand.

We could try:

• Not signing DMGs to avoid the mounting integrity checks
• Doing DMG without any compression at all and see how that would go

However, FUSE can ensure that the file system is always unmounted post-execution, which is not the case with DMGs...

[jviotti]

As per the call today, let's park this thread for the time-being. We'll focus on making VFS implementations possible from user-land, and we can revisit the problem of a blessed VFS implementations later on. The reason for this strategic decision is that designing a truly universal VFS seems be a very complex problem. We can focus on the other parts of SEAs for now instead of being blocked on VFS for a long time.