Direct writes to local disk images

[ethanaobrien]

I was looking around the settings in google chrome and I saw there was a File Editing option in the site settings
chrome://settings/content/siteDetails?site=http%3A%2F%2Fcopy.sh

Could we use this to read/write directly to the disk file so we don't have to press get hard disk image every time we want to save?

It would be a big change but I think it would speed up the emulator a lot and it would use a lot less memory.

[copy]

Someone correct me if I'm wrong, but I don't think it's possible. File looks like it's read-only, while FileSystem only gives you access to a virtual filesystem.

[hello-smile6]

Yes, @copy @ethanaobrien . It is possible, using the Filesystem Access API. In fact, it should be implemented, as the current implementation (storing the disk in memory) defeats the purpose of swap.

[hello-smile6]

Could it be implemented? All you'd need is to change the disk image from being stored in memory to being written to disk.

[hello-smile6]

What's the timeline for this?

[copy]

@hello-smile6 I was expecting your contribution ...

[ethanaobrien]

@copy I got a chance to play around with this for a bit. Here's a small demo on how to use that api.

<button id="cf">Choose file</button>
<script>
    // See https://blog.merzlabs.com/posts/native-file-system/
    // and https://developer.mozilla.org/en-US/docs/Web/API/Window/showOpenFilePicker
    // and https://developer.mozilla.org/en-US/docs/Web/API/FileSystemFileHandle/createWritable
    // and https://developer.mozilla.org/en-US/docs/Web/API/FileSystemWritableFileStream
    // For some more information/exmaples
    cf.addEventListener('click', async () => {
        if (!window.showOpenFilePicker) {
            // Show file picker requires https
            // and is only currently supported in chrome, edge, and opera (See https://developer.mozilla.org/en-US/docs/Web/API/Window/showOpenFilePicker#browser_compatibility)
            console.error("showOpenFilePicker not supported!");
            return;
        }
        // Data to write
        const data = "Data to write";

        // Options for the file picker
        const pickerOpts = {
            // Types that can be chosen
            types: [
                {
                    description: 'Plain Text',
                    accept: {
                        'text/*': ['.txt']
                    }
                },
            ],
            // Removes the option to select types not specified above
            excludeAcceptAllOption: true,

            // Allow selecting multiple files
            multiple: false
        };
        // Open the picker
        const files = await showOpenFilePicker(pickerOpts);

        // Get the file
        let file = await files[0].getFile();
        const fileLength = file.size;
        console.log(files[0], file);

        // Get (and log) the current file contents
        console.log("Current file contents:", await file.text());

        // Obtain write permissions
        const writable = await files[0].createWritable();

        // Write the data to the file
        // It is (theoretically) possible too seek in the file
        // see https://developer.mozilla.org/en-US/docs/Web/API/FileSystemWritableFileStream/seek
        await writable.write(data);

        // Close the writable
        await writable.close();

        // Get the file again (its the same method on the same object,
        // but since the file was updated we will get new file contents)
        // We cannot just call file.text again.
        file = await files[0].getFile();

        // And then get (and log) the new file contents
        const contents = await file.text();
        console.log("New file contents:", contents);

        console.assert(data === contents);
    });
</script>

Hope this can help!

Edit: Edited to add comments

[Pixelsuft]

NodeJS way - a simple file buffer that automaticly writes back to the image (may unsafe):

/**
 * Asynchronous buffer with writeback support
 *
 * @constructor
 * @param {string} filename Name of the file to download
 * @param {boolean} writeback Allow writeback
 * @param {number|undefined} size
 */
function AsyncNodeJSBuffer(filename, writeback, size) {
  this.fs = require("fs");
  this.filename = filename;
  this.writeback = writeback;
  this.byteLength = size || this.fs.statSync(filename).size;

  this.fd = 0;

  this.onload = undefined;
  this.onprogress = undefined;
}

AsyncNodeJSBuffer.prototype.load = function() {
  this.fs.open(this.filename, this.writeback ? "r+" : "r", (error, fd) => {
    if (error) {
      throw new Error("Cannot load: " + this.filename + ". " + error);
    } else {
      this.fd = fd;
      this.onload && this.onload(Object.create(null));
    }
  });
}

AsyncNodeJSBuffer.prototype.destroy = function() {
  this.fs.close(this.fd, error => {
    if (error) {
      throw new Error("Cannot close: " + this.filename + ". " + error);
    }
  });
}

/**
 * @param {number} offset
 * @param {number} len
 * @param {function(!Uint8Array)} fn
 */
AsyncNodeJSBuffer.prototype.get = function(offset, len, fn) {
  const buffer = new Uint8Array(len);
  this.fs.read(this.fd, buffer, 0, len, offset, (error, bytesRead, buffer) => {
    if (error) {
      throw new Error("Cannot read: " + this.filename + ". " + error);
    } else {
      fn(buffer);
    }
  });
}

/**
 * @param {number} start
 * @param {!Uint8Array} data
 * @param {function()} fn
 */
AsyncNodeJSBuffer.prototype.set = function(start, data, fn) {
  if (!this.writeback)
    return;
  console.assert(start + data.byteLength <= this.byteLength);
  this.fs.write(this.fd, data, 0, data.byteLength, start, (error, bytesWritten, data) => {
    if (error) {
      throw new Error("Cannot write: " + this.filename + ". " + error);
    } else {
      fn();
    }
  });
};

AsyncNodeJSBuffer.prototype.get_buffer = function(fn) {
  fn();
};

AsyncNodeJSBuffer.prototype.get_state = function() {
  // All changes should be written to the disk
  return [];
};

AsyncNodeJSBuffer.prototype.set_state = function(state) {
  return;
};

[ethanaobrien]

NodeJS way - a simple file buffer that automaticly writes back to the image (may unsafe):

We are running this in the browser, so we can't use this

[nklayman]

I'm gonna give this issue a go, any suggestions before I start working on it? I was thinking I'd make a class that serves as a drop-in replacement for a standard File object, so the buffer could be initialized to the new class (or a normal File still) and work the same way everywhere it's used. Then, whenever it's written to (ie in ide.js), the write would actually go to disk instead of in-memory if the new class was used. It seems like the class would only need to support get, set, and byteLength which should be doable with the FileSystemFileHandle api. For reading, the API gives a standard File object so that could be kept the same, and for writing you can get a seekable write stream to the file. Does this sound like a good plan?

[ethanaobrien]

@nklayman I could never get a seakable writable stream to work properly. I recommend experimenting with it before trying this. When writing data to a file at a specific byte, it would erase the rest of the file. Let me know if you manged to get around this, I'd like to know how. Thanks!

[copy]

@nklayman That sounds like a good approach, although there seems to be some confusion around which class to implement: File is a browser API that can be read using FileReader, which will be difficult to shim. You'll probably want to implement a class similar to AsyncFileBuffer, which is a v86-specific class that implements the get, set methods you mentioned. I'd suggest not implementing read caching, and also skip get_state/set_state.

[nklayman]

@ethanaobrien I also found that in-place writes don't work, but they do when using SyncAccessHandle. This has the downside of only working within the Origin Private File System, but that should be okay (I was able to allocate an 8gb file there with no issue). The only downside is you would have to manually save the file to a user-accessible location on disk, but this should still heavily reduce memory usage by storing the disk file on the host disk instead of in memory.

@copy AsyncFileBuffer seems like the better class to implement, I'll do that.

[nklayman]

Here's a working read-only implementation: https://github.com/nklayman/v86/tree/origin-private-file-system. I copy the file into the origin private file system, then can read from (and hopefully write to) it. The file operations have to be in a web worker due to API limitations. My current test setup (linux.iso) doesn't create an AsyncFileBuffer for the disk image so it gets no writes. Any suggestions on how to configure the app so that it issues writes to that?

Also, what's the best way to get/generate a large disk image to test the performance/memory usage of my implementation?