Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to speedup writeFile? #115

Open
iliakan opened this issue Feb 2, 2023 · 1 comment
Open

How to speedup writeFile? #115

iliakan opened this issue Feb 2, 2023 · 1 comment

Comments

@iliakan
Copy link

iliakan commented Feb 2, 2023

I have a code to read a real folder from File Access API into lightningFS.

The repo is about 800kb, has 300 files.

The code works very slowly, because of writeFile (~1.3 sec).

I've read that indexedDB throttles writeFile, is that so?
How to speedup?

Here's the code, it recursively reads all dirs/files and uses fs.promises.writeFile to write them to lightningFS.

This writeFile call is the main reason for the delay, even though the data is very small.

  let fs = new LightningFS('fs', {wipe: true});

  let relPath = [''];

  async function handle(dirHandle) {
    console.time(dirHandle.name);

    for await (const entry of dirHandle.values()) {
      if (entry.kind === "file") {
        const file = await entry.getFile();
        let data = new Uint8Array(await file.arrayBuffer());
        let filePath = [...relPath, file.name].join('/');
        await fs.promises.writeFile(filePath, data);
      }

      if (entry.kind === "directory") {
        const newHandle = await dirHandle.getDirectoryHandle( entry.name, { create: false } );
        relPath.push(entry.name);
        let dirPath = relPath.join('/');
        await fs.promises.mkdir(dirPath);
        await handle(newHandle);
        relPath.pop();
      }
    }
  }

P.S. Is there any other backend for lightningFS? I need a simple in-memory strorage.
It's quite ironic that lightningFS is so sluggish for a tiny test repo.
Maybe I'm doing something wrong?

@jcubic
Copy link
Contributor

jcubic commented Oct 24, 2023

I have no idea how I've missed this issue.
I have one in memory backend implementation. It uses Map object instead of key value pair with indexedDB

https://github.com/jcubic/fake-linux-terminal

But the code is bit convoluted, it contain in memory implementation with option to persist the files in key value pair from lightning-fs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants