backFile(): add http-backed files to superblock

[fuzzyTew]

Provides for HTTP backing of files when there is no .superblock.txt file or when it is missing content.

These changes allow for a crude alternative to 'dumb' HTTP git remotes as in isomorphic-git/isomorphic-git#672 . A third-party server folder can now back the filesystem directly and be used as a copy-on-write git store, with this PR and some basic boilerplate such as:

let fs = new LightningFS('httpgit', {
	wipe: true,
	url: url,
	urlauto: true
})
let pfs = fs.promises
let packs = await pfs.readFile('/objects/info/packs', { encoding: 'utf8' })
packs = packs.match(/pack-.{40}\.pack/g)
for (let pack of packs) {
	await pfs.backFile('/objects/pack/' + pack.slice(0, 45) + '.idx')
}
git.plugins.set('fs', fs)

[fuzzyTew]

I removed my old comment after noticing a promise that wasn't waited for, but I still do not see a reason for this test failure. I tried testing the master branch on azure and it also failed.

[billiegoose]

Maybe it's a permissions thing. I'll try re-running it 🤷‍♂

[fuzzyTew]

The tests pass fine locally for me. If the PR looks good, would you be willing to try the tests on your system? Even your test-passing commits to master are failing for me on azure, with the same error.

Mine are on the left, yours on the right, click to see larger:

[billiegoose]

Yes, I think it's a CI configuration thing... the environment variables aren't available to PRs created on forks. It's surprisingly tricky to make that work right without accidentally making it so anyone can steal your npm or Github tokens by opening a PR.

I actually do like this addition, so I think I will merge it! Don't worry about the CI, I'll figure that out tomorrow and take a closer look at this. Mostly I'm wondering if I can make backFile and the HTTP backend more powerful. For instance, it would be nice if you could specify multiple URLs. Maybe fs.mount('/imgs', 'https://i.static.foo.com') or something. I just want to think about it a bit before committing to an API.

[isomorphic-git-bot]

🎉 This PR is included in version 3.4.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[billiegoose]

So I've been thinking about it, and I have too many ideas. I want to add a "hooks" API so that users can register interceptor functions to run before or after each method. Then you could do something like:

fs.after('readFile', async (path, content) => {
  // Lazily detect "index" files when they are read by the application
  if (path.endsWith('/objects/info/packs')) {
    let root = path.replace(/\/objects\/info\/packs$/, '')
    // Parse their contents
    for (let pack of content.match(/pack-.{40}\.pack/g)) {
      // Inject new files and directories into the file system
      await this.addInode({
        filepath:`${root}/objects/pack/${pack.slice(0, 45)}.idx`,
        type: 'file',
        size: await fetch({ url, method: 'HEAD' }).headers.contentLength
      })
    }
  }
})

But clearly my designing and scheming would take too long and slow you down - you've already made 3 PRs! So screw it, we'll settle for backFile for now, and worry about directories and multiple HTTP roots and mounting and stuff later.

[fuzzyTew]

Well, the other 2 PRs didn't depend on this one =) But thank you! Looking forward to the newer API when you put all the pieces together. I'm thinking the simple generic approach might be to make mount plugins, and HTTP and IDB would be separate ones.