[WIP] feat: load IPLD formats lazily from IPFS

[alanshaw]

I saw that @hugomrdias's PR #1795 will actually remove all the exotic IPLD formats from the browser build and at first I thought that was a good idea, but then I thought maybe not, and then I thought well maybe we can just do what @daviddias suggested in the first place and load the IPLD formats from IPFS.

This PR does that.

I created https://github.com/alanshaw/ipld-formats as a tool to get the browser builds of IPLD formats onto IPFS...and this PR uses the loadFormat option of IPLD to lazily load the module from IPFS. It simply ipfs.cats the module from a CID, runs the code and returns the module to IPLD.

View this video for more info: https://youtu.be/ADA_HT2whok

In my local build I found the following bundle size improvement:

Before: 2,565,129 bytes (minified)
After: 1,864,080 bytes (minified)

~27% smaller

If y'all like the sound of this I'll pull the ipfsJsLoader into an external module and clean this up some more.

I like that the bundle size is smaller but I don't like that the versions for these modules are now not in package.json and instead are in the code...I'm not sure what to do about that. Suggestions?

cc @vmx @mikeal @ipfs/wg-gui-team

[daviddias]

This is really fantastic! This idea has been on the table for a while and it is great to see it implemented and working.

That said, I believe we need some sort of strategy to accommodate users that will be in a disjoint network (either not connected to main IPFS or not connected to main Internet at all). How will they be able to add these formats? Is there a config option to ensure that they get packaged up?

The IPLD Tutorial can/should be augmented to cover this.

[vmx]

What are the implications of running the test suite in the Browser? Would I need some additional step if I make a change to and IPLD Format?

[hugomrdias]

i would love to support this, load all things everywhere from ipfs thats the dream right ? but i can't right now ! there's still a long road ahead.

here we are entering the dapps dev environment and devs, browsers, tooling still are based on http and others old and ugly tech :)

imo doing this would bring rivers of issues and problems.

my idea on the bundle-size PR is to give the dapp dev the same level of configuration for IPLD as they already have for libp2p, i.e. "i need zcash? yes! ok simple i require it and pass it in the ipfs config ipld prop. done!" small surface, easy to understand and explain, full flexibility.

this proposal brings magic i.e. it just works but at what cost ?

• devs no longer own their production env (stuff gets lazy loaded and they have no control)
• this lazy loading can have unexpected results specially when debugging we would need to make sure we inline sourcemaps for dev and external for production. External sourcemaps would be really hard or probably impossible to make the browser load them.
• can we make sure this ipfs.cat will always work ? can we guarantee delivery? (not there yet)
• this brings overhead and non standard steps to the release process (hard coded cid and package versions in the code can be cumbersome to update)
• using new Function(<code string from ipfs>) will opt-out the dapp dev from browser and v8 optimizations

we could try this behing an experimental flag to explore the idea and make sure we have some nice visible warnings to the dapp dev explaining what's happening (only in DEV environment process.env.NODE_ENV !== production)

also we can explore this using plain http instead of ipfs

var script = document.createElement('script');
script.onload = function () {
    //do stuff with the script
};
script.src = 'url to ipls-git';

document.head.appendChild(script);

still what would be a reliable url ? unpkg.com, ipfs gateway ? can we guarantee delivery ?

[lidel]

I feel this is the future 🛰️, but like others mentioned, really rough around the edges today.

Some thoughts about raised issues:

• can we make sure this ipfs.cat will always work?: I assume pinning assets will be part of release dance. Still, if cat fails for some reason, we could fallback to HTTP by loading library from our public gateway or URL specified by developer.
• devs no longer own their production env: valid concern, but if someone cares about full control it may be enough to have a mechanism to opt-out from lazy-loading by providing a hardcoded list of ipld-formats.
• external sourcemap: would sourceMappingURL pointing at .map at our public gateway do the trick?
• ensuring versions from ipld-formats-browser.js are in sync with package.json: this sounds like something a build script could validate: look at browser section of package.json and throw an error if the right side does not match versions currently resolving for the left side.
  • To make this less cumbersome: perhaps code could be left untouched, but the logic responsible for injecting lazy-loading from IPFS moved to aegir or a webpack plugin?
    • My intuition is that it should be transparent for developer, hidden behind "optimizeBundleSizeByLoadingDepsFromIPFS: true" flag or something. Build could detect browser env and the flag and replace first calls to specific imports with lazy-load logic, and build & add them to IPFS behind the scenes. I feel if we don't reach that level of automation, maintenance cost may be too high.

[achingbrain]

The consensus is that this will make the runtime non-deterministic so may result in unpredictable behaviour.

However if this sort of thing is desired it can be done without code changes to core, you just need to know which CID a given codec name/code should map to:

const { create } = require('ipfs-core')
const toBuffer = require('it-to-buffer')
const { toString } = require('uint8arrays/to-string')

const formatLookup = {
  // resolves to a js file that implements a block codec for 'my-custom-format'
  1337: CID.parse('bafyfoo')
}

const ipfs = await create({
  ipld: {
    async loadCodec (code) {
      if (!formatLookup[code]) {
        throw new Error('Unknown format ' + code)
      }

      const buf = await toBuffer(ipfs.cat(formatLookup[code]))

      return eval(toString(buf))
    }
  }
})

// cause our custom format to be pulled from ipfs
const node = await ipfs.dag.get(CID.createV1(1337, digest))