Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Content-Addressable Asset Storage (IPFS, etc.) #2322

Open
lindner opened this Issue Jan 15, 2019 · 0 comments

Comments

Projects
None yet
1 participant
@lindner
Copy link

lindner commented Jan 15, 2019

The current implementation of Known stores most media assets in a local filesystem and creates UUIDs to reference them. As an alternate I would like to propose supporting Content-Addressable storage. That is, storage based on the Hash of the content. This is used by IPFS and other systems. It can be applied to both user uploaded content and especially for embed content that is fetched and rendered.

With this in place it will be possible to use a public IPFS gateway instead of a CDN (Content Delivery Network) Read more on how CloudFlare supports this in their blog post.

Ideal scenario is that core has the right hooks and then an IPFS plugin (or DAT, or Swarm, or other future system) can use these for to do their job.

Proposed Solution

  1. Replace or augment the UUID handling for uploaded content/images to use consistent hashing of the content for URLs.
  • IPFS uses the hash of the content, or the hash of a container that holds references to other content.
  • The same content always has the same hash values, this means that this content is deduplicated between instances and scales with usage.
  1. Allow for an alternate URL prefix for content serving.

    • Internal file-based, using the /file/ prefix, but use consistent hashing for paths that 'look like' hashes, due to encoding they will start with Qm
    • Internal path-mapped IPFS, use /ipfs/ prefix on same server
    • External IPFS content service like https://cloudflare-ipfs.com/ipfs/
  2. On upload allow for hashing the content, storing content in an external system, and storing the resulting consistent hash code

  3. Handle Deletes, since there is no deleting content once it's out there.

  4. Support Content-Addressable assets in imageproxy. Instead of storing the encoded URL, store the content hash and pin the content locally. This allows for the same image to be shared among multiple instances.

  5. Bonus points: store generated js/css assets in IPFS as well. On server startup find the static assets used and add them to the content-addressable backend and use the resulting hash codes to construct the serving URLs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.