Repository navigation
Manifest files for unixfs loaded via HTTP Gateway #257
Description
Activity
- added
on Nov 18, 2021 Related are notes from the 2021-11-16 IPFS operators meeting: https://hackmd.io/MiYYUCR-RWqhAKfJ_B15wA
Reacted by Jay Freeman (saurik)Hi @lidel. I started working at Fission this year and have recently been tasked with implementing redirects support in go-ipfs. @cbrake started working on this in this PR 🙏 and I plan on continuing the work in this PR.
If you don't mind, I'd like to make sure I understand the process here as far as specs and issue management go so I don't cause any annoying friction.
From the issues I've read through it appears that it was decided to split out redirects support from manifests support when Cloudflare announced Cloudflare Pages with
_redirectssupport, similar to Netlify's_redirectssupport. This issue I'm commenting on appears to be specific to spec'ing out manifest support. Should redirects support get its own spec and therefore spec issue or were you thinking it would be included in the same spec?Regardless of whether or not manifests and redirects are separate specs, my assumption is that the redirects portion of the spec will both inform and be informed by the go-ipfs redirects support tentative implementation. Does that sound right? Is there any preferred order to tackling this (spec vs implementation first)? I'm currently assuming I should start with essentially implementing what Cloudflare supports and see what feedback others provide, adjust for feedback, and then eventually work backward from there into text for the spec.
Assuming I'm on the right track, I was planning on submitting 1) an issue in go-ipfs to track this implementation work vs just my draft PR, and 2) possibly an issue in this specs repo to track the redirects portion of the spec, unless it should all be included in this spec issue.
Thank you in advance for any feedback and guidance. If you have any other tips on the code base or any plans you had for how the code should be reorged (I see you've been doing some refactoring recently) I'd love to hear them.
Thank you! 🙏
Reacted by Jay Freeman (saurik) and Alexander MangelHi @justincjohnson. Thanks for reaching out here. @lidel is great at seeing GitHub notifications and processing them. I don't know how many others are though. If you don't get a response in the next day or so, feel free to start a thread in #ipfs-operators in FIL Slack or #ipfs-dev in IPFS Discord.
Reacted by Justin Johnson and Jay Freeman (saurik)Hi @justincjohnson, really appreciate reaching out!
I believe tldr from the prior discussions is that we translated requirements into two sibling features that will improve the devexp around unixfs website hosting using IPFS gateway:
_redirects(prior art in netlify docs, also discussed in Proposal: native redirects config kubo#7392 (comment))- does not seem to be controversial, ready to be worked on
- mvp should support:
- relative and absolute redirects
- "catch-all" redirect for PWAs
_headersfile (prior art in netlify docs)- bit more tricky, as we need to be very careful which headers are allowed where
- comes with some open questions around which headers could be specified without
X-fooprefix.
Feel free to work on both, but focusing on
_redirectsfirst is a good idea. I am happy to guide / help with reviews. 👍the redirects portion of the spec will both inform and be informed by the go-ipfs redirects support tentative implementation. Does that sound right? Is there any preferred order to tackling this (spec vs implementation first)? I'm currently assuming I should start with essentially implementing what Cloudflare supports and see what feedback others provide, adjust for feedback, and then eventually work backward from there into text for the spec. [..]
I was planning on submitting 1) an issue in go-ipfs to track this implementation work vs just my draft PR, and 2) possibly an issue in this specs repo to track the redirects portion of the spec, unless it should all be included in this spec issue.Yes, this plan sounds good.
There are caveats around both headers and redirects that we will fully understand while writing sharness tests for them, so specs may change multiple times before we finish the mvp implementation. It is fine to keep discussion around go-ipfs PR and wait with specs until the dust settles.
Should redirects support get its own spec and therefore spec issue or were you thinking it would be included in the same spec?
We have various gateway improvements in flight, so it is better to have separate specs for now.
When you feel like writing the spec, just open a PR againstipfs/specsthat adds it as a standalone file undergateways/REDIRECTS_FILE.md– this way we can get it done without being blocked on anything.ps. I left some quick comments in ipfs/kubo#8816
Reacted by Daniel NormanReacted by Jay Freeman (saurik)- changed the title
[-]Manifest for unixfs loaded via HTTP Gateway[/-][+]Manifest files for unixfs loaded via HTTP Gateway[/+]on Mar 24, 2022 Fabulous! Thanks for the guidance @lidel. I'll get started.
Reacted by Jay Freeman (saurik) and Marcin RatajFor the benefit of anyone coming here looking for information on redirects support, see ipfs/kubo#8890.
@lidel a
_headersfile would be fantastic. At eth.limo, we've had multiple users request the addition/removal/modification of the following (user defined) headers:User defined headers
Header Rationale content-security-policyuseful not only for iframing protection but also for better dependency/resource security and XSS defense. Ideally dApps/dWebsites should have whitelisted domains and in-line script hashes within the html bundle. Encouraging better version pinning could help defend against things like the Ledger JS library compromise from last year. cross-origin-resource-policybest practices for cross-origin isolation. Additionally, advanced features such as SharedArrayBufferrequire one or more cross-origin-* headers to be properly configured. See: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffercross-origin-opener-policy^^^ cross-origin-embedder-policy^^^ permissions-policyShould be customizable by the content publisher, depending on what features are required. x-frame-optionsConfigurable depending upon the need for iframing. However, this can also be addressed via the content-security-policyframe-ancestordirective.x-xss-protectionlegacy, but can provide a "belt and suspenders" approach for XSS protections in conjunction with a well defined content-security-policy. At a minimum, only the1; mode=blockdirective should be configurable due to potential vulnerabilities arising from botched filtering: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection#vulnerabilities_caused_by_xss_filteringreferrer-policyuseful for dApps/dWebsites interested in some privacy mitigations for public gateways. This would greatly improve the functionality of static dWebsites delivered locally, or through a gateway.
Gateway operator defined headers
There are many more, but this is a good starting point. These should not be configurable by the end user as they can globally impact the operation and quality of the service.
Header Rationale strict-transport-securityas mentioned elsewhere, this is really designed for the gateway operator to configure. It has global ramifications and should not be configurable by content owners. clear-site-dataanother header that can potentially alter top-level origin settings. Ideally the gateway operator would configure this to automatically wipe cookies issued by other subdomains in the absence of inclusion on the Public Suffix List (PSL) cache-controllikely something the gateway operator should control since caching is directly related to resource usage (bandwidth, etc...) Reacted by Marcin Rataj and Joel ThorstenssonThank you, useful feedback.
Agree, allowing arbitrary headers is risky, especially in the future when new things are added to the web platform.To get this moving froward, we could start with a safelist of allowed headers.
Ones that allow website owners to adjust security for their Origin sgtm.For next steps here, interested parties could
- open an IPIP similar to one that added
_redirects:- https://specs.ipfs.tech/ipips/ipip-0002/
- https://specs.ipfs.tech/http-gateways/web-redirects-file/
- It would add
_headersto web gateways with Origin isolation (subdomains, DNSLink gws), andip[fn]s://handlers), similar to one that added_redirects, and have it a list of headers that are allowed to be customized.
- write reference implementation for boxo/gateway library (which ships in IPFS Desktop/Kubo/rainbow/Brave)
If anyone is interested in kicking this off and providing implementation and specs, I am willing to lend necessary review time. 👍
Sponsoring this work via https://ipshipyard.com/contact-us is also an option.- open an IPIP similar to one that added
Two things worth recording here.
The pattern is proven.
_redirectsfrom IPIP-0002 is implemented in boxo, so Kubo and Rainbow, and in@helia/verified-fetch, which is what the Service Worker Gateway runs. A file in the content root that changes gateway behaviour works across both stacks.This is also the home for the browser-side asks now: ipfs/in-web-browsers#152, no way for a publisher to override
Content-Type, and ipfs/in-web-browsers#176, website manifest, were both closed pointing here.What is missing is the headers half. #364 tracks the other shape, carrying the media type in the UnixFS DAG itself. Either way it needs an IPIP and tests in gateway-conformance.
This is a placeholder issue for creating a spec for "content root's manifest files" which would be part of the data published on IPFS (eg.
/ipfs/CID/_somethingor/ipfs/CID/.well-known/ipfs/foo) that provides hints to HTTP Gateway around things like redirects, content types and custom HTTP headers.Related reading:
Possible reuse of:
_redirects(prior art in netlify docs_headersfile (prior art in netlify docs and cloudflare docs)Feel free to discuss/post ideas below, or open PRs with IPIP drafts that could be reviewed independently.