Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Manifest files for unixfs loaded via HTTP Gateway #257

Open
1 of 2 tasks
lidel opened this issue Nov 18, 2021 · 8 comments
Open
1 of 2 tasks

Manifest files for unixfs loaded via HTTP Gateway #257

lidel opened this issue Nov 18, 2021 · 8 comments
Assignees
Labels
dif/expert Extensive knowledge (implications, ramifications) required help wanted kind/discussion Topical discussion; usually not changes to codebase kind/enhancement A net-new feature or an improvement to an existing feature kind/maintenance Work required to avoid breaking changes or harm to project's status quo need/community-input Needs input from the wider community

Comments

@lidel
Copy link
Member

lidel commented Nov 18, 2021

This is a placeholder issue for creating a spec for "content root's manifest files" which would be part of the data published on IPFS (eg. /ipfs/CID/_something or /ipfs/CID/.well-known/ipfs/foo) that provides hints to HTTP Gateway around things like redirects, content types and custom HTTP headers.

Related reading:

Possible reuse of:

Feel free to discuss/post ideas below, or open PRs with IPIP drafts that could be reviewed independently.

@lidel lidel added help wanted dif/expert Extensive knowledge (implications, ramifications) required kind/discussion Topical discussion; usually not changes to codebase kind/enhancement A net-new feature or an improvement to an existing feature kind/maintenance Work required to avoid breaking changes or harm to project's status quo need/community-input Needs input from the wider community labels Nov 18, 2021
@BigLep
Copy link
Contributor

BigLep commented Dec 19, 2021

Related are notes from the 2021-11-16 IPFS operators meeting: https://hackmd.io/MiYYUCR-RWqhAKfJ_B15wA

@justindotpub
Copy link
Contributor

Hi @lidel. I started working at Fission this year and have recently been tasked with implementing redirects support in go-ipfs. @cbrake started working on this in this PR 🙏 and I plan on continuing the work in this PR.

If you don't mind, I'd like to make sure I understand the process here as far as specs and issue management go so I don't cause any annoying friction.

From the issues I've read through it appears that it was decided to split out redirects support from manifests support when Cloudflare announced Cloudflare Pages with _redirects support, similar to Netlify's _redirects support. This issue I'm commenting on appears to be specific to spec'ing out manifest support. Should redirects support get its own spec and therefore spec issue or were you thinking it would be included in the same spec?

Regardless of whether or not manifests and redirects are separate specs, my assumption is that the redirects portion of the spec will both inform and be informed by the go-ipfs redirects support tentative implementation. Does that sound right? Is there any preferred order to tackling this (spec vs implementation first)? I'm currently assuming I should start with essentially implementing what Cloudflare supports and see what feedback others provide, adjust for feedback, and then eventually work backward from there into text for the spec.

Assuming I'm on the right track, I was planning on submitting 1) an issue in go-ipfs to track this implementation work vs just my draft PR, and 2) possibly an issue in this specs repo to track the redirects portion of the spec, unless it should all be included in this spec issue.

Thank you in advance for any feedback and guidance. If you have any other tips on the code base or any plans you had for how the code should be reorged (I see you've been doing some refactoring recently) I'd love to hear them.

Thank you! 🙏

@BigLep
Copy link
Contributor

BigLep commented Mar 23, 2022

Hi @justincjohnson. Thanks for reaching out here. @lidel is great at seeing GitHub notifications and processing them. I don't know how many others are though. If you don't get a response in the next day or so, feel free to start a thread in #ipfs-operators in FIL Slack or #ipfs-dev in IPFS Discord.

@lidel
Copy link
Member Author

lidel commented Mar 24, 2022

Hi @justincjohnson, really appreciate reaching out!

I believe tldr from the prior discussions is that we translated requirements into two sibling features that will improve the devexp around unixfs website hosting using IPFS gateway:

  • _redirects (prior art in netlify docs, also discussed in Proposal: native redirects config kubo#7392 (comment))
    • does not seem to be controversial, ready to be worked on
    • mvp should support:
      • relative and absolute redirects
      • "catch-all" redirect for PWAs
  • _headers file (prior art in netlify docs)
    • bit more tricky, as we need to be very careful which headers are allowed where
    • comes with some open questions around which headers could be specified without X-foo prefix.

Feel free to work on both, but focusing on _redirects first is a good idea. I am happy to guide / help with reviews. 👍

the redirects portion of the spec will both inform and be informed by the go-ipfs redirects support tentative implementation. Does that sound right? Is there any preferred order to tackling this (spec vs implementation first)? I'm currently assuming I should start with essentially implementing what Cloudflare supports and see what feedback others provide, adjust for feedback, and then eventually work backward from there into text for the spec. [..]
I was planning on submitting 1) an issue in go-ipfs to track this implementation work vs just my draft PR, and 2) possibly an issue in this specs repo to track the redirects portion of the spec, unless it should all be included in this spec issue.

Yes, this plan sounds good.

There are caveats around both headers and redirects that we will fully understand while writing sharness tests for them, so specs may change multiple times before we finish the mvp implementation. It is fine to keep discussion around go-ipfs PR and wait with specs until the dust settles.

Should redirects support get its own spec and therefore spec issue or were you thinking it would be included in the same spec?

We have various gateway improvements in flight, so it is better to have separate specs for now.
When you feel like writing the spec, just open a PR against ipfs/specs that adds it as a standalone file under gateways/REDIRECTS_FILE.md – this way we can get it done without being blocked on anything.

ps. I left some quick comments in ipfs/kubo#8816

@lidel lidel changed the title Manifest for unixfs loaded via HTTP Gateway Manifest files for unixfs loaded via HTTP Gateway Mar 24, 2022
@justindotpub
Copy link
Contributor

Fabulous! Thanks for the guidance @lidel. I'll get started.

@justindotpub
Copy link
Contributor

For the benefit of anyone coming here looking for information on redirects support, see ipfs/kubo#8890.

@eth-limo
Copy link

@lidel a _headers file would be fantastic. At eth.limo, we've had multiple users request the addition/removal/modification of the following (user defined) headers:

User defined headers

Header Rationale
content-security-policy useful not only for iframing protection but also for better dependency/resource security and XSS defense. Ideally dApps/dWebsites should have whitelisted domains and in-line script hashes within the html bundle. Encouraging better version pinning could help defend against things like the Ledger JS library compromise from last year.
cross-origin-resource-policy best practices for cross-origin isolation. Additionally, advanced features such as SharedArrayBuffer require one or more cross-origin-* headers to be properly configured. See: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer
cross-origin-opener-policy ^^^
cross-origin-embedder-policy ^^^
permissions-policy Should be customizable by the content publisher, depending on what features are required.
x-frame-options Configurable depending upon the need for iframing. However, this can also be addressed via the content-security-policy frame-ancestor directive.
x-xss-protection legacy, but can provide a "belt and suspenders" approach for XSS protections in conjunction with a well defined content-security-policy. At a minimum, only the 1; mode=block directive should be configurable due to potential vulnerabilities arising from botched filtering: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection#vulnerabilities_caused_by_xss_filtering
referrer-policy useful for dApps/dWebsites interested in some privacy mitigations for public gateways.

This would greatly improve the functionality of static dWebsites delivered locally, or through a gateway.

Gateway operator defined headers

There are many more, but this is a good starting point. These should not be configurable by the end user as they can globally impact the operation and quality of the service.

Header Rationale
strict-transport-security as mentioned elsewhere, this is really designed for the gateway operator to configure. It has global ramifications and should not be configurable by content owners.
clear-site-data another header that can potentially alter top-level origin settings. Ideally the gateway operator would configure this to automatically wipe cookies issued by other subdomains in the absence of inclusion on the Public Suffix List (PSL)
cache-control likely something the gateway operator should control since caching is directly related to resource usage (bandwidth, etc...)

@lidel
Copy link
Member Author

lidel commented Mar 13, 2024

Thank you, useful feedback.
Agree, allowing arbitrary headers is risky, especially in the future when new things are added to the web platform.

To get this moving froward, we could start with a safelist of allowed headers.
Ones that allow website owners to adjust security for their Origin sgtm.

For next steps here, interested parties could

If anyone is interested in kicking this off and providing implementation and specs, I am willing to lend necessary review time. 👍
Sponsoring this work via https://ipshipyard.com/contact-us is also an option.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dif/expert Extensive knowledge (implications, ramifications) required help wanted kind/discussion Topical discussion; usually not changes to codebase kind/enhancement A net-new feature or an improvement to an existing feature kind/maintenance Work required to avoid breaking changes or harm to project's status quo need/community-input Needs input from the wider community
Projects
No open projects
Status: 🏃‍♀️ In Progress
Development

No branches or pull requests

4 participants