Proposal: native redirects config

[cwaring]

Native redirects for Web Apps on IPFS

Problem

When hosting a website or application on IPFS served via a gateway you will often need to manage redirects to ensure links are not broken for previously referenced urls, these can appear in search indexes or external sites. If a path is changed and you are not manually proxying results from IPFS via a web-server with configured redirects then a user is presented an unhelpful 404 error, such as:

ipfs resolve -r /ipns/<DOMAIN>/<PATH>/: no link named "<RESOURCE>" under <CID>

This is currently true for results redirected to a local node via IPFS Companion or for IPFS hosting services such as Fleek.

Proposal

We allow a plain text root _redirects config that could be checked to transparently redirect to the new resource, with optional response codes for correct routing.

---

How could this work?

Each deploy could include a _redirects config at the root of the CID tree which could be checked by go-ipfs in attempt to locate the new resource or follow a pattern to gracefully direct users to the new location.

A few examples:

# Redirect with a 301
/expired-path  /             301

# Redirect with a 302
/my-redirect  /              302

# Show a custom 404 for this path
/store    /store-closed       404

# Rewrite a path
/pass-through /index.html    200

Splats

/news/*  /blog/:splat

# redirect all paths under /jobs to /careers/
/jobs/*  /careers/:splat

Placeholders

/news/:year/:month/:date/:slug  /blog/:year/:month/:date/:slug

Client-side apps with history pushstate (React, Vue, Angular, etc)

When developing a single page app with client-side routing you often wish the delegate all routing to the application using a rule like this.

/*    /index.html   200

Custom 404 pages

If a resource can not be resolved then fallback to matching a root /404.html or a custom option by matching a path

/en/* /en/404.html 404
/zh/* /zh/404.html 404

These examples are based on syntax from the Netlify _redirect configuration file, which I believe is a great pattern to follow as it is simple and would allow easy migration for existing apps hosted on Netlify.

Related Issues

#6214
ipfs-inactive/docs#405

If there is other existing work, issues or solutions that I have missed please add them to the discussion below and I'll include them in this proposal.

[terichadbourne]

Thanks so much for this proposal @cwaring! That splat option was particularly useful when we replaced the Filecoin docs site content on Netlify and @andyschwab has confirmed that the forwarding options are much more limited as he tries to move to IPFS on Fleek. Big +1 for this proposal to help developers use familiar patterns when moving their sites to IPFS.

[Stebalien]

404 pages will be supported in 0.6.0 (an ipfs-404.html somewhere upstream of a missing page will do it). It's now technically possible to implement meta redirects with this custom 404 page, but we can do better.

I'd like to just implement #6214, but there's a performance tradeoff: we'd need to lookup that file for every request to process things like mime types. The nice thing about your proposal is that we only need to perform redirects when content isn't found.

[ItalyPaleAle]

there's a performance tradeoff: we'd need to lookup that file for every request to process things like mime types

Would caching help here?

• The first time a folder is requested, retrieve the manifest file. The file then gets stored in the local IPFS object cache. At that point, it's just one file I/O away.
• The file can also be cached in memory for a short amount of time (say, 5 minutes?) to avoid having to read it from disk at all.

[lidel]

Cloudflare just announced Cloudflare Pages available for free with built-in support for _redirects files.

It may now be worth evaluating if redirects could be handled separately from the general-purpose manifest, which is tracked in #6214

By going with _redirects we would simplify our stack and leverage existig mindshare, SEO, and the docs (netlify, cloudflare) which improves both onboarding and ux/dx.

[Stebalien]

That sounds like a great idea!

[lidel]

Some related feedback form https://github.com/Velenir/nextjs-ipfs-example#readme by @Velenir:

IPFS doesn't support automatic redirect to index for 404 routes, which is commonly employed when hosting SPA

If we implement _redirect file, it will cover this use case, as described in https://docs.netlify.com/routing/redirects/rewrites-proxies/#history-pushstate-and-single-page-apps

[bmann]

Yeah, this looks good!

[cwaring]

Yes I'd love to see this happen and will even easier to port existing static apps to IPFS.

Currently Cloudflare only support simple 301/302 redirects but it looks like they will add splats, placeholders etc in the future. We can combine that with an ipfs-404.html fallback for now for soft client-side routing.

[bmann]

Talked about this during the IPFS Operators group. @lidel what do we need to do to move this forward?

I think:

• share config for supporting this in Nginx, so it works today for those that run proxies
• assign someone to implement in go-ipfs

There are a bunch of considerations of how a particular operator supports this as part of a publish process to make it work with their system, and we're happy to talk about how we would do it.

I think it should be a goal to have this work on arbitrary IPFS gateways for maximum portability.

[lidel]

Let's do this in go-ipfs, avoid custom behaviors at nginx level: this makes sense only if it is the default behavior on all HTTP gateways, including local ones run by Brave users. The absolute / in _redirect file must map to the content root (/ipfs/$CID/), to work with Origin-based isolation on the web (in dnslink and subdomain gateways, native URIs).

I am happy to review / advise if someone has bandwidth to submit a PR draft.

[ilyaigpetrov]

Sorry, I've just skim-read this topic, but still have a message to share.

1. IPFS-gateway must be abstract, it doesn't need to know about http-redirects. It's purpose is to give access to files in IPFS network via HTTP(S) and any other purpose must be implemented in a plugin for the gateway.
2. Redirects on the gateway may be generated from IPFS/IPLD files that take a role similar to (sym)link. Ideally, files on IPFS should know nothing about HTTP redirects (ipfs is not a http gateway), so even if you mount IPFS node into a filesystem (without any browser) then there should be no http-redirects but some files serving (sym)link-like role.

[wyc]

Agreed, I think the success condition for IPFS gateways should be output protocol agnosticism--it should work whether it's across HTTP, HTTPS, SCP, FTP, rsync/ssh, etc. without the transport protocol leaking into IPFS itself. After we figure out serving files, I'd be interested in semantic and protocol-agnostic ways to write files to the server.

[lidel]

Those are valid concerns. Agree on transport agnosticism: there will always be a mode on gateway to fetch raw block or DAG without any post-processing. We will be adding even more explicit way to do this in #8234

The UX/DX improvement discussed here is only relevant when we talk about unixfs with website data that was specifically crafted for consumption via web browsers. A catch-all redirect set up for single-page-application makes little sense when represented on filesystem.

Future-proofing websites

I believe our main concern around "website hosting" should be future-proofing.
Every time we add a feature to gateways, and people start building websites that rely on it, the same behavior will be expected to work in all the future web browsers with native IPFS support. They need to re-implement the same feature set, otherwise websites put on IPFS will break. In 2022 gateway will get its own spec and a compliance test suite, to ensure our basic implementations do not diverge.

Just to be sure we are on the same page when it comes to the line in sand, some additional context below.

Porcelain already provided by go-ipfs gateway

Gateways already do various things to make hosting websites more pleasant:

• set various HTTP headers based on data read from IPFS (guess content-type, set cache-control, etag)
• return index.html instead of directory listing, if such file is present
• redirect folders to URL that ends with /
• detect Host header and return content behind DNSLink

We already have two ways of doing redirects on websites published via ipfs2http gateways:

• creating a dummy .html file with <meta http-equiv="refresh" content="0;URL='http://somethingelse.example.com/'" />
  • this is fully independent of gateway software, redirect happens on the client
• creating ipfs-404.html file in the root dir and using it as a catch-all
  • this requires gateway software to support the catch-all via custom 404 Not Found page (but then the actual redirect happens on the client, either via meta header or some JS)
  • (personally I wish we had something with wider utility like _redirect discussed here or something that also allows controlling content-type (Proposal: manifest file for the IPFS gateway #6214), but ipfs-404.html is a thing and here to stay)

Are redirects enough?

The next here is to decide if support for _redirects files brings enough value on top of what already exists.
Afaik added value is not just splats and placeholders, but removing the need for polluting the filesystem with dummy .html files that only have <meta redirect.

[ilyaigpetrov]

I believe our main concern around "website hosting" should be future-proofing.

What about introducing gateway plugins and making "Static site hosting" plugin enabled by default which will provide already existing porcelain?

[mathew-cf]

Something to consider is that Netlify has support for various options (status code, redirect based on header, etc.) _redirects. Not sure how many of these are wanted/needed for IPFS
https://docs.netlify.com/routing/redirects/redirect-options/

[lidel]

Either way, we need specs first. Let's consolidate and continue in ipfs/specs#257

[2color]

In case folks come across this, this was resolved in the following issue: #8890

The spec can be found here: https://github.com/ipfs/specs/blob/main/IPIP/0002-gateway-redirects-file.md

[cwaring]

So excited to see this land, thanks everyone for the hard work to make it happen! Extra kudos to @lidel, @justincjohnson ❤️ + @TMoMoreau & @johnnymatthews (for docs)

[ianconsolata]

The documentation for this can be found here: https://docs.ipfs.tech/how-to/websites-on-ipfs/redirects-and-custom-404s/#evaluation

This issue shows up highly when searching for redirects on IPFS, so wanted to make sure folks can find the actual docs on it if they end up here.