[🚢 SHIPPED] SXG republisher service, web extension, remote and local proxy

[jimpick]

This idea came to me based on experimentation here and here:

• Signed/Bundled HTTP Exchanges and WebPackage ipfs/in-web-browsers#121
• https://github.com/jimpick/signed-exchange-test

I posted it to the #ipfs-in-web-browsers IRC channel on Freenode:

i dusted off https://github.com/jimpick/signed-exchange-test last night ... bought a wildcard cert for *.ipfs.v6z.me from digicert ($13 for 7 days)

i'm going to experiment to see if i can use a proxy instead of the service worker ... theory being that i can serve up the .sxg files for any url under *.ipfs.v6z.me if i do that

i could make a tiny service to allow people to republish static IPFS sites under -.ipfs.v6z.me

on a regular (daily?) basis, it would ingest an IPFS/IPNS site, and publish .sxg files for the all the content. (need to republish regularly because sxg files expire). then i could serve those out on an IPNS feed for each subsite

i could push the IPNS feeds into dns txt records, and maybe publish a master feed
maybe a web extension could point to a public proxy? and if somebody wants offline, they could install an npm module that would run a proxy / ipfs mirror on localhost that subscribed to what they wanted

too crazy?

(bonus: it's almost the same pattern that a package manager would use)

[jimpick]

If I could get this working by next week, there are a number of people at csv,conf in Portland (which I'm attending) that might be interested:

https://csvconf.com/speakers/

[jimpick]

I made a quick little proof-of-concept chrome extension to see if I could load an SXG file that way - just generate a redirect when accessing *.ipfs.v6z.me ... seems to work great!

Really simple:

https://github.com/jimpick/sxg-republisher-extension/blob/50389b0f93d48d9ecd48d85d67c6a1459c10863d/background.js

[lidel]

Exciting! cc @ipfs/wg-web-browsers

• The republishing part could be a prototype of decentralized alternative to Real URLs for AMP by Cloudflare
  • eg. if we provide a service similar to hashbase.io, seamlessly generated .sxg could be one of its perks
• Using WebExtension and redirecting via onBeforeRequest or setting proxy via proxy.onRequest won't be as versatile as a service worker, but may be worth checking out.
  • It could act as good demonstration of how self-bootstraping .sxg websites could look like, making a case for tweaking SXG spec to solve SW problems
• An open problem I see is when you point DNSLink at an address of .sxg: when you open it in browser without SXG support, it will just download .sxg package, but won't render it.
  • Would it be feasible to point IPNS/DNSLink at an intermediate page with small JS snippet that redirects to regular website if SXG is not supported?

[jimpick]

I think it would be a cool progressive enhancement to have a fallback option ... where the static version of all the files are available for loading via http/https if somebody doesn't have the web extension installed - pretty much the same thing that Cloudflare already provides for IPFS.

Previously, I did try the Service Worker approach, and got that working as well. For some reason, back in the fall, I couldn't load the Service Worker from an SXG file directly, so I had to host it on the underlying website. That wasn't too difficult, but wouldn't work for first load in an offline situation. It may be that it is now possible to load a Service Worker from an SXG now ... they do mention that as a use case for WebPackage (but we're only using Signed HTTP exchanges so far).

For now, I think it makes sense to just prototype the Web Extension + SXG without the fallback. That keeps it a bit simpler, and would be extremely cheap to host on the server side.

The most expensive part will be the periodic batch job to convert an IPFS cid or IPNS link to a set of SXG files ... but it's probably not that bad since it only needs to run once a week (max timeout on SXG is currently 7 days), and it's pretty fast.

With the wildcard domain, I could use GitHub authentication, and hand out domains such as <mysite>-<github_id>.ipfs.v6z.me. I was thinking about a Hashbase inspired interface as well where people could submit an IPFS cid or an IPNS name.

A more advanced service might offer a 'bring your own domain name' option, which would include the special SSL certs needed. Right now, only Digicert is selling the certs, but they are somewhat pricey and the approval process is strict, compared to something like Let's Encrypt. I imagine Cloudflare is generating their own certs for their AMP service. If people have their own DNS and certs, the service could look for dnslink txt records on DNS instead of having a web admin interface.

The service would probably want to have a tie-in to ipfs-cluster to pin the generated SXG files. It would be a no brainer for the service to also act as an extra pinner for the original source as well.

[jimpick]

Another thought I had for an enhancement would be to automatically map <base32-cid>.ipfs.v6z.me urls to SXG files ... the initial web request could block until the SXG files were generated (which should be quite fast relative to the time it takes to find the data on the IPFS network).

[jimpick]

I made a little video to show what I've managed to get working so far...

https://ipfs.io/ipfs/Qmdn8biDeT1sWdyLCPs7nqfssPfzqLeWRHHpp9WKbH42aM/http-signed.mp4

[jimpick]

I'll need to host some batch processes somewhere to periodically republish the SXG files. I tried out https://cloud.google.com/run/ and it seems like it might be a nice fit. It's serverless, but you can use a Docker image.

[jimpick]

I started working on a bit of a "worker" for publishing the SXG files. So far, it just updates some dnslink DNS records using AWS Route53.

https://github.com/jimpick/sxg-republisher-worker/blob/master/index.js

This lets us lookup IPFS CIDs via DNS and the public http gateway, eg.

$ curl 'https://ipfs.io/api/v0/dns/ipfs-docs-jimpick.ipfs.v6z.me?r=true'
{"Path":"/ipfs/QmXSHCEUJfFhGvw6U1aSuMtSTSnCmWcwvnCHdSZwbfRyer"}
$ curl 'https://ipfs.io/api/v0/dns/sxg.ipfs-docs-jimpick.ipfs.v6z.me?r=true'
{"Path":"/ipfs/Qma2smSLfpgsTk3sJhA7oukLCk4wXDYy9UpyTTCyWLYaeU"}

I can do that from within the web extension to quickly resolve where to get the SXG and asset files from via IPFS.

[lidel]

Lookup via SXG subdomain is a cool solution to the "opening in browser without SXG support" problem, feels like it could be matching existing subdomain conventions:

• _dnslink.foo.tld - an alternative way to add dnslink to foo.tld without polluting the main domain
• _dnsaddr.foo.tld - the multiaddr-equivalent of dnslink: we put multiaddrs into a DNS records, and /dnsaddr/{fqdn}/ lets us resolve through them
  (example: dig +short TXT _dnsaddr.bootstrap.libp2p.io)
• _sxg.foo.tld - SXG-republished version of the website

[jimpick]

The standalone publisher service now works well enough to demo. Here's a short video clip:

Both @lidel and @autonome were able to install the extension (from source) and use the publisher to convert a static HTML website on IPFS to one that can be loaded using signed exchanges!

Higher resolution video on IPFS

[jimpick]

Tweeted the video from above.

https://twitter.com/jimpick/status/1136717976734797824

[jimpick]

I added a tiny bit of a README to the extension:

https://github.com/jimpick/sxg-republisher-extension

I made an icon and tried submitting it to the Google Web Store but it was flagged for manual review, which takes several business days.

[jimpick]

I gave a demo during Kinuko's 5 minute lightning talk at IPFS Camp where I showed the publishing demo.

I thought it would be nice to publish the extension so a few more people could try it out without having to manually install the extension as a developer. A number of Protocol Labs employees tried it out, but it is unreasonable to expect any community adoption with such a high friction installation procedure.

My first attempt at submitting the extension was rejected as it "did not comply with our program policies", listed here:

https://developer.chrome.com/webstore/program_policies

The rejection email didn't mention which policy was violated. Quite likely the reviewer didn't understand the purpose of the extension. I think I would need to escalate via the forums or some other backchannel to figure out how to get the extension published.

In order to keep the demo alive, I'd need to:

• purchase a new wildcard certificate from Digicert every 90 days for $114 each time
• generate a new CBOR file every 7 days and resign all the content

Also, the extension uses the webRequest and webRequestBlocking extension APIs which Chrome is deprecating:

https://blog.chromium.org/2019/06/web-request-and-declarative-net-request.html

It's quite likely that signed exchanges will undergo changes in the future as well.

With all that, I think the best move for now is to let the certificates expire, and close this experiment. I'm encouraged that there will be a path for good IPFS support in mainstream browsers in the future.