Address blockers to enabling DNSlink and allowing discovery via IPFS Companion

[terichadbourne]

We're about to switch ProtoSchool hosting to Fleek without DNSlink enabled so that IPFS Companion doesn't discover that ProtoSchool is available on IPFS. The reasoning for this is that we have a number of unsolved issues to address before an IPFS-hosted version of the site will function properly. This issue will serve as a place to document those issues and discuss solutions before enabling DNSlink.

I'll let @zebateira add more context and then it would be great to meet with @lidel & @hugomrdias for more context to ensure we approach these challenges in the most effective manner.

• Folks who originally accessed the site via traditional means and now access it via IPFS will be dealing with a new localStorage instance and will lose their progress history.
• localStorage on IPFS will be shared with every other site hosted on IPFS, so we'll need more specific field names to avoid conflicts.
• Redirects don't work on IPFS (see issue)
• URLs on IPFS are "beautiful" (aka ugly) - what's the least ugly option we can come up with?

[zebateira]

Note about relative paths deployment (ipfs gateway): since fleek has deployment setup to subdomains, this will not be an issue.

[lidel]

Quick brain dump before today's meeting:

• Companion redirects DNSLink websites to local gateway as the default, but opt-out from IPFS integrations per website can be done:
  • by user via browser action menu or Preferences
  • by developer, by appending ?x-ipfs-companion-no-redirect or #x-ipfs-companion-no-redirect to the URL of a resource that should not be redirected
    • Note: I would love to introduce a better opt-out method for website developers, but we are heavily constrained by what is possible in browser extension running in Chromium :( I'm happy to brainstorm ideas for better programmatic opt-out to see if they are feasible given existing APIs.

Regarding "new localStorage" and losing progress history:

• There are bit hairy solutions for sharing localStorage cross-origin (zendesk/cross-storage), but this won't work in Safari, and will most likely break over time, as browser vendors move towards hardened isolation to improve privacy.
• Perhaps lo-fi solution would be better?
  • it should be possible to detect that window.location.origin is not https://proto.school and if there is no progress data in localStorage display modal informing user that the progress will be tied to current origin, and if they see progress missing, suggest disabling gateway redirect for protoschool and resume at https://proto.school
• Even more lo-fi: we can just disable redirect for proto.school by default in the next release of Companion (user would have option to opt-in)

[terichadbourne]

Is there a way to have Cypress simulate the experience of having IPFS Companion installed and toggling it on and off? Wondering how we’ll build tests for when DNSlink is enabled but only a subset of users are accessing the site that way.

[zebateira]

Is there a way to have Cypress simulate the experience of having IPFS Companion installed and toggling it on and off? Wondering how we’ll build tests for when DNSlink is enabled but only a subset of users are accessing the site that way.

Apparently yes: https://www.npmjs.com/package/cypress-browser-extension-plugin
But I think we will need to have a workaround to load the extension because this package loads the extension from the file system (maybe we can load it from the URL when cypress loads?), otherwise we will need to add the extension to version control and keep updating it from time to time.

[terichadbourne]

Notes from 2020-09-11 call with @zebateira @andyschwab @lidel @hugomrdias re challenges and UX options for enabling DNSlink for ProtoSchool:

• Currently testing on Fleek - backend on IPFS but delivered with user’s browser unaware that it’s on IPFS, can’t discover on Companion until DNSlink is set up

• Redirect issues

  • Issue Chris opened re native redirects in IPFS (via HTTP Gateway) is not expected to land in 0.8, not anytime soon, at least 2 releases from now

    • Proposal: native redirects config ipfs/kubo#7392

  • Have ipfs-404.html feature but can’t set up manifest

    • Proposal: manifest file for the IPFS gateway ipfs/kubo#6214

    • This issue affects ability to:

      • control content type
      • redirect
      • return custom headers (TBD, needs analysis of security ramifications)

  • Doing redirects in SEO-safe way (header, sitemap)

    • https://en.wikipedia.org/wiki/Canonical_link_element
    • https://www.w3.org/TR/WCAG20-TECHS/H76.html (client-side redirect)

  • You get penalized for client-side redirects (not as bad w/o timers/ads)

  • Using ipfs-404.html file for Fleek to address /*

  • Non-404 redirects (eg chapters -> events): Fleek can do some server-side stuff but better to address it from IPFS side - shortcoming of IPFS, not Fleek

• Persistence issues with user history in localStorage

  • Current issues with ProtoSchool + DNSlink (once enabled)

    • If you load ProtoSchool from local IPFS node (companion or otherwise) you’ll get a different origin (not proto.school) and lose your localStorage history (lessons completed, cached code, etc.)
    • URL when enabled in IPFS Companion will be proto.school.ipns.localhost:8080 (no other options, no CID displayed)
    • Don’t expect origin used for loading from local gateway to change anytime soon (unless user changes the port)
    • This gateway does separate based on subdomain so we won’t be sharing localStorage and don’t need to worry about fixing field names to be more specific to our site
    • History could revert if they uninstall Companion

  • UX options for lost history issue:

    • OPTION 1: Automatically switch to IPFS version w/o messaging on why history disappears

      • Current state if we enabled DNSlink with no action (see challenges above)

    • OPTION 2: Automatically switch to IPFS version and pop up a message about why their history is lost

      • No changes needed to how DNS link works
      • Detect in ProtoSchool that it’s an IPFS URL with no progress tracked in localStorage (eg first time visiting on IPFS) and give them heads up about why their history is missing, option to disable to get back to Web2 version with existing history.
      • We won’t know whether they’re an old or first-time user.

    • OPTION 3: Opt out of Companion by default and let them toggle on, with no ProtoSchool-specific messaging

      • Companion could have a set of websites for which user isn’t automatically redirected but makes that choice consciously (new feature)

      • Right now there’s a toggle enable/disable IPFS integration - currently always enabled with a deny list - so we’d be adding protoschool to the deny list for all users unless they remove it

      • Deny List Option A: internal deny list that would require a new release of Companion to change the deny list for a new site

      • Deny List Option B: serve deny list through IPNS and updatable from, say, a GitHub repo (so anyone could add themselves to the list, and it wouldn’t require publishing a new release)

        • If you did it you’d risk someone ask questions during review (who doesn’t understand IPFS)
        • Easier than new releases to quickly add sites to the list

    • OPTION 4: Opt out of Companion by default and let user toggle on, warning them in advance that it will delete their history

      • See section above for deny list options

      • Can website detect that a user is running a local IPFS gateway in order to pop up a warning before they use the option to toggle on?

        • Can’t use window.ipfs anymore
        • Could try to serve an image on IPFS and see if it fails

  • Root question of persistence - what would be the web3 equivalent of localStorage?

    • Identity with keys to know who they are (
    • Otherwise always connected to origin (localStorage, etc.) and would be affected by domain changes

• NEXT ACTIONS:

  • For now, skip DNSlink until they land Companion with the deny list / toggle fix (Q4)
  • @lidel - Open an issue for adding the deny list to Companion (to enable options 3 or 4 above) – done: Opt-out proto.shool from the DNSLink redirect by default ipfs/ipfs-companion#921
  • @lidel - Experiment with whether you can detect that a user is running a local IPFS gateway (to enable option 4 above) – done: poc at https://bafkreic34lfh2ox3q2ccrohhxzy6zjxweqic323gesnvtbdinyhsajz6vu.ipfs.dweb.link/
  • @zebateria - Comment on manifest issue and describe our use case to try to get that issue higher priority

[terichadbourne]

@lidel This is a belated response regarding the PR to add the deny list in IPFS Companion. I really appreciate your work on that and the great wording tweaks from @jessicaschilling! My notes here are probably a mix of suggestions that could potentially apply to everyone in Companion and things that are just me confirming I know where we're at. 😂 Happy to move any suggestions over to your repo if merited.

My one uncomfortable gut reaction to that PR was about that I think of "manual" meaning I exerted effort to do it myself, and wouldn't expect automatic things to be under that label. Do we want to explain how something could be in your supposedly manual opt-out list when you didn't manually put it there yourself? One way to do it might be a text link under the box that says "Don't recognize one of these sites?" and if you click it, it says something longer like "Some web developers disable [insert right words] by default because certain features may be affected when accessing their site via [insert right words]. Before removing a site you use from this opt-out list, you may want to check their website for a guide to functionality differences". But shorter and with words like DNSLink or IPFS Companion or local gateway inserted as appropriate. 😂 (Sorry, still working on getting the right vocab on this one.)

Also just confirming I understand the flow and options here under the new system based on the change you just merged. Is my interpretation below correct?

• Currently, with DNSLink disabled for proto.school in Fleek, users of IPFS Companion will not see the toggle for it at all. (And I believe Fleek made a change at our request recently so people will no longer accidentally discover the CID when DNSLink is disabled?)
• If we were to enable DNSLink in Fleek as normal (removing it from the opt out list you added it to), they would see the toggle and it would be turned on by default.
• If we were to enable DNSLink in Fleek with proto.school on the opt out list (as it now is), they would see the toggle but it would be automagically turned off by default, and the user wouldn't know why it was turned off by default unless they went into the settings. If we wanted to warn them about a negative experience they'd have when toggling it on, we'd need to do it in our own site's code by detecting that they're on a local gateway and popping something up faster than we expect them to hit the toggle. (But can we still detect that even though they're not yet accessing OUR site through that local gateway?) There's no way for Companion itself to pop up that functionality change warning near the toggle when a developer asks to be on the opt-out list? Could we pass a warning message in the http header for you to read and apply in Companion?

If I'm understanding correctly, we've arrived (from the Companion side) at Option 3, Deny List Option A from the previous comment. There's an open issue to switch to Deny List Option B, which won't affect the ProtoSchool team moving forward. And it's up to the ProtoSchool team to get us to Option 4 using by mimicking Lidel's POC. Does that sound accurate?

[jessicaschilling]

Hi @terichadbourne - I can't weigh in on anything after your second paragraph, but just a note that subsequent commits in ipfs/ipfs-companion#929 got rid of the "manual" wording and explained that you can also add/remove items on those two lists by using the toggle in the browser bar menu. 😊

[lidel]

@terichadbourne thank you for this thoughtful analysis 🙏

We indeed implemented Option 3A, but instead of 3B (centralized opt-out list) we plan to let website owners to control default behavior somehow (ipfs/ipfs-companion#930) (no central list to manage, every site controls their own fate). Option 4 is up to you, but see my last note below.

Overall, things are a bit simpler than what you described and I believe the gist is:

• The toggle is always present, we show it on every website, does not matter if DNSLink is present or not
  (this is because regular sites can still have subresources loaded from /ipfs/<cid> etc)
• In the future, when DNSLink is enabled for proto.school, we can make a new release of Companion and remove it from the "default opt-out list":
  • Existing users will keep their explicit setting, so nobody will lose their progress due to Origin change
  • New users will be redirected to local gateway
• Yes, you can always detect when window.location.origin === 'https://proto.school' and display some infobox informing user that their progress will be tied to window.location.origin
  • However, the subset of users running local gateway and then the sub-subset of users who are at risk of changing the origin and losing progress is probably small in practice, and you should not invest too much time in UI that is specific to this niche user flow
  • Instead, you could add "academic transcript" feature, where user can export their progress to a JSON file, and then import it later. This would not only cover the niche problem of a technical user changing their Gateway origin, but would also be useful for people who use school/library computer for study.
    Just an idea, but I feel its probably the better way of tackling this problem space.

[lidel]

Some updates:

• IPFS Companion v2.15 shipped with Opt-out proto.shool from the DNSLink redirect by default ipfs/ipfs-companion#921. Chromium users should get it later this week. This will ensure the Origin for proto.school stays the same, unless user explicitly made a choice to redirect.
• I also discussed this with Fleek and they are working on removing x-ipfs-path header unless DNSLink is set up, to avoid in-between state for other websites where human-readable domain gets redirected to immutable CID.

[terichadbourne]

@lidel Wanted to check in and ask whether either the recent Brave integration or the DNSLink / TLS updates you shared today will change anything about the effects of us enabling DNSLink?

[lidel]

@terichadbourne loading ipns://proto.school will have the same effect as loading from localhost gateway: it will get a different Origin than https://proto.school (separate storage and users will lose progress)