Wiki Chrome Extension planning

[rynomad]

In regards to the wiki extension I showed briefly on the last hangout. For those not present:

The wiki-client code, with it's localStorage capabilities (and possible migration to indexedDB), is ripe encapsulation in a dedicated chrome extension. In essence, creating a wiki server that lives inside the browser. Preliminary hacking suggests a number of exciting features:

• 'fork to local' becomes 'fork to extension' to save pages to a single location independent of origin,
• the extension can register itself to the neighborhood when viewing remote wikis, so ones own pages can be referenced easily within the context of any other wiki's domain.
• extension can serve as an offline reading app (fun story: this is actually the original use case for me; my apartment internet went down for a while and I wanted to be able to conveniently save things to read at home while at school... it worked wonderfully)
• if we rename wiki page .json files to something more app specific (.wik, for example) opening them in the browser can trigger the opening of the chrome extension (similar to a browser-based pdf reader)

my work-in-progress is view-able on my chromeExt branch of wiki-client code. There are some small coordinated changes in wiki-node-server to support allowing the extension to 'hook-in' (just a meta tag in static.html that the extension checks for the presence of on page load). Other than that, here's a high level of the approach so far:

• Our client.js is created by a grunt task. Use another grunt task coupled with task-specific file aliases for browserify to create client.chromeExtension.js which re-uses much but not all of the same code as client.js. Gruntfile.js should be the first stop in my branch for anyone perusing the code.
• refactor wiki-client such that one module handles the fetching of all non-static resources (page json, plugin code, favicons, images, sitemaps...) this module is the most significant chunk of non-shared code between the extension and normal client., though there are others.
• treat a modified static.html as index.html of the chrome-extension. which can therafter be accessed via the browser with it's chrome-extension:// url

There's more but it's late now and typing is getting really slow. I'll post a few followups tomorrow and remove some bit rot from earlier attempts so as not to be confusing.

I'm planning to spend Sunday afternoon hacking on this if anyone is available to G-hanghout and collaborate. I was thinking 12pm Colorado time until I run out of steam. Additionally, this might take a few sessions, so I'll post here every time I'm beginning/planning a significant coding session, and when each ends, I'll post a run-down of the work.

Cheers!

[WardCunningham]

I'll be there.
https://plus.google.com/hangouts/_/g32fblr7vpwniyyv6sazxaoswqa

[rynomad]

Cool! A few additional notes as I re-familiarize myself with the changes:

• I initially stated that Gruntfile.js should be the first stop when looking at the difference between client.js and client.chromeExtension.js. Manifest.json should be the first stop for examining the operation of the extension, specifically background and content_scripts section
  • background contains the steward.js file (and a jquery file for ajax), These scripts are "always-on" and ready to handle forks from any arbitrarily visited wiki, and (potentially) publishing to a wiki owned by the user
  • content_scripts contains two patterns to inject scripts.
    • The first pattern (file:///.wik) injects bootStrap.js, a simple script which scrapes the text of the .wik (.json) file, parses it, and sends it to the steward for storage. Upon recieving an ack from the extension, bootStrap creates and clicks on a link to the extension, including a location.hash value to view whatever page was just imported. This write action is currently haphazard. There is no check for an existing page, so you always overwrite any existing page
    • The second pattern (http:///) injects checkwiki.js, which checks the title tag of each web page visited for class wiki. upon finding it, checkwiki changes the class to 'wiki-extension-enabled' so as to alert the client.js from the website that it should outsource fetching/saving semantics to the extension and register it to the neighborhood.
      • This is a bit of a terrifying level of access for the features we want, but it's the simplest way I can think of at the moment to ensure that we can detect whether a site is a wiki without keeping a global roster of wikis in the manifest of the chrome extension, or requiring some sort of unified host pattern in order for a wiki site to be compatible with extension.
• Having had some time to let the current approach stew in the back of my mind, and recognizing that I'm currently 65 commits behind fedwiki/master and should probably start fresh, it occurs to me to map out explicitly the contexts which will occur for someone who has the extension and is browsing wiki's. Nothing written below should be considered set in stone, but merely an exploration and reflection upon work thus far.
  1. Direct to Extension: User clicks a chrome-extension button which takes them to the chrome-extension:// url of the extension. Extension acts as "origin server". in effect, and operates offline, just like a wiki server running on localhost.
  2. Open a .wik file: User get's an email with a .wik file, opens it with browser. First approach injected wiki-client into the file://* tab. New approach imports and redirects to 1)
  3. Wiki that User doesn't own: Without extension, fork to local. With extension, fork to extension. Should we register extension from the neighborhood in this case? It seems useful to be able to pull up one's own pages in the context of a remote wiki.
  4. Wiki that User does own: fork from local turns into fork from extension. Should fork from remote duplicate pages on server and extension? It seems reasonable that the extension sitemap can be a superset of sitemaps that are stored on public facing sites, replicating the local-edit semantics often used with localhost wiki's. Also, for offline reading/editing, it seems handy to have extension stay synced with sites I own (but not necessarily have those sites sync with the extension).
• Some thoughts for case 3 and 4 above: In either case, to get the webpage version of client.js to talk to the extensions steward.js requires multiple levels of message passing (first a custom event to get from client.js to checkwiki.js, second a chrome.runtime.sendMessage to get to steward. and then back again) which get's messy fast, notably when managing callbacks to handle responses in the form of custom message events between checkwiki.js and client.js.
  • One possible solution would be to enforce an "always browse wikis from extension" policy, whereby a semantic similar to 2) above would simply note that a wiki was being visited, grab the lineup order, and convert it into a hash url to feed into a chrome-extension redirect. This has the advantage of simplicity in code, but there may be good reasons not to enforce the extension as the only gateway to all wikis. That said, given the ludicrous permission of http:/// in manifest.json, redirecting to the extension could be a good way of insuring that no site tries to contaminate our app without our knowing, which would be possible if the steward is simply listening to messages from any site that identifies itself as a wiki.

I've tried to remove as much of the high level cruft from manifest.json and client/chrome/ but there's undoubtedly still some confusing bit rot in lib/

T-minus 1.5 hours till Hangout. See you there :)

[rynomad]

Yesterdays discussion was fruitful. There were a few digressions into the larger context of fedwiki's future, as well as a few practical considerations having to do with this specific issue/feature/evolution. Documented here are a few big ideas in question/possible answer format.

If a user has write-access to several wiki's, each within the local context, how does the client code decide which to push edits/forks to?

Currently, one only has two choices: the origin server (if logged in) or localStorage. One must manually open a new tab for any wiki in order to make edits to a new origin. A common pattern is to host a wiki on localhost to edit and refine pages until they are "ready for the world." Similarly, one may log out of their own server to edit from local storage and then log in and fork. A wiki extension has the potential to unify/replace these patterns by functioning similar to a localhost server, while potentially being able to establish credentials with several public-facing wikis simultaneously. We discussed the notion that a sensible default action might be to always save data to whatever location is "closest to the screen" (localStorage/extension/app) and pair this semantic with a more explicit push/publish button which would ask which of your write-enabled wikis (if more than one) you would like to publish to. This has the twofold advantage of being able to preserve the localhost privacy of ideas in progress as well as support multiple publish points with a minimum of confusion.

Is there a way to simplify the sometimes circuitous paths one must take to link wiki's together, Or to switch between viewing a remote wiki and editing one's own? Currently, if I email someone a link to my wiki, and they begin reading from my origin, notice that they'd like to fork my pages to theirs or make a new page based on some spontaneous inspiration, they must open a new tab, log in to their wiki, drag links/urls. One potential benefit of an extension/app is to unify the browsing/editing experience across multiple origins. That is, the extension becomes the default handler for all wikis, local or remote, and handles it all within a single tab.

These and other discussions of usability, as well as the code complexity referenced in the previous comment, suggest a more in-depth exploration of "wiki as an installed app" semantics. Note that we say here exploration so as to remind the reader that the federated wiki they know and love will continue to be able to function as a traditional webapp as this exploration is made. Ideally, the two use patterns would co-exist (a good analogy might be that of an installed email client paired with webmail, configured such that you can use whichever you have access to at a given moment, and many users may prefer to use only the webmail client, with the extension/app being more oriented towards power users).

I will likely spend a little time here and there exploring some more of the chrome/firefox extension/app ecology over this week, and hope to have another session next Sunday.

[rynomad]

For those interested, the new pass at the chrome extension is located in my 'chrome' branch of wiki-client:

https://github.com/rynomad/wiki-client/tree/chrome

In a nutshell:

• Packaged app
• ducktype detection of wiki HTML structure forwards to viewing that wiki as a remote page from app origin
• history persisted using localStorage, launching wiki brings up the last used lineup
• very minimal code changes, able to redirect require calls via packageChrome task specific options in Gruntfile
• all standard plugins are bundled, until we find our holy grail of package distribution

[judell]

Thanks Ryan. Is there an install recipe I didn't find?

[rynomad]

ah yes! Though the reason you didn't find it is because I forgot to include it...

• in chromes url bar, go to chrome://extensions
• toggle "developer mode" to ON
• click "load unpacked extension"
• navigate to the /client/
• start browsing
• when you want it off, disable from the chrome://extensions page

It's worth mentioning that there's no publish ability as of yet, and other things you would expect from a local server are lacking (sitemaps, favicons). Definitely a minimum viable build, not good for much more than storing pages for offline reading/private journaling at the moment, but I'm liking the app-centered approach a lot. Much simpler.

[WardCunningham]

Ryan's repurposing of our client code as a plugin drives home the lessons learned from the node community: write and publish small modules that do not presume to know how they will be applied.

Ryan's work has been harder than it might have been had the client side application been developed from day one with this radical modularity as its primary organizing principle. Instead, his willingness to forge ahead informs our ongoing modularization. Thank you Ryan.

[judell]

I think this opens the door to use of alternative local stores, including the wildly interesting and capable PouchDB, which in turn opens the way to CouchDB sync.

[rynomad]

I haven't looked much into PouchDB yet, but I do have some experience with levelUP on-top of indexedDB, by way of level-js. It works quite well.

between the ability to store large amounts of data and webRTCs data channels for peer to peer browser connections, I'm hopeful that the packaged app environment can be nearly as feature complete as a public facing server. Chrome even allows apps to run a background page, akin to having a wiki tab always open. If we put page storage and serving in that background page, it could be accessible to the federation as long as the browser was running.

My pie in the sky idea here is that eventually the notion of running a large wiki farm can be supplemented with the action of running a wiki 'hub' that helps app users connect directly to each other.