[META] server-side component

[hamilton]

@mdboom @bcolloran @wlach @djbarnwal ---

This issue is meant to work through the particulars and roadmap for the iodide server(s).

Iodide has a few challenges as a client-side product -

• saving / exporting notebooks strictly from the client-side view is not particularly ergonomic.
• sharing is not easy. The lowest-friction method right now is to export your notebook, upload it on gist.github.com, get the slug, and then feed it back into an iodide URL with ?gist=<slug>. Lots of steps here.

So, a server component would fix these two issues, and open up the possibilities of a number of other useful pieces of functionality. The vision, in the long-term, is that users will probably prefer a hosted solution over a locally-run one, so we should be building toward that.

1. Lowest-Hanging Fruit - @wlach's server branch

This is a node server that is meant to 1.) make uploading your notebook as a gist (and getting the URL) very straightforward, which will allow easy sharing, and 2.) some light filesystem-saving API, similar to how Jupyter works now. This is pretty close to being together. As a long-term solution for a hosted thing, we're skeptical this is the right approach, but as a lightweight server that ships with the iodide repository, this seems like it'd open up a lot of simple workflows for folks expecting a jupyter server type setup to get real work done.

2. GH-heavy Fruit, higher up the tree, maybe not worth it

This option would probably be similar to (1) (and is not mutually exclusive to it), but would allow greater functionality around using Github as our main "filesystem", allowing for versioning, branching, and uploading notebooks directly to repositories.

We talked about the many issues around this tight integration with another company's API in the long-run. Github is great, but they may make major API changes or restrict access at some point for whatever reason, and we don't really want to build on top of a third party solution that we don't control. The solution proposed in (1), furthermore, would make it simple to use git for versioning of one's notebooks (saved with a filesystem api), so this may also simply not be useful.

3. build our own barebones db-powered server component

This option would probably be the better long-run bet, but has a bigger up-front cost. We could build an app with python + django + postgres that'd allow a.) authentication & user creation, b.) versioning of notebooks, c.) granting certain permissions to notebooks, etc.

An MVP would look like this:

• figure out the barebones data model
• figure out how we want to handle user creation / authentication
• user view - user has a set of notebooks
• notebook view - serve the static notebook via templating
• allow saving functionality with static notebook back to database
• provide a simple way to share a notebook via a url pointing to iodide.io

@wlach believes this barebones hosted solution wouldn't be that difficult to build, but it is not trivial, either.

---

I'm hoping that we can discuss this during the meeting on Thursday.

[bcolloran]

I'm confused about that option 1 -- is that meant to refer to the localhost server we discussed? @wlach, my understanding was that the server component you have been working on just enables the uploading to gist stuff, but not the file system saving / Jupyterish functionality?

re option 2: i'm not super worried about building around github's API in the near term; i think their v3 rest API has been pretty stable for many years, and is unlikely to shift so quickly that it creates problems for us anytime soon, so that part feels like speculative concern that is in anycase not something we'd need to confront for quite some time. and as a fast path to getting more usage (including dogfooding), my intuition is that github offers familiarity and a huge set of features that we could build a pretty nice workflow around pretty quickly.

i like the idea of option 3 in abstract, but it scares me. i think actually building a backend that supports the workflows we want is a huge task, and for creating user value and driving usage in the short term (particularly internal dogfooding) i think it's a much slower path to reaching featureful enough workflows that people will actually want to try Iodide for their work.

Anyway, after sleeping on it, I guess i'm leaning towards taking @wlach's server and, starting from the gist functionality, extending it bit-by-bit with more github APIs to enable a few richer workflows. We don't have to do everything (or commit to building on top of github forever), but even being able to update a notebook in a repo would get us a long way towards some of the key workflows we talked about yesterday ( https://developer.github.com/v3/repos/contents/#update-a-file ).

[mdboom]

Maybe another way to look at it is:

• develop a generic interface around the operations we need (save, load, getHistoricalVersion etc.)
• develop the server as a delegator to interchangable storage backends, the first two of which we'll probably start with Github/Gist and local filesystem
  Then it should be easier to just add more of these as the landscape changes without significant changes to the front end.

[hamilton]

Yep, sorry if it wasn't clear before - option 1 was largely about extending @wlach's node server that at the moment will come bundled with the iodide repository to also include filesystem saving, which will allow a bunch of obvious jupyter like workflows for many analysts, so we're not totally held up by not having full git integration right now. I would argue that this would probably be enough to enable dogfooding by others while we figure out the right roadmap going forward.

@mdboom's got the right idea - a generic interface around the operations we need feels like the right solution, which will allow all sorts of different integrations. I'd be very open to having Github integration with that in mind vs. building specifically for Github, or perhaps the real solution is to have an interface for git generally. Again, no preference, as long as we're not building ourselves into a corner.

But that brings up a bigger question: one aspect of my convo w/ @wlach left out of the OP was whether or not we expand his work building a barebones node server to include a bunch of different backends, vs. aim for a technology stack that is more proven / boring / easier to find resources to build within Mozilla and beyond and leave that simple node server for casual analyst use (primarily python + django). I have no particular preference on this, but feel it would be good to discuss the options here before we move forward.

At any rate, I'm sure we'll have a lively discussion about this tomorrow.

[bcolloran]

following from @mdboom's suggestion, here's a stab at what i think are the most important workflow steps to de-frictionify in the near term. (not that we have to build fully robust solutions in this order; it may be the case that some of the items higher on the list have acceptable low-fi work-arounds). To deliver on on zero-friction value proposition, we obvi want to get all these actions down to like one click/keystroke.

These are the workflow tasks that I can think of that I believe people will want/need to accomplish, roughly ordered in terms of what people will need to move from easily dropping in to test out Iodide to a robust workflow for doing real work. Please add any that i'm forgetting!

0. viewing published notebooks -- right now, if i want to share a notebook, the easiest way to do that is with a gist and https://iodide-project.github.io/master/?gist=user/gistnumber. (this is pretty ok actually, and not a big source of friction)
   0.1) viewing existing demo/example/documentation notebooks -- right now, access to our existing notebooks is rather clunky, you have to dig around in the GH repos, there is no landing page that automatically updated. We need an easy on-ramp for people to see our examples so that people can play with enough notebooks to see the value of our approach.

1. saving notebooks while writing them -- saving to the filesystem in local mode is clunky, saving to a server when working at https://iodide-project.github.io/master/ doesn't exist, and saving to LocalStorage is a weird trap of a workflow. We need a simple way for people to save their notebooks in a way that is understandable. ctrl-s should just work to save your NBs where you want.
   1.1) autosave should also just work, and should put backups some place you can find and reopen them easily (not scattered across domain-restricted LocalStorage buckets)

2. simple publishing of notebooks -- publishing notebooks is a pain. you have to save them locally, make sure that you are pointing at the correct resources (it's a hassle to change between dev and master), either create a gist or push to a repo, and then manually retrieve either the gist link or the correct link from https://iodide-project.github.io/iodide-examples/ .
   2.1) simple updating of existing notebooks by original author -- if you get feedback or discover a problem and want to make a change or update to a notebook you have published, you have to go through the rigmarole above again.

3. collaboration features
   3.1) easily saving a copy of a notebook (non-forking) -- for permitted collaborators, it should be easy to load and update someone else's notebok
   3.2) easily saving a copy of a notebook (forking) -- for non-permitted collaborators, it should be easy to open an existing notebook and save your own changes
   3.3) suggest changes to someone else's notebook -- it should be easy to do something like a "PR" when you have suggested edits to someone else's notebook
   3.3.1) view diff of suggested changes

4. history management
   4.1) see/retrieve historical versions
   4.2) branch from historical versions
   4.3) view diffs between historical versions

Also for reference, here is my personal view on the hierarchy of (expected value of) user friction:

zero: just show up at a web site, and all work flow options are available
minor: sign in with a tier 1 web property-- facebook or google
minor+: sign in with a tier 2-3 web property-- github
medium: create a new login with a new website
pretty high: install an addon **
high: install a single fully self-contained binary application locally **
very high: install a command line program that requires you to have installed other command line prereqs (e.g. node) **

** [may not be an option on locked-down systems]

[wlach]

Relevant to this discussion: I put up a basic heroku MVP in #602

[bcolloran]

closing, superseded by more recent issues