Initial Substance Hub Specs

[michael]

So the first thing we'd need, is a service that receives, stores and renders Substance Documents.

1. Store multiple versions (snapshots) of a document using the Substance App.

POST /documents

A post service that takes a substance document as the body. The substance doc includes an id field, so we can either create a first or subsequent version in the hub database.

This is how the document looks like:

{
  "id": "some-doc-id",
  "properties": {},
  "nodes": {
    "section:1": {
      "content": "Hello world!",
      "id": "section:1",
      "type": "section",
      "prev": null,
      "next": "text:2"
    },
    "text:2": {
      "content": "Hey there.",
      "id": "text:2",
      "type": "text",
      "prev": "section:1",
      "next": null
    }
  },
  "annotations": {
    "suggestion:1": {
      "id": "suggestion:1",
      "type": "suggestion",
      "node": "text:2",
      "pos": [
        0,
        3
      ]
    }
  },
  "comments": {
    "comment:a": {
      "id": "comment:a",
      "node": "text:2",
      "content": "A way of saying hello.",
      "annotation": "suggestion:1"
    }
  },
  "head": "section:1",
  "tail": "text:2"
}

1. Render latest version

GET /some-doc-id

HTML-rendered version of latest document version

1. Access JSON version

GET /some-doc-id.json

Technology:

• Node.js/Express - perfect
• I was considering PostgreSQL as a database backend for the hub, since we can easily ensure referential integrity (e.g. users, networks, network subscriptions etc) this client seems to work well and is actively maintained (https://github.com/brianc/node-postgres)
• We stopped using SASS/Less CSS-Precompilers for various reasons, but if want to keep using it, it's really no problem.

[yuchi]

Ideas straight out of my mind.

Functional

ID creation

Identification should be made by the hubs which the client connected to. In the first project iteration you expressed the desire to have multiple hubs, a git like distribution, if I'm not wrong. That means also the there should some kind of protocol to transmit documents and histories, so the private development of Substance Hub will not hurt future Open Source solutions for that. If this is still the case, there should be a Client ID, for local work, and a Server ID, one per hub, IMHO.

Even without the multiple hub thing, we should evaluate collision problems. Should the client only use a local identifier?

History

I'm simply unsure, is it going to store the whole OT history or just snapshots?

PGSQL

Postgres is perfect, I love it (while not expert). I'm going to do some research on it, but should we use the JSON data type? Looks promising to me. If someone here knows something about it please let me know, so I can make up my mind faster.

Rendering

Pandoc? JS only?

Development

I use to write CSS in stylus only to remove vendor prefixes transparently, but given that we're going to use it as the main endpoint for readers I'm going to create a pretty complex structure for responsiveness, and I'd like to minimize the burden 😄

[michael]

Ad ID Creation: Collisions should be very rare. as the clients generate unique ids just like Git does it. It would be a good idea to check for existing guids on the server, just to ensure we're not overwriting a previously published doc.

History: The most important thing we need for the new Substance release is a way to simply publish snapshots. Once we have that thinking about ways to distribute the whole history makes sense (also with respect to collaborative editing, which is not supported yet).

PGSQL: Yeah, have a look at it, definitely. I'm not sure if node_postgres already supports it.

Rendering: I think we should integrate Letterpress (using Pandoc) later, and start with a simple HTML-Renderer for the hub.

[michael]

Oh and regarding IDs: Once we have the authentication system and networks setup we can make documents available by username/docname or networkname/docname.

[yuchi]

About IDs: that's true!

About PG's JSON: looks like it's just a different semantic for TEXT, so when you export a row as JSON it's not converted to a string, but it's interpreted as a JSON object. http://blog.redfin.com/devblog/2012/03/json_in_postgres.html This article is old, but they did it without the offical support so...

[michael]

Nice. Give it a shot. I mean.. it's not a big deal if we use just the TEXT for now and do the serialization/deserialization ourselves (if this saves you some time).

Thaanks! :)

[yuchi]

This night I'll push some code. Once you guys give me directions and approve a milestone we can put on a working Heroku app or similar to start some real tests.

Goals:

• Authentication
• Initial API structure (full document CRUD)

Next step:

• OAuth 1.0a (I'm not an OAuth 2 fan) authorization steps
• Initial work on display

[vectorsize]

That is great news @yuchi !! I'm really looking forward to see the hub progressing!

[yuchi]

Sorry, didn't do in time. I'll push things tomorrow.

Edit: did -> do

[michael]

Sounds good. Took some time off yesterday but I'm back in business.. :)

Do we really need OAuth at this stage? username+password (over ssl) should do it for now?

[yuchi]

Seemed like a good idea at the moment :) I usually don't like the client to send authentication every request, nor using cookies and persistent connections. Now that I think of it, we could also work with RSA and public keys...

BTW: the most important thing is to put up the structure and let you guys do some work to store the docs and authenticate, for the next auth processes I'll wait for your needs then.

[michael]

Yeah, I'm ready to publish some documents here. :) Just let me know how to auth against the hub. Again, we can add a security as we go. I just can't wait to test the whole workflow. ;)

[vectorsize]

push! push! push! push! :)

[Integral]

Hi all!
Just if you don't know there is excellent passport.js which is already implement 40+ authentication strategies and works very good with express.

[yuchi]

Passport is the way to go. If we choose to implement OAuth as providers then we can drop in https://github.com/jaredhanson/passport-http-oauth

[vectorsize]

👍 (thanks @Integral )

[Integral]

It's my pleasure!
@yuchi, you can check an example. I don't know how the things is going with OAuth 1, but as I know you have no way to auth with OAuth2 in localhost. So if it's true for other APIs (like OAuth 1) we need to keep in mind this fact and decide what to do with authentication in local app.

UPD: Just found this https://developers.google.com/accounts/docs/OAuthForInstalledApps

Anyway, I think we need to think ahead with keeping in mind local app authentication capabilities.

[yuchi]

In fact!

Sorry for the delay, actually I thought I had almost everything ready to be delivered here, I just needed to wire things up.

Actually I realized I had nothing ready: pgsql not always correctly connected (ident auth), promises api wrapper I built aroung node-postgres pretty useless, hand made authentication which could be a pain to maintain and extend, and a simply wrong API to store documents.

Once I have at least the boiler plate structure for authentication and api in place I swear I'll push it :)

BTW passport is very easy to plug in.

[Integral]

Sorry, just read about JSON in postgres. Note that the JSON datatype is in pgSQL out of the box from september.
Node-postgres support this data-type.

[michael]

Can't wait. :)

[michael]

Btw: Did you guys see the updated composer? http://interior.substance.io/substance

[Integral]

@michael

[michael]

Hey @yuchi. Just got back from London. Let's sync up regarding the hub some time next week. Oliver and me are working on backend integration for the coming two weeks. Once we have that stable, we should really look into implementing a simple publishing workflow using the hub.

Talk soon!

-- Michael

[vectorsize]

yay! welcome back @michael !

[michael]

@yuchi? Pls ping us if u're still alive. :)

[yuchi]

Here I am! Let's schedule a team skype talk this week. Tuesday and Thursday I'm free very late (11 PM CET), today and Wednesday and Friday from 8 PM CET.

@vectorsize @michael

[michael]

Hey! :)

Team Skype sounds good. Today/Tomorrow I'm busy in the evening. But Wednesday/Friday would work for me. @vectorsize @oliver----

Just a quick update. We almost have a functional build of the native substance app. So the coming weeks would be perfect for connecting a hub to it. ;)

In order to have a look at the app you need to build it yourself for now (see instructions here: http://interior.substance.io/modules/substance.html), but we're close to shipping it as a ready-to-use package for OSX. Probably later this week.

Also we launched our campaign, and we have one confirmed sponsor already (donation not yet captured at the campaign).

See here: http://pledgie.com/campaigns/18902

[vectorsize]

Allright...Im kindof flexible this week, wednesday might be a bit more complicated during the day but after 8 should be fine :)

[vectorsize]

Is it confirmed @yuchi ? meeting tonight from 20? looking fwd to it
EDIT: went ahead of myself here, sorry, @michael wont be online until later maybe, lets try anyways...my skype username is vectorsize

[yuchi]

I can reach you only after 1 AM, I'm stuck at a reunion..

[michael]

Let's try tomorrow. Or sync up via email. No worries...

On 19.12.2012, at 23:08, Pier Paolo Ramon notifications@github.com wrote:

I can reach you only after 1 AM, I'm stuck at a reunion..

—
Reply to this email directly or view it on
GitHubhttps://github.com//issues/1#issuecomment-11551024.

[yuchi]

Ok, guys. I'm finally home and quite free (remember I'm Italian so I'm stuck with xmas lunches and dinners till the 26th 🎄 😞 )

I'm going to be on skype for few hours (you can find me as yuchi_it)

[vectorsize]

So after talking to Pier on skype today we decided to try and have a kickoff meeting sometime next week.

Correct me if Im wrong Pier, but I think the suggestions where perhaps wednesday in the morning or sometime thursday or friday. Im available anytime so it's your call @michael – of course everyone else is welcome to join if you wish.

See you soon!

[michael]

Nice. I can't do Wednesday but Friday would work I think? What are your plans?

My focus currently is on the composer side, polishing the layout and fixing bugs that occur during editing. Also we need to push the campaign somehow (we're not too good at marketing). I thought of sending emails to all existing Substance.io users, which make around 4000 ppl. Do you guys know about good services for handling such email tasks?

Will checkout Mailchimp for that. Talk soon.

[Integral]

Mailchimp is good. They have free option with mailchimp banner.

On 24 December 2012 13:48, Michael Aufreiter notifications@github.comwrote:

Nice. I can't do Wednesday but Friday would work I think? What are your
plans?

My focus currently is on the composer side, polishing the layout and
fixing bugs that occur during editing. Also we need to push the campaign
somehow (we're not too good at marketing). I thought of sending emails to
all existing Substance.io users, which make around 4000 ppl. Do you guys
know about good services for handling such email tasks?

Will checkout Mailchimp for that. Talk soon.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1#issuecomment-11656982.

[vectorsize]

Friday I'm in, just pick a time!

Also...true, Mailchimp seems to be ok if we dont want to roll our own solution.