Streaming API

[jonasschneider]

The initial points outlined by @titanous:

• The API should be read-only. Pipelined/keep-alive HTTP POSTs are fine for post creation.
• We need to keep shared hosting providers in mind. This means that clients should support a 30x redirect to a different server.
• The streaming API will not be an optional part of the protocol, so there's no need to specify the endpoint in the profile.
• I don't want to switch to a framework like Goliath that depends on EventMachine. While EM is useful in some situations, I'm not a fan of using it for tentd.
• Sinatra supports streaming out of the box on most Rack servers, we should look into how that works.
• To keep deployment simplicity, we should not require spinning up a separate process to do streaming. We can support an optional separate process deployment, but this must not be required.

This is the first draft of what I came up with. Check it out and run

curl -H "Accept: application/vnd.tent.v0+length-prefixed-json-stream" http://localhost:5000/posts

The concurrency is utterly broken and completely open to DoS and probably countless other attacks, and the performance sucks. Yesyes. I'm open to everything.

Things still to be done:

• extract serialization into TentD::Serializer or similar
• use an external thread pool library, or manage the streaming threads manually in a sensible way (i.e. limit to N per app/IP or similar, clean up occasionally)
• use a shared DB connection and process-internal broadcast
• check if that JSON format is right
• add SSE format (text/event-stream, http://dev.w3.org/html5/eventsource/).
• clean up the raw HTTP fail
• maybe consolidate the Post.fetch_with_permissions/fetch_all API
• The endpoint should support all of the filtering parameters that the current /posts endpoint does.
• We need to sort out advisory events/backpressure for clients that do not read quickly enough (we should not be buffering posts infinitely for slow clients).
• mighty specs!!
• Hijack rack/rack#481

[titanous]

Use \r\n for line breaks.

[titanous]

Overall this is a good start. Post comments with updates as it evolves, and I'll jump in.

[raggi]

Ewwww.... you don't want to do this...

[raggi]

What happens when this forwards an exception to the main thread in the middle of an ensure block?

[jonasschneider]

Are you looking to implement this feature in 0.3?

[jvatic]

Streaming has been deferred until 0.4 and tentd has been completely rewritten for 0.3.