Protocols

Protocols

External Commenters

• http://stuartsierra.com/2008/07/10/thrift-vs-protocol-buffers: ancient, from 2008

• http://predic8.com/protobuf-etch-thrift-comparison.htm: has a list of protocols and brief comments about them

Cap'N Protocol

• Home

Update to Protocol Buffers, by one of its original authors.

Protocol Buffers

• Home
• Format: yes
• Transport: no
• C Support:
  • https://code.google.com/p/protobuf-c/
  • http://koti.kapsi.fi/jpa/nanopb/
  • https://github.com/haberman/upb

Looks like we could do an rpc-like (or even RESTy) protocol on top of pbufs streamed directly over tcp. Code size for dependencies would be very small, would have to write the C/Java directly ourselves.

nanopb

Doesn't use malloc, no dependencies, looks VERY memory efficient, and it's runtime is less than a thousand lines of code.

The price is that it's API can be a bit heavy going, you need a seperate callback function for every dynamically sized message member...

For example, support for the addressbook.proto tutorial example is almost unreadable as-is. In order to make it more friendly, you need to add a nanopb extension comment on strings to the .proto definition, and to other dynamically sized message members, giving maximum sizes, then it will create the fixed-size buffers in the struct for you. See http://koti.kapsi.fi/jpa/nanopb/docs/concepts.html for descriptions of max_size and max_count.

Interestingly, since protobuf encoding has internal size delimiting, it looks like it might be possible to stream them directly over TCP, or more reasonably, over SSL. This avoids the extra overhead of HTTP, or any other transport protocol.

• addressbook example, in C: pbuf/addressbook.pb.h, pbuf/addressbook.pb.c
  • Note how it all uses callbacks, because the .proto file wasn't annotated with max buffer sizes for strings.
• client/server example, in C:
  • pbuf/nanopb-example/fileproto.proto proto with file annotations
  • pbuf/nanopb-example/fileproto.pb.h generated struct
  • pbuf/nanopb-example/client.c client code, show use of api
  • pbuf/nanopb-example/server.c server code, ditto
  • pbuf/nanopb-example/common.c shows how a pb stream is bound to a socket

Thrift

• Home
• Format: yes (various, including json)
• Protocol: yes (but no C support)

Funny, its got lots of language support (D!), but no C support other than some incomplete support using glib (which is too heavy a dependency, I'm not porting glib to VxWorks!).

Probably good for data centre environment, don't see how porting it to VxWorks would be a good idea.

Avro

• C docs
• JSON or IDL (experimental) for protocol spec
• minimal external dependencies for C
• no C/C++ RPC support
• schema is encapsulated with message. For dynamic languages this could make deploying new pieces a bit easier
• has some sort of weird Avro-specific string representation

Looks like the message wire format is pretty simple, just a bunch of uin32 length/value pairs. However, the schema is also in the message, and maybe needs to be handled.

I think we could use this, but it would require some C-side work. If using Thrift was really compelling from the point of view of the rest of the system, we could get it working.

JSON-REST

I still like this idea.

I can use joyent's http-parser, do the sockets myself (or use libevent, maybe, if I can strip out some small part for just tcp handling), and use some kind of json lib (TBD on that).

C support:

• libjson:
  • This library looks quite good. Uses malloc, so supports a reasonable API, but still quite small and efficient.
• jsmn: no malloc, single pass, no deps, MIT
  • Underdocumented API, see http://alisdair.mcdiarmid.org/2012/08/14/jsmn-example.html and https://github.com/alisdair/jsmn-example/blob/master/github.c
• libsyck is a yaml parser, but yaml is superset of json...
• yajl: streaming
• libjson: can push data chunk-wise, callback based, but LGPL.
• cjson
• jsonsl, cson, json-parser
• jansson
• json-c: "reference counting", oh no
• ccan:

REST

• http://blog.feedly.com/2009/05/06/best-practices-for-building-json-rest-web-services
• http://marakana.com/s/post/1221/designing_a_beautiful_rest_json_api_video
• http://kinderman.net/2010/06/23/approaching-pure-rest-learning-to-love-hateoas

Java support:

• Jersey
• RESTEasy (from stackoverflow)
• spring?

C support:

• http-parser: need to do sockets on top
  • lua bindings for above: https://github.com/creationix/lhttp_parser
• libevent: has an HTTP server (and SSL?) support, but looks like the quality isn't good, see below.
• libevhtp: hm.

Documenting JSON-REST APIs

It looks to me (SR) that using either JSON message examples, with inline comments, would be a decent way, maybe using docco to pull docs out.

Also, could use json schema files, with inline docs.

• Twitter api documentation, an example of how to do it
• enunciate, Java WS stuff.... useful?
• iodocs, another generator of client-js for APIs, as well as node.js-side server stubs?
• swagger
• wadl and do we need it?
• review of above, and more, with some discussion of various approaches
• restdoc, another spec for machine readable rest api description, note the ref to node.js... maybe our server endpoints should use node.js? :-) especially simple aggregators... so many technologies, so many choices.

Also, maybe "spore"? https://github.com/agladysh/api-description

JSON-RPC

• Home

Not REST, but if we aren't using HTTP, REST is maybe not the way to go?

TIPC

• Home: http://tipc.sourceforge.net/doc/Programmers_Guide.txt

Alternative to TCP, lots of cool features... we can't use it. Needs kernel support, and won't make it through firewalls. Sigh.

Zeugma was a heavy user, apparently.

SOAP/HTTP

VxWorks is supposed to support SOAP... the only saving grace of considering this.

Etch

• Home

Has some interesting features, like bi-directional communication.

C binding depends on apr, the Apache portable runtime. That's an unacceptably large dependency (even if it has been ported to VxWorks).

Eclipse TCF

• Home

TCF is a vendor-neutral, lightweight, extensible network protocol mainly for communicating with embedded systems (targets).

Initially, looked really promising, but after looking closer, it appears like this is for development tool support (thus, the involvement of the Eclipse foundation), it allows things like downloading images, getting stack traces, etc. Its not a run-time management protocol.

OPC-UA

Possible :-)

ICE

• Home

No support for C

MessagePack

• Home
• Wikipedia

Seems to be a binary equivalent of JSON - no IDL, no generated code, just binary representation of dict-like recursive data structues.

• Java documentation is API only - no description
• message-focused, not model focused
• couldn't find docs on MessagePack-RPC
• relatively simple

ActiveMQ

• C support may be fairly rough
• no JSON support