Compressible onions for rendez-vous routing

[cdecker]

This is a draft of the compressible onion being discussed on the mailing list at the moment. It includes generation, compression, serialization and decompression of the compressible onions in the devtools/onion tool.

There are still a number of rough edges that I'll address before un-drafting this:

• Move some more of the shared secret logic into sphinx.c. With this I also intend to introduce a new type compressed_onion that can be derived from a full onion, and that can be decompressed into a full onion. Currently this is handed around as a u8*.
• There seems to be a stack-smashing bug somewhere, that I need to trace down.
• Some of the commits need to be rephrased.

[rustyrussell]

Nice work! Agree that a struct compression_onion would wrap it nicely with a bow, but it's usable as-is for my tests!

[cdecker]

Reworked the API for handling compressed sphinx onions by adding the following functions:

• serialize_compressed_onion
• sphinx_decompress
• sphinx_compress
• sphinx_compressed_onion_deserialize
• sphinx_compressed_onion_serialize

This should allow transforming the compressed onion from/to both full onions and serialized formats.

I also found the stack smashing issue, so this should be good to go :-) Keeping as a draft until the spec solidifies.

[rustyrussell]

Please take "read_buffer" and throw it into the sun where future generations can't be exposed to it.

sphinx_compressed_onion_deserialize() MUST be safe against all inputs! If you used a pull_ function that would happen naturally, but also it shouldn't assert() on its inputs.

(The first thing I did is use this on an incoming TLV field, which would have been v. bad!)

[rustyrussell]

Please make decompress function robust!

[rustyrussell]

Still one deserialization bug, and gave you more work to do :)

[rustyrussell]

This assert is true, but can we be more explicit?

I wonder if we'd be better with the hmac at the front for compressed onions? Makes it easier to pull apart if the variable record is at the end?

Either way, let's introduce a fromwire_tal_bytes() which does the tal_arr and fromwire_u8_array, IFF there is sufficient room in the msg (otherwise, don't even allocate). That prevents a whole class of errors, and means we can simply skip the routelen > ROUTING_INFO_SIZE check:

fromwire_pubkey(&cursor, &max, &dst->ephemeralkey);
dst->routinginfo = fromwire_talarr(&cursor, &max, max - HMAC_SIZE);
fromwire_u8_array(&cursor, &max, dst->mac, HMAC_SIZE);

if (!cursor)
    return tal_free(dst);

assert(max == 0);

Note that you still should check that cursor is not NULL: that pubkey could be malformed!

[cdecker]

Good point, I like fromwire_tal_bytes, however it doesn't address the full issue, since that check isn't just there to avoid allocating a huge array, but also to enforce the logic that an onion that is larger than an uncompressed onion can't possibly be a compressed onion.

I added an initial check that the total compressed onion length is smaller than the total length of an uncompressed one, no longer computing routelen

[rustyrussell]

Ack: ebe9f9c

Rebase onto master, rename -devtool/onion option.

[rustyrussell]

Removed final commit which broke tests; we can do a separate PR for that.