BOLT-04: modify Sphinx packet construction to use starting random bytes

[Roasbeef]

In this commit, we modify the existing instructions to create the Sphinx
packet to no longer start out with a zero initialize set of 1366 bytes.
Instead, we now instruct the sender to use random bytes derived from a
CSPRG. This fixes a recently discovered privacy leak that allows an
adversarial exit hop to ascertain a lower bound on the true path length.

Note that this doesn't affect packet processing, so this is a backwards
compatible change. Only clients need to update in order to avoid this
privacy leak.

After this change is applied, the test vectors as is don't match the
spec, as they're created using the original all zero starting bytes. We
can either update these with our specified set of random bytes, or leave
them as is, as they're fully deterministic as is.

An alternative path would be to generate more random bytes from the
session key as we do elsewhere (the chacha based CSPRNG).

[rustyrussell]

Ack e116441

[cfromknecht]

Misspelled words in 04-onion-routing.md: CSPRNG

[cdecker]

fwiw we have decided to generate the initial content of the routing packet using ChaCha20 keyed with a key derived from the session_key using the pad type:

https://github.com/rustyrussell/lightning/blob/d8c386a141beccc67fc9d30de70cf027331de089/common/sphinx.c#L534-L542

This would allow us to maintain reproducibility, at the cost of slightly more computation to generate the stream (then again CSPRGs are likely also not free).

One notable difference to this PR is that we just fill the entire routing packet instead of offsetting from the end of the last hop's payload.

[Roasbeef]

So i've updated our implementation on the lnd end to allow the caller to specify their own filler: lightningnetwork/lightning-onion#40 (comment)

In the end, all that matters is that the callers fill out the packet with random bytes, the precise method can actually be left up to the sender. The receiver won't be able to distinguish how the bytes were generated, just that they're indistinguishable from a random set of bytes.

[Roasbeef]

One notable difference to this PR is that we just fill the entire routing packet instead of offsetting from the end of the last hop's payload.

The PR just fills the entire set of bytes now. Skipping the first few bytes is really just an optimization.

See my comment above w.r.t how we can still maintain reproducibility yet let the caller choose between the two...

If we go with this route, we'd need to add a comment to the test vectors. Alternatively, we can update the not-really-psuedo-code to do something similar to this: https://github.com/lightningnetwork/lightning-onion/blob/088b62e7296023531f54595092a03672278acadc/sphinx.go#L189

[Roasbeef]

On the other hand, it's nice for there just to be "one correct way" in the end. This would mean less guess work for implementors and more "plug and chug" (just implement the spec and you'll be fine) specification. Going the route of re-using the chacha stream (extracting more bytes under a different key) would give us that IMO.

[t-bast]

I think it would be useful to update the official test vectors for that change, especially if we're all leaning towards spec-ing one way to do it (I like @cdecker's proposal to reuse ChaCha20).

Applying this change here is what I'm getting as new bolt 4 test vectors. Let me know if you find the same results.

onion-test-v0.json

{
  "comment": "This is a simple testcase in which we only use v0 payloads, and all hops have single frame payloads",
  "generate": {
    "session_key": "4141414141414141414141414141414141414141414141414141414141414141",
    "associated_data": "4242424242424242424242424242424242424242424242424242424242424242",
    "hops": [
      {
        "realm": 0,
        "pubkey": "02eec7245d6b7d2ccb30380bfbe2a3648cd7a942653f5aa340edcea1f283686619",
        "hop_shared_secret": "53eb63ea8a3fec3b3cd433b85cd62a4b145e1dda09391b348c4e1cd36a03ea66",
        "payload": "0000000000000000000000000000000000000000000000000000000000000000",
        "rhokey": "ce496ec94def95aadd4bec15cdb41a740c9f2b62347c4917325fcc6fb0453986",
        "mukey": "b57061dc6d0a2b9f261ac410c8b26d64ac5506cbba30267a649c28c179400eba",
        "hmac": "ac73e62f2cc382a1356e06f2762f9cdbcec7c7771037e824b0c1071f4f93d614"
      },
      {
        "realm": 0,
        "pubkey": "0324653eac434488002cc06bbfb7f10fe18991e35f9fe4302dbea6d2353dc0ab1c",
        "hop_shared_secret": "a6519e98832a0b179f62123b3567c106db99ee37bef036e783263602f3488fae",
        "payload": "0101010101010101000000000000000100000001000000000000000000000000",
        "rhokey": "450ffcabc6449094918ebe13d4f03e433d20a3d28a768203337bc40b6e4b2c59",
        "mukey": "05ed2b4a3fb023c2ff5dd6ed4b9b6ea7383f5cfe9d59c11d121ec2c81ca2eea9",
        "hmac": "9350c058fe33b70d9cea125d2d68ba02d81af4609c916d51a575c8db2d5f1de5"
      },
      {
        "realm": 0,
        "pubkey": "027f31ebc5462c1fdce1b737ecff52d37d75dea43ce11c74d25aa297165faa2007",
        "hop_shared_secret": "3a6b412548762f0dbccce5c7ae7bb8147d1caf9b5471c34120b30bc9c04891cc",
        "payload": "0202020202020202000000000000000200000002000000000000000000000000",
        "rhokey": "11bf5c4f960239cb37833936aa3d02cea82c0f39fd35f566109c41f9eac8deea",
        "mukey": "caafe2820fa00eb2eeb78695ae452eba38f5a53ed6d53518c5c6edf76f3f5b78",
        "hmac": "ef3860bc4d742ca9a6c2818d03c83f034d5c22776232a87ca049ec79b63627b9"
      },
      {
        "realm": 0,
        "pubkey": "032c0b7cf95324a07d05398b240174dc0c2be444d96b159aa6c7f7b1e668680991",
        "hop_shared_secret": "21e13c2d7cfe7e18836df50872466117a295783ab8aab0e7ecc8c725503ad02d",
        "payload": "0303030303030303000000000000000300000003000000000000000000000000",
        "rhokey": "cbe784ab745c13ff5cffc2fbe3e84424aa0fd669b8ead4ee562901a4a4e89e9e",
        "mukey": "5052aa1b3d9f0655a0932e50d42f0c9ba0705142c25d225515c45f47c0036ee9",
        "hmac": "22b8aa7917b4e2c87fc5678bab10b450454fbb5e70708c3d49f0670a616085ea"
      },
      {
        "realm": 0,
        "pubkey": "02edabbd16b41c8371b92ef2f04c1185b4f03b6dcd52ba9b78d9d7c89c8f221145",
        "hop_shared_secret": "b5756b9b542727dbafc6765a49488b023a725d631af688fc031217e90770c328",
        "payload": "0404040404040404000000000000000400000004000000000000000000000000",
        "rhokey": "034e18b8cc718e8af6339106e706c52d8df89e2b1f7e9142d996acf88df8799b",
        "mukey": "8e45e5c61c2b24cb6382444db6698727afb063adecd72aada233d4bf273d975a",
        "hmac": "f4bc4473ec0a55d5f3ededdba4a8e303f9128391a0f1f54d8adacf2221c04e1b"
      }
    ]
  },
  "onion": "0002eec7245d6b7d2ccb30380bfbe2a3648cd7a942653f5aa340edcea1f283686619e5f14350c2a76fc232b5e46d421e9615471ab9e0bc887beff8c95fdb878f7b3a71d215fe077311013884fabc09318c23a2435069ff1c40e90d2c150084eb8eb59ed10f7d61ab590531cf08000178a333a347f8b4072e216400406bdf3bf03865979313da96e66517a37b5c485744275a73b8eab6f43e3d56a2a59764eb12617efbe0ce721caad69320c3a469a202f3e468c67eaf7a7cda226d0fd32f7b48084dca885d3a3451e1a7368bb5edac5c665a61f6796b288bc1f12cd3b9fbaf915bc732ee749a1ec866abe044a9ad635778ba61fc0776dc832b39451bd5d35072d2269cf9b04040caeaeed42e3eb4adf6146661e3708accfeeb02d174de7effeaa8040417c8b49ed4c7f0218ff8c6c7dd7221d189c65b3b9aaa71a01484b122846c7c7b57e02e679ea8469b70e14fe4f70fee4d87b910cf144be6fe48eef24da475c0b0bcc6565ac90cc68a2cfd94a8a93dc787d6a074e539903b1d36e8d5d45655aa59208a20f3f600ef6b05d963ab796cdf33748c944795403626cdff6c0f114586f1754fb6d64800beeda3e18c04f109b7a91c9049ba2967316f088a8b45c57ea708768ed1cc99f06886ee3089d6c8e9ea3ce884b8dfffed4d4cdc425bf4a8740d03cfcdc95861300a03f5913bf284daa96b9c81dac1ec47cd72bd1049256f41d3710bb47d6151bc6d2dca3851363b4c499f53cbfd445bbd11e669f6b868b35b2e2cc950b1dcde4517b548066ac7543d9f9ac78393b61d1df632c1e7e048427d36227a51f795049c9b89cfdf25b9982e8d79f0da84caf54f7904e8731af45bcbf558eeefd885e2907ce00a3f2124421a704c1907a966e26215b26d2fa7558406a767c0e7218e1f9ac26b012971eba585be112b4e84abbab93a6d76f626b744c1f86df334dcc5d5277b2ce89e0c9869be3fd952ccfac0d3ab40eecd8dd5356155a8b0b186e650b9bbac561a9b5f309b18d31d3bfdd6dac9169b7e38f82b300e3598eb142fb5dd99612d3cd8e1f720551142f5460736e6da0d1158b238a626afa86076c64851694651d2bf6808de85104ef995ddc159d2efbe67c50a2462c10864400bf01ef387d63dd06b21ddebc4ddfb12b9ff2ad2816bcabf737ed8e89f8749a5179ab26faf5796c80b2364af41dc0384507b160c1e7bd93cc6062d82fb87572b9822f1400222c9517a5f4a89fe8fbe39b54efcca1d40bc8627d6b1f43e621c3297ea5ab9076824d90cfb86e5025aad6160cbff1d62c1e46d86a43dce76e09bc1833def9deeb42b02e96b2e423c3acf305e02d4c2c942fddcba269ffcdaef2b4dd6514b427e3236cee79bc28a26adf83af2137a9c7c6453a8571a65d86146c6a93fea20fe91d64f4e3eda8c1980d68e4d9d22a8dd20d5b5d36abda9a7a32bc0711802bd68a19a5837d6e45cf21ae674e291f9ab6846c9c0440017b47630f67cba07a90bfc98b4acd9bef2f8b379357f0e91c3dd04ee819214cfb4f50e35e9a6428d072c09240bb550b225ad312698a827da5fad2f1ae11a474d02872a3554822ff12045b230276f3ad5bd57e5b1e71705076ce6c234b32543cd86a8bec4a1da37b02ffeceb4ebc98311c2c1938458ce8e390e8b702032efc2d30326ffd9a16bbee7cd198b442e22a5210152c6b08c13972b50f5c4d3fd8e393d752463b9192f66cb4be243737ae167c044e4517b4d5efd6af68ab040477acd144fbd42d60725ed2a3e0d83af620e800c6c55e628bef178d4490bc8fa0e0be00f309183c75a402fa501ec8253530199d74bd360932281c3cc5a5d33906e695c927f49b169604959aec1978a369784bdeaecfb79281996b1fc377841ac73e62f2cc382a1356e06f2762f9cdbcec7c7771037e824b0c1071f4f93d614",
  "decode": [
    "4141414141414141414141414141414141414141414141414141414141414141",
    "4242424242424242424242424242424242424242424242424242424242424242",
    "4343434343434343434343434343434343434343434343434343434343434343",
    "4444444444444444444444444444444444444444444444444444444444444444",
    "4545454545454545454545454545454545454545454545454545454545454545"
  ]
}

onion-test-multi-frame.json

{
  "comment": "A testcase for a variable length hop_payload. The third payload is 256 bytes long.",
  "generate": {
    "session_key": "4141414141414141414141414141414141414141414141414141414141414141",
    "associated_data": "4242424242424242424242424242424242424242424242424242424242424242",
    "hops": [
	{
	    "type": "legacy",
	    "pubkey": "02eec7245d6b7d2ccb30380bfbe2a3648cd7a942653f5aa340edcea1f283686619",
	    "payload": "0000000000000000000000000000000000000000"
	},
	{
	    "type": "tlv",
	    "pubkey": "0324653eac434488002cc06bbfb7f10fe18991e35f9fe4302dbea6d2353dc0ab1c",
	    "payload": "0101010101010101000000000000000100000001"
	},
	{
	    "type": "raw",
	    "pubkey": "027f31ebc5462c1fdce1b737ecff52d37d75dea43ce11c74d25aa297165faa2007",
	    "payload": "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff"
	},
	{
	    "type": "tlv",
	    "pubkey": "032c0b7cf95324a07d05398b240174dc0c2be444d96b159aa6c7f7b1e668680991",
	    "payload": "0303030303030303000000000000000300000003"
	},
	{
	    "type": "legacy",
	    "pubkey": "02edabbd16b41c8371b92ef2f04c1185b4f03b6dcd52ba9b78d9d7c89c8f221145",
	    "payload": "0404040404040404000000000000000400000004"
	}
    ]
  },
  "onion": "0002eec7245d6b7d2ccb30380bfbe2a3648cd7a942653f5aa340edcea1f283686619e5f14350c2a76fc232b5e46d421e9615471ab9e0bc887beff8c95fdb878f7b3a71068850554818be051c5fb6e47df0f24ab88623abf089afc95db4f653cfdc4e3fc50f7d61ab590531cf08000178a333a347f8b4072ecac76d4ab27f190828a68222a6bb6c6f23fc671295d5d4fe37b9f3dacaa629d6404bae9a8e8d7178977d7094a1ae549f89338c0777551f874159eb42d3a59fb9285ad4e24883f27de23942ec966611e99bee1cee503455be9e8e642cef6cef7b9864130f692283f8a973d47a8f1c1726b6e59969385975c766e35737c8d76388b64f748ee7943ffb0e2ee45c57a1abc40762ae598723d21bd184e2b338f68ebff47219357bd19cd7e01e2337b806ef4d717888e129e59cd3dc31e6201ccb2fd6d7499836f37a993262468bcb3a4dcd03a22818aca49c6b7b9b8e9e870045631d8e039b066ff86e0d1b7291f71cefa7264c70404a8e538b566c17ccc5feab231401e6c08a01bd5edfc1aa8e3e533b96e82d1f91118d508924b923531929aea889fcdf0568b9af020955bf28f2296f547bec0773d4f76e19351ceda5ad381e6a79b57a5a658149e7e9ced4edde5d38c9b8f92e16f6b4ab13d704721bb12be5941e1b6620be32870100cb1000cf05b7562be8dbfb7fdb4da45b2f3bca8e6fe47eb45fbdd3be21a8a8d200797eae3c9a0497132f92410d804977408494dff49dd3d8bce248e0b74fd9e6f0f7102c25ddfa02bd9ad9f746abbfa337cea1082d8ae639b52346b18c05b8248d9b30df256d432ab365b732a58b04d7d0d6f6dd20b810c52b2457544c629ca0911c35ff1292523f980f13ebd87d1627e9759abf8b17e54955835fa70c3d9217d21dd2a7e33bea072be9170265c9465d5b4c54a1df3c15513332dfb2123c27498af3bfa116a140d28d43661a3088c6f2ae995bd921801d43ee31a353f2e164704b1593136e84815e2d304515ddf45ef3f70b2169718c811fbe697a1c0d97032e83dec858b6672be484244127e23e2ca8fa055e8e34165c3d02219fb588c7a221722e45e2b54f6b9fa4a6013906198cdf72c8d5222155cfc64f1f322f7785d84727fdd2f9b66627cbe8c7ecf5d944ffcb9266ac815ebc4b4301ed6bc1a765bfeade304cac32e5841340ded04f1f9ea4e2fa7ceea533f8b5253a940f842176c672443af37f03a03f7219df1c4062c51e6e79cf859d07eece8cb7583643e698cf76c04cc6f4c01ca9378046e1763818c616c54f640b26a424fbc0b37f6765f57ce1ade6c8c1eef6ab12df9fb6df6125d2d8bb665a4033d1386525eebc5af407155a5bbc9f8719b836f58044a8f5d1256ca627a592c0419af0bbe4886b9f97667c1bb1712f003f9ac8c53e5ceeaa2182260bab8f5ea50951529c4bda2e58ac649bff4447b585e07f8ca3e35379e969dc71f015557fc588eef1282f7da45f57d94f9316898c904217c3a39ceb2e20c4878c6c196ad11aee10b7ed0b047220930aa1d6463d60058dbf1c36f12eb767d5dc09b9c9d1df628766a4de995fa7e85040533087133ba8892b2cd207da65b99e5d95575e4d7d5ba2666de8f9d5582c1f8799e43e25aa1ad7b945bfb97a953e09a9341c49d1346901af9eb30486bd270d750493dfb69b467a24757e5b3a7198f6794549725bccdc69303f6ccaa48f825e5f83d42faba8f32895148c18db0ec3f6411b969674a507d39a4cb723bba876b766bc2219b164a0f32fd9a8c198695343cbc7c48dda1d36ce69efd07a3d7d46275583e50b9fd1a6d95ce3454da8fe754ab58ee4b5ed48a020f514287e900076d389d6266c627f1e0e22a4b511dfa5c4891eac4d8fe7e9f0332cc26af9a5f467b4acea6da597f27fbb58511ffb8a200d68f09aa9e83b401dc012",
  "decode": [
    "4141414141414141414141414141414141414141414141414141414141414141",
    "4242424242424242424242424242424242424242424242424242424242424242",
    "4343434343434343434343434343434343434343434343434343434343434343",
    "4444444444444444444444444444444444444444444444444444444444444444",
    "4545454545454545454545454545454545454545454545454545454545454545"
  ]
}

[Roasbeef]

@t-bast I've updated my implementation to generate the random bytes re-using our existing chacha format. However, I'm getting a distinct ending packet compared to what you've posted above. My final packet is:

0002eec7245d6b7d2ccb30380bfbe2a3648cd7a942653f5aa340edcea1f283686619e5f14350c2a76fc232b5e46d421e9615471ab9e0bc887beff8c95fdb878f7b3a71e87f9aab8f6378c6ff744c1f34b393ad28d065b535c1a8668d85d3b34a1b3befd10f7d61ab590531cf08000178a333a347f8b4072e216400406bdf3bf038659793a1f9e7abc789266cc861cabd95818c0fc8efbdfdc14e3f7c2bc7eb8d6a79ef75ce721caad69320c3a469a202f3e468c67eaf7a7cda226d0fd32f7b48084dca885d014698cf05d742557763d9cb743faeae65dcc79dddaecf27fe5942be5380d15e9a1ec866abe044a9ad635778ba61fc0776dc832b39451bd5d35072d2269cf9b040a2a2fba158a0d8085926dc2e44f0c88bf487da56e13ef2d5e676a8589881b4869ed4c7f0218ff8c6c7dd7221d189c65b3b9aaa71a01484b122846c7c7b57e02e679ea8469b70e14fe4f70fee4d87b910cf144be6fe48eef24da475c0b0bcc6565a9f99728426ce2380a9580e2a9442481ceae7679906c30b1a0e21a10f26150e0645ab6edfdab1ce8f8bea7b1dee511c5fd38ac0e702c1c15bb86b52bca1b71e15b96982d262a442024c33ceb7dd8f949063c2e5e613e873250e2f8708bd4e1924abd45f65c2fa5617bfb10ee9e4a42d6b5811acc8029c16274f937dac9e8817c7e579fdb767ffe277f26d413ced06b620ede8362081da21cf67c2ca9d6f15fe5bc05f82f5bb93f8916bad3d63338ca824f3bbc11b57ce94a5fa1bc239533679903d6fec92a8c792fd86e2960188c14f21e399cfd72a50c620e10aefc6249360b463df9a89bf6836f4f26359207b765578e5ed76ae9f31b1cc48324be576e3d8e44d217445dba466f9b6293fdf05448584eb64f61e02903f834518622b7d4732471c6e0e22e22d1f45e31f0509eab39cdea5980a492a1da2aaac55a98a01216cd4bfe7abaa682af0fbff2dfed030ba28f1285df750e4d3477190dd193f8643b61d8ac1c427d590badb1f61a05d480908fbdc7c6f0502dd0c4abb51d725e92f95da2a8facb79881a844e2026911adcc659d1fb20a2fce63787c8bb0d9f6789c4b231c76da81c3f0718eb7156565a081d2be6b4170c0e0bcebddd459f53db2590c974bca0d705c055dee8c629bf854a5d58edc85228499ec6dde80cce4c8910b81b1e9e8b0f43bd39c8d69c3a80672729b7dc952dd9448688b6bd06afc2d2819cda80b66c57b52ccf7ac1a86601410d18d0c732f69de792e0894a9541684ef174de766fd4ce55efea8f53812867be6a391ac865802dbc26d93959df327ec2667c7256aa5a1d3c45a69a6158f285d6c97c3b8eedb09527848500517995a9eae4cd911df531544c77f5a9a2f22313e3eb72ca7a07dba243476bc926992e0d1e58b4a2fc8c7b01e0cad726237933ea319bad7537d39f3ed635d1e6c1d29e97b3d2160a09e30ee2b65ac5bce00996a73c008bcf351cecb97b6833b6d121dcf4644260b2946ea204732ac9954b228f0beaa15071930fd9583dfc466d12b5f0eeeba6dcf23d5ce8ae62ee5796359d97a4a15955c778d868d0ef9991d9f2833b5bb66119c5f8b396fd108baed7906cbb3cc376d13551caed97fece6f42a4c908ee279f1127fda1dd3ee77d8de0a6f3c135fa3f1cffe38591b6738dc97b55f0acc52be9753ce53e64d7e497bb00ca6123758df3b68fad99e35c04389f7514a8e36039f541598a417275e77869989782325a15b5342ac5011ff07af698584b476b35d941a4981eac590a07a092bb50342da5d3341f901aa07964a8d02b623c7b106dd0ae50bfa007a22d46c8772fa55558176602946cb1d11ea5460db7586fb89c6d3bcd3ab6dd20df4a4db63d2e7d52380800ad812b8640887e027e946df96488b47fbc4a4fadaa8beda4abe446fafea5403fae2ef

To generate this key, I first serialize the session private key as a big-endian number (padding to a length of 32 bytes if needed), then feed it in as the data to sha256-hmac, using "pad" as the key (appears to also match @cdecker's code).

[Roasbeef]

Updated the PR (other than the test vectors) to use the new pad key to deterministically generate a set of random bytes.

[t-bast]

However, I'm getting a distinct ending packet compared to what you've posted above.

Good thing I shared mine, it spot an issue :).
I was using the shared_secret with the recipient instead of the session_key to compute the padding key!
I updated that and I find the same onion as you do, so you can now update the test vectors and we should have a match.

[Roasbeef]

Thanks for the review @t-bast! The latest push includes the set of updated test vectors, as well as those typo fixes :p.

@cdecker can you verify that CL outputs the same final onion as specified in this PR? Thanks.

[t-bast]

Thanks @Roasbeef! One small change is missing: in the onion-test-v0.json the hmacs in hop are now incorrect. They should be:

• b8640887e027e946df96488b47fbc4a4fadaa8beda4abe446fafea5403fae2ef
• a93aa4f40241cef3e764e24b28570a0db39af82ab5102c3a04e51bec8cca9394
• 5d1b11f1efeaa9be32eb1c74b113c0b46f056bb49e2a35a51ceaece6bd31332c
• 19ca6357b5552b28e50ae226854eec874bbbf7025cf290a34c06b4eff5d2bac0
• 16d4553c6084b369073d259381bb5b02c16bb2c590bbd9e69346cf7ebd563229

[Roasbeef]

Nice catch! Of course the HMAC changes as well, I overlooked it, will update the set of test vectors. `lnd` 0.9 (to be release soon), will start to generate onions according to the guidelines in this PR.

…

On Mon, Jan 13, 2020 at 1:38 AM Bastien Teinturier ***@***.***> wrote: Thanks @Roasbeef <https://github.com/Roasbeef>! One small change is missing: in the onion-test-v0.json the hmacs in hop are now incorrect. They should be: - b8640887e027e946df96488b47fbc4a4fadaa8beda4abe446fafea5403fae2ef - a93aa4f40241cef3e764e24b28570a0db39af82ab5102c3a04e51bec8cca9394 - 5d1b11f1efeaa9be32eb1c74b113c0b46f056bb49e2a35a51ceaece6bd31332c - 19ca6357b5552b28e50ae226854eec874bbbf7025cf290a34c06b4eff5d2bac0 - 16d4553c6084b369073d259381bb5b02c16bb2c590bbd9e69346cf7ebd563229 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#697>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHTWLWBIG7ANU5PSCFJYUDQ5QZC3ANCNFSM4JJOTXVQ> .

[Roasbeef]

Updated, also fixed a spelling error caught by the Travis spellcheck.

[t-bast]

ACK 2c45eba, thanks @Roasbeef

[cdecker]

I'm missing the updates to the multi-frame test-vector, which stumped me for a bit. This results in the following failure when running the test-vector:

devtools/onion runtest ../lightning-rfc/bolt04/onion-test-multi-frame.json
onion: Generated does not match the expected onion:
generated: 0002eec7245d6b7d2ccb30380bfbe2a3648cd7a942653f5aa340edcea1f283686619e5f14350c2a76fc232b5e46d421e9615471ab9e0bc0393985e79ca785dfc3b4f506f38e0b228b21e9eaa403231155af3bbd299303a9795d5d9bb1b44e09a8a525c2a6edf4d366dca91fdf4f91788a678da958a4f41afd21754d7677195e25fe7cea974695531f2e4ff39d03745e85d7b6e11e5be22dafaa2984dc5dc7e31fbfc187301fd50a04e768e5a8e37d668a2c657e830562b4fbcfe54c69469b5cbe0b30c72c76e04addaf5ad190c367037e7539df17044dfd2170bf0066756c90d46ac2469e7869b3527cb06475bb6ff273c2cc602a3164c8909a693e6a9a94fd38aecedd5ff27a45e6b9e68351ed1f34f7196bf39df628fdb2e6d673d1f5bfa6501883e57de93e95cf331c028fade01aaa18212b84e53604914232075d11b50e14aa9e7481efe8e90fec9f2e7bca437c7d84ba567377c11facfe6d925a13503565792a859720b443ceed5b91aaf48a40fca90926ea8fac2d40847e18267504eb8cde102ba4df8a7028f7e7832d3aada0e3b4d9f01bd6dd49ef94f7265c73997f09fef8b124c298209699487671056ef4a426df5583f152b5dee669212d63bb2386fb52d153d0ae2f5fa68ad533d552b1559b7a2df6e4353de187e5cf06648c8e4578190297e3a0baf031af264e8a156dd00bdc78844d6c974a890cbc195ae05175efbdf0cc0f2d41c19b885644c689957fbe686f6025e26742d39b3cb3c8f9a0520fcea294a514b6c0fc32bf6f78b3dd80357d611897162e5af088853847dbc11a701c0b2d5d692ced5dbb206954a396f3b6c504b5056c7f5d1e2c3a2a8011319ad684f0ccaeb2ab8db9af0ea9a97cc4f7efdad6b7bacd29b2d08ccbe69e877ef2c3a4b4e64265965cfea0dd70cf3c0e91440f5aee82c9f6de5c6a386c2b9fe15b286d0fe18f126fee1929d3a43ec687923cd79564abeec1313a490ff1fde0807c7a9dc5852576924cfeffb5e38f13ed3b9e620c78fb855dfc35a5952feda1ea78d91e43421c4e72e684f1728ffe420c72abe128074108e194fc153d52c88641bc2756b42b72c5072be3677f675e5d9236899573b42827bf79258db2650b1a6910cceab30bbaa05300dc29a7142646961aff2450c073695e1c0910fb8c2c47703f72165f460ea0f0e9e23c46a126b82b46d4ee81b6797358e943329bbc030d085bd4d6a225613bef2f1ba38c9bb01deb4ce575972fbe87affbee860670fb6819955c5d821e095eb3f44c14ca3b133194d31bcf2c8042ef3bf5995ae0b8920aad6c1863b3a059f4f8adf16abc93100c3a3dd733217258f3482633a4d0a671ed82fdf2204fa145d873756713d548c0e97432473e5cf85e82e5d2a8654f75f1980e53db4118439498765da4e5f5fe5f971a681215deb6681f43803e3052e33729bcf003aad45f00c2bf40e576c50177c11ef9f0e37ddcc0c7d08ca106349cba304663bcba57fac995ea965f763bcb3f408d8b6fe7695df82e2fb3c75f74289a4fb18a5cefb159a69ef570825f3620c957ed81699d8ebbfe3d6656ebab438122f26acd8f593d55bcd50d09c36507a075ebf9d77c64311ee93f18f81c48397b5eecfe7bf18db53d7b905bc0dac35dca408ef0c848eeda3029c92908a2e99cae009e969934851cc1b5ccf36cca584edb11f5036408bb6001822e237b2173476c6331408bc7e9393b9283ec8803009630da3704c6302507818aeb79099b0d5b1e2cfbc4c3d8c5652e0f81169628b08393ad5a2e8baf16ba7122646ebefee99d24a613960aadc268c5432fd4177d936523b999901f5bb1cf64c38d2eb7c227864182d5e4e655049f7ba426c9af2d916f4e766050a17c77e04ccfcd87aac40f9c78c4109cde53e7f72f2308386d33951763d994c14032282d60b70
expected : 0002eec7245d6b7d2ccb30380bfbe2a3648cd7a942653f5aa340edcea1f283686619e5f14350c2a76fc232b5e46d421e9615471ab9e0bc887beff8c95fdb878f7b3a710f8eaf9ccc768f66bb5dec1f7827f33c43fe2ddd05614c8283aa78e9e7573f87c50f7d61ab590531cf08000178a333a347f8b4072e1cea42da7552402b10765adae3f581408f35ff0a71a34b78b1d8ecae77df96c6404bae9a8e8d7178977d7094a1ae549f89338c0777551f874159eb42d3a59fb9285ad4e24883f27de23942ec966611e99bee1cee503455be9e8e642cef6cef7b9864130f692283f8a973d47a8f1c1726b6e59969385975c766e35737c8d76388b64f748ee7943ffb0e2ee45c57a1abc40762ae598723d21bd184e2b338f68ebff47219357bd19cd7e01e2337b806ef4d717888e129e59cd3dc31e6201ccb2fd6d7499836f37a993262468bcb3a4dcd03a22818aca49c6b7b9b8e9e870045631d8e039b066ff86e0d1b7291f71cefa7264c70404a8e538b566c17ccc5feab231401e6c08a01bd5edfc1aa8e3e533b96e82d1f91118d508924b923531929aea889fcdf057f5995d9731c4bf796fb0e41c885d488dcbc68eb742e27f44310b276edc6f652658149e7e9ced4edde5d38c9b8f92e16f6b4ab13d710ee5c193921909bdd75db331cd9d7581a39fca50814ed8d9d402b86e7f8f6ac2f3bca8e6fe47eb45fbdd3be21a8a8d200797eae3c9a0497132f92410d804977408494dff49dd3d8bce248e0b74fd9e6f0f7102c25ddfa02bd9ad9f746abbfa3379834bc2380d58e9d23237821475a1874484783a15d68f47d3dc339f38d9bf925655d5c946778680fd6d1f062f84128895aff09d35d6c92cca63d3f95a9ee8f2a84f383b4d6a087533e65de12fc8dcaf85777736a2088ff4b22462265028695b37e70963c10df8ef2458756c73007dc3e544340927f9e9f5ea4816a9fd9832c311d122e9512739a6b4714bba590e31caa143ce83cb84b36c738c60c3190ff70cd9ac286a9fd2ab619399b68f1f7447be376ce884b5913c8496d01cbf7a44a60b6e6747513f69dc538f340bc1388e0fde5d0c1db50a4dcb9cc0576e0e2474e4853af9623212578d502757ffb2e0e749695ed70f61c116560d0d4154b64dcf3cbf3c91d89fb6dd004dc19588e3479fcc63c394a4f9e8a3b8b961fce8a532304f1337f1a697a1bb14b94d2953f39b73b6a3125d24f27fcd4f60437881185370bde68a5454d816e7a70d4cea582effab9a4f1b730437e35f7a5c4b769c7b72f0346887c1e63576b2f1e2b3706142586883f8cf3a23595cc8e35a52ad290afd8d2f8bcd5b4c1b891583a4159af7110ecde092079209c6ec46d2bda60b04c519bb8bc6dffb5c87f310814ef2f3003671b3c90ddf5d0173a70504c2280d31f17c061f4bb12a978122c8a2a618bb7d1edcf14f84bf0fa181798b826a254fca8b6d7c81e0beb01bd77f6461be3c8647301d02b04753b0771105986aa0cbc13f7718d64e1b3437e8eef1d319359914a7932548c91570ef3ea741083ca5be5ff43c6d9444d29df06f76ec3dc936e3d180f4b6d0fbc495487c7d44d7c8fe4a70d5ff1461d0d9593f3f898c919c363fa18341ce9dae54f898ccf3fe792136682272941563387263c51b2a2f32363b804672cc158c9230472b554090a661aa81525d11876eefdcc45442249e61e07284592f1606491de5c0324d3af4be035d7ede75b957e879e9770cdde2e1bbc1ef75d45fe555f1ff6ac296a2f648eeee59c7c08260226ea333c285bcf37a9bbfa57ba2ab8083c4be6fc2ebe279537d22da96a07392908cf22b233337a74fe5c603b51712b43c3ee55010ee3d44dd9ba82bba3145ec358f863e04bbfa53799a7a9216718fd5859da2f0deb77b8e315ad6868fdec9400f45a48e6dc8ddbaeb3

The HMACs I get are the following (in ascending order, matching the test-vector order):

• d87aac40f9c78c4109cde53e7f72f2308386d33951763d994c14032282d60b70
• 8be877a6b095a3da734075212e7ddeeda590a8abb2509c6509be6a1b49d3ada5
• dd159c7047bc5801803de47c37f9bb08fe6155680bc942ea8c7849ab1ff562fa
• ca4b47e5170a5ffc11c33cae982df57af8698dc4a51702e55a9a72f701596114
• 83ac47b6fe39a31b11b3e403010832e6e49e3db2dc5825bb29f1762a0cbf4266

I'm pretty confident they'll match up with your tests, since I can reproduce the v0 payload test-vector without issues.

Full trace of intermediate steps can he found here

Ping @t-bast @Roasbeef and @rustyrussell 😉

[t-bast]

@cdecker I get the same value for Packet after applying pad stream but our values differ for Packet after adding payload for hop 4.

I think that for some reason you're using only the first 20 bytes of the hmac (instead of 32 bytes).
Your payload for hop 4 before encryption starts with:

000404040404040404000000000000000400000004000000000000000000000000 (actual test payload) 0000000000000000000000000000000000000000 (trimmed hmac???) 77b5a170... (pad stream).

Mine is:

000404040404040404000000000000000400000004000000000000000000000000 (actual test payload) 0000000000000000000000000000000000000000000000000000000000000000 (empty hmac) 77b5a170... (pad stream).

[cdecker]

Doh, I had removed the legacy encoding (padding of the payload with 12 0x00 bytes). Now the test vector works fine. It was just the test-vector tool that was borked...

Nevermind about the hmacs in the multi-frame test-vector, they're missing altogether :-)

My test-vector tool was broken...

[cdecker]

Reproduced in Eclair, lnd, and c-lightning (ElementsProject/lightning#3427). We decided to merge this during the 2020/01/20 spec meeting, however I fixed a JSON typo and the length of the padding that is generated. So this requires sign-off from @t-bast and @Roasbeef. Merging once I have those ACKs here :-)

ACK 43f18ef

[t-bast]

ACK 43f18ef, thanks!