Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BOLT-04: modify Sphinx packet construction to use starting random bytes #697

Open
wants to merge 3 commits into
base: master
from

Conversation

@Roasbeef
Copy link
Member

Roasbeef commented Nov 6, 2019

In this commit, we modify the existing instructions to create the Sphinx
packet to no longer start out with a zero initialize set of 1366 bytes.
Instead, we now instruct the sender to use random bytes derived from a
CSPRG. This fixes a recently discovered privacy leak that allows an
adversarial exit hop to ascertain a lower bound on the true path length.

Note that this doesn't affect packet processing, so this is a backwards
compatible change. Only clients need to update in order to avoid this
privacy leak.

After this change is applied, the test vectors as is don't match the
spec, as they're created using the original all zero starting bytes. We
can either update these with our specified set of random bytes, or leave
them as is, as they're fully deterministic as is.

An alternative path would be to generate more random bytes from the
session key as we do elsewhere (the chacha based CSPRNG).

In this commit, we modify the existing instructions to create the Sphinx
packet to no longer start out with a zero initialize set of 1366 bytes.
Instead, we now instruct the sender to use _random_ bytes derived from a
CSPRG. This fixes a recently discovered privacy leak that allows an
adversarial exit hop to ascertain a lower bound on the true path length.

Note that this doesn't affect packet processing, so this is a backwards
compatible change. Only clients need to update in order to avoid this
privacy leak.

After this change is applied, the test vectors as is don't match the
spec, as they're created using the original all zero starting bytes. We
can either update these with our specified set of random bytes, or leave
them as is, as they're fully deterministic as is.

An alternative path would be to generate more random bytes from the
shared secret as we do elsewhere (the chacha based CSPRNG).
Copy link
Collaborator

rustyrussell left a comment

Ack e116441

rustyrussell added a commit to rustyrussell/lightning that referenced this pull request Nov 6, 2019
rustyrussell added a commit to rustyrussell/lightning that referenced this pull request Nov 6, 2019
04-onion-routing.md Outdated Show resolved Hide resolved
@@ -402,7 +402,7 @@ The construction returns a single 1366-byte packet along with the first receivin
The packet construction is performed in the reverse order of the route, i.e.
the last hop's operations are applied first.

The packet is initialized with 1366 `0x00`-bytes.
The packet is initialized with 1366 _random_ bytes derived from a CSPRNG.

This comment has been minimized.

Copy link
@cfromknecht

cfromknecht Nov 6, 2019

Collaborator

this doesn't match the code below. the mix header is only 1300 bytes, of which we initialize 1300-firstHopSize bytes via the CSPRNG. since we copy the hop payloads directly into the mix header, we can specify that all 1300 should be random, and leave it as a minor optimization for implementations to not randomize the firstHopSize bytes.

the overall size is 1366 bytes including the session key, next mac, and version, but this is suggesting that those should be initialized via the CSPRNG as well.

This comment has been minimized.

Copy link
@rustyrussell

rustyrussell Nov 13, 2019

Collaborator

Yes, in our code we generated the 1300 byte stream using the session key and the 3-byte string "pad" (I know, not a greek letter, but meh).

This comment has been minimized.

Copy link
@Roasbeef

Roasbeef Nov 22, 2019

Author Member

Updated to just fill the set of bytes directly.

04-onion-routing.md Outdated Show resolved Hide resolved
@cfromknecht

This comment has been minimized.

Copy link
Collaborator

cfromknecht commented Nov 6, 2019

Misspelled words in 04-onion-routing.md: CSPRNG
cdecker added a commit to rustyrussell/lightning that referenced this pull request Nov 7, 2019
cdecker added a commit to rustyrussell/lightning that referenced this pull request Nov 7, 2019
rustyrussell added a commit to rustyrussell/lightning that referenced this pull request Nov 8, 2019
lightningnetwork/lightning-rfc#697
https://lists.linuxfoundation.org/pipermail/lightning-dev/2019-November/002288.html

We generate it from an hmac using the session secret.  It's not
clear that this will be useful for reproducing test vectors though,
since we don't generate the first 66 bytes, which is what the
spec says to do.

Reported-by: @Roasbeef
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
cdecker added a commit to ElementsProject/lightning that referenced this pull request Nov 8, 2019
lightningnetwork/lightning-rfc#697
https://lists.linuxfoundation.org/pipermail/lightning-dev/2019-November/002288.html

We generate it from an hmac using the session secret.  It's not
clear that this will be useful for reproducing test vectors though,
since we don't generate the first 66 bytes, which is what the
spec says to do.

Reported-by: @Roasbeef
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
@cdecker

This comment has been minimized.

Copy link
Collaborator

cdecker commented Nov 8, 2019

fwiw we have decided to generate the initial content of the routing packet using ChaCha20 keyed with a key derived from the session_key using the pad type:

https://github.com/rustyrussell/lightning/blob/d8c386a141beccc67fc9d30de70cf027331de089/common/sphinx.c#L534-L542

This would allow us to maintain reproducibility, at the cost of slightly more computation to generate the stream (then again CSPRGs are likely also not free).

One notable difference to this PR is that we just fill the entire routing packet instead of offsetting from the end of the last hop's payload.

@Roasbeef

This comment has been minimized.

Copy link
Member Author

Roasbeef commented Nov 22, 2019

So i've updated our implementation on the lnd end to allow the caller to specify their own filler: lightningnetwork/lightning-onion#40 (comment)

In the end, all that matters is that the callers fill out the packet with random bytes, the precise method can actually be left up to the sender. The receiver won't be able to distinguish how the bytes were generated, just that they're indistinguishable from a random set of bytes.

@Roasbeef

This comment has been minimized.

Copy link
Member Author

Roasbeef commented Nov 22, 2019

One notable difference to this PR is that we just fill the entire routing packet instead of offsetting from the end of the last hop's payload.

The PR just fills the entire set of bytes now. Skipping the first few bytes is really just an optimization.

See my comment above w.r.t how we can still maintain reproducibility yet let the caller choose between the two...

If we go with this route, we'd need to add a comment to the test vectors. Alternatively, we can update the not-really-psuedo-code to do something similar to this: https://github.com/lightningnetwork/lightning-onion/blob/088b62e7296023531f54595092a03672278acadc/sphinx.go#L189

@Roasbeef

This comment has been minimized.

Copy link
Member Author

Roasbeef commented Nov 22, 2019

On the other hand, it's nice for there just to be "one correct way" in the end. This would mean less guess work for implementors and more "plug and chug" (just implement the spec and you'll be fine) specification. Going the route of re-using the chacha stream (extracting more bytes under a different key) would give us that IMO.

TheBlueMatt added a commit to TheBlueMatt/rust-lightning that referenced this pull request Nov 25, 2019
This avoids at least the trivial hop count discovery attack, though
other obvious ones remain and are slightly harder to avoid.

See lightningnetwork/lightning-rfc#697
TheBlueMatt added a commit to TheBlueMatt/rust-lightning that referenced this pull request Nov 25, 2019
This avoids at least the trivial hop count discovery attack, though
other obvious ones remain and are slightly harder to avoid.

See lightningnetwork/lightning-rfc#697
TheBlueMatt added a commit to TheBlueMatt/rust-lightning that referenced this pull request Nov 28, 2019
This avoids at least the trivial hop count discovery attack, though
other obvious ones remain and are slightly harder to avoid.

See lightningnetwork/lightning-rfc#697
TheBlueMatt added a commit to TheBlueMatt/rust-lightning that referenced this pull request Dec 2, 2019
This avoids at least the trivial hop count discovery attack, though
other obvious ones remain and are slightly harder to avoid.

See lightningnetwork/lightning-rfc#697
@t-bast

This comment has been minimized.

Copy link
Collaborator

t-bast commented Dec 10, 2019

I think it would be useful to update the official test vectors for that change, especially if we're all leaning towards spec-ing one way to do it (I like @cdecker's proposal to reuse ChaCha20).

Applying this change here is what I'm getting as new bolt 4 test vectors. Let me know if you find the same results.

onion-test-v0.json

{
  "comment": "This is a simple testcase in which we only use v0 payloads, and all hops have single frame payloads",
  "generate": {
    "session_key": "4141414141414141414141414141414141414141414141414141414141414141",
    "associated_data": "4242424242424242424242424242424242424242424242424242424242424242",
    "hops": [
      {
        "realm": 0,
        "pubkey": "02eec7245d6b7d2ccb30380bfbe2a3648cd7a942653f5aa340edcea1f283686619",
        "hop_shared_secret": "53eb63ea8a3fec3b3cd433b85cd62a4b145e1dda09391b348c4e1cd36a03ea66",
        "payload": "0000000000000000000000000000000000000000000000000000000000000000",
        "rhokey": "ce496ec94def95aadd4bec15cdb41a740c9f2b62347c4917325fcc6fb0453986",
        "mukey": "b57061dc6d0a2b9f261ac410c8b26d64ac5506cbba30267a649c28c179400eba",
        "hmac": "ac73e62f2cc382a1356e06f2762f9cdbcec7c7771037e824b0c1071f4f93d614"
      },
      {
        "realm": 0,
        "pubkey": "0324653eac434488002cc06bbfb7f10fe18991e35f9fe4302dbea6d2353dc0ab1c",
        "hop_shared_secret": "a6519e98832a0b179f62123b3567c106db99ee37bef036e783263602f3488fae",
        "payload": "0101010101010101000000000000000100000001000000000000000000000000",
        "rhokey": "450ffcabc6449094918ebe13d4f03e433d20a3d28a768203337bc40b6e4b2c59",
        "mukey": "05ed2b4a3fb023c2ff5dd6ed4b9b6ea7383f5cfe9d59c11d121ec2c81ca2eea9",
        "hmac": "9350c058fe33b70d9cea125d2d68ba02d81af4609c916d51a575c8db2d5f1de5"
      },
      {
        "realm": 0,
        "pubkey": "027f31ebc5462c1fdce1b737ecff52d37d75dea43ce11c74d25aa297165faa2007",
        "hop_shared_secret": "3a6b412548762f0dbccce5c7ae7bb8147d1caf9b5471c34120b30bc9c04891cc",
        "payload": "0202020202020202000000000000000200000002000000000000000000000000",
        "rhokey": "11bf5c4f960239cb37833936aa3d02cea82c0f39fd35f566109c41f9eac8deea",
        "mukey": "caafe2820fa00eb2eeb78695ae452eba38f5a53ed6d53518c5c6edf76f3f5b78",
        "hmac": "ef3860bc4d742ca9a6c2818d03c83f034d5c22776232a87ca049ec79b63627b9"
      },
      {
        "realm": 0,
        "pubkey": "032c0b7cf95324a07d05398b240174dc0c2be444d96b159aa6c7f7b1e668680991",
        "hop_shared_secret": "21e13c2d7cfe7e18836df50872466117a295783ab8aab0e7ecc8c725503ad02d",
        "payload": "0303030303030303000000000000000300000003000000000000000000000000",
        "rhokey": "cbe784ab745c13ff5cffc2fbe3e84424aa0fd669b8ead4ee562901a4a4e89e9e",
        "mukey": "5052aa1b3d9f0655a0932e50d42f0c9ba0705142c25d225515c45f47c0036ee9",
        "hmac": "22b8aa7917b4e2c87fc5678bab10b450454fbb5e70708c3d49f0670a616085ea"
      },
      {
        "realm": 0,
        "pubkey": "02edabbd16b41c8371b92ef2f04c1185b4f03b6dcd52ba9b78d9d7c89c8f221145",
        "hop_shared_secret": "b5756b9b542727dbafc6765a49488b023a725d631af688fc031217e90770c328",
        "payload": "0404040404040404000000000000000400000004000000000000000000000000",
        "rhokey": "034e18b8cc718e8af6339106e706c52d8df89e2b1f7e9142d996acf88df8799b",
        "mukey": "8e45e5c61c2b24cb6382444db6698727afb063adecd72aada233d4bf273d975a",
        "hmac": "f4bc4473ec0a55d5f3ededdba4a8e303f9128391a0f1f54d8adacf2221c04e1b"
      }
    ]
  },
  "onion": "0002eec7245d6b7d2ccb30380bfbe2a3648cd7a942653f5aa340edcea1f283686619e5f14350c2a76fc232b5e46d421e9615471ab9e0bc887beff8c95fdb878f7b3a71d215fe077311013884fabc09318c23a2435069ff1c40e90d2c150084eb8eb59ed10f7d61ab590531cf08000178a333a347f8b4072e216400406bdf3bf03865979313da96e66517a37b5c485744275a73b8eab6f43e3d56a2a59764eb12617efbe0ce721caad69320c3a469a202f3e468c67eaf7a7cda226d0fd32f7b48084dca885d3a3451e1a7368bb5edac5c665a61f6796b288bc1f12cd3b9fbaf915bc732ee749a1ec866abe044a9ad635778ba61fc0776dc832b39451bd5d35072d2269cf9b04040caeaeed42e3eb4adf6146661e3708accfeeb02d174de7effeaa8040417c8b49ed4c7f0218ff8c6c7dd7221d189c65b3b9aaa71a01484b122846c7c7b57e02e679ea8469b70e14fe4f70fee4d87b910cf144be6fe48eef24da475c0b0bcc6565ac90cc68a2cfd94a8a93dc787d6a074e539903b1d36e8d5d45655aa59208a20f3f600ef6b05d963ab796cdf33748c944795403626cdff6c0f114586f1754fb6d64800beeda3e18c04f109b7a91c9049ba2967316f088a8b45c57ea708768ed1cc99f06886ee3089d6c8e9ea3ce884b8dfffed4d4cdc425bf4a8740d03cfcdc95861300a03f5913bf284daa96b9c81dac1ec47cd72bd1049256f41d3710bb47d6151bc6d2dca3851363b4c499f53cbfd445bbd11e669f6b868b35b2e2cc950b1dcde4517b548066ac7543d9f9ac78393b61d1df632c1e7e048427d36227a51f795049c9b89cfdf25b9982e8d79f0da84caf54f7904e8731af45bcbf558eeefd885e2907ce00a3f2124421a704c1907a966e26215b26d2fa7558406a767c0e7218e1f9ac26b012971eba585be112b4e84abbab93a6d76f626b744c1f86df334dcc5d5277b2ce89e0c9869be3fd952ccfac0d3ab40eecd8dd5356155a8b0b186e650b9bbac561a9b5f309b18d31d3bfdd6dac9169b7e38f82b300e3598eb142fb5dd99612d3cd8e1f720551142f5460736e6da0d1158b238a626afa86076c64851694651d2bf6808de85104ef995ddc159d2efbe67c50a2462c10864400bf01ef387d63dd06b21ddebc4ddfb12b9ff2ad2816bcabf737ed8e89f8749a5179ab26faf5796c80b2364af41dc0384507b160c1e7bd93cc6062d82fb87572b9822f1400222c9517a5f4a89fe8fbe39b54efcca1d40bc8627d6b1f43e621c3297ea5ab9076824d90cfb86e5025aad6160cbff1d62c1e46d86a43dce76e09bc1833def9deeb42b02e96b2e423c3acf305e02d4c2c942fddcba269ffcdaef2b4dd6514b427e3236cee79bc28a26adf83af2137a9c7c6453a8571a65d86146c6a93fea20fe91d64f4e3eda8c1980d68e4d9d22a8dd20d5b5d36abda9a7a32bc0711802bd68a19a5837d6e45cf21ae674e291f9ab6846c9c0440017b47630f67cba07a90bfc98b4acd9bef2f8b379357f0e91c3dd04ee819214cfb4f50e35e9a6428d072c09240bb550b225ad312698a827da5fad2f1ae11a474d02872a3554822ff12045b230276f3ad5bd57e5b1e71705076ce6c234b32543cd86a8bec4a1da37b02ffeceb4ebc98311c2c1938458ce8e390e8b702032efc2d30326ffd9a16bbee7cd198b442e22a5210152c6b08c13972b50f5c4d3fd8e393d752463b9192f66cb4be243737ae167c044e4517b4d5efd6af68ab040477acd144fbd42d60725ed2a3e0d83af620e800c6c55e628bef178d4490bc8fa0e0be00f309183c75a402fa501ec8253530199d74bd360932281c3cc5a5d33906e695c927f49b169604959aec1978a369784bdeaecfb79281996b1fc377841ac73e62f2cc382a1356e06f2762f9cdbcec7c7771037e824b0c1071f4f93d614",
  "decode": [
    "4141414141414141414141414141414141414141414141414141414141414141",
    "4242424242424242424242424242424242424242424242424242424242424242",
    "4343434343434343434343434343434343434343434343434343434343434343",
    "4444444444444444444444444444444444444444444444444444444444444444",
    "4545454545454545454545454545454545454545454545454545454545454545"
  ]
}

onion-test-multi-frame.json

{
  "comment": "A testcase for a variable length hop_payload. The third payload is 256 bytes long.",
  "generate": {
    "session_key": "4141414141414141414141414141414141414141414141414141414141414141",
    "associated_data": "4242424242424242424242424242424242424242424242424242424242424242",
    "hops": [
	{
	    "type": "legacy",
	    "pubkey": "02eec7245d6b7d2ccb30380bfbe2a3648cd7a942653f5aa340edcea1f283686619",
	    "payload": "0000000000000000000000000000000000000000"
	},
	{
	    "type": "tlv",
	    "pubkey": "0324653eac434488002cc06bbfb7f10fe18991e35f9fe4302dbea6d2353dc0ab1c",
	    "payload": "0101010101010101000000000000000100000001"
	},
	{
	    "type": "raw",
	    "pubkey": "027f31ebc5462c1fdce1b737ecff52d37d75dea43ce11c74d25aa297165faa2007",
	    "payload": "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff"
	},
	{
	    "type": "tlv",
	    "pubkey": "032c0b7cf95324a07d05398b240174dc0c2be444d96b159aa6c7f7b1e668680991",
	    "payload": "0303030303030303000000000000000300000003"
	},
	{
	    "type": "legacy",
	    "pubkey": "02edabbd16b41c8371b92ef2f04c1185b4f03b6dcd52ba9b78d9d7c89c8f221145",
	    "payload": "0404040404040404000000000000000400000004"
	}
    ]
  },
  "onion": "0002eec7245d6b7d2ccb30380bfbe2a3648cd7a942653f5aa340edcea1f283686619e5f14350c2a76fc232b5e46d421e9615471ab9e0bc887beff8c95fdb878f7b3a71068850554818be051c5fb6e47df0f24ab88623abf089afc95db4f653cfdc4e3fc50f7d61ab590531cf08000178a333a347f8b4072ecac76d4ab27f190828a68222a6bb6c6f23fc671295d5d4fe37b9f3dacaa629d6404bae9a8e8d7178977d7094a1ae549f89338c0777551f874159eb42d3a59fb9285ad4e24883f27de23942ec966611e99bee1cee503455be9e8e642cef6cef7b9864130f692283f8a973d47a8f1c1726b6e59969385975c766e35737c8d76388b64f748ee7943ffb0e2ee45c57a1abc40762ae598723d21bd184e2b338f68ebff47219357bd19cd7e01e2337b806ef4d717888e129e59cd3dc31e6201ccb2fd6d7499836f37a993262468bcb3a4dcd03a22818aca49c6b7b9b8e9e870045631d8e039b066ff86e0d1b7291f71cefa7264c70404a8e538b566c17ccc5feab231401e6c08a01bd5edfc1aa8e3e533b96e82d1f91118d508924b923531929aea889fcdf0568b9af020955bf28f2296f547bec0773d4f76e19351ceda5ad381e6a79b57a5a658149e7e9ced4edde5d38c9b8f92e16f6b4ab13d704721bb12be5941e1b6620be32870100cb1000cf05b7562be8dbfb7fdb4da45b2f3bca8e6fe47eb45fbdd3be21a8a8d200797eae3c9a0497132f92410d804977408494dff49dd3d8bce248e0b74fd9e6f0f7102c25ddfa02bd9ad9f746abbfa337cea1082d8ae639b52346b18c05b8248d9b30df256d432ab365b732a58b04d7d0d6f6dd20b810c52b2457544c629ca0911c35ff1292523f980f13ebd87d1627e9759abf8b17e54955835fa70c3d9217d21dd2a7e33bea072be9170265c9465d5b4c54a1df3c15513332dfb2123c27498af3bfa116a140d28d43661a3088c6f2ae995bd921801d43ee31a353f2e164704b1593136e84815e2d304515ddf45ef3f70b2169718c811fbe697a1c0d97032e83dec858b6672be484244127e23e2ca8fa055e8e34165c3d02219fb588c7a221722e45e2b54f6b9fa4a6013906198cdf72c8d5222155cfc64f1f322f7785d84727fdd2f9b66627cbe8c7ecf5d944ffcb9266ac815ebc4b4301ed6bc1a765bfeade304cac32e5841340ded04f1f9ea4e2fa7ceea533f8b5253a940f842176c672443af37f03a03f7219df1c4062c51e6e79cf859d07eece8cb7583643e698cf76c04cc6f4c01ca9378046e1763818c616c54f640b26a424fbc0b37f6765f57ce1ade6c8c1eef6ab12df9fb6df6125d2d8bb665a4033d1386525eebc5af407155a5bbc9f8719b836f58044a8f5d1256ca627a592c0419af0bbe4886b9f97667c1bb1712f003f9ac8c53e5ceeaa2182260bab8f5ea50951529c4bda2e58ac649bff4447b585e07f8ca3e35379e969dc71f015557fc588eef1282f7da45f57d94f9316898c904217c3a39ceb2e20c4878c6c196ad11aee10b7ed0b047220930aa1d6463d60058dbf1c36f12eb767d5dc09b9c9d1df628766a4de995fa7e85040533087133ba8892b2cd207da65b99e5d95575e4d7d5ba2666de8f9d5582c1f8799e43e25aa1ad7b945bfb97a953e09a9341c49d1346901af9eb30486bd270d750493dfb69b467a24757e5b3a7198f6794549725bccdc69303f6ccaa48f825e5f83d42faba8f32895148c18db0ec3f6411b969674a507d39a4cb723bba876b766bc2219b164a0f32fd9a8c198695343cbc7c48dda1d36ce69efd07a3d7d46275583e50b9fd1a6d95ce3454da8fe754ab58ee4b5ed48a020f514287e900076d389d6266c627f1e0e22a4b511dfa5c4891eac4d8fe7e9f0332cc26af9a5f467b4acea6da597f27fbb58511ffb8a200d68f09aa9e83b401dc012",
  "decode": [
    "4141414141414141414141414141414141414141414141414141414141414141",
    "4242424242424242424242424242424242424242424242424242424242424242",
    "4343434343434343434343434343434343434343434343434343434343434343",
    "4444444444444444444444444444444444444444444444444444444444444444",
    "4545454545454545454545454545454545454545454545454545454545454545"
  ]
}
t-bast added a commit to ACINQ/eclair that referenced this pull request Dec 10, 2019
@t-bast t-bast mentioned this pull request Dec 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.