Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BOLT7: extend channel range queries with optional fields #557

Merged
merged 11 commits into from Sep 16, 2019

Conversation

@sstone
Copy link
Collaborator

commented Jan 23, 2019

This is a new pull requests that supersedes #519 .

It addresses issues with the original proposal, mainly that it defined a new set of messages, adding complexity to a simple gossip protocol that we knew was limited in the first place.

This proposal does not add new messages, or feature bits, and is fully compatible with existing implementations. Instead of defining new messages it extends existing ones with additional data, that will be ignored by nodes which do not implement extended queries (see BOLT #1).

Nodes that support extended queries will append an additional extended query flag to their query_channel_range queries. If the receiver supports extended queries and understands this flag, it will append the requested additional data to its reply_channel_range message.

There is currently only one type of additional data: one timestamp and one checksum per channel_update.
The checksum is a simple Adler32 checksum computed over the channel_update with timestamp and signature omitted.
Together they can be used to avoid querying channel_updates that are older than the ones you already have, or that are newer but don't include new information.

Nodes can then append additional data to their query_short_channel_ids messages, which consists in one flag per short channel id and specifies what they would like to receive (channel_announcement, or/and one channel_update or both`).

@sstone

This comment has been minimized.

Copy link
Collaborator Author

commented Jan 24, 2019

For completeness, there is even a simpler solution: leave query_channel_range as is (no extra flag to signify that you want extended data), and if the receiver supports extended queries they always include extended data in their reply_channel_range.

I chose instead to explicitly signal support for extended queries in query_channel_range, which is more bandwidth efficient (replies won't include extended data unless you ask for it).

07-routing-gossip.md Outdated Show resolved Hide resolved
@pm47

This comment has been minimized.

Copy link
Collaborator

commented Mar 11, 2019

@rustyrussell does this correspond to feature option_fec_gossip in #571?
edit: nevermind

@pm47

This comment has been minimized.

Copy link
Collaborator

commented Mar 11, 2019

For those interested, this feature is now implemented on endurance: 03933884aaf1d6b108397e5efe5c86bcf2d8ca8d2f700eda99db9214fc2712b134@34.250.234.192:9735

@sstone

This comment has been minimized.

Copy link
Collaborator Author

commented Mar 12, 2019

There is no need for a specific feature bit for this: additional "extended" data appended to query messages will simply be ignored by nodes which do not implement extended queries (see BOLT 1). It's one of the reasons why this PR is much better than my first proposal: extended queries are completely optional and compatible with current implementations and deployed nodes.

option_fec_gossip refers to new messages to efficiently sync routing tables based on set reconciliation techniques (IBLT, minisketch, ...) that have not been defined yet.

@pm47

This comment has been minimized.

Copy link
Collaborator

commented Mar 19, 2019

Here are a few test vectors, hope that helps:

{
  "msg" : {
    "type" : "QueryChannelRange",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "firstBlockNum" : 100000,
    "numberOfBlocks" : 1500
  },
  "hex" : "01070f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206000186a0000005dc"
}
{
  "msg" : {
    "type" : "QueryChannelRange",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "firstBlockNum" : 35000,
    "numberOfBlocks" : 100,
    "extendedQueryFlags_opt" : "TIMESTAMPS_AND_CHECKSUMS"
  },
  "hex" : "01070f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206000088b80000006401"
}
{
  "msg" : {
    "type" : "ReplyChannelRange",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "firstBlockNum" : 756230,
    "numberOfBlocks" : 1500,
    "complete" : 1,
    "shortChannelIds" : {
      "encoding" : "UNCOMPRESSED",
      "array" : [ "0x0x142", "0x0x15465", "0x69x42692" ]
    },
    "optionExtendedQueryFlags_opt" : "TIMESTAMPS_AND_CHECKSUMS"
  },
  "hex" : "01080f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206000b8a06000005dc01001900000000000000008e0000000000003c69000000000045a6c401"
}
{
  "msg" : {
    "type" : "ReplyChannelRange",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "firstBlockNum" : 1600,
    "numberOfBlocks" : 110,
    "complete" : 1,
    "shortChannelIds" : {
      "encoding" : "COMPRESSED_ZLIB",
      "array" : [ "0x0x142", "0x0x15465", "0x4x3318" ]
    },
    "optionExtendedQueryFlags_opt" : "TIMESTAMPS_AND_CHECKSUMS"
  },
  "hex" : "01080f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206000006400000006e01001601789c636000833e08659309a65878be010010a9023a01"
}
{
  "msg" : {
    "type" : "ReplyChannelRange",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "firstBlockNum" : 122334,
    "numberOfBlocks" : 1500,
    "complete" : 1,
    "shortChannelIds" : {
      "encoding" : "UNCOMPRESSED",
      "array" : [ "0x0x12355", "0x7x30934", "0x70x57793" ]
    },
    "optionExtendedQueryFlags_opt" : "TIMESTAMPS_AND_CHECKSUMS",
    "extendedInfo_opt" : {
      "array" : [ {
        "timestamp1" : 164545,
        "checksum1" : 1111,
        "timestamp2" : 948165,
        "checksum2" : 2222
      }, {
        "timestamp1" : 489645,
        "checksum1" : 3333,
        "timestamp2" : 4786864,
        "checksum2" : 4444
      }, {
        "timestamp1" : 46456,
        "checksum1" : 5555,
        "timestamp2" : 9788415,
        "checksum2" : 6666
      } ]
    }
  },
  "hex" : "01080f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e22060001ddde000005dc01001900000000000000304300000000000778d6000000000046e1c1010030000282c100000457000e77c5000008ae000778ad00000d0500490ab00000115c0000b578000015b300955bff00001a0a"
}
{
  "msg" : {
    "type" : "ReplyChannelRange",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "firstBlockNum" : 500,
    "numberOfBlocks" : 100,
    "complete" : 1,
    "shortChannelIds" : {
      "encoding" : "COMPRESSED_ZLIB",
      "array" : [ "0x18x54897", "0x74x47820", "0x69x42692" ]
    },
    "optionExtendedQueryFlags_opt" : "TIMESTAMPS_AND_CHECKSUMS",
    "extendedInfo_opt" : {
      "array" : [ {
        "timestamp1" : 164545,
        "checksum1" : 1111,
        "timestamp2" : 948165,
        "checksum2" : 2222
      }, {
        "timestamp1" : 489645,
        "checksum1" : 3333,
        "timestamp2" : 4786864,
        "checksum2" : 4444
      }, {
        "timestamp1" : 46456,
        "checksum1" : 5555,
        "timestamp2" : 9788415,
        "checksum2" : 6666
      } ]
    }
  },
  "hex" : "01080f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206000001f40000006401001a01789c63600002a16b85208ac16bd71930edbaec08002c7804d9010030000282c100000457000e77c5000008ae000778ad00000d0500490ab00000115c0000b578000015b300955bff00001a0a"
}
{
  "msg" : {
    "type" : "QueryShortChannelIds",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "shortChannelIds" : {
      "encoding" : "UNCOMPRESSED",
      "array" : [ "0x0x142", "0x0x15465", "0x69x42692" ]
    }
  },
  "hex" : "01050f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206001900000000000000008e0000000000003c69000000000045a6c4"
}
{
  "msg" : {
    "type" : "QueryShortChannelIds",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "shortChannelIds" : {
      "encoding" : "COMPRESSED_ZLIB",
      "array" : [ "0x0x4564", "0x2x47550", "0x69x42692" ]
    }
  },
  "hex" : "01050f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206001801789c63600001c12b608a69e73e30edbaec0800203b040e"
}
{
  "msg" : {
    "type" : "QueryShortChannelIds",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "shortChannelIds" : {
      "encoding" : "UNCOMPRESSED",
      "array" : [ "0x0x12232", "0x0x15556", "0x69x42692" ]
    },
    "queryFlags_opt" : {
      "encoding" : "COMPRESSED_ZLIB",
      "array" : [ 1, 2, 4 ]
    }
  },
  "hex" : "01050f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e22060019000000000000002fc80000000000003cc4000000000045a6c4000c01789c6364620100000e0008"
}
{
  "msg" : {
    "type" : "QueryShortChannelIds",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "shortChannelIds" : {
      "encoding" : "COMPRESSED_ZLIB",
      "array" : [ "0x0x14200", "0x0x46645", "0x69x42692" ]
    },
    "queryFlags_opt" : {
      "encoding" : "COMPRESSED_ZLIB",
      "array" : [ 1, 2, 4 ]
    }
  },
  "hex" : "01050f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206001801789c63600001f30a30c5b0cd144cb92e3b020017c6034a000c01789c6364620100000e0008"
}
@pm47 pm47 referenced this pull request Mar 19, 2019
07-routing-gossip.md Outdated Show resolved Hide resolved
@pm47

This comment has been minimized.

Copy link
Collaborator

commented Mar 20, 2019

@rustyrussell

This comment has been minimized.

Copy link
Collaborator

commented Apr 1, 2019

OK, so I've implemented this. I've kept the existing logic to send node_announcements for any channel_announcement we send (uniqified). Am testing the vectors now...

@rustyrussell

This comment has been minimized.

Copy link
Collaborator

commented Apr 1, 2019

OK, I've got a fixup which I applied to your INV gossip PR, which should have gone here instead. Will push. Also, rebased onto v1.0, otherwise I get unrelated regressions...

@rustyrussell rustyrussell force-pushed the sstone:bolt7-extended-channel-queries branch from 1e9483f to 8eeb1c5 Apr 1, 2019

@rustyrussell

This comment has been minimized.

Copy link
Collaborator

commented Apr 1, 2019

... and another proposed cleanup (no semantic changes, just integrating the requirements better, I think).

@sstone

This comment has been minimized.

Copy link
Collaborator Author

commented Apr 4, 2019

This PR has been discussed in our dev meetings a few times and there are still open points, I'll try and summarise them here. They were raised mostly by @Roasbeef and @cfromknecht , If I've missed anything please add a question/comment which I'll try and address here.

Why do we need this ? We could use heuristics, be more optimistic on the sender's side and work with a "good enough" routing table without trying to get all updates asap. Better heuristics could also be use to broadcast enable/disable updates and reduce "flapping" gossip.

It's very true. But this PR really is a fix for current channel queries which are very good for learning about channels you don't have, but almost unusable for updating the ones you already have. Basically they're just a way of reconciling your list of channel ids with your peer's. Then what ? You've learned nothing about what has changed. If you do nothing, you will get updates as your payments fails with an "update" error, but it's bad UX, especially for mobile nodes when users turn them on, make a single or just a few payments, then turn them off again. And gossip filters (which say “send me everything that is more recent than timestamp X”) don’t really help when you’re offline most of the time since you don’t know what you’re missing to begin with.

I think adding timestamps, and a flag to query specific items and not everything, should not be controversial, especially since now they're completely optional. Do we all agree on that ?

Why do we also need a checksum ? We should use better heuristics to fix the flapping channels issue.

Because then you have the option of ignoring updates that don't change routing policies (or simply flagging them as low priority). Again, better heuristics for broadcasting these updates, as well as stricter peer selection/banning heuristics, would help a lot, but having the option to know that an update does not really change anything * before * you've downloaded it is very useful (and this proposal very cheap from a bandwitdh/cpu point of view).

Should we use TLV for the new optional fields ?

I'm now having second thoughts about this one. Current proposal is simple and consistent, I think that using TLV would just slow things down and would not brings us anything ? But I'll give it a try.

07-routing-gossip.md Outdated Show resolved Hide resolved

@sstone sstone force-pushed the sstone:bolt7-extended-channel-queries branch from 9a747a3 to f322ffc Apr 26, 2019

@sstone

This comment has been minimized.

Copy link
Collaborator Author

commented Apr 26, 2019

I've updated this PR:

  • optional fields now use TLV format
  • timestamps and checksums are now independent

These changes heavily borrow on Rusty's tlv branch (mistakes are mine of course :))

Here's an updated test vector:

{
  "msg" : {
    "type" : "QueryChannelRange",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "firstBlockNum" : 100000,
    "numberOfBlocks" : 1500,
    "extensions" : [ ]
  },
  "hex" : "01070f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206000186a0000005dc"
}
{
  "msg" : {
    "type" : "QueryChannelRange",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "firstBlockNum" : 35000,
    "numberOfBlocks" : 100,
    "extensions" : [ "WANT_TIMESTAMPS | WANT_CHECKSUMS" ]
  },
  "hex" : "01070f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206000088b800000064010103"
}
{
  "msg" : {
    "type" : "ReplyChannelRange",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "firstBlockNum" : 756230,
    "numberOfBlocks" : 1500,
    "complete" : 1,
    "shortChannelIds" : {
      "encoding" : "UNCOMPRESSED",
      "array" : [ "0x0x142", "0x0x15465", "0x69x42692" ]
    }
  },
  "hex" : "01080f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206000b8a06000005dc01001900000000000000008e0000000000003c69000000000045a6c4"
}
{
  "msg" : {
    "type" : "ReplyChannelRange",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "firstBlockNum" : 1600,
    "numberOfBlocks" : 110,
    "complete" : 1,
    "shortChannelIds" : {
      "encoding" : "COMPRESSED_ZLIB",
      "array" : [ "0x0x142", "0x0x15465", "0x4x3318" ]
    }
  },
  "hex" : "01080f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206000006400000006e01001601789c636000833e08659309a65878be010010a9023a"
}
{
  "msg" : {
    "type" : "ReplyChannelRange",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "firstBlockNum" : 122334,
    "numberOfBlocks" : 1500,
    "complete" : 1,
    "shortChannelIds" : {
      "encoding" : "UNCOMPRESSED",
      "array" : [ "0x0x12355", "0x7x30934", "0x70x57793" ]
    },
    "timestamps" : {
      "encoding" : "UNCOMPRESSED",
      "timestamps" : [ {
        "timestamp1" : 164545,
        "timestamp2" : 948165
      }, {
        "timestamp1" : 489645,
        "timestamp2" : 4786864
      }, {
        "timestamp1" : 46456,
        "timestamp2" : 9788415
      } ]
    },
    "checksums" : {
      "checksums" : [ {
        "checksum1" : 1111,
        "checksum2" : 2222
      }, {
        "checksum1" : 3333,
        "checksum2" : 4444
      }, {
        "checksum1" : 5555,
        "checksum2" : 6666
      } ]
    }
  },
  "hex" : "01080f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e22060001ddde000005dc01001900000000000000304300000000000778d6000000000046e1c1011900000282c1000e77c5000778ad00490ab00000b57800955bff031800000457000008ae00000d050000115c000015b300001a0a"
}
{
  "msg" : {
    "type" : "ReplyChannelRange",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "firstBlockNum" : 122334,
    "numberOfBlocks" : 1500,
    "complete" : 1,
    "shortChannelIds" : {
      "encoding" : "COMPRESSED_ZLIB",
      "array" : [ "0x0x12355", "0x7x30934", "0x70x57793" ]
    },
    "timestamps" : {
      "encoding" : "COMPRESSED_ZLIB",
      "timestamps" : [ {
        "timestamp1" : 164545,
        "timestamp2" : 948165
      }, {
        "timestamp1" : 489645,
        "timestamp2" : 4786864
      }, {
        "timestamp1" : 46456,
        "timestamp2" : 9788415
      } ]
    },
    "checksums" : {
      "checksums" : [ {
        "checksum1" : 1111,
        "checksum2" : 2222
      }, {
        "checksum1" : 3333,
        "checksum2" : 4444
      }, {
        "checksum1" : 5555,
        "checksum2" : 6666
      } ]
    }
  },
  "hex" : "01080f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e22060001ddde000005dc01001801789c63600001036730c55e710d4cbb3d3c080017c303b1012201789c63606a3ac8c0577e9481bd622d8327d7060686ad150c53a3ff0300554707db031800000457000008ae00000d050000115c000015b300001a0a"
}
{
  "msg" : {
    "type" : "QueryShortChannelIds",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "shortChannelIds" : {
      "encoding" : "UNCOMPRESSED",
      "array" : [ "0x0x142", "0x0x15465", "0x69x42692" ]
    },
    "extensions" : [ ]
  },
  "hex" : "01050f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206001900000000000000008e0000000000003c69000000000045a6c4"
}
{
  "msg" : {
    "type" : "QueryShortChannelIds",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "shortChannelIds" : {
      "encoding" : "COMPRESSED_ZLIB",
      "array" : [ "0x0x4564", "0x2x47550", "0x69x42692" ]
    },
    "extensions" : [ ]
  },
  "hex" : "01050f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206001801789c63600001c12b608a69e73e30edbaec0800203b040e"
}
{
  "msg" : {
    "type" : "QueryShortChannelIds",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "shortChannelIds" : {
      "encoding" : "UNCOMPRESSED",
      "array" : [ "0x0x12232", "0x0x15556", "0x69x42692" ]
    },
    "extensions" : [ {
      "encoding" : "COMPRESSED_ZLIB",
      "array" : [ 1, 2, 4 ]
    } ]
  },
  "hex" : "01050f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e22060019000000000000002fc80000000000003cc4000000000045a6c4010c01789c6364620100000e0008"
}
{
  "msg" : {
    "type" : "QueryShortChannelIds",
    "chainHash" : "0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206",
    "shortChannelIds" : {
      "encoding" : "COMPRESSED_ZLIB",
      "array" : [ "0x0x14200", "0x0x46645", "0x69x42692" ]
    },
    "extensions" : [ {
      "encoding" : "COMPRESSED_ZLIB",
      "array" : [ 1, 2, 4 ]
    } ]
  },
  "hex" : "01050f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206001801789c63600001f30a30c5b0cd144cb92e3b020017c6034a010c01789c6364620100000e0008"
}
@cfromknecht
Copy link
Collaborator

left a comment

great work @sstone! really like the direction this is taking and addition of distinct bits for timestamps and checksums. as a whole the proposal looks to be in a good place, i've left some minor comments in line. looking forward to implementing this :)

07-routing-gossip.md Outdated Show resolved Hide resolved
07-routing-gossip.md Outdated Show resolved Hide resolved
07-routing-gossip.md Outdated Show resolved Hide resolved
07-routing-gossip.md Show resolved Hide resolved
07-routing-gossip.md Outdated Show resolved Hide resolved
07-routing-gossip.md Outdated Show resolved Hide resolved
07-routing-gossip.md Show resolved Hide resolved
07-routing-gossip.md Outdated Show resolved Hide resolved
07-routing-gossip.md Outdated Show resolved Hide resolved

@sstone sstone added this to Scheduled in Specification Meeting Agenda Jun 10, 2019

@cdecker cdecker moved this from Scheduled to Accepted in Specification Meeting Agenda Jun 10, 2019

@t-bast

This comment has been minimized.

Copy link
Collaborator

commented Jul 5, 2019

ACK 1978561

@sstone

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 5, 2019

Since #607 has been approved (yes !), I've implemented a suggestion by @t-bast to use minimally-encoded varints instead of single bytes for query flags. Given the number of options atm they will all still be encoded on a single byte, and test vectors remain valid. I think we're good to go now and hope this will be merged with the TLV PR.

@rustyrussell

This comment has been minimized.

Copy link
Collaborator

commented Aug 6, 2019

OK, so meeting agreed to remove compression flag and use a straight array for checksums:

http://www.erisian.com.au/meetbot/lightning-dev/2019/lightning-dev.2019-08-05-20.03.html

So I'm going to merge this for c-lightning (only with --enable-experimental-features) for next release: ElementsProject/lightning#2900

I'm not completely opposed to removing the EXPERIMENTAL_FEATURES conditional in the next few days (we tag -rc1 on the 10th of alternate months) if spec is sorted, feature flag is added, and Eclair confirms interoperation in practice. But that's a tight timeline!

sstone and others added 7 commits Jul 17, 2019
BOLT7: query_short_channel_ids: allow requesting specific information…
… (folded)

Nodes can append additional data to their `query_short_channel_ids`
messages, which consists in one flag per short channel id and
specifies what they would like to receive (`node_announcement`,
`channel_announcement`, or/and one `channel_update` or both).
BOLT7: query_channel_range: allow requesting timestamps & checksums (…
…folded)

Nodes that support extended queries will append an additional extended query flag to
their `query_channel_range` queries. If the receiver supports extended queries and
understands this flag, it will append the required additional data to its
`reply_channel_range` message.

There is currently only one type of additional data: one timestamp and one checksum
per `channel_update`.
The checksum is a CRC32 checksum computed over the `channel_update`
with `timestamp` and `signature` omitted.

Along with query_short_channel_ids extension, this can be used to
avoid querying `channel_updates` that are older than the ones you
already have, or that are newer but don't include new information.
BOLT 7: fix up TLV formatting for modern parser.
Formatting changes only.

This make tools/extract-formats.py work (well, it misses some stuff
until the tlv-testcases merge, but then it's OK).

We use `tlvs` (for tlv stream), and we refer to TLV records as "being
included" rather than re-using the TLV name.

We even use subtypes for the pairs of checksums and timestamps.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
BOLT 7: fixup: add missing encoding_type in checksums_tlv.
Since some can be zero (missing updates), it's probably worth
doing the compression thing optionally.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
BOLT 7: fixup: specify exactly which version of CRC32.
This is the one in SSE4, FWIW, and the iSCSI RFC contains test
vectors.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

@sstone sstone force-pushed the sstone:bolt7-extended-channel-queries branch from b726493 to c11c35a Aug 6, 2019

sstone added 3 commits Aug 6, 2019
TLV Formatting changes
We use the more tool-friendly `...*` description for TLV extensions.
Checksums are now serialized as raw arrays, as using zlib compression here would not help.
@sstone

This comment has been minimized.

Copy link
Collaborator Author

commented Aug 6, 2019

@rustyrussell I've changed the formatting and added a JSON test vector. Its format is just a bit different from the one you used but I could get your test to pass with a few changes

@ysangkok

This comment has been minimized.

Copy link
Contributor

commented on bolt07/extended-queries.json in e52e8ec Aug 6, 2019

why is this encoded in a string?

This comment has been minimized.

Copy link
Collaborator Author

replied Aug 7, 2019

This way people can check that they interpret the flag correctly. It's similar to what you find in some Bitcoin Core JSON test vectors.

rustyrussell added a commit to rustyrussell/lightning that referenced this pull request Aug 10, 2019
gossipd: test vectors for lightningnetwork/lightning-rfc#557
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
rustyrussell added a commit to ElementsProject/lightning that referenced this pull request Aug 10, 2019
gossipd: test vectors for lightningnetwork/lightning-rfc#557
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

@t-bast t-bast moved this from Accepted to Scheduled in Specification Meeting Agenda Aug 19, 2019

sstone added a commit to ACINQ/eclair that referenced this pull request Aug 22, 2019
Use extended range queries on regtest and testnet
We will use them on mainnet as soon as lightningnetwork/lightning-rfc#557 has been merged.
pm47 added a commit to ACINQ/eclair that referenced this pull request Aug 22, 2019
Extended Queries: use TLV format for optional data (#1072)
* Extended Queries: use TLV format for optional data

Optional query extensions now use TLV instead of a custom format.
Flags are encoded as varint instead of bytes as originally proposed. With the current proposal they will all fit on a single byte, but will be
much easier to extends this way.

* Move query message TLVs to their own namespace

We add one new class for each TLV type, with specific TLV types, and encapsulate codecs.

* Optional TLVs are represented as a list, not an optional list

TLVs that extend regular LN messages can be represented as a TlvStream and not an Option[TlvStream] since we don't need
to explicitely terminate the stream (either by preprending its length or using a specific terminator) as we do in Onion TLVs.

No TLVs simply means that the TLV stream is empty.

* Update to match  BOLT PR

Checksums in ReplyChannelRange now have the same encoding as short channel ids and timestamps: one byte for
the encoding type (uncompressed or zlib) followed by encoded data.

* TLV Stream: Implement a generic "get" method for TLV fields

If a have a TLV stream of type MyTLV which is a subtype of TLV, and MyTLV1 and MYTLV2 are both
subtypes of MyTLV then we can use stream.get[MyTLV1] to get the TLV record of type MYTLV1 (if any)
in our TLV stream.

* Extended range queries: Implement latest BOLT changes

Checksums are just transmitted as a raw array, with optional compression as it would be useless here.

* Use extended range queries on regtest and testnet

We will use them on mainnet as soon as lightningnetwork/lightning-rfc#557 has been merged.

* Address review comments

* Router: rework handling of ReplyChannelRange

We remove the ugly and inefficient zipWithIndex we had before

* NodeParams: move fee base check to its proper place

* Router: minor cleanup
BOTL 7: fixup: clarify when to reply with a `node_announcement`
Do not reply with a node_announcement if the query includes an optional query flag that does not request it.
The current wording could be interpreted as "always follow with node announcements whenever
you reply with a channel announcements" which defeats the point of using query flags (if you want the node
announcements just set the corresponding bits).
@rustyrussell

This comment has been minimized.

Copy link
Collaborator

commented Aug 27, 2019

OK, I found a bug in my code while implementing the protocol tests; my checksums are wrong.

I assume the crc32 is not supposed to cover the 2 type bytes at the start of the channel_update? ie. our code (now!):

	assert(tal_count(channel_update) > 2 + 64 + 32 + 8 + 4);
	sum = crc32c(0, channel_update + 2 + 64, 32 + 8);
	sum = crc32c(sum, channel_update + 2 + 64 + 32 + 8 + 4,
		     tal_count(channel_update) - (64 + 2 + 32 + 8 + 4));

And for a specific test cases:

  1. signature=76df7e70c63cc2b63ef1c062b99c6d934a80ef2fd4dae9e1d86d277f47674af3255a97fa52ade7f129263f591ed784996eba6383135896cc117a438c80293282 chain_hash=06226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f short_channel_id=103x1x0 timestamp=1565587763 message_flags=0 channel_flags=0 cltv_expiry_delta=144 htlc_minimum_msat=0 fee_base_msat=1000 fee_proportional_millionths=10
    010276df7e70c63cc2b63ef1c062b99c6d934a80ef2fd4dae9e1d86d277f47674af3255a97fa52ade7f129263f591ed784996eba6383135896cc117a438c8029328206226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f00006700000100005d50f933000000900000000000000000000003e80000000a
    crc32 = 0x1112fa30

  2. signature=06737e9e18d3e4d0ab4066ccaecdcc10e648c5f1c5413f1610747e0d463fa7fa39c1b02ea2fd694275ecfefe4fe9631f24afd182ab75b805e16cd550941f858c chain_hash=06226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f short_channel_id=109x1x0 timestamp=1565587765 message_flags=1 channel_flags=0 cltv_expiry_delta=48 htlc_minimum_msat=0 fee_base_msat=100 fee_proportional_millionths=11 htlc_maximum_msat=100000
    010206737e9e18d3e4d0ab4066ccaecdcc10e648c5f1c5413f1610747e0d463fa7fa39c1b02ea2fd694275ecfefe4fe9631f24afd182ab75b805e16cd550941f858c06226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f00006d00000100005d50f935010000300000000000000000000000640000000b00000000000186a0
    crc32 = f32ce968

@sstone

This comment has been minimized.

Copy link
Collaborator Author

commented Aug 27, 2019

@rustyrussell thanks! we do skip the first 2 bytes of the encoded channel_update and find the same checksums, I'll make new test vectors and include your data.

pm47 added a commit to ACINQ/eclair that referenced this pull request Aug 28, 2019
Extended queries optional (#899)
This is the implementation of lightningnetwork/lightning-rfc#557.

* Correctly handle multiple channel_range_replies

The scheme we use to keep tracks of channel queries with each peer would forget about
missing data when several channel_range_replies are sent back for a single channel_range_queries.

* RoutingSync: remove peer entry properly

* Remove peer entry on our sync map only when we've received
a `reply_short_channel_ids_end` message.
* Make routing sync test more explicit

* Routing Sync: rename Sync.count to Sync.totalMissingCount

* Do not send channel queries if we don't want to sync

* Router: clean our sync state when we (re)connect to a peer

We must clean up leftovers for the previous session and start the sync process again.

* Router: reset sync state on reconnection

When we're reconnected to a peer we will start a new sync process and should reset our sync
state with that peer.

* Extended Queries: use TLV format for optional data

Optional query extensions now use TLV instead of a custom format.
Flags are encoded as varint instead of bytes as originally proposed. With the current proposal they will all fit on a single byte, but will be
much easier to extends this way.

* Optional TLVs are represented as a list, not an optional list

TLVs that extend regular LN messages can be represented as a TlvStream and not an Option[TlvStream] since we don't need
to explicitely terminate the stream (either by preprending its length or using a specific terminator) as we do in Onion TLVs.

No TLVs simply means that the TLV stream is empty.

* TLV Stream: Implement a generic "get" method for TLV fields

If a have a TLV stream of type MyTLV which is a subtype of TLV, and MyTLV1 and MYTLV2 are both
subtypes of MyTLV then we can use stream.get[MyTLV1] to get the TLV record of type MYTLV1 (if any)
in our TLV stream.

* Use extended range queries on regtest and testnet

We will use them on mainnet as soon as lightningnetwork/lightning-rfc#557 has been merged.

* Channel range queries: send back node announcements if requested (#1108)

This PR adds support for sending back node announcements when replying to channel range queries:
- when explicitly requested (bit is set in the optional query flag)
- when query flags are not used and a channel announcement is sent (as per the BOLTs)

A new configuration option `request-node-announcements` has been added in the `router` section. If set to true, we
will request node announcements when we receive a channel id (through channel range queries) that we don't know of.
This is a setting that we will probably turn off on mobile devices.

* Extended Channel Queries: add CL interop test
rustyrussell added a commit to ElementsProject/lightning-rfc-protocol-test that referenced this pull request Sep 2, 2019
FIXME: temporary formats.csv file with extensions under test.
In particular, this is effectively a merge of lightningnetwork#557 and lightningnetwork#655, so
you can run all the protocol tests at once.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>

@t-bast t-bast moved this from Scheduled to Accepted in Specification Meeting Agenda Sep 3, 2019

@t-bast

This comment has been minimized.

Copy link
Collaborator

commented Sep 3, 2019

CL and Eclair have implemented this and correctly inter-operate.
LL said implementation will come later, but concept acked.
@cfromknecht can you tell us if you're ok with the current state of the PR and merging it as-is?

@cfromknecht
Copy link
Collaborator

left a comment

Final version LGTM!! 😀

@sstone sstone merged commit c8e53fe into lightningnetwork:master Sep 16, 2019

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.