BOLT 7: add extended channel queries

Extended channel queries include extra timestamps (one for each channel_update), which
allows nodes that are often online to easily sync their routing tables.

.aspell.en.pws

@@ -260,6 +260,7 @@ aa
 df
 versa
 timestamp
+timestamps
 metadata
 Bitcoin's
 Versioning

07-routing-gossip.md

@@ -727,6 +727,139 @@ is simple to implement.
 In the case where the `channel_announcement` is nonetheless missed,
 `query_short_channel_ids` can be used to retrieve it.
 
+## Extended Query Messages
+
+The query messages defined above do not cover the use case of nodes
+that are offline most of the time and are missing a timestamps that can
+be used to detect obsolete channel updates.
+
+Here we define a new set of channel queries, with additional timestamps (one
+timestamp per channel update).
+
+A new feature bit is used to specify which type of queries a node will support. If 
+both nodes support extended queries then the first message that they will send once they're
+connected will be `query_channel_range_with_timestamps` and they will only use extended queries 
+from then on.
+
+Encoding types are the same as above, with a single byte prefix (`0` or `1`)
+
+### The `query_short_channel_ids_with_timestamps`/`reply_short_channel_ids_with_timestamps_end` Messages
+
+1. type: 1011 (`query_short_channel_ids_with_timestamps`) (`gossip_queries`)
+2. data:
+    * [`32`:`chain_hash`]
+    * [`2`:`len`]
+    * [`len`:`encoded_short_ids_and_flag`]
+
+This message encodes an ordered list of [short channel id (8 bytes) | flag (1 byte)] where flag
+specifies what the sender wants to receive for a given short channel id and can be a combination of:
+
+* `1`: include `channel_update` for node 1
+* `2`: include `channel_update` for node 2
+* `4`: include `channel_announcement`
+
+1. type: 1012 (`reply_short_channel_ids_with_timestamps_end`) (`gossip_queries`)
+2. data:
+    * [`32`:`chain_hash`]
+    * [`1`:`complete`]
+
+This is a general mechanism which lets a node query for the
+`channel_announcement` and `channel_update` messages for specific channels
+(identified via `short_channel_id`s). This is usually used either because
+a node sees a `channel_update` for which it has no `channel_announcement` or
+because it has obtained previously unknown `short_channel_id`s
+from `reply_channel_range`.
+
+#### Requirements
+
+The sender:
+  - MUST NOT send `query_short_channel_ids_with_timestamps` if it has sent a previous `query_short_channel_ids_with_timestamps` to this peer and not received `reply_short_channel_ids_with_timestamps_end`.
+  - MUST set `chain_hash` to the 32-byte hash that uniquely identifies the chain
+  that the `short_channel_id`s refer to.
+  - MUST set the first byte of `encoded_short_ids_and_flag` to the encoding type.
+  - MUST encode a whole number of `short_channel_id`s and timestamps to `encoded_short_ids_and_flag`
+  - MAY send this if it receives a `channel_update` for a
+   `short_channel_id` for which it has no `channel_announcement`.
+  - SHOULD NOT send this if the channel referred to is not an unspent output.
+
+The receiver:
+  - if the first byte of `encoded_short_ids_and_flag` is not a known encoding type:
+    - MAY fail the connection
+  - if `encoded_short_ids_and_flag` does not decode into a whole number of `short_channel_id` and flags:
+    - MAY fail the connection.
+  - if it has not sent `reply_short_channel_ids_with_timestamps_end` to a previously received `query_short_channel_ids_with_timestamps` from this sender:
+    - MAY fail the connection.
+  - MUST respond to each known `short_channel_id` with a `channel_announcement` and/or the latest `channel_update` for each end, according to what the sender specified in the `flag` byte
+	- SHOULD NOT wait for the next outgoing gossip flush to send these.
+  - MUST follow with any `node_announcement`s for each `channel_announcement`
+	- SHOULD avoid sending duplicate `node_announcements` in response to a single `query_short_channel_ids_with_timestamps`.
+  - MUST follow these responses with `reply_short_channel_ids_with_timestamps_end`.
+  - if does not maintain up-to-date channel information for `chain_hash`:
+	- MUST set `complete` to 0.
+  - otherwise:
+	- SHOULD set `complete` to 1.
+
+#### Rationale
+
+Future nodes may not have complete information; they certainly won't have
+complete information on unknown `chain_hash` chains.  While this `complete`
+field cannot be trusted, a 0 does indicate that the sender should search
+elsewhere for additional data.
+
+The explicit `reply_short_channel_ids_with_timestamps_end` message means that the receiver can
+indicate it doesn't know anything, and the sender doesn't need to rely on
+timeouts.  It also causes a natural ratelimiting of queries.
+
+### The `query_channel_range_with_timestamps` and `reply_channel_range_with_timestamps` Messages
+
+1. type: 1013 (`query_channel_range_with_timestamps`) (`gossip_queries`)
+2. data:
+    * [`32`:`chain_hash`]
+    * [`4`:`first_blocknum`]
+    * [`4`:`number_of_blocks`]
+
+1. type: 1014 (`reply_channel_range_with_timestamps`) (`gossip_queries`)
+2. data:
+    * [`32`:`chain_hash`]
+    * [`4`:`first_blocknum`]
+    * [`4`:`number_of_blocks`]
+    * [`1`:`complete`]
+    * [`2`:`len`]
+    * [`len`:`encoded_short_ids_and_timestamps`]
+
+This allows a query for channels within specific blocks.
+
+#### Requirements
+
+The sender of `query_channel_range_with_timestamps`:
+  - MUST NOT send this if it has sent a previous `query_channel_range_with_timestamps` to this peer and not received all `reply_channel_range_with_timestamps` replies.
+  - MUST set `chain_hash` to the 32-byte hash that uniquely identifies the chain
+  that it wants the `reply_channel_range_with_timestamps` to refer to
+  - MUST set `first_blocknum` to the first block it wants to know channels for
+  - MUST set `number_of_blocks` to 1 or greater.
+
+The receiver of `query_channel_range_with_timestamps`:
+  - if it has not sent all `reply_channel_range_with_timestamps` to a previously received `query_channel_range_with_timestamps` from this sender:
+    - MAY fail the connection.
+  - MUST respond with one or more `reply_channel_range_with_timestamps` whose combined range
+	cover the requested `first_blocknum` to `first_blocknum` plus
+	`number_of_blocks` minus one.
+  - For each `reply_channel_range_with_timestamps`:
+    - MUST set with `chain_hash` equal to that of `query_channel_range_with_timestamps`,
+    - MUST encode a `short_channel_id` and 2 timestamps, one for each `channel_update`, for every open channel it knows in blocks `first_blocknum` to `first_blocknum` plus `number_of_blocks` minus one.
+    - MUST limit `number_of_blocks` to the maximum number of blocks whose
+      results could fit in `encoded_short_ids_and_timestamps`
+    - if does not maintain up-to-date channel information for `chain_hash`:
+      - MUST set `complete` to 0.
+    - otherwise:
+      - SHOULD set `complete` to 1.
+
+#### Rationale
+
+A single response might be too large for a single packet, and also a peer can
+store canned results for (say) 1000-block ranges, and simply offer each reply
+which overlaps the ranges of the request.
+
 ## Initial Sync
 
 If a node requires an initial sync of gossip messages, it will be flagged

09-features.md

@@ -26,6 +26,7 @@ These flags may only be used in the `init` message:
 | 3  | `initial_routing_sync` | Indicates that the sending node needs a complete routing information dump | [BOLT #7](07-routing-gossip.md#initial-sync) |
 | 4/5  | `option_upfront_shutdown_script` | Commits to a shutdown scriptpubkey when opening channel | [BOLT #2](02-peer-protocol.md#the-open_channel-message) |
 | 6/7  | `gossip_queries`           | More sophisticated gossip control | [BOLT #7](07-routing-gossip.md#query-messages) |
+| 16/17  | `extended_gossip_queries`           | Gossip queries with extra timestamps | [BOLT #7](07-routing-gossip.md#extended-query-messages) |
 
 ## Assigned `globalfeatures` flags