invoices: expose custom tlv records from the payload #3742

joostjager · 2019-11-19T12:33:49Z

This PR extracts the received custom records from the hop payload and stores them in the invoice database. Users can retrieve the custom records through the usual invoice queries / subscriptions.

Roasbeef · 2019-11-20T01:43:08Z

tlv/record.go

+// TypeSet is an unordered set of Types. The map values are byte slices. If the
+// byte slice is nil, the type was successfully parsed. Otherwise the value is
+// byte slice containing the encoded data.
+type TypeSet map[Type][]byte


This seems to be really overloading the type beyond its original intended usage. It's only meant to be used in order to determine violations w.r.t known/unknown types. We already return the full set of unparsed types after decoding, so not sure why this is needed.

Also have you benched marked this? As is, it appears we'll end up copying all the values from the main stream twice.

Yes it is overloading the type, but we can deviate from an original intention.

I haven't benchmarked. If we want to process the raw bytes of unrecognized records, we need to copy anyway. Just in case our intention is to really ignore them, the copy is unnecessary. Afaik, for db tlv that isn't expected. And for the tlv payload it may happen with an unrecognized standard field (the custom ones we probably want to store somewhere). To me it seems premature to optimize for.

I am open to different ways of implementing this. What do you prefer?

Where do you see the copy twice btw? Afaik we only read bytes from the stream and store them in a new byte slice that is a value of the TypeSet map.

Renamed to TypeMap and preallocated the buffer.

Roasbeef · 2019-11-20T01:48:57Z

htlcswitch/hop/payload.go

@@ -157,14 +164,22 @@ func NewPayloadFromReader(r io.Reader) (*Payload, error) {
 		mpp = nil
 	}

+	for t, parseResult := range parsedTypes {


Feels like there's another way to do this with less copying.

In line with comment above, I don't think it matters much. If you want the tlv stream to return a map that we can use directly, we need to add more controls to the stream to specify what exactly we expect.

suggestion from @cfromknecht: delete items from the map

It seems it isn't possible to cast map[Type][]byte to map[uint64][]byte though

Talked about this offline with @Roasbeef. We do want to stop tlv.Type from being used anywhere. Copying the map (but not the values) looks to be ok.

joostjager · 2019-11-20T10:44:21Z

channeldb/invoices.go

+			return err
+		}
+
+		records = append(records, customRecords...)


Offline comment from @cfromknecht: not sure if it is a good idea to mix the custom records with the db types here.

Discussed this further. It may be better to create a single tlv record for which the value is a tlv stream with just the custom records. As a simple insurance against accidental modification of standard fields.

Discussed offline. There seems to be more support for a flat structure, so leaving it as is.

I am still somewhat concerned about mingling minimally validated user data with known records. I think it is possible to do it safely, but i think it forces us to be stricter in how we handle migrations.

Consider the scenario where we add a field to the invoice body but do not bump the db version. If a user downgrades, this will now appear in the custom record set even tho its type is beneath the cutoff. I think it's possible that data gets written back, but it's difficult imo to ascertain whether this can lead to unforeseen bugs.

We could prevent this by filter the custom records when reading as we do in the hop, then the data is actually dropped when writing back. This would maintain the current behavior in that scenario.

joostjager · 2019-11-20T10:45:14Z

channeldb/invoices.go

+
+		records = append(records, customRecords...)
+
+		tlvStream, err := tlv.NewStream(records...)


Question open as to whether we want to bump the db version, to prevent users running master from downgrading and loosing their custom records

Discussed this in the context of another PR. We don't give this guarantee to users running master generally.

tlv/stream.go

htlcswitch/hop/payload.go

htlcswitch/hop/payload_test.go

channeldb/invoices.go

invoices/interface.go

channeldb/invoices.go

cfromknecht · 2019-12-05T23:09:19Z

channeldb/invoices.go

+			return err
+		}
+
+		records = append(records, customRecords...)


I am still somewhat concerned about mingling minimally validated user data with known records. I think it is possible to do it safely, but i think it forces us to be stricter in how we handle migrations.

Consider the scenario where we add a field to the invoice body but do not bump the db version. If a user downgrades, this will now appear in the custom record set even tho its type is beneath the cutoff. I think it's possible that data gets written back, but it's difficult imo to ascertain whether this can lead to unforeseen bugs.

We could prevent this by filter the custom records when reading as we do in the hop, then the data is actually dropped when writing back. This would maintain the current behavior in that scenario.

joostjager · 2019-12-06T08:55:53Z

We could prevent this by filter the custom records when reading as we do in the hop, then the data is actually dropped when writing back. This would maintain the current behavior in that scenario.

Yes that is a nice safe guard that doesn't require a different db struct. It should be in place now

Roasbeef

LGTM 🧹

Left one comment which is mainly concerned with consistency of the API w.r.t TypeMap.

Roasbeef · 2019-12-07T00:52:39Z

tlv/stream.go

+			// Record the successfully decoded type if the caller
+			// provided an initialized TypeMap.
+			if parsedTypes != nil {
+				parsedTypes[typ] = nil


Why don't we also want to provide the caller w/ the raw bytes of the type just as we do for unknown+odd types?

In that case we would copy the data twice for the known types. And need another data structure to signal which ones were decoded. We can also rethink that when the need arises to access the raw bytes of known types, this is all in memory.

cfromknecht

LGTM 🤘

cfromknecht · 2019-12-10T01:56:14Z

needs rebase

joostjager · 2019-12-10T05:57:02Z

rebased

joostjager requested a review from Roasbeef as a code owner November 19, 2019 12:33

Roasbeef requested changes Nov 20, 2019

View reviewed changes

joostjager requested review from Roasbeef and cfromknecht November 20, 2019 10:42

joostjager commented Nov 20, 2019

View reviewed changes

joostjager mentioned this pull request Nov 20, 2019

Allow custom data fields in the htlc payload #3670

Closed

joostjager added this to WIP in v0.9.0-beta Nov 20, 2019

joostjager self-assigned this Nov 22, 2019

joostjager force-pushed the expose-custom-tlv branch 8 times, most recently from a4cf7d8 to bd6cd63 Compare December 4, 2019 15:42

joostjager added the v0.9.0 label Dec 4, 2019

joostjager added this to the 0.9.0 milestone Dec 4, 2019

joostjager moved this from WIP to Needs Review in v0.9.0-beta Dec 4, 2019

joostjager force-pushed the expose-custom-tlv branch from bd6cd63 to 4e70485 Compare December 5, 2019 11:00

joostjager mentioned this pull request Dec 5, 2019

lncli+invoices: experimental key send mode #3795

Merged

joostjager force-pushed the expose-custom-tlv branch 2 times, most recently from 963132e to ae79e2c Compare December 5, 2019 13:56

cfromknecht reviewed Dec 5, 2019

View reviewed changes

joostjager force-pushed the expose-custom-tlv branch from ae79e2c to df22474 Compare December 6, 2019 08:56

joostjager requested a review from cfromknecht December 6, 2019 08:56

Roasbeef approved these changes Dec 7, 2019

View reviewed changes

cfromknecht approved these changes Dec 8, 2019

View reviewed changes

v0.9.0-beta automation moved this from Needs Review to Approved Dec 8, 2019

joostjager added 3 commits December 10, 2019 06:54

tlv: expose unknown values

cbe213f

hop: store custom records from payload

37258c4

invoices: store custom records in invoice database

5f4bd13

joostjager force-pushed the expose-custom-tlv branch from df22474 to 5f4bd13 Compare December 10, 2019 05:55

joostjager merged commit 699bb19 into lightningnetwork:master Dec 10, 2019

v0.9.0-beta automation moved this from Approved to Done Dec 10, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

invoices: expose custom tlv records from the payload #3742

invoices: expose custom tlv records from the payload #3742

joostjager commented Nov 19, 2019 •

edited

Roasbeef Nov 20, 2019

Roasbeef Nov 20, 2019

joostjager Nov 20, 2019

joostjager Dec 3, 2019

joostjager Dec 4, 2019

Roasbeef Nov 20, 2019

joostjager Nov 20, 2019

joostjager Nov 22, 2019

joostjager Dec 3, 2019

joostjager Dec 4, 2019

joostjager Nov 20, 2019

joostjager Nov 22, 2019

joostjager Dec 4, 2019

cfromknecht Dec 5, 2019

joostjager Nov 20, 2019

joostjager Dec 4, 2019

cfromknecht Dec 5, 2019

joostjager commented Dec 6, 2019

Roasbeef left a comment

Roasbeef Dec 7, 2019

joostjager Dec 7, 2019

cfromknecht left a comment

cfromknecht commented Dec 10, 2019

joostjager commented Dec 10, 2019


		records = append(records, customRecords...)

		tlvStream, err := tlv.NewStream(records...)

invoices: expose custom tlv records from the payload #3742

invoices: expose custom tlv records from the payload #3742

Conversation

joostjager commented Nov 19, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joostjager commented Dec 6, 2019

Roasbeef left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cfromknecht left a comment

Choose a reason for hiding this comment

cfromknecht commented Dec 10, 2019

joostjager commented Dec 10, 2019

joostjager commented Nov 19, 2019 •

edited