Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stitching multiple readable buffers together #1060

Open
Drawaes opened this issue Dec 12, 2016 · 12 comments

Comments

Projects
None yet
4 participants
@Drawaes
Copy link
Contributor

commented Dec 12, 2016

In TLS there is two layers of framing. The Record frame and the protocol frame. A record frame is wrapped inside the protocol frame. However the record frame can be much larger than the record frame. Therefore I need to be able to unwrap and decrypt the record fragments in place.

I then need to slice off the headers, and trailers to get the actual record data. At this point if I don't have the complete frame I will need to preserve and check multiple of these. It would be nice if at this point I didn't need to construct a collection or my own linked list but instead could append the start cursor of the second list to the end cursor of the first.

I am not sure if this is possible today but as I believe they are linked lists of buffers internally this should be possible.

@davidfowl

This comment has been minimized.

Copy link
Contributor

commented Dec 12, 2016

Code samples would help me understand better here. Is this across multiple calls to ReadAsync?

   ReadAsync()                ReadAsync()

[ header | payload ] -> [ header | payload ]

Is that what it looks like? You want to string multiple payloads together into a single readable buffer?

@Drawaes

This comment has been minimized.

Copy link
Contributor Author

commented Dec 12, 2016

Yeah but it's even more complex.

ReadAsync ReadAsync ReadAsync ReadAsync
[Header1][Payload1.1] [Header2][Payload1.2]

Payload1.1 + Payload1.2 == Payload1

The "Header" record can be max 2^16-1 long (16k) but the internal message can be 2^24-1 long (again go figure?) Each of Payload1.1 and Payload1.2 is independently encrypted so will be decrypted at that layer, however they need to be combined to make a single message to process.... the only way I can currently think of doing this is to have a listener internally that does the decryption, and then puts them with another custom header injected and put them back on another pipeline, which seems a bit over the top.

Instead I want something like

AsyncRead, AsyncRead (until I have a whole frame according to [Header])

Then decrypt, Preserve the decrypted chunk, and save that as state in the reading loop, then go back for another frame, once I have another frame, preserve that and append it to the other readable buffer and process the completed [Payload]

@Drawaes

This comment has been minimized.

Copy link
Contributor Author

commented Dec 12, 2016

The other problem with the internal channel is that the process needs to be synchronised. Async won't work because the current payload may change the encryption state which will change the way the next [header] frame will be handled (encrypted or not, renegotiated keys etc)

One way to solve it but seems unclean is to hold a list of buffers that makes up the current payload. Obviously then I have to enumerate that and the readables which is a mess.

@Drawaes

This comment has been minimized.

Copy link
Contributor Author

commented Dec 12, 2016

@davidfowl does this kinda help (P.S. don't mention spacing, var it's mock code)
Gist of Demo
😆

@Drawaes

This comment has been minimized.

Copy link
Contributor Author

commented Dec 15, 2016

What would be a solution is if You could build an I SEQUENCE from a bunch of preserved readablebuffers. So in effect you would end up with a selectmany to use linq terminology

@davidfowl

This comment has been minimized.

Copy link
Contributor

commented Dec 22, 2016

(P.S. don't mention spacing, var it's mock code)

My eyes

@Drawaes

This comment has been minimized.

Copy link
Contributor Author

commented Dec 22, 2016

@Drawaes

This comment has been minimized.

Copy link
Contributor Author

commented Dec 22, 2016

Managed TLS using an internal handshake channel

Shows how I have used an internal channel. This might actually be preferable if a new handshake happens during communication (renegotiation for certs etc).

@Drawaes

This comment has been minimized.

Copy link
Contributor Author

commented Jan 27, 2017

@KrzysztofCwalina is this solved with the multi span reader that you have made over readonly bytes?

@KrzysztofCwalina

This comment has been minimized.

Copy link
Member

commented Jan 27, 2017

I would say the reader is trying to solve it. It will be solved when the pipeline APIs use the same abstraction to represent multi-segmented buffers as the reader. Today, the reader operates over ReadOnlyBytes and the pipeline APIs use ReadableBuffer.

@davidfowl

This comment has been minimized.

Copy link
Contributor

commented Jan 31, 2017

Yep, as soon as we get through this push converting Kestrel to pipelines, we'll look at merging the APIs. We're very very close.

@ahsonkhan ahsonkhan added this to the 2.1.0 milestone Apr 6, 2017

@davidfowl davidfowl removed their assignment Feb 7, 2018

@ahsonkhan

This comment has been minimized.

Copy link
Member

commented May 31, 2018

Does ReadOnlySequence resolve this issue?

@ahsonkhan ahsonkhan modified the milestones: 2.1.0, 2.2.0 Jun 2, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.