Composite Metadata Extension: data Mime/type definition per-stream

[OlegDokuka]

Problematic

For now, we have the only way to define data mime/type at the very beginning of the connection setup in the SetupPayload. For many cases, it is fine to have a homogeneous mime-types (unified "format"), or frankly speaking a mime/type per connection. However, when the case comes to fronted, gateway, or broker, it is necessarily needed to define what data is packed within a logical stream.

Proposal

That discussion has already been raised at #117, and it is clear that we can define what data is in the payload using metadata. However, it could be tricky having an unspecified way to do that and, especially today, when we have a Composite Metadata came in, it becomes pretty straight forward on how to solve that problem.
For instance, we can define a specific WellKnownType for doing an override of data mime-type.

1. SetupPayload still has a mandatory field for data mime-type, so at the first connection setup, we define a general mime-type.
2. If for interaction within a connection we define metadata as a CompositMetadata then it becomes possible to override a data mime per stream
3. If a requester during the interaction does not specify a mime-type using composite metadata, then the default mime-type is a mime-type for that stream
4. If a requester specifies mime-type using composite metadata, then the default mime-type is overridden by one specified in composite metadata for that stream

[rstoyanchev]

We had a similar request in spring-projects/spring-framework#23431.

Overriding the data mime type from the SETUP frame per stream in composite medatadata seems a straight-forward enough idea. Doesn't that also naturally lead to the possibility for separate input and output data mime types per stream?

Maybe there could be two possible per-stream overrides, one for input and one for output. If only one is specified, the other assumes the connection default.

[OlegDokuka]

Yeah, sounds reasonable, at least proven, but long-lived HTTP. Thus does not have any objection. Work is in progress. I will ping you @rstoyanchev once have something

[linux-china]

@OlegDokuka almost, a stream contains data encoding & accepted encodings, just like a HTTP Request. We can follow the Composite Metadata design with following structure.

    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |M| MIME ID/Len |   Data Encoding MIME Type                ...
    +---------------+---------------+---------------+---------------+
    |M| MIME ID/Len |   Accepted Encoding MIME Type            ...
    +---------------+---------------+---------------+---------------+
    |M| MIME ID/Len |   Accepted Encoding MIME Type            ...
    +---------------+---------------+---------------+---------------+

Data Encoding Metadata (message/x.rsocket.data.encoding.v0), contains 1 field of data encoding and 0 -N fields of accepted encodings.

Requester side

Of course, data encoding should be filled. if accepted encodings absent, it means the requester accepts data with any encoding format.

Response side

just fill data encoding field, and indicate data format for the payload.

[OlegDokuka]

@linux-china great idea about the list of accepted mime-types. Totally agree with you, except I guess it will be better to sperate data mime-type and accepted mime-types

[linux-china]

@OlegDokuka yes. it's good to sperate data encoding & accept encodings, and similar to HTTP and easy for understanding. the names for data encoding: message/x.rsocket.data.encoding.v0 and message/x.rsocket.accept.encodings.v0

[OlegDokuka]

@linux-china I have created a PR with a draft version of the spec, so do not hesitate to leave your review :D

[maxim-bandurko-lsvt]

@OlegDokuka Will it include authentication mime type? As it is convenient to authenticate the user later (in situation of anonymous connection) and do actual authorize of user on server once it is needed.

[OlegDokuka]

Hey, @maxim-bandurko-lsvt!

What is the use case for including Auth mimetype in response message?

BTW, the draft PR is there -> #285

[maxim-bandurko-lsvt]

Hello @OlegDokuka

I think the most best use case for this mime type would be simple shopping cart and starting actual ordering process. Not authorized user have some items in cart and now want to finish the purchase by having option to make it not authorized or authorized with entering credentials for his account. Calling requestResponse with passed credentials is going to authorize user at server without dropping already established connection transport through websocket. And it would be just new stream that is going to be processed by server.

Other use cases could be like switching between multiple accounts with same transport. Can come with more use cases if it is needed.

And have one more idea about having encrypted stream option to any mime type or at just cm, that can use some basic encryption algorithm for stream by certain "salt" key. But perhaps, this probably should go in a new topic of enhancements ideas.

[OlegDokuka]

@maxim-bandurko-lsvt, I guess I'm a little bit confused about the use case. Are we talking about the auth per-stream or are we talking about having auth in the response payload?

What I understood from the mentioned post is auth per-request and it is more than legit use case even right now (see -> https://github.com/rsocket/rsocket/blob/master/Extensions/Security/Authentication.md).

Why this topic is about payload data mime-type is because in the spec (see -> https://github.com/rsocket/rsocket/blob/master/Protocol.md#setup-frame-0x01) there is a clear definition that mime-type is defined once per connection.

In contrast, composite-metadata is per-connection as well as per-stream

[maxim-bandurko-lsvt]

@OlegDokuka Sorry for confusing. My idea was auth per stream inside data. In use case that I described there are no auth with initial request done, as user is anonymous. Idea was to be able to do actual auth within stream per need (when user decided to provide credentials). In this case all existing streams are not interrupted, metadata mime type is cm, and user just got authorized at server later by credentials passed in data.

From https://github.com/rsocket/rsocket/blob/master/Extensions/Security/Authentication.md is told that it will be possible to have auth per stream in metadata payloads. But thinking about overriding of mime types for data per stream, could be convenient to keep composite metadata and have auth mime type for data inside a stream. Probably I had to explain better my question. And, may be, it is too early to think about such option for auth extension.

[OlegDokuka]

@maxim-bandurko-lsvt

Idea was to be able to do actual auth within stream per need (when user decided to provide credentials).

stream is a logical unit within a connection (see https://github.com/rsocket/rsocket/blob/master/Protocol.md#terminology)

if you mean stream as a physical connection, then there is no limitation to make authentication in the post connection phase. In fact, I can imaging RSocket broker which does the multiplexer/router role without being able to send auth in already created connection.

CompositeMetadata is a definition of metadata per-streams as well as per-connection. What is necessary is to define the metadata mimetype at the connection phase.

if you would like to make auth per payload - then it would be too overhead and you can always check your logical requestStream or requestChannel on requestResponse which will let you do auth kind of per payload

Feel free to add more if I misunderstood you

[maxim-bandurko-lsvt]

Hello @OlegDokuka

I think that explained idea in very confusing way, especially coming from incubation phase for auth extension. Sorry about that. And have more concerns, that it has to go to auth extension discussion, but at the same time have it connected with mime type per stream discussion thread too.

Any way, here is a full explanation (it got to be very big comment btw).

Some times auth shouldn't be initialized in setup. That happens mostly with front end implementation of rsocket, when there is a website that is using same websocket transport connection for data transfer of public data and private data (website at browser window has only one connection established on site initialization phase). Like javascript frontend api, this could be something like simple online store. Once user enters this website, browser loads all assets and api establishes websocket transport with server with no auth in setup (server has no idea about user and allows to use publically accessed routes for websocket messages, like streams of items from some categories, etc.). User browse the store and add some items to shopping cart (server on that moment store selected items to some temporary reactive storage that is identified by browser session (in that situation, if user will open one more window of this website, server recognizes those websocket request and pins it to same anonymous session, and allows to subscribe second window to the stream of shopping cart items that were added in first window, in other words, both windows are subscribed to same data streams)). When, later, user is done with adding items to shopping cart, and decides to start with checkout process. For this user has two options, continue in anonymous way by filling all checkout details or sign in/register to the store. With this second option comes the moment, that is needed to do auth of user and enable authorized user access routes at server. If we have auth only per connection request, api has to re-establish the transport connection to pass the credentials in setup. But that is not convenient for front end api, as after re-connection is needed to re-stream everything at front end (second window will have to do almost the same btw, but not using credentials). Also breaking connection with new one may lead to data loss from broker, that requires to re-check all passed data from server with first unauthorized connection. Idea that I had in mind, is to allow the front end do requestStream with mime type auth for data, and that message will auth user at server, and server will identify the connection sessions for both windows to certain user account, and server will automatically update the routes access to allow private ones based on user roles, etc. Both windows will have the same "streams" logic established, and user can continue with checkout using any window he want (similar to logic result we are getting with basic server side rendering, that user can open any amount of windows and do any different operations inside them).

That was one use case for website, another one for mobile app could be something like online banking app, that stores user token in it's storage once user did first sign in and user can see for example balance of his account without typing credentials when open app next time at his device (or will be getting push notifications from server to his device). In this situation we are passing user token in connection setup. But when it comes to moment that user would like to do some transactions with his account, app requires user to enter credentials, as to verify the user. In this situation we can use requestStream with auth mime type for data that will verify and re-authorize user at server without dropping existing rsocket connection (just auth in stream).

There could be more use cases, like social network website app, that not authorized user can browse posts and user profiles, but when it comes to moment that user wants to do a post, user should auth at app.

Key point is keeping same only one connection up, and not re-create new one only because of allowed auth only at connection setup. Much convenient having requestResponse to do auth procedure inside stream.

I understand, that auth extension is still in process, and having such can say "reactive" and "lazy" auth on server side is not implemented on default even at spring's security, but idea can be interesting from the part of implementation and beneficial for developers once it be inside frameworks like spring. As most biggest issue in migration at website development from ajax requests to persistent connection using websocket is authentication with sessions. Developers had to implement the logic how to keep front end (with multiple windows at browser) synced to same auth session at server.

Huge progress on that is doing spring with spring boot, but again, developers have to implement custom version of "reactive security" and "reactive sessions" for websocket broker based on their app requirements. Especially when it comes with horizontal scaling of brokers to multiple servers. Hope that in future, such implementation will be in spring done.

[tomasz-galuszka]

What about a case when in the SETUP frame we do an authentication with metadata mime type: message/x.rsocket.authentication.v0. After the successful authentication then send some FNF requests with metadataMimeType: message/x.rsocket.routing.v0 - to different logical endpoints.

SETUP: https://github.com/rsocket/rsocket/blob/5920ed374d008abb712cb1fd7c9d91778b2f4a68/Extensions/Security/Authentication.md

REQUEST_FNF: https://github.com/rsocket/rsocket/blob/5920ed374d008abb712cb1fd7c9d91778b2f4a68/Extensions/Routing.md

I guess with the fixed metadata mimetype currently it is not possible. The only way I could achieve that is to use composite metadata with only one item inside, but with different mime types (but I feel it is a bit over-kill).

[OlegDokuka]

Hi @maxim-bandurko-lsvt @tomasz-galuszka!

I guess there is some misunderstanding on how these mime-types should be used. The mime-typed mentioned in this issue, as well as auth mime-type and others, are intended to be used only within CompositeMetadata.

That said, if you define metadata-mime-type at rsocket setup as message/x.rsocket.authentication.v0 - none of the existing frameworks (e.g. spring-messaging or rsocket-rpc) will handle it properly.

what you want to do instead is to define metadata-mime-type as message/x.rsocket.composite-metadata.v0 and then include your authentication metadata as a part of SetupPayload.

Same valid for Routing, once the connection metadata-mimetype is set to message/x.rsocket.composite-metadata.v0, you MUST follow the structure of composite metadata all the way down and that is the only way how you may define routing within your composite metadata.

Coming back to @maxim-bandurko-lsvt case, if you need to make deferred authentication - go and do that, your user may do actions for a while and then call request-response providing the required composite-metadata with auth payload encoded in it - and that is basically it.

Just to clarify, this issue aimed to add another well-known mime-type for composite metadata which let one redefine data mime-type for a particular request. The data mime-type does not affect metadata mime-type.

Also, please find a rsocket-js example which shows how metadata and mime-types are intended to be used https://github.com/rsocket/rsocket-js/blob/master/packages/rsocket-examples/src/CompositeMetadataExample.js#L47

[OlegDokuka]

closed since the initial PR was merged into master

[maxim-bandurko-lsvt]

@OlegDokuka Thanks for this update!