An SSE Proposal

[robertlagrant]

Introduction

Server-Sent Events are an HTTP-based way for servers to push data to clients as "events", blessed with an official specification in Javascript on the client side. Unlikely websockets or gRPC it can't do bidirectional communication, but it also sticks to text-based HTTP rather than "upgrading" to a binary protocol.

The latter point I believe makes it suitable for description as a REST resource. For example it allows the description of complex resources such as "progress" - the server performs a long-running activity; this activity's progress can be modelled as a REST resource using SSE.

Given SSE is a free-text medium, just as HTTP is, I believe it would be wise to add certain opinions to the OpenAPI way of specifying it. This proposal is of what I think those opinions could be and why, and a possible resulting OpenAPI description.

The proposed opinions

• While SSE can send any text data, it should be limited here to JSON. This allows the usual OpenAPI descriptors to be used in describing the shape of event data. In the future XML could be added if it were popular.
• SSE has optional event names to accompany a particular data push. This allows clients to parse different event types differently, based on the event name. Event names should be mandatory in the OpenAPI specification, to allow simple generation and use of different possible events on the client side.
• SSE has an event ID to send. My feeling is this should be made available in clients if it is present, but as it is more for clients to request from a certain point in a message stream, it does not need to be specified in OpenAPI.

The proposed format

The format tries to incorporate standard OpenAPI semantics.

The only deviation is how the event type is specified, as a key under the anyOf node. I considered an x- property, such as x-event-name in the component schema, but that would be difficult with any resources that are used in multiple event types. It also makes the event name appear optional, which I think would be a mistake. Open to alternatives, of course.

responses:
  "200":
    description: "Activity in progress"
    content:
      text/event-stream:
        anyOf:
          "event_in_progress":
            schema:
              $ref: "#/components/schemas/EventProgess"
          "event_completed":
            schema:
              $ref: "#/components/schemas/EventWithCompletionData"

Additional comments:

• The use of anyOf may not be what that is intended for, but it appears to be the best fit to my current understanding.
• I wonder if each event could have its own content-type within the SSE format. That would allow any type of data in each event - clunky example:

  anyOf:
    "event_in_progress":
      eventContentType: "application/json"
      schema:
        $ref: "#/components/schemas/EventProgess"
    "event_completed":
      eventContentType: "text/plain"
      schema:
        $ref: "#/components/schemas/EventWithCompletionData"

Thanks for reading

I'd love feedback on this proposal. Very happy to work asynchonously in the Github-style, but also happy to join the calls and talk through it at some point, if there's time available in them.

P.s.

There was an issue opened on this already. I believe I've read the entire thing, but thought a new discussion might be a better (re)-starting place. Many thanks to nikojpapa's comment that the proposed format was originally based on.

[lornajane]

I'm not familiar with server-sent events, can you say something about why something like AsyncAPI for websockets doesn't serve this use case?

Is a server-sent event "response" (from server to client) sent in response to a request? If not, does the existing webhook feature work for this use case?

[robertlagrant]

Why not websockets/webhooks?

I'm not familiar with server-sent events, can you say something about why something like AsyncAPI for websockets doesn't serve this use case?

Is a server-sent event "response" (from server to client) sent in response to a request? If not, does the existing webhook feature work for this use case?

Sure! SSE is a different protocol and format to websockets, so using a websocket interface description wouldn't allow servers and clients to build SSE-specific tooling. Same with webhooks. As far as I understand it, webhooks are a regular HTTP request; just initiated by the server. SSE works slightly differently to both of these.

What is SSE?

SSE is a normal client-initiated request with multiple events being pushed down the same response. You could think of it like old-school HTTP long polling, but with a formalised API in Javascript and HTTP, so tooling can be built on top of it.

Browsers have support for it already, with the EventSource API. The API allows connection, autoreconnection if network conditions change, resumption of processing from the point the client last saw, and interpretation of SSE-formatted messages into Javascript-land.

SSE's format

The client initiates an HTTP GET request, e.g. to /progress_monitor?task_id=12345}.

The endpoint responds with a content-type: text/event-stream response. This lets the client know to expect SSE.

The SSE format at its most basic looks like this:

data: my_data

Or multi-line:

data: my_data
data: some more data
data: even more data

In each case a message is one or more data: lines, with a blank space after.

Messages can have an ID (see my first post; I would recommend that a specification such as OpenAPI mandates this, to simplify tooling creation - then a client can interpret multiple types of message based on event type):

event: progress
data: { "progress_percentage": 98}

event: progress_complete_success
data: {"new_resource_id": "12345"}

# or
event: progress_complete_failure
data: {"failure_reason": "Splines did not reticulate fast enough"}

Rationale

Why SSE at all?

• only pushes server->client, so is much simpler for this use case than a general-purpose message procotol
• has a nice and simple API on the web browser / Javascript side to consume and resume messages if a connection drops or transfers
• removes the need for long polling/comet/regular polling without totalling changing from a REST API to websockets or similar
• registers against a URL, so it can be used to describe RESTful resources - can be more structured than the more free-form general messaging cases such as websockets/gRPC
• still HTTP, so all headers work, such as accept-language and authorization
• client-initiated, so all the usual firewall rules that an application needs already will also let it through
• lighter than HTTP polling, as headers aren't needing to be resent or re-received

Why should OpenAPI support this (and why OpenAPI and not AsyncAPI):

• just regular text-based HTTP, not a separate protocol like websockets (which needs HTTPS and is a binary protocol)
• more RESTy than messaging-y
• fills a gap - describes resource types regular HTTP/REST currently cannot without polling, e.g. creating a resource that changes over time such as a progress monitor

[LasneF]

adding this level would be a kind of breaking change .

i am even wondering in fact OAS does not already supports SSE

the content type specify that it is server side event , in a way , then the schema could then just be the anyOf , one of the schema (or oneOf may be ) that would defined the schema of the data part of a SSE

that said it would requires promotion about SSE support , what is covered,
having a particular section for this specific media type , a kind of what exists for XML

responses:
  "200":
    description: "Activity in progress"
    content:
      text/event-stream:
            schema:
              anyOf: 
                - $ref: "#/components/schemas/EventProgess"
                - $ref: "#/components/schemas/EventWithCompletionData"
                

that said it is does not make it so clear ,

[handrews]

@LasneF I agree with you- this is a similar challenge to supporting streaming JSON formats (e.g. application/json-seq), which just need to be treated as a JSON array where each array element is a document in the stream.

In other words, both of these are solved by defining a mapping between the media type and the JSON Schema data model. We already do this (in a much more complex way involving the Encoding Object) for application/x-www-form-urlencoded and multipart/form-data message bodies.

Because the streaming JSON mapping is so simple, I've proposed including it in OAS 3.2 (and adding more comprehensive yet simplified mapping support in 3.3). It looks like text/event-stream is similarly straightforward, where it could just be modeled as a JSON object. Honestly, I can't tell why they didn't do that, except perhaps that JSON doesn't support comments. But for OAS, we don't care about comments either way, so treating it as if it were a JSON object that doesn't have any nesting should work.

So that could go in 3.2 easily enough, and might be sufficient to solve this problem?

[robertlagrant]

Thanks both for the commentary. @LasneF would you mind expanding on how this would be a breaking change?

[handrews]

This would be slightly more complex than I said earlier as text/event-stream allows multiple lines with the same field name. But that's also true of other things we need to model such as HTTP headers and their parameters. And technically, application/x-www-form-urlencoded is ordered and allows the same key to be used multiple times. So we might need to handle this in 3.3 when we tackle those modeling problems rather than being able to do a quick simple thing in 3.2.

[robertlagrant]

To quickly follow up on the call - thanks all for discussing this relatively niche feature!

I thought it would be good to gather the above options for how to describe SSE.

Assume JSON messages; infer event type

E.g.

responses:
  "200":
    description: "Activity in progress"
    content:
      text/event-stream:
        schema:
          anyOf: 
            - $ref: "#/components/schemas/EventProgess"
            - $ref: "#/components/schemas/EventWithCompletionData"

This is lovely and simple. The issue it has is it does not have a way to associate an event type with a particular message format. This could be perhaps convention-based, so #/components/schemas/EventProgess has event type EventProgess, although that has the small downside of making clients potentially a little brittle to changes in the API internals.

Assume JSON messages; specify event type

E.g. (may not be idiomatic OpenAPI, but hopefully it gets the point across)

responses:
  "200":
    description: "Activity in progress"
    content:
      text/event-stream:
        anyOf:
          "event_in_progress":
            schema:
              $ref: "#/components/schemas/EventProgess"
          "event_completed":
            schema:
              $ref: "#/components/schemas/EventWithCompletionData"

This specifies event types, but is limited to JSON.

Specify content-type; specify event type

E.g. (again may not be idiomatic OpenAPI)

responses:
  "200":
    description: "Activity in progress"
    content:
      text/event-stream:
        anyOf:
          "event_in_progress":
            eventContentType: "application/json"
            schema:
              $ref: "#/components/schemas/EventProgess"
          "event_completed":
            eventContentType: "text/plain"
            schema:
              $ref: "#/components/schemas/EventWithCompletionData"

This is fully flexible, but probably has the downside of clients having to decide how to deal with different content types in messages!

[handrews]

@robertlagrant how is this different from any other API's typed messages? Even if SSE is the thing defining the type, the problem of designating a particular schema as a "type" (rather than a set of constraints with no additional meaning) is a common one for APIs. I don't see why SSE should get special handling over anyone else's type system.

[disintegrator]

Hi there, wanted to share that we (Speakeasy) added an SSE feature to our code generator driven through OpenAPI. We were able to describe this kind of API neatly without funky extensions with OpenAPI and JSON Schema. You can read about it here: https://www.speakeasy.com/docs/runtime/server-sent-events#modeling-sse-in-openapi

[kevinswiber]

@disintegrator Thanks for this! I believe contentMediaType and contentEncoding were added in JSON Schema draft-07 while contentSchema was added in 2019-09. I think this means the proposed solution will only work with the OpenAPI 3.1 release line. Is that right, @handrews?

[handrews]

@kevinswiber yes, 3.1 (and be sure to read the draft 2020-12 versions- these keywords took a bit to get the explanation right).

(speaking to folks in general) We really can't do anything about 3.0. The solution to 3.0 problems is 3.1.

[handrews]

@disintegrator ironically, those keywords get confusing around multipart because contentMediaType overlaps with the Encoding Object's contentType (Encoding Object wins), and contentEncoding at least in theory could also be handled by the Encoding Object's headers field with Content-Transfer-Encoding, which is actually deprecated for multipart/form-data anyway....

This is one of many reasons we want to offer a new, refactored system for non-JSON data modeling in 3.3 (too big of a feature for 3.2). It's just too many weird bits and pieces that don't entirely fit together right now.

[handrews]

To be clear: 3.2 and 3.3 will be strictly compatible with 3.1. The existing data modeling isn't going anywhere, but we can't keep putting endless duct tape on something that's gotten rather rickety already. As seen by needing to add a ton of new explanation in 3.0.4 and 3.1.1 to deal with confusion that people have faced.

[handrews]

(speaking to folks in general) We really can't do anything about 3.0. The solution to 3.0 problems is 3.1.

You know, I said this, but now that I think about it, since we're talking to people implementing tools... JSON Schema ignores unknown keywords, so you can just throw the content* keywords into a draft-04 schema if you want. You don't get annotation results so you'd have to associate them with the correct instance locations yourself (or internally run a tool like alterschema and convert it to 2020-12), but there's no reason you couldn't just build a tool that uses those keywords if present.

Although... hmm.. OAS 3.0's schema object requires the x- prefix, doesn't it... so you'd have to use x-contentMediaType , etc, 🤦 (have I mentioned lately how much I dislike the x- prefix requirement?)

Eh, just upgrade to 3.1 if possible!

[handrews]

@disintegrator I'm going to start a new thread here so that the previous one can keep going on contentMediaType or other smaller things.

It's clear that you put a lot of thought into this, and have proven its utility through implementation. That definitely carries weight.

However... TL;DR: You've done a great job of modeling the EventSource Web API (which I'll call ESW), but the OAS Media Type Object does not operate at that level. It only operates at the level of documents (even indefinite-length ones) based on the media type format (text/event-stream in isolation, not in the larger context of SSE).

---

I'll go into more detail here, but do also read the last bit after the next horizontal divider as I talk about some other possibilities.

SSE has both an application-level interface (ESW) and a wire-level interface (text/event-stream, and maybe some HTTP usage conventions — I can't make heads or tails of WHATWG specs because they're all written exclusively for browser implementors in pseudocode).

The OAS is only concerned with the wire-level interface. We do model data based on its in-memory, parsed form, so there is some distinction between that and the wire format, most notably for form data. But even there, the Media Type Object models documents (whether finite or indefinite) of a given media type. Hence the name.

The only place where we model at a different level is the Parameter Object, and that has generally been a source of pain that we hope to address in 3.3 (modeling application/x-www-form-urlencoded should look the same whether it is in a query string or a message body, among other inconveniences).

SSE as a whole, and ESW in particular, are among many higher-level protocols/APIs/conventions built on top of HTTP. Encapsulating such higher-level usage is a great thing for an SDK to do, but the OAS does not model any of these things, and it would be an enormous undertaking and expansion of scope to do so. On the other hand, AsyncAPI is much more about these sorts of higher interfaces. My impression is that AsyncAPI isn't about non-HTTP things, it's about async things, which might be implemented on HTTP or some other protocol. If you want to model events, that's AsyncAPI.

Getting back to the OAS perspective, the only new thing for us to model is documents of the text/event-stream media type. It's quite clear that the text/event-stream format is a sequence of sets of name-value pairs (the appearance of multiple data: fields for data that spans newlines is a serialization detail and irrelevant to the data modeling, just as OAS does not distinguish between multiple HTTP headers on separate lines or with comma-separated values).

So we really would model text/event-stream as an array, which would pretty much just involve taking your schema and stuffing it under an items keyword. We would leave it to a higher level change from the document view (a sequence of events grouped by whole or partial HTTP resposne) to an event view. This is the same approach we are taking for streaming JSON formats: While the stream is just a transport packaging format (you use the JSON documents in the stream, not the stream itself), application/json-seq, application/jsonl and application/x-ndjson are sequences of JSON documents, not JSON documents.

Media Type Objects need to model media types, not an application interface built on top of a media type.

---

That said if AsyncAPI is not suitable for SSE (and I really do not keep up with that project so I'm not clear on that point), someone could make a spec that sits on top of the OAS and describes a higher-level interface. The Arazzo workflows specificaton essentially does just that, and it is better as a separate spec than as part of the OAS.

We're only just starting to figure out how to build an ecosystem of API specs that work together to model what people need, preferably without a lot of redundancy. We would not want a system that requires modeling the low-level details of text/event-stream in order to re-state most of it in a more application-level form. So there would have to be some thought as to what would be suitable there.

[handrews]

I'm also glad we got it almost right and it seems we only have a couple of small course corrections to make if your proposal described so far moves forward.

Indeed- I'm pretty sure that the direction we'll go in is to formalize treating it as an array and using the content* keywords for things like JSON or other non-plain-text values, and using null for that [DONE] token. I'll open an issue for that, and we might be able to get it into 3.2.

[kevinswiber]

@disintegrator @handrews Thanks for having this discussion. I do think it's important for OAS to represent the text/event-stream media type. It's very common to use in conjunction with more traditional API media types. It's deliberately simple. Even though AsyncAPI can model SSE, I think there's a strong case for OpenAPI Specification to support it, as well, and take advantage of the benefits the authors provided us by keeping it simple. (WebSockets would be more difficult because of the sub-protocol, but that's a different discussion.)

[handrews]

@kevinswiber definitely agree that we should have text/event-stream support. It's only the higher-level EventSource Web API interface that I think is more AsyncAPI-esque.

We can also just state clearly that a schema of items: {anyOf: [...]} (or oneOf) indicates exactly the same thing that Speakeasy's current implementation (without the items part) does. If there are use cases for not treating it like that, we may want to figure out what the relationship is. But I think the only real change from what Speakeasy has is that the "official" solution would be more verbose and more flexible (because of the ability to have more embedded data formats than JSON)

[disintegrator]

Somewhat tangential but a few months ago we had a user that wanted to generate an SDK to consume an API that streamed lines of json. Each line represented a structured log message/event.
Instead of supporting this feature directly, we advised them to adopt text/event-stream which involved prepending each line with data: and an extra newline between each line. With encoding properties I think they would have ended with a nice, complete description of that endpoint.

I think solving this modelling problem out for SSE should unblock several other streaming HTTP API formats.

context here: speakeasy-api/speakeasy#832 (comment)

[handrews]

@disintegrator is that data: [DONE] sentinel event part of a specification somewhere? I can't find it in WHATWG's specs but that's hardly surprising as I find their documents completely unreadable since I'm not writing a browser and their pseudocode is meaningless for me.

If it's not part of a specification, I hesitate to put special handling for it in the OAS. Instead, people who use such special values would be expected to model them as data, and handle them at the application level.

I fully realize that that is not ideal, and that they really make more sense as control instructions. But if they are not standardized, we can't implement that directly because someone else might use the string "[DONE]" in some other way.

[kevinswiber]

One thing that occurs to me is that, even with text/event-stream, most of the time, we're looking to model a sub-protocol, even if SSE doesn't make that explicit like WebSockets does. We might be able to leverage prefixItems to validate a sequence of messages, but really, OAS isn't exactly equipped to model sub-protocols. Thoughts?

[handrews]

@kevinswiber Yeah, I think modeling sub-protocols is out of scope. Or, if we chose to take it on, it would be a big expansion.

prefixItems and other techniques can be used to do a certain amount, and I'd say let's see how much people try to do and what limitations they hit before expanding our scope that much.

[robertlagrant]

@kevinswiber I was hoping here that the OpenAPI spec wouldn't directly be modelling the prefixes, but would instead be communicating the slightly higher-level information: what documents could you expect from this endpoint, and (perhaps) what event IDs would result in what documents. The nitty gritty of the protocol itself I totally agree is up to implementors, and given it's a simple spec, I thought it should be pretty robust.

[robertlagrant]

But, I take handrews' point from somewhere that OpenAPI is meant to be modelling what actually gets sent down the wire, and not higher-level things. It just leaves SSE in a funny spot where it'd be really easy to build tooling that interoperates, because the standard is pretty simple, but there's no machine spec standard to describe it.

[handrews]

@robertlagrant

and (perhaps) what event IDs would result in what documents.

This is the sort of thing that would fit under Moonwalk (4.0)'s concept of "signatures." But it's too big of a change to go into a near-term 3.x release. There are a lot of complexities that we're wrestling with right now in the Moonwalk meetings.

[HavenDV]

responses:
  "200":
    description: "Activity in progress"
    content:
      text/event-stream:
        schema:
          oneOf:
            - $ref: '#/components/schemas/EventProgess'
            - $ref: '#/components/schemas/EventWithCompletionData'
          discriminator:
            mapping:
              event_in_progress: '#/components/schemas/EventProgess'
              event_completed: '#/components/schemas/EventWithCompletionData'

This will keep backwards compatibility and look good.

[robertlagrant]

I was wondering about using discriminator after I posted my examples above. This looks clean; @handrews what do you think?

[LasneF]

given the fact discriminator is not "restrictive" and not determinitic toward implementation, i would avoid usage of it and keep to "pure" json schema

[handrews]

@robertlagrant discriminator is a pain because it has two forms: the anyOf/oneOf form that is more widely supported but inverts the way you usually arrange schemas (meaning that using the "child", schema does not automatically incorporate the "parent" schema, but using the "parent" schema includes all "children"), OR the allOf form that matches typical data modeling arrangements but is not as well-supported, especially if your OpenAPI Description is spread over multiple documents.

There are ways to make it work, and it's not wrong here, but I hesitate to pin more functionality on it. And as @LasneF notes, it's intended as an optimization. It doesn't really change the validation outcomes.

If you're using 3.1 and have a JSON Schema implementation supporting annotation collection, you can use annotations to determine which subschema was successfully applied, which is another way to solve the problem discriminator attempts to address.

[robertlagrant]

Yes - I think I need to ponder it a bit more. I see what you mean - it's not even just a subschema (well, maybe it is) but it's a protocol, with it's event: or data: prefixes, and then there's another schema inside that. Or even several schema, potentially of different formats!

[handrews]

@karenetheridge proposed the following in PR #4518, which I think warrants taking down that PR and starting over from this perspective:

---

Given an OAD that looks like:

...
  responses:
    '200':
      content:
        <some media type here>:
          schema:
            ...

The content under the schema keyword here MUST describe the structure of the HTTP response body as a whole, after it has been decoded into the json document model using the mechanism prescribed for this media type. The body MUST be finite, and given the assumption one has enough computing resources available (disk, RAM etc), MUST be capable of being read all at once and decoded all at once. That is, we can expect that it is possible to parse the entire HTTP body, decode it, and apply that decoded content against the json schema for validation or other purposes that json schemas make possible.

Certain media types MAY allow for the possiblity of reading in just a portion of the HTTP response body, decode that portion, and apply that portion of the content to the json schema. However, this should not be assumed -- there are many ways to construct a json schema that would make this possibility impossible (for example, "type":"array","maxItems": 10 means that we need to see the end of the content so we can be sure that the maximum number of array items does not exceed 10. Also, some media types require seeing the end of the document before we can know if the entire content is valid -- for example application/json itself and an array or object requires seeing the final ] or } character at the end of the data).

Therefore, I conclude:

• some streaming media types and content MAY be describable with the current syntax available to us in OAS v3.1.1. Users and implementers are welcome to go ahead and do so if they see a way forward for their particular usecase.
• we do not currently have a way of modelling data in which a chunk is read from a stream, decoded and validated, and then we read another chunk and validate that, and so on -- whether the stream is finite or infinite.

An alternative modelling is to have the schema model purely the JSON within the stream, and to validate the schema against each entry.

I agree we should try to do that, but not within the framework of the {"content": { <media_type>: { "schema": { ... } } } } structure. schema here MUST always describe the expected structure of the entire content; repurposing this to describe an individual parsed item in some contexts would be confusing (both to users and to tools).

So, what we really need is:

• we need to specify the media type of the HTTP request/response body itself (as found in the Content-Type header)
• we may want to specify some constraints on the body as a whole (for example, "type":"array", "minItems": 3 is easily verifiable for a finite stream, by evaluating this schema after the stream has been fully read from
• most importantly, we want to provide a schema for each individual item in the stream (not the same as a chunk, as we may read more or less than one item in each chunk, and the calling application must be responsible for concatening chunks together to create coherent items that can be validated), and this schema MIGHT be specified in addition to (not instead of) the schema of the content as a whole

Might all of the above be satisfied simply by adding a keyword itemSchema adjacent to schema? If this keyword is present, then the calling application can presume that the media type is one that can be decoded into coherent items, and that each item shall be evaluated against the itemSchema. Do we need a "streamed": true indicator to live alongside it, to indicate to the application that it should attempt to read the content in a chunked fashion, rather than wait until the end of the body is seen -- we might be able to infer that from the Content-Type itself? [edited to add: it appears that this can be determined unambiguously from the Transfer-Encoding header.]

Example:

...
responses:
  description: logging messages. may be transmitted in chunks.
  '200':
    content:
      application/jsonl:
        schema:
          type: array   # this is mandatory for the JSONL type, but is here just for clarity for the user
          maxItems: 5   # this is okay to use, because as soon as we pass 5 items this is true and can never become false again
        itemSchema:
          type: object
          properties:
            timestamp:
              type: string
            message:
              type: string

---

I've copied the proposal here so we can have any necessary further discussion before posting a new PR.

[hudlow]

I assume that each sequential media type in the registry would define what structure defined by the media type itself maps to an itemSchema in an OpenAPI document?

[handrews]

@hudlow yes.

[mikekistler]

I like the proposal for itemSchema. My only reservation is that this means there is no way to describe these streaming media types in OpenAPI 3.1 or earlier (yes, there are still some people that use OpenAPI 3.0.x and (shudder) OpenAPI v2).

[handrews]

On the plus side, this could help drive 3.2 adoption, particularly for users and tools already on 3.1 as the delta will be quite small.

Another option (since technically it would be a vendor-specific extension either way as we can't add a requirement to respect the media type regsitry in a patch release) is that vendors can use an x-itemSchema extension, just as x-webhooks was popular as a 3.0 backport of the 3.1 proposal.

[handrews]

I'm copying @hudlow's comment from the PR here so I can reply to it:

I'm also good with @karenetheridge's proposed direction although I also sort of wonder about a chunkSchema that would be modeled as an array schema and that you could use if you wanted to guarantee a whole number of items would be in a chunk and make further assertions like minimum or maximum items in each chunk.

[handrews]

@hudlow idk if you saw on Slack where I speculated about a more generalized eventSchema rather than just itemSchema (where "event" means parsing events of some sort, along the lines of event-based XML parsing, but a bit less granular as not all XML parsing events have a sensible correspondence to a schema).

eventSchema would be a map of events to schemas, or possibly for some events, a map of events to a map of qualifiers to schemas.

application/jsonl:
  eventSchema:
    sequenceItem: {this is the equivalent of itemSchema}
    chunk: {this would be the equivalent of your `chunkSchema`}
application/xml:
  eventSchema:
    elements:
      address: {schema for "address" elements}

The downside (other than the added complexity) is that for something non-trivial like XML elements, this is insufficient to specify how to serialize a value.

For chunks (either with eventSchema or chunkSchema), this would have to be tied to the HTTP chunking mechanism, and it's not entirely clear to me that people would understand that clearly. Not the least because there are several different ways that APIs can structure reading more from a stream, so it's not clear to me if we would need other options or what.

[hudlow]

Well, it won't take a lot of convincing to convince me that this additional complexity isn't something we should pursue unless someone is asking for it.

[handrews]

@hudlow the main use case I see would be for XML documents that are used similar to sequential JSON formats for streaming data, particularly as @karenetheridge's proposal formalizes (correctly, in my view) the idea that the schema applies to the entire HTTP content.

So if you have indefinite-length chunked XML content, which is not guaranteed to send well-formed documents (or even well-formed elements) aligned with chunk boundaries, then an element hook would let you process the items in the XML stream with the same behavior as the sequential media types.

My impression from a small amount of research is that such use cases are done with regular application/xml, so there is not a media type that we could associate with a generic itemSchema.

I would be interested in hearing from folks with more XML experience (@ralfhandl ?) on this. If it is a real use case, we could put some boundaries around how much complexity needs to be supported- we certainly do not need to implement a fully event-based parser for XML AFAICT.

[handrews]

@hudlow here's an example of what I'm talking about with XML, under the "streaming" hybrid model. This documentation notes that a single complete record needs to be built as a DOM tree, but each of those records can be processed independently. For this use case, an element schema based on the record element would allow validating the records as a stream without requiring the entire stream to be loaded into memory before a Schema Object can be applied.

A simple concrete example (separate from this IBM processing mode documentation) is shown here.

[handrews]

@disintegrator @ThomasRooney Regarding "sentinel" events, I am still reluctant to model more than the media type and HTTP. The itemSchema (or eventSchema) proposal relies on the media type definition, while chunkSchema is tied to an HTTP feature. Sentinel events such as data: [DONE] are not standardized. I would tend to expect something like:

text/event-stream:
  itemSchema:
    oneOf:
    - {your normal event schema}
    - const: {data: "[DONE]"}

and if you want to figure out whether you matched the sentinel or regular schema, since they are mutually exclusive you could use an annotation with a JSON Schema implementation that collects annotations to determine which was matched.

[thecheatah]

If we have an api that is returning streamed multipart/mixed responses, how would I indicate that each mixed response is application/json?

[handrews]

Your first problem is that we don't have a good way to model multipart/mixed although you can kinda fake it if you can manage a slightly non-standard use of the Content-Disposition: form-data; name=... per-part header, giving the schema property name in the name= parameter.

I tried to figure out a better way to do unnamed multipart by treating it as an array for the purpose of the Schema Object in the Media Type Object's schema field, but the problem is that the Encoding Object (which is how you set the application/json part) is not set up to work with arrays, and changing the encoding field of the Media Type Object to allow arrays is problematic. It's currently slated to be part of a larger data modeling update in OAS 3.3.

Given the difficulty with multipart/mixed, I had not thought about streaming vs non-streaming. It does not quite fit the itemSchema use case, as that depends on the parts being the same sort of value, and while you can do that with multipart/mixed, it's more common that the parts are, well... mixed.

Can you give me a bit more details on your use case? Are you using multipart/mixed as a way to stream multiple JSON documents or is it more of a mixed media type use case?

[thecheatah]

In our use case, we are returning json in each multipart response. We are not mixing media types.

[handrews]

@thecheatah it would seem to be straightforward to add itemEncoding that is to itemSchema as encoding is to schema. This would support your use case of streaming uniform parts. This could also support multiart/byteranges (see #3725) and byte range streams.

A sensible expansion to support multipart/mixed (see #3721) in general would be to also have a prefixEncoding that applies positionally (with itemEncoding applying to any parts after the last position in prefixEncoding). This is needed (while prefixSchema is not) because the encoding field lacks the flexibility of schema.

This has several advantages:

1. It follows our new precedent of itemSchema
2. It does not change the actual Encoding Object behavior, it just provides a new way to correlate Encoding Objects with data

The one wrinkle I can think of is the special-casing of using array values to produce multiple multipart/form-data parts with the same name. This comes from how web forms work and doesn't make as much sense for sequential multipart, so I'm inclined to restrict it to the encoding field. This is a bit annoying because it would appear to change how the Encoding Object works, but we could clean up the wording so that the apply-to-what rules (including the handling of arrays) go with the fields in the Media Type Object's section rather than in the Encoding Object's.