Protocol: Support for HTTP trailers

[sm-Fifteen]

It doesn't seem like the current ASGI specification allows applications to send HTTP trailers in their response, since it is indicated that servers should ignore any further messages on a channel once mode_body becomes false in a http.response.body message. This prevents application that stream their content from sending additionnal response information as trailers when it may not be known before the body has begun transfering, regarding things like caching, timing (Server-Timing), integrity check (Content-MD5, obsolete), digital signature (Signature, draft), etc.

WSGI doesn't support it and I'm reading that overall support for them in various user agents tends to be flaky most of the time, but I couldn't find any information regarding the omission of trailers from ASGI (whether deliberate or not) and I would like to know if there are any plans to add them at some point in the future, perhaps as an extension.

[andrewgodwin]

I considered adding them into the spec when I was writing it, but as you say, the user agent support is very spotty. ASGI in general is meant to be a spec for "95% of web interactions" (much like WSGI before it), and I think trailers are definitely less than 5% usage out in the wild.

If it had any significant usage I'd consider adding it, but as it stands, it's such a specific feature unused by anyone that if someone wants it, they probably want low enough control over HTTP that they need to be doing their own HTTP transport code anyway. Given that, I don't really want to add it into ASGI.

[sm-Fifteen]

Yeah, I figured as much. Thanks for giving me the official word on this.

[pgjones]

I've been asked about trailing headers as a Hypercorn issue - do you think it could be supported as an ASGI extension?

[andrewgodwin]

@pgjones If gRPC uses them as that issue suggests, then that would make me think there's some value in adding support.

[cetanu]

@pgjones If gRPC uses them as that issue suggests, then that would make me think there's some value in adding support.

This doc may assist:
https://chromium.googlesource.com/external/github.com/grpc/grpc/+/HEAD/doc/PROTOCOL-HTTP2.md

[synodriver]

It would be great if asgi have grpc support. I'm considering writing an extension for that. Probably like

"scope": {
    ...
    "extensions": {
        "http.trailing_headers.send": {},
    },
}

then use await send({"type":"http.trailing_headers.send","headers":[(b"grpc-status",b"0")]})

Is it ok to add an extension like this?

[simonw]

I only just found out HTTP trailers are a thing - I have a few use-cases for them in mind for Datasette now.

• Datasette can use streaming for large CSV responses, but an interesting question there is what should happen if an error happens later in the response (malformed data in row 10,000 of a table that is being streamed out, perhaps). Being able to use a trailing header to indicate that the stream was invalid would be interesting - see also: https://simonwillison.net/2021/Jun/25/streaming-large-api-responses/#challenge-errors
• I wanted to add timing information about how long the page took to render. I came up with a devious hack where I ASGI attach a <script> block at the end of the page, but a trailing header would be a nice complement or alternative to this: https://github.com/simonw/datasette-total-page-time
• Generally it feels like there are all sorts of interesting debugging/diagnostic things one could do with trailing headers

[andrewgodwin]

Trailers are a really, really gnarly part of the HTTP spec, though. I resisted adding them to ASGI for all this time as they're something 99% of applications won't need, and I don't even know if they will work right with proxies (much like why I held out on raw query strings for so long).

If we can get some good background information on them being reliably transported, useful enough they're worth implementing, at least one server implementation to say it's not that hard to write, and a decent proposed API for them, then it might make sense to have them in ASGI.

[synodriver]

Well, if you really need that, I have already made a personal fork of hypercorn to support trailers. And some of my home-made extensions. Just look into the code . I use it to serve asgi app together with grpc services.

[Kludex]

@andrewgodwin Given your last comment, should we reopen this issue?

[andrewgodwin]

Yes, the use of them by GRPC is probably enough of a reason at this point - trust those folks to go resurrect a dusty old part of the HTTP spec :)