x/net/http: support content negotiation

[kevinburke]

Content negotiation is, roughly, the process of figuring out what content type the response should take, based on an Accept header present in the request.

An example might be an image server that figures out which image format to send to the client, or an API that wants to return HTML to browsers but JSON to command line clients.

It's tricky to get right because the client may accept multiple content types, and the server may have multiple types available, and it can be difficult to match these correctly. I think this is a good fit for Go standard (or adjacent) libraries because:

• there's a formal specification for how it should behave: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
• it's annoying to implement yourself, correctly; you have to write a mini-parser.
• it would take one function to implement, which makes it annoying to import an entire third party library for (assuming you find the right one)

I've seen people hack around this in various ways:

• checking whether the Accept header contains a certain string
• checking for the first matching value,
• returning different content types based on the User-Agent
• requiring different URI's to get different content.

There's a sample implementation here with a pretty good API: https://godoc.org/github.com/golang/gddo/httputil#NegotiateContentType

// NegotiateContentType returns the best offered content type for the request's
// Accept header. If two offers match with equal weight, then the more specific
// offer is preferred.  For example, text/* trumps */*. If two offers match
// with equal weight and specificity, then the offer earlier in the list is
// preferred. If no offers match, then defaultOffer is returned.
func NegotiateContentType(r *http.Request, offers []string, defaultOffer string) string

offers are content-types that the server can respond with, and can include wildcards like text/* or */*. defaultOffer is the default content type, if nothing in offers matches. The returned value never has wildcards.

So you'd call it with something like

availableTypes := []string{"application/json", "text/plain", "text/html", "text/*"}
ctype: = NegotiateContentType(req, availableTypes, "application/json")

If the first value in offers is text/* and the client requests text/plain, NegotiateContentType will return text/plain. This is why you have to have a default - you can't just return the first value in offers because it might include a wildcard.

In terms of where it could live, I'm guessing that net/http is frozen at this point. Maybe one of the packages in x/net would be a good fit? There is also a similar function for parsing Accept-Language headers in x/text/language. Open to ideas.

[quentinmit]

Funny you should mention this, I was just looking for this for use in x/perf. Your proposed API is reasonable, but you should explicitly document what happens in exceptional circumstances:

• What happens when multiple content types end up with the same q? I think I would expect the first one in offers to be chosen.
• What happen if the Accept header(s) cannot be parsed?

[kevinburke]

What happens when multiple content types end up with the same q? I think I would expect the first one in offers to be chosen.

That sounds reasonable to me.

What happen if the Accept header(s) cannot be parsed?

The spec includes a grammar but doesn't really say what happens if you can't match it, other than you should return a 406 if you can't match anything. I think we should just return the default?

[kevinburke]

@bradfitz suggested (if we want to go forward with this proposal) it could live in net/http/httputil.

[bradfitz]

I also mentioned that it seems like defaultOffer string could go away and we say that the first element in the slice was the default.

[kevinburke]

Sorry, I tried to edit the description to cover that case.

I also mentioned that it seems like defaultOffer string could go away and we say that the first element in the slice was the default.

I think the problem with this would be, if you had "text/*" as the first element in offers, it would match if the client sent text/plain and text/plain would be returned from the function.

If we return the first value in offers as the default, then the response value would contain a wildcard ("text/*"), which it doesn't in any other circumstance. Maybe that's okay.

[jimmyfrasche]

@kevinburke What's wrong with returning text/plain if the first offer is text/*? I don't see why it has to return the exact string passed in—don't you need to know the negotiated type to set the header in the response?

Also, if negotiation fails does it return the empty string? I'd rather have an error to distinguish between expected failure, negotiation failed, and unexpected failure, bad request.

[bradfitz]

@kevinburke, any response to @jimmyfrasche? This seems fine if the simplified signature works. You want to prepare a CL?

[bradfitz]

Also, we like to make sure we append "; charset=utf-8" to Content-Types to be explicit. Can we make sure that that's still automatic?

[cellfish]

offers are content-types that the server can respond with, and can include wildcards like text/* or */*

I don't think wildcards should be allowed in the offers. The server knows exactly what content types it supports and returning a wildcard content type seems weird. Nor do I think the RFC says wildcards are OK in the content-type header.
Content-Type (https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.17) mentions media-type (https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7) that does not have wild cards.
Accept (https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.1) mentions media-range which includes wild cards.

I agree with @jimmyfrasche that returning an error seems better so that the caller can know the reason if a default (first offered) is returned. maybe even not return a default at all. If there is a default the caller can handle that rather than build it as part of this method. For example there is a difference between not finding a match in an existing accept header and when the accept header is missing IMO.

Since the RFC for Accept supports multiple parameters (such as charset) how about returning something more than just a string like:

type NegotiatedContentType struct {
   ContentType string
   Params map[string]string
}

func NegotiateContentType(r *http.Request, offers []string) (NegotiatedContentType, error)

[bradfitz]

@kevinburke, what's the status here?

It can also live in x/net if you want.

It's probably too late for Go 1.10, though.

[kevinburke]

It's too late and I'm buried in work stuff unfortunately, I probably won't have time to try to offer an implementation. I remember when I tried to implement it I ran into non-trivial problems with the design and some kinds of inputs.

[agnivade]

Go for it. No need to ask for permission. :)

[rothskeller]

Permit me to note that Accept is not the only header that needs this sort of parsing. Accept-Encoding and Accept-Language have nearly the same syntax and need virtually the same parsing. A generic implementation would be welcome.

[fgm]

Notice that there is already such a negotiation in the x/text/language package, in the ParseAcceptLanguage function

[twifkak]

Has anybody made any progress on this? I'd be interested in using some of this functionality. A suggestion for an incremental improvement that could be merged first: Make a variant of mime.ParseMediaType that just parses the first media type from a list, and returns a slice containing the unparsed portion. (Basically, check for , here.)

[markusthoemmes]

I've just ported goautoneg which is used by Kubernetes and Prometheus to parse accept headers into a Github repository at https://github.com/markusthoemmes/goautoneg.

Would it be a valuable first step to add a ParseAccept function to httputil to get this going? We need to parse it first to get negotiation going anyway and it seems like these projects could take advantage of such an interim step.

[flimzy]

I'm interested in working on this, but I wonder if it makes sense to follow x/text/language's example, and provide a Matcher type, so that the provided CTs needn't be parsed for every HTTP request.

Any objections to this approach?

If so, what should we call it? AcceptMatcher?

[leighmcculloch]

@markusthoemmes I see in your port of goautoneg that you're proposing in its readme be internalized into the stdlib you dropped the Negotiate function. Many web apps are also going to need to match against a content-type they have as a supported content type and shouldn't just use the results of the parser verbatim. Providing a Matcher like @flimzy suggested would go a long way and might be a good replacement for goautoneg's Negotiate if it's going to be left out.

As an example we use the goautoneg Negotiate function here in a way that I think many web apps would.

[elnormous]

Since Go standard library (and no library I could find) does not support content negotiation that follows RFC 7231, I wrote my own: https://github.com/elnormous/contenttype
I would be happy if you could take a look at it and consider including the functionality in the standard library.

[jamietanna]

I've also written a library for this - which is BSD-3 licensed (like Go) and handles a lot of the edge cases I've hit before when writing these (more info on the blog)

gitlab.com/jamietanna/content-negotiation-go

I'd be happy to look at what we'd need to do to make this work within Go's codebase and follow the standard library's style?

[awagner-mainz]

Just for the record, I have written (in Go) a content negotiation plugin for the caddy webserver:

https://github.com/mpilhlt/caddy-conneg

It relies on @elnormous's library mentioned above, adding stuff from go's own x/text/language libraries. (Will have a look at @jamietanna's library, too - when I find the time to do so, that is.)

Thanks to you all for this issue and discussion, and for all your efforts, obviously. I am posting this here so that it may eventually provide some ideas about use cases and parameters that one would maybe like to throw at the library/libraries.

[fgm]

FWIW, I just noticed Gin provides another implementation for this, with the gin.Context.NegotiateFormat(accepted ...string) string function

https://pkg.go.dev/github.com/gin-gonic/gin#Context.NegotiateFormat

It does not handle quality factors, AFAICS, though.

[vault-thirteen]

Many people have been writing the same or similar code for their projects. Me also is not an exception. I remember writing a library for this functionality. Unfortunately, it was done for a closed-source project which is not controlled by me, so all the code is lost there. But this is not the case. I am surprised how long does it take for developers of this language to accept offers made by people across the Internet. We see that there are a lot of examples of already written code which is already working, but this issue is still opened and we do not see this fuctionality in the standard Go language.

[iamcalledrob]

I ended up here trying to find this in the standard library. Echoing what others have said, I expected it to be part of the standard library because:

1. Many applications of nontrivial complexity will need this eventually
2. The spec is tricky to implement properly

Many of the 3rd party packages linked here have a complicated API or don't implement the spec as-per the RFC, e.g. the quality scores.

For my own use-cases, the API I'd most like to see would be a single func along the lines of:

func ResponseType(accept string, offers ...string) string

// or

var r *http.Request
r.ResponseType(offers ...string) string

Rationale:

1. This makes switching on the result possible
2. Default/zero behaviour is delegated to the caller
3. Doesn't introduce errors to handle, as some packages do
4. (method on *http.Request) Doesn't require the caller to manually extract the "Accept" header value, which is tedious.

A common use case might look like:

switch r.ResponseType("text/html", "application/json") {
case "text/html":
    writeHtmlResponse(w, ...)
case "application/json":
    writeJsonResponse(w, ...)
default:
    w.WriteHeader(http.StatusNotAcceptable)
}

In case it helps anyone, I use a shim for this for my own projects as follows, using github.com/golang/gddo/httputil which seems to be a solid implementation:

import "github.com/golang/gddo/httputil"

func ResponseType(r *http.Request, offers ...string) string {
    return httputil.NegotiateContentType(r, offers, "")
}

[wwaites]

Hi there. Original author of the goautoneg library here. My apologies @markusthoemmes, I never saw your bug reports and only just today learned that it got used in things like Kubernetes and Prometheus (because I was wondering if Caddy could do autoneg). As Go developed more sophisticated and somewhat opinionated packaging infrastructure, now long ago, I probably got tired of keeping up with it. I also no longer had the original use case that made me write it to begin with.

Anyways, though it's a bit late now, but just to say that I am perfectly happy for it to get used wherever you like under some sort of free license. Sorry for stuffing license information into the README, that was silly! It would make me happy if some small piece of it made it into Go's system libraries.