media-params: get rid of string/token distinction

[seliopou]

From RFC7231§3.1.1.1:

A parameter value that matches the token production can be transmitted either as a token or within a quoted-string. The quoted and unquoted values are equivalent. For example, the following examples are all equivalent, but the first is preferred for consistency:

text/html;charset=utf-8
text/html;charset=UTF-8
Text/HTML;Charset="utf-8"
text/html; charset="utf-8"

The parser now simply returns a string and serializes to a token or a quoted string depending on whether the parameter includes characters that require escaping.

[rgrinberg]

@dsheets this is your territory I believe

[dsheets]

Unfortunately, I believe that Stringext.quote is not the right function and this code has never been correct with respect to a total function for encoding. https://tools.ietf.org/html/rfc2045#section-5.1 production tspecials gives the set of characters that must be in a quoted-string production and cannot be in a token production. I believe quoted-string from https://tools.ietf.org/html/rfc822#section-3.3 is still the normative specification for quoted string text and I don't believe the parser/printer currently follows that grammar correctly.

Beyond that, I think Postel's Law is applicable and we should be ready to accept (or allow users to accept) either a token or a quoted string for parameters that can be tokens or are specified as tokens (as RFC 7231 specifies for HTTP). On the flip side, I think the user should have control of the encoding of parameters and be able to specify that even parameters that could satisfy token get encoded as quoted-string. Perhaps this is not relevant in the context of modern HTTP but, as far as I can tell from my brief search, there is no general media type requirement to allow token parameters when quoted-string is unnecessary. I believe media type registrations (or unregistered real world use) can require some parameter values to always be quoted.

Finally, I'd really love to see a stand-alone library just to handle media types. @dinosaure's MrMime also contains parser/printer/types for media types and it looks like it does distinguish strings and tokens at the OCaml type level. Though I do love these specs and really want a highly focused media-type opam package, I don't think I'll have time until July or August to even consider writing one. I will definitely contribute and heavily use such a library in my own work if someone creates it (maybe me in August...).

@seliopou thanks for your interest in improving this functionality. Unfortunately, I think it's always been defective (and is still defective with your patch) and really needs to be factored out of cohttp and MrMime into a stand-alone component that is definitely correct, user friendly, and has no unnecessary dependencies or functionality. Hope this helps.

[dinosaure]

Hmmhmm, I think it's possible to extract from MrMime the part about the RFC 822 and the RFC 2045 (and extract the Content-Type). The good thing is: MrMime use the same abstraction than angstrom and this part of MrMime don't want to know the type definition of Angstrom.t (in some cases of MrMime, it's mandatory).

So, I don't know the plan about cohttp and httpaf but if @dsheets wants a light library just to compute the Content-Type, I can create that (with the angstorm dependancy) - however, I can't make a common part with MrMime because I need to know the type Angstrom.t (and in angstrom, it's an abstract type).

[dinosaure]

See https://github.com/oklm-wsh/TypeBeat.

It's not available on opam but if you want, I can make a release. I'm lazy to integrate the library in CoHTTP but the API was thinked to be easy to use (I think).

[avsm]

Please do make an opam release @dinosaure!

[mseri]

Typebeat has been released (see https://github.com/ocaml/opam-repository/tree/master/packages/typebeat). Does this PR need refresh and inclusion? Or should we just close it?

[dinosaure]

For my perspective, I agree that we should (according to the RFC) split the MIME type, typebeat on the other side is dead (and I have some other plans about this specific part). I can try to revive this PR if you want.

[mseri]

I am happy wither way. We can also postpone this change if you think it would take too long and it needs to be rethought carefully.

[dinosaure]

Close by #725