Appendix on RFC6570-derived behavior + allowReserved (3.0.4)

[handrews]

Rendering with all currently open 3.0.4 changes (as of the timestamp of this edit - use the dropdown on the word "edited" in the bar for this comment) can be found here.

NOTE: The rendering will have mismatched table of contents lettering, which will be fixed when I merge the various appendix PRs and resolve conflicts along the way.

Addresses:

• How are conflicts between allowReserved and RFC 3986 handled? #3759 (by explaining that allowReserved does not make illegal reserved characters legal)
• rfc6570 template clarification requested for the parameter styles, cookie use case unclear #2940
• Mixed type array with array of objects in location path  #2276
• Parameter style definitions inconsistent with the examples #1945 (at least partially)
• More and complete examples for serialization styles learn.openapis.org#100 (just the part about manual assembly because ?+ is not a valid operator)

This is the first of several appendixes to be added to clarify the most obscure details of Parameter Object and Encoding Object serialization. Future PRs will add more on serializing headers and cookies, and on the incredibly convoluted topic of percent-encoding.

Why an appendix? Because these are really fiddly details, and technically they all follow from the existing requirements. There will be other clarifications in the main text. I just need to get this really fiddly stuff out of the way so it's easier to write the major parts.

Why not put this on the Learn site? Because this is part of a larger set of serialization issues that took me several weeks to sort out, despite being very comfortable researching and reconciling RFCs in general, and being very familiar with the relevant RFCs in particular. It's too high of a bar to relegate to secondary documentation. It needs clarity in the spec, IMNSHO. [EDIT: However, the directly RFC6570-analogus parts are probably the most straightforward part of this- the complexity is in the exceptions and in related areas that I have not yet posted as PRs.]

This clarifies the correspondence between OAS fields and RFC6570 operators, and acknowledges that some field values and combinations do not have analogues. It provides further guidance for how to use RFC6570 implementations to support these configurations.

This includes a SHOULD directive regarding using RFC6570 expansion with the non-RFC6570 styles, as the use of "explode" and "allowReserved" does not otherwise make any sense. It perhaps could be a MUST.

[ralfhandl]

+1, one phrasing suggestion

[handrews]

@darrelmiller after thinking about your comment in OAI/learn.openapis.org#100 I'm wondering if maybe I'm not giving the right guidance on the combination of style: form, allowReserved: true.

In this PR, I basically suggest using single-variable templates like "{+foo}" and assembling the query string with the results. Alternatively, you could assemble a template using a mixture of RFC6570's ?, &, and + operators, and pass that whole thing through to an RFC6570s process. Which you could also do for pipeDelimited, spaceDelimited, and deepObject.

I'm going to move this to a draft PR because I think this needs some consideration.

[darrelmiller]

RFC 6570 refers to explode as a modifier because it is a suffix added to the variable as opposed to an operator which appears as a prefix.

[handrews]

Ah, yes. I was just going by the characters, not the placement, but you're right that it should be more clear.

[handrews]

I changed the headings to make things more clear in the latest reworking of this PR.

[darrelmiller]

{?foo*}{&bar}
Is this even a valid URI Template? If foo is empty/undefined then the resulting href would be invalid.

[handrews]

Yeah that's kind of my point. I mean, it's a valid URI Tempate in the sense that it's syntactically legal (I think?) it just might not produce a valid URI. The URI Template spec is pretty intentionally blind to the context outside of the variables.

But perhaps I could note that that's why you need to do it the {?foo*,bar} way.

[handrews]

I think I've addressed this in the latest, much-expanded version.

[handrews]

Based on @darrelmiller 's comments and trying to think this through further, I have significantly expanded this and changed the recommendation regarding how to construct query strings with configurations that don't quite match RFC6570.

I've also rewritten the allowReserved section and incorporated that aspect in the appendix. I had planned to address that in another appendix on percent-encoding, but this allows readers to find all query string-related RFC6570 content in one place, and will let the percent-encoding appendix focus on the confusing interoperability landscape there.

Finally, the appendix now has several examples showing both easy and difficult cases.

[darrelmiller]

Applications are still responsible for percent-encoding reserved characters that are not allowed in the query string

I'm not sure how to interpret this. Is it saying applications are responsible for characters that only reserved in the query string? Or is it talking about parts of the query string that are are not parameter values?

[handrews]

The upshot is that if you use allowReserved: true, you still need to manually percent-encode the following reserved characters:

• # because it is the delimiter between the query and fragment
• [ and ] because while they are reserved, they are forbidden in the query by RFC3986 (yes, deepObject examples are wrong, see Appendix for percent-encoding concerns (3.0.4) #3859 for more on this)
• + because form-urlencoded treats it as a space
• & and = because form-urlencoded uses them as delimiters

You can leave any of the others unencoded, and you MUST leave a + that represents a space unencoded. Which maybe I did not show in the example and should have... hm....

[handrews]

@darrelmiller I'm open to suggestion on how to better state this. At one point I had a short paragraph after thist table going into a bit more detail, and then moved it all to the appendix. Perhaps a short paragraph is warranted, though.

[darrelmiller]

I don't think it is necessary to set allowReserved to true.

            var template = new UriTemplate("http://example.org/{?foo%2Fid}");
            template.SetParameter("foo%2Fid", "bar/");
            var uriString = template.Resolve();
            Assert.Equal("http://example.org/?foo%2Fid=bar%2F", uriString);

I tried this using my Tavis.UriTemplates library and it passed without having to use allowReserved.

[handrews]

Yes, I tried two Python implementations and they agree, sse also this comment about the larger context

[darrelmiller]

It isn't clear to me why we need the + for the formula names. Is this intended to cover the deep object scenario where someone uses [ ] characters in the names.

[handrews]

It is not, it was about names that aren't valid template variable names. Which would include the deepObject-style names and, yeah, that could be interesting, but... UGH. deepObject. idek know what to do about that. See #1322 - I can't entirely figure out how it's even supposed to work. I think I'm going to ignore it for now as I'll need to fix several deepObject things as it is. I've been putting it off to the last thing.

[darrelmiller]

I'm not sure I understand the advantage of that manually constructed URI template over

?words={wv0} {wv1}

[handrews]

It's a good question and relates to your other two comment about not needing to set allowReserved for names with percent encoding and not needing the + on the formula names.

My original thought was "simulate the normal behavior of parameter names in form by putting them through the same process as the values" which led to "if you need pre-percent encoded values in names then you'll have to set allowReserved: true to get them.

But since that seems not to be true (two Python URI Template implementations both agree with you on that), then there's no need to use allowReserved and therefore no need to make the names template variables since they have to be pre-encoded anyway. It's simpler to just copy them as part of the literal portion of the template. I'll make an update taking all of this into account.

[darrelmiller]

This is a really interesting exploration of doing "late generation" of URI templates based on the parameter values to work around the features in OpenAPI that are not supported by 6570.

[handrews]

@darrelmiller I've added another commit that hopefully addresses all of your concerns.

[handrews]

@ralfhandl I've started a pre-release checklist and put the RFC links on it:

• Style guide / release checklist for the specification #3785 (comment)

I'm not going to change them in this PR because it would just be added noise as we don't know the target yet (and I do not want to hold this PR up for that decision- we need to do that all at once, like fixing the section links).

[handrews]

This now has 2 tsc approvals, one of which was cleared by the merge conflict resolution which was trivial. So I'm going to merge this.