Support deep objects for query parameters with deepObject style

[bajtos]

Background

Many applications expect deeply nested objects in input parameters, see the discussion in swagger-ui starting from this comment: swagger-api/swagger-ui#4064 (comment) In LoopBack, we are running into this problem too, see loopbackio/loopback-next#1679.

Consider a filter parameter defined as follows:

parameters:
 filterParam:
   in: query
   name: filter
   schema:
     type: object
   style: deepObject
   explode: true
   description: Options for filtering the results
   required: false

Let's say the user wants to provide the following value, for example by entering a JSON into the text area rendered by swagger-ui:

{
  "name": "ivan",
  "birth-date": {
    "gte": "1970-01-01"
  }
}

At the moment, the OpenAPI Specification v3 does not describe how to encode such value in a query string. As a result, OpenAPI clients like swagger-ui don't know how to handle this input - see the discussion in swagger-api/swagger-js#1385 for an example.

Proposal

The following query string should be created by swagger-js client for the input value shown above.

filter[name]=ivan&filter[birth-date][qte]=1970-01-01

The proposed serialization style is supported by https://www.npmjs.com/package/qs, which is used by http://expressjs.com as the default query parser, which means that a vast number of Node.js API servers are already expecting this serialization style.

I am not sure if there is any existing formal specification of this format, I am happy to look into that once my proposal gets considered as acceptable in principle.

Additional information

Existing issues that are touching a similar topic:

• style: form + object looses the parameter name? style: form + object looses the parameter name? #1006
• Are indexes in the query parameter array representable? Are indexes in the query parameter array representable? #1501

Two older comments from swagger-js that may be relevant:

swagger-api/swagger-js#1140

Limitations:
deepObject does not handle nested objects. The specification and swagger.io documentation does not provide an example for serializing deep objects. Flat objects will be serialized into the deepObject style just fine.

swagger-api/swagger-js#1140 (comment)

As for deepObject and nested objects - that was explicitly left out of the spec, and it's ok to just Not Support It™.

[earth2marsh]

Are there any other examples of these nested deepObjects outside of Express? The more widespread a pattern, the more likely it is to be considered. Myself, I have an aversion to passing such a complicated object in the query string. Any insight into why Express even landed on this pattern?

[darrelmiller]

We were a bit stuck when allowing the deepObject style. We understood that users wanted this capability but there is no standard definition of what that serialization format looks like. We had a few choices, allow it and define our own standard and hope implementations followed it. Don't allow it because there are no standards, or add it in, say nothing about its format and hope that a default serialization format emerges.

If we can get some confidence that the qs package is becoming a defacto standard and we can create an accurate description of the serialization, then I have no issue recommending that we include that description in a future minor release of the spec.

[rmunix]

What about the URL max length limits? I think one of the reasons why a standard for object serialization in the URL is hard to materialize is because of the URL max length problem, the URL just wasn't intended to pass data in this way. Depending on the browser and/or server software being used the URL max length varies but in general it is small when compared to how much data can be transmitted with other methods like POST or PUT. It certainly would work for small objects but people tend to inadvertently abuse these kinds of APIs by passing bigger than allowed payloads.

[handrews]

@rmunix regarding URL max length problems, I was very happy to see the HTTP SEARCH method draft RFC revived last month: https://tools.ietf.org/html/draft-snell-search-method-01

[louisl]

Are there any other examples of these nested deepObjects outside of Express? The more widespread a pattern, the more likely it is to be considered. Myself, I have an aversion to passing such a complicated object in the query string. Any insight into why Express even landed on this pattern?

http://esbenp.github.io/2016/04/15/modern-rest-api-laravel-part-2/

I use a modified version of https://github.com/esbenp/bruno referenced in the article above in a few apis, it's extremely useful for including related data and search filtering. I'm not really sure how to define those in a spec. I do appreciate that the url strings could get silly long to the point of failing if abused, but without this sort of thing searches and such would have to be actioned as POST requests or limited to basic GET request params. It seems to me no ones really come up with a holy grail API solution for complex search queries so it's a bit of a free for all at the moment.

[bajtos]

Ruby on Rails

Seems to use the same approach.

Reference: https://edgeapi.rubyonrails.org/classes/Hash.html#method-i-to_query
Source code: https://github.com/rails/rails/blob/b5302d5a820b078b6488104dd695a679e5a49623/activesupport/lib/active_support/core_ext/object/to_query.rb#L61-L86

Example code:

require "activesupport"

data = {
  "name" => "David",
  "nationality" => "Danish",
  "address" => {
    "street" => "12 High Street",
    "city" => "London",
  },
  "location" => [10, 20],
}
print data.to_query("person")

Produces the following query string, I have urldecoded and reformatted it for better readability:

person[address][city]=London&
person[address][street]=12+High+Street&
person[location][]=10&
person[location][]=20&
person[name]=David&
person[nationality]=Danish

Notice that array items are using an empty index, i.e. person[location][]=10, instead of person[location][0]=10.

[bajtos]

Python 2.7

AFAICT, Python does not support nested objects in query parameters.

Example code:

from urllib import urlencode
from urlparse import parse_qs

data = {
  'person': {
    'name': 'David',
    'nationality': 'Danish',
    'address': {
      'street': '12 High Street',
      'city': 'London',
    },
    'location': [10, 20],
  }
}

print urlencode(data)

Produces the following query string, I have urldecoded and reformatted it for better readability:

person={
  'nationality':+'Danish',+
  'location':+[10,+20],+
  'name':+'David',+
  'address':+{
    'city':+'London',+
    'street':+'12+High+Street'
  }
}

Interestingly enough, the roundtrip does not preserve the original data.

print parse_qs(urlencode(data))

Outcome:

{'person': ["{'nationality': 'Danish', 'location': [10, 20], 'name': 'David', 'address': {'city': 'London', 'street': '12 High Street'}}"]}

Another example:

print parse_qs('foo[bar]=1&foo[baz]=2')
# {'foo[baz]': ['2'], 'foo[bar]': ['1']}

[louisl]

Not API specific, but jQuery can generate nested array url params from objects.

http://api.jquery.com/jquery.param/

[Stratus3D]

JSONAPI sparse fieldsets require this: https://jsonapi.org/format/#fetching-sparse-fieldsets

[benhaynes]

Hey @earth2marsh and @darrelmiller ... We (Directus team) have been trying to use OpenAPI 3.0 for a while now, but the lack of support for nested deepObjects has kept us from using this spec. We have a dynamic API that allows for relatively complex filtering, for example: filter[<field-name>][<operator>]=<value>

GET /items/users?filter[category][eq]=vip

Our API Reference for this filtering

Is there any hope for this being supported in the future or should we "move on"?

[ewrayjohnson]

@bajtos: On October 11, 2018 you wrote "I'll try to find some time to fix swagger-js in the next few weeks." What is your status on this?

[bajtos]

Eh, I didn't even started 😢 Feel free to contribute the fix yourself.

[darrelmiller]

@benhaynes Sorry if this comes across as snarky, it's not intended to, it's just I'm in a rush and I don't know how to ask this in a friendly/sincere way.

What would you like us to do? Pick a winner from the many options? Design our own format? If we pick a format that is incompatible with what you currently use, would you switch? Should we find a way of supporting many different formats?

[benhaynes]

Hey @darrelmiller — not snarky at all, I sincerely appreciate the response as it maintains some momentum in the discussion!

We're certainly not trying to force the spec to follow our format, and understand your position of not wanting to blaze ahead without a standard to follow. To answer your question honestly, if the option you choose is incompatible with our method, then we wouldn't be able to use OpenAPI since we can't introduce a breaking change into our platform's filtering. Still, we'd support your team's decision if they think a different direction is a better solution.

I'm not sure how "extensible" your spec/codebase is, but support for multiple (even optional) serialization formats seems the most promising. Perhaps leaving these as unofficial until a "winner" eventually surfaces. In our experience, industry-leading frameworks offering solutions is the most efficient way for a de facto standard to emerge.

Our proposal is to support a deeply nested structure, such as: param[nest1][nest2]=value, where there can be 1,2,3,n levels of nesting. The comments here might be biased, but it seems that most are either already using this pattern or are recommending it.

Thanks again. I'd love to hear your (or anyone else's) thoughts on this approach!

[wellingguzman]

Hey @darrelmiller, I would like to understand the OpenAPI position on this. Is the reason to not support the outcome /items/users?filter[category][eq]=vip because it's not a standard or because the parameter definition format is not part of the standard?

Also, in the technical side, will this bring a complex/breaking change and it requires much more time?

In term of where else this format is supported I would like to add another example.

PHP

Example:

<?php

$data = [
  'person' => [
    'name' => 'David',
    'nationality' => 'Danish',
    'address' => [
      'street' => '12 High Street',
      'city' => 'London',
    ],
    'location' => [10,20],
  ]
];

echo http_build_query($data);

Result:

person[name]=David&
person[nationality]=Danish
&person[address][street]=12+High+Street
&person[address][city]=London
&person[location][0]=10
&person[location][1]=20

The result is urldecoded.

Also PHP automatically parse these parameters into an array. Passing the result above into query string will result in the original array.

[ardalis]

I think I'm running into this same issue. I have a simple C# type that needs two integers. I can define an API endpoint like this just fine:
public IActionResult Foo(int x, int y) {}

and it works but if I use my binding model type with those same two integers as properties:
public IActionResult Foo(FooModel model) {}

then my /swagger endpoint wants to generate JSON+Patch stuff and has no idea how to generate a URL with the appropriate querystring values to bind to that model. Will complex / deep object support help me in this (very simple) case? If not is there a known good way to support it currently?

[apimon]

Arrrgh, hit that wall, too.... great example of a not so obvious limitation that might kill your whole project dev setup and workflow.

[okirmis]

Ran into the same problem (using Rails), also with a parameter allowing for dynamic filtering.

The funny thing is, that swagger-editor does generate deeper nested objects as placeholder text for the parameters text area when clicking "try it out":

      - in: query
        name: filter
        schema:
          type: object
          properties:
            boolean_param:
              type: boolean
              nullable: true
            some_enum_of_types:
              type: array
              items:
                type: string
              nullable: true
          default: {}
        style: deepObject

results in

{
  "boolean_param": true,
  "some_enum_of_types": [
    "string"
  ]
}

which will be silently ignored. I know that this is actually a bug in swagger-editor, but it shows that it would be more consistent to allow deeper nested objects.

[rijkvanzanten]

What needs to be done to move on with this @earth2marsh? You asked for a couple examples outside of Express, which I think have been provided in the messages above. I can help out writing some of the needed documents for this in a PR if that helps.

[ElectroBuddha]

Five years later, this feature request is still relevant. Any news on this ?

[jakguru]

I'll also chime in that this is a common method of building complex queries for elasticsearch. While in their case, they're expecting the body to be passed via POST, the ability to deeply nest objects in arrays allows for a huge level of flexibility in generating queries.

Also, AdonisJS uses qs style parsing

[mbraz]

waiting for "qs style" serialization feature

[darrelmiller]

@mbraz You are waiting for the feature in what tooling? Docs, testing, code generators? Here's how you could describe it in the specification.

parameters:
 filterParam:
   in: query
   name: filter
   schema:
     type: object
   style: deepObject
   x-deepObject-style: qs
   explode: true
   description: Options for filtering the results
   required: false

I just made that extension up. That's how we are going to move forward here.

The best way to get a feature into the specification is to propose an extension to a tooling creator, or PR a tool with support. Adoption of the the extension will be what demonstrates the value and allows us to bring the feature into the specification.

If you really don't want to go the extension route, you could do this as an alternative.

parameters:
 filterParam:
   in: query
   name: filter
   content:
     text/vnd.qs-serialization: {}
   description: Options for filtering the results
   required: false

In this case I invented a new text based media type for describing the contents of the filter value. Media types are not usually used for describing query parameter values, but it is technically allowed by OAS.

The important point here is that the real work is to go convince the tooling creators that this is an important feature to implement. The blocker is not the specification.

[mbraz]

Hi, wait in swagger to use in NestJS applications. I'll try text/vnd.qs-serialization... Em dom., 5 de mar. de 2023 às 14:34, Darrel ***@***.***> escreveu:

…

@mbraz <https://github.com/mbraz> You are waiting for the feature in what tooling? Docs, testing, code generators? Here's how you could describe it in the specification. parameters: filterParam: in: query name: filter schema: type: object style: deepObject x-deepObject-style: qs explode: true description: Options for filtering the results required: false I just made that extension up. That's how we are going to move forward here. The best way to get a feature into the specification is to propose an extension to a tooling creator, or PR a tool with support. Adoption of the the extension will be what demonstrates the value and allows us to bring the feature into the specification. If you really don't want to go the extension route, you could do this as an alternative. parameters: filterParam: in: query name: filter content: text/vnd.qs-serialization: {} description: Options for filtering the results required: false In this case I invented a new text based media type for describing the contents of the filter value. Media types are not usually used for describing query parameter values, but it is technically allowed by OAS. — Reply to this email directly, view it on GitHub <#1706 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA23VEFTQ4RDD3XPBOU4H5DW2TFDTANCNFSM4F257XBQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[handrews]

I think the simplest way to allow folks to experiment with different approaches would be to implement #1502 (Support for arbitrary query strings). That would let folks define entirely custom serialization, perhaps based on media types as @darrelmiller suggests, without having to work around the spec's mechanisms for supporting more common serialization forms.

[POMXARK]

https://github.com/abbasudo/laravel-purity

This is extremely necessary! I can't convert to format query parameter

filters[is_publish][$contains]=true

Terrible. An outrageous flaw.
And no one has undertaken to fix this with at least a third-party package

[dafeder]

@POMXARK let's keep it respectful, the maintainers don't owe you anything. You're welcome to submit a PR or make and share a third-party package if this is so important to you, we're all just doing our best here.

FWIW I'm not a JS developer but I would still strongly vote for standardizing around "qs style" as it's very intuitive and close enough to other serialization methods to work with in different languages. It should be recognized though that there is really no way to do nested query objects in a GET query string that will be straightforward to document.

[handrews]

@dafeder thanks for your comment!

It should be recognized though that there is really no way to do nested query objects in a GET query string that will be straightforward to document.

Yeah, this is why I'm pushing for #1502 in version 3.2 of the spec. It's not ideal, but it gets the OpenAPI spec details out of the way and lets folks do their own serialization for the entire query string, including by just setting the whole thing to content: {application/x-www-form-urlencoded: {...}} and serializing the way we do application/x-www-form-urlencoded bodies. Which would also allow creating 3rd-party extensions to handle alternate query string formats, which is not possible right now because there is no place in any Object where you could easily put such an extension.

[dafeder]

@handrews yes that makes a lot of sense, the problem is really more on the SwaggerUI side. As a spec I can see this being the right way to do it (and clarifying this will probably help Swagger handle it better).

[DustinCai]

Any updates on this?

[handrews]

@DustinCai see my comment above about 3.2. We're getting 3.0.4 and 3.1.1 out the door right now, then 3.2 should be a relatively quick release as we want to keep it small (and then do a 3.3 if 3.2 is well-received). I'd hoped to get 3.2 done by the end of the year, but 3.0.4 and 3.1.1 have taken a bit longer so maybe January or (hopefully) at worst February.

I know that solution (#1502) might not be what you're looking for, but there are too many possible different formats and too many competing demands. Allowing people to define their own query string formats (media types) for the whole query string will unblock experimentation, particularly if combined with a media type registry per #3771.