Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support deep objects for query parameters with deepObject style #1706

Open
bajtos opened this issue Oct 11, 2018 · 34 comments
Open

Support deep objects for query parameters with deepObject style #1706

bajtos opened this issue Oct 11, 2018 · 34 comments
Labels

Comments

@bajtos
Copy link

@bajtos bajtos commented Oct 11, 2018

Background

Many applications expect deeply nested objects in input parameters, see the discussion in swagger-ui starting from this comment: swagger-api/swagger-ui#4064 (comment) In LoopBack, we are running into this problem too, see strongloop/loopback-next#1679.

Consider a filter parameter defined as follows:

parameters:
 filterParam:
   in: query
   name: filter
   schema:
     type: object
   style: deepObject
   explode: true
   description: Options for filtering the results
   required: false

Let's say the user wants to provide the following value, for example by entering a JSON into the text area rendered by swagger-ui:

{
  "name": "ivan",
  "birth-date": {
    "gte": "1970-01-01"
  }
}

At the moment, the OpenAPI Specification v3 does not describe how to encode such value in a query string. As a result, OpenAPI clients like swagger-ui don't know how to handle this input - see the discussion in swagger-api/swagger-js#1385 for an example.

Proposal

The following query string should be created by swagger-js client for the input value shown above.

filter[name]=ivan&filter[birth-date][qte]=1970-01-01

The proposed serialization style is supported by https://www.npmjs.com/package/qs, which is used by http://expressjs.com as the default query parser, which means that a vast number of Node.js API servers are already expecting this serialization style.

I am not sure if there is any existing formal specification of this format, I am happy to look into that once my proposal gets considered as acceptable in principle.

Additional information

Existing issues that are touching a similar topic:

  • style: form + object looses the parameter name? #1006
  • Are indexes in the query parameter array representable? #1501

Two older comments from swagger-js that may be relevant:

swagger-api/swagger-js#1140

Limitations:
deepObject does not handle nested objects. The specification and swagger.io documentation does not provide an example for serializing deep objects. Flat objects will be serialized into the deepObject style just fine.

swagger-api/swagger-js#1140 (comment)

As for deepObject and nested objects - that was explicitly left out of the spec, and it's ok to just Not Support It™.

@earth2marsh

This comment has been minimized.

Copy link
Member

@earth2marsh earth2marsh commented Oct 29, 2018

Are there any other examples of these nested deepObjects outside of Express? The more widespread a pattern, the more likely it is to be considered. Myself, I have an aversion to passing such a complicated object in the query string. Any insight into why Express even landed on this pattern?

@darrelmiller

This comment has been minimized.

Copy link
Member

@darrelmiller darrelmiller commented Oct 29, 2018

We were a bit stuck when allowing the deepObject style. We understood that users wanted this capability but there is no standard definition of what that serialization format looks like. We had a few choices, allow it and define our own standard and hope implementations followed it. Don't allow it because there are no standards, or add it in, say nothing about its format and hope that a default serialization format emerges.

If we can get some confidence that the qs package is becoming a defacto standard and we can create an accurate description of the serialization, then I have no issue recommending that we include that description in a future minor release of the spec.

@rmunix

This comment has been minimized.

Copy link

@rmunix rmunix commented Oct 29, 2018

What about the URL max length limits? I think one of the reasons why a standard for object serialization in the URL is hard to materialize is because of the URL max length problem, the URL just wasn't intended to pass data in this way. Depending on the browser and/or server software being used the URL max length varies but in general it is small when compared to how much data can be transmitted with other methods like POST or PUT. It certainly would work for small objects but people tend to inadvertently abuse these kinds of APIs by passing bigger than allowed payloads.

@handrews

This comment has been minimized.

Copy link
Contributor

@handrews handrews commented Nov 1, 2018

@rmunix regarding URL max length problems, I was very happy to see the HTTP SEARCH method draft RFC revived last month: https://tools.ietf.org/html/draft-snell-search-method-01

@louisl

This comment has been minimized.

Copy link

@louisl louisl commented Nov 13, 2018

Are there any other examples of these nested deepObjects outside of Express? The more widespread a pattern, the more likely it is to be considered. Myself, I have an aversion to passing such a complicated object in the query string. Any insight into why Express even landed on this pattern?

http://esbenp.github.io/2016/04/15/modern-rest-api-laravel-part-2/

I use a modified version of https://github.com/esbenp/bruno referenced in the article above in a few apis, it's extremely useful for including related data and search filtering. I'm not really sure how to define those in a spec. I do appreciate that the url strings could get silly long to the point of failing if abused, but without this sort of thing searches and such would have to be actioned as POST requests or limited to basic GET request params. It seems to me no ones really come up with a holy grail API solution for complex search queries so it's a bit of a free for all at the moment.

@bajtos

This comment has been minimized.

Copy link
Author

@bajtos bajtos commented Nov 16, 2018

Ruby on Rails

Seems to use the same approach.

Reference: https://edgeapi.rubyonrails.org/classes/Hash.html#method-i-to_query
Source code: https://github.com/rails/rails/blob/b5302d5a820b078b6488104dd695a679e5a49623/activesupport/lib/active_support/core_ext/object/to_query.rb#L61-L86

Example code:

require "activesupport"

data = {
  "name" => "David",
  "nationality" => "Danish",
  "address" => {
    "street" => "12 High Street",
    "city" => "London",
  },
  "location" => [10, 20],
}
print data.to_query("person")

Produces the following query string, I have urldecoded and reformatted it for better readability:

person[address][city]=London&
person[address][street]=12+High+Street&
person[location][]=10&
person[location][]=20&
person[name]=David&
person[nationality]=Danish

Notice that array items are using an empty index, i.e. person[location][]=10, instead of person[location][0]=10.

@bajtos

This comment has been minimized.

Copy link
Author

@bajtos bajtos commented Nov 16, 2018

Python 2.7

AFAICT, Python does not support nested objects in query parameters.

Example code:

from urllib import urlencode
from urlparse import parse_qs

data = {
  'person': {
    'name': 'David',
    'nationality': 'Danish',
    'address': {
      'street': '12 High Street',
      'city': 'London',
    },
    'location': [10, 20],
  }
}

print urlencode(data)

Produces the following query string, I have urldecoded and reformatted it for better readability:

person={
  'nationality':+'Danish',+
  'location':+[10,+20],+
  'name':+'David',+
  'address':+{
    'city':+'London',+
    'street':+'12+High+Street'
  }
}

Interestingly enough, the roundtrip does not preserve the original data.

print parse_qs(urlencode(data))

Outcome:

{'person': ["{'nationality': 'Danish', 'location': [10, 20], 'name': 'David', 'address': {'city': 'London', 'street': '12 High Street'}}"]}

Another example:

print parse_qs('foo[bar]=1&foo[baz]=2')
# {'foo[baz]': ['2'], 'foo[bar]': ['1']}
@louisl

This comment has been minimized.

Copy link

@louisl louisl commented Nov 16, 2018

Not API specific, but jQuery can generate nested array url params from objects.

http://api.jquery.com/jquery.param/

@Stratus3D

This comment has been minimized.

Copy link

@Stratus3D Stratus3D commented Dec 17, 2018

JSONAPI sparse fieldsets require this: https://jsonapi.org/format/#fetching-sparse-fieldsets

@benhaynes

This comment has been minimized.

Copy link

@benhaynes benhaynes commented Jan 18, 2019

Hey @earth2marsh and @darrelmiller ... We (Directus team) have been trying to use OpenAPI 3.0 for a while now, but the lack of support for nested deepObjects has kept us from using this spec. We have a dynamic API that allows for relatively complex filtering, for example: filter[<field-name>][<operator>]=<value>

GET /items/users?filter[category][eq]=vip

Our API Reference for this filtering

Is there any hope for this being supported in the future or should we "move on"?

@ewrayjohnson

This comment has been minimized.

Copy link

@ewrayjohnson ewrayjohnson commented Jan 26, 2019

@bajtos: On October 11, 2018 you wrote "I'll try to find some time to fix swagger-js in the next few weeks." What is your status on this?

@bajtos

This comment has been minimized.

Copy link
Author

@bajtos bajtos commented Jan 29, 2019

Eh, I didn't even started 😢 Feel free to contribute the fix yourself.

@darrelmiller

This comment has been minimized.

Copy link
Member

@darrelmiller darrelmiller commented Jan 29, 2019

@benhaynes Sorry if this comes across as snarky, it's not intended to, it's just I'm in a rush and I don't know how to ask this in a friendly/sincere way.

What would you like us to do? Pick a winner from the many options? Design our own format? If we pick a format that is incompatible with what you currently use, would you switch? Should we find a way of supporting many different formats?

@benhaynes

This comment has been minimized.

Copy link

@benhaynes benhaynes commented Jan 29, 2019

Hey @darrelmiller — not snarky at all, I sincerely appreciate the response as it maintains some momentum in the discussion!

We're certainly not trying to force the spec to follow our format, and understand your position of not wanting to blaze ahead without a standard to follow. To answer your question honestly, if the option you choose is incompatible with our method, then we wouldn't be able to use OpenAPI since we can't introduce a breaking change into our platform's filtering. Still, we'd support your team's decision if they think a different direction is a better solution.

I'm not sure how "extensible" your spec/codebase is, but support for multiple (even optional) serialization formats seems the most promising. Perhaps leaving these as unofficial until a "winner" eventually surfaces. In our experience, industry-leading frameworks offering solutions is the most efficient way for a de facto standard to emerge.

Our proposal is to support a deeply nested structure, such as: param[nest1][nest2]=value, where there can be 1,2,3,n levels of nesting. The comments here might be biased, but it seems that most are either already using this pattern or are recommending it.

Thanks again. I'd love to hear your (or anyone else's) thoughts on this approach!

@wellingguzman

This comment has been minimized.

Copy link

@wellingguzman wellingguzman commented Jan 30, 2019

Hey @darrelmiller, I would like to understand the OpenAPI position on this. Is the reason to not support the outcome /items/users?filter[category][eq]=vip because it's not a standard or because the parameter definition format is not part of the standard?

Also, in the technical side, will this bring a complex/breaking change and it requires much more time?

In term of where else this format is supported I would like to add another example.

PHP

Example:

<?php

$data = [
  'person' => [
    'name' => 'David',
    'nationality' => 'Danish',
    'address' => [
      'street' => '12 High Street',
      'city' => 'London',
    ],
    'location' => [10,20],
  ]
];

echo http_build_query($data);

Result:

person[name]=David&
person[nationality]=Danish
&person[address][street]=12+High+Street
&person[address][city]=London
&person[location][0]=10
&person[location][1]=20

The result is urldecoded.

Also PHP automatically parse these parameters into an array. Passing the result above into query string will result in the original array.

@ardalis

This comment has been minimized.

Copy link

@ardalis ardalis commented Feb 5, 2019

I think I'm running into this same issue. I have a simple C# type that needs two integers. I can define an API endpoint like this just fine:
public IActionResult Foo(int x, int y) {}

and it works but if I use my binding model type with those same two integers as properties:
public IActionResult Foo(FooModel model) {}

then my /swagger endpoint wants to generate JSON+Patch stuff and has no idea how to generate a URL with the appropriate querystring values to bind to that model. Will complex / deep object support help me in this (very simple) case? If not is there a known good way to support it currently?

@apimon

This comment has been minimized.

Copy link

@apimon apimon commented May 5, 2019

Arrrgh, hit that wall, too.... great example of a not so obvious limitation that might kill your whole project dev setup and workflow.

@mikena-truss mikena-truss mentioned this issue May 29, 2019
0 of 3 tasks complete
@bajtos bajtos mentioned this issue May 30, 2019
0 of 7 tasks complete
@okirmis

This comment has been minimized.

Copy link

@okirmis okirmis commented Jun 3, 2019

Ran into the same problem (using Rails), also with a parameter allowing for dynamic filtering.

The funny thing is, that swagger-editor does generate deeper nested objects as placeholder text for the parameters text area when clicking "try it out":

      - in: query
        name: filter
        schema:
          type: object
          properties:
            boolean_param:
              type: boolean
              nullable: true
            some_enum_of_types:
              type: array
              items:
                type: string
              nullable: true
          default: {}
        style: deepObject

results in

{
  "boolean_param": true,
  "some_enum_of_types": [
    "string"
  ]
}

which will be silently ignored. I know that this is actually a bug in swagger-editor, but it shows that it would be more consistent to allow deeper nested objects.

@rijkvanzanten

This comment has been minimized.

Copy link

@rijkvanzanten rijkvanzanten commented Jul 9, 2019

What needs to be done to move on with this @earth2marsh? You asked for a couple examples outside of Express, which I think have been provided in the messages above. I can help out writing some of the needed documents for this in a PR if that helps.

@darrelmiller

This comment has been minimized.

Copy link
Member

@darrelmiller darrelmiller commented Jul 9, 2019

Does someone have a proposal of how we can fix the OpenAPI specification to allow it to tell tooling which flavour of deepNesting object serialization to use?

If someone wants to tackle this, here are some things to consider when creating the proposal:

  • Which serialization flavors exist? Will they be enumerated in the spec?
  • Will it be required that all tooling support all flavours?
  • If not, what should tooling do when it hits a flavour it doesn't support?
  • Do we have commitment from tooling implementors to implement this feature?
@rijkvanzanten

This comment has been minimized.

Copy link

@rijkvanzanten rijkvanzanten commented Jul 9, 2019

Which serialization flavors exist? Will they be enumerated in the spec?

I hope I'm understanding what you're asking correctly, but based on the comments above

input

parameters:
 filterParam:
   in: query
   name: filter
   schema:
     type: object
   style: deepObject
   explode: true
   description: Options for filtering the results
   required: false
{
  "name": "ivan",
  "birthdate": {
    "gte": "1970-01-01"
  }
}

should result in

output

filter[name]=ivan&filter[birthdate][qte]=1970-01-01

The main difference seems to be in how nested arrays are handled. There seem to be slight differences between PHP, Node and Ruby on Rails.

Both PHP and qs* seem to favor the &numbers[0]=5&numbers[1]=10 syntax

$params = array(
  'filter' => array(
      'numbers' => array(5, 10)
  )
);

echo http_build_query($params);
// => filter%5Bnumbers%5D%5B0%5D=5&filter%5Bnumbers%5D%5B1%5D=10
var qs = require('qs');

var result = qs.stringify({
  filter: {
    numbers: [0, 10]
  }
});

console.log(result);
// => filter%5Bnumbers%5D%5B0%5D=0&filter%5Bnumbers%5D%5B1%5D=10

In normal non-nested use the Node built-in querystring and Swagger favor &numbers=5&numbers=10:

var querystring = require("querystring");

var result = querystring.encode({
  numbers: [5, 10]
});

console.log(result);
// => numbers=5&numbers=10

Combined by the comment @bajtos' comment about Ruby's implementation, this leaves us in a scenario where there are three different outputs for this given input for filter:

{
  "name": "ivan",
  "birthdate": {
    "gte": "1970-01-01"
  },
  "numbers": [5, 10]
}
&filter[name]=ivan
&filter[birthdate][gte]=1970-01-01
&filter[numbers]=5
&filter[numbers]=10
&filter[name]=ivan
&filter[birthdate][gte]=1970-01-01
&filter[numbers][0]=5
&filter[numbers][1]=10
&filter[name]=ivan
&filter[birthdate][gte]=1970-01-01
&filter[numbers][]=5
&filter[numbers][]=10

Seeing that different languages and implementations are already out there, I don't think we can pick one over the other. (Eg PHP's parse_str doesn't work with 1, while Ruby's won't parse 2).

qs seems to support all three of these flavors in parsing.

Re: your comment Oct 29 2018

If we can get some confidence that the qs package is becoming a defacto standard and we can create an accurate description of the serialization, then I have no issue recommending that we include that description in a future minor release of the spec.

Seeing the ~22 million weekly downloads, I think it's safe to say that qs has become the defacto query string parser in Node JS.

I don't know enough about the governance of the project nor the extended OpenAPI family of projects to make any comments about the other questions mentioned.

@philleepflorence

This comment has been minimized.

Copy link

@philleepflorence philleepflorence commented Jul 10, 2019

I concur and second, @rijkvanzanten...

&filter[name]=ivan
&filter[birthdate][gte]=1970-01-01
&filter[numbers][]=5
&filter[numbers][]=10

This format is rather important! It would allow for an array of arrays (PHP) or an array of objects (Node).

@br3nt

This comment has been minimized.

Copy link

@br3nt br3nt commented Jul 24, 2019

@darrelmiller , it seems query params could be defined in the same way as body objects by specifying a schema for a property. I'll add my thoughts about serialization after this example.

So, as an example:

params = {
  some_query_object: {
    hello: "world",
    foo: [
      { value: "blerg" },
      { value: "blah" }
    ],
    bar: [1, 2, 3]
  }
}

jQuery.param(params)

Serializes to this query string:

 some_query_object[hello]=world
&some_query_object[foo][0][value]=blerg
&some_query_object[foo][1][value]=blah
&some_query_object[bar][]=1
&some_query_object[bar][]=2
&some_query_object[bar][]=3

And, I would expect to be able to write something like this:

paths:
  "/api/example":
    post:
      produces:
        - application/json
      parameters:
        - name: some_query_object
          in: query
          description: The object in the query string
          required: false
          schema:
            "$ref": "#/definitions/an_object"
definitions:
  an_object:
    type: object
    properties:
      hello:
        type: string
      foo:
        type: array
        items:
          "$ref": "#/definitions/nested_object"
      bar:
        type: array
        items:
          type: string
  nested_object:
    type: object
    properties:
      value:
        type: string

Using this method, it would be easy to add multiple properties defined by a schema definition.

In regards to query string serialization, the only case OpenAPI really needs to account for is the various array assignment types (The python example above is clearly type: string so can be ignored).

I would suggest adding a single option arrayAssignmentType.
It only applies to arrays. Not objects/maps.
I'm not sure if it should be a global option or per property. Global should be fine as its generally enforced by frameworks/libraries.

arrayAssignmentType would have the the following options:

direct (or traditional like in jQuery param docs), where the values are assigned directly to the key:

item=a&item=b
OR
obj[item]=a&[item]=b

accessor, where values are assigned via an [] or [int] accessor:

item[]=a&item[]=a
OR
obj[item][1]=a&item[2]=b
OR
obj[item][]=a&obj[item][]=b
OR
obj[item][1]=a&[item][2]=b
OR
obj[][value]=a&obj[][value]=b
OR
obj[1][value]=a&obj[2][value]=b

The naming can be worked out later.

In regards to the accessor option, it would be interesting to see if various frameworks support both formats ([] vs [1]). I would expect the answer is both based on how JQuery serializes params:

$.param({ a: [1, 2, 3] })
"item[]=1&item[]=2&item[]=3 "

$.param({ item: [{ x: 'a' }, { x: 'b'}, { x: 'c'}] })
"item[0][x]=a&item[1][x]=b&item[2][x]=c"

I think the default value should be arrayAssignmentType: accessor just because that format is more explicit in how arrays are defined (...also "to accommodate modern scripting languages and frameworks such as PHP and Ruby on Rails" — jQuery docs lols)

@br3nt

This comment has been minimized.

Copy link

@br3nt br3nt commented Jul 24, 2019

Ok, I just tested how Rails handle the different array accessors in the query string:

?item=1&item=2&item=3
# { 'item' => '3' }

?item[]=1&item[]=2&item[]=3
# { 'item' => ['1', '2', '3'] }

?item[1]=1&item[2]=2&item[3]=3
# { 'item' => { '1' => '1', '2' => '2', '3' => '3' } }

So, @darrelmiller, following on from my previous comment, perhaps there can be 4 options for arrayAssignmentType... direct (or traditional), emptyAccessor, numericAccessor, dynamicAccessor. Again, the naming is up for debate, it's more about the concept.

Likewise, perhaps arrayAssignmentType should be both a global option and a per-property option, to account for all scenarios.

@bajtos

This comment has been minimized.

Copy link
Author

@bajtos bajtos commented Jul 30, 2019

Thank you all for pushing this discussion forward ❤️

@br3nt's proposal seems very reasonable to me.

perhaps there can be 4 options for arrayAssignmentType... direct (or traditional), emptyAccessor, numericAccessor, dynamicAccessor. Again, the naming is up for debate, it's more about the concept.

It's not clear to me what is the difference between numericAccessor and dynamicAccessor, but that's a minor detail that can be figured out while working on the actual spec proposal.

perhaps arrayAssignmentType should be both a global option and a per-property option, to account for all scenarios.

+1

Do we want this option to be per-property (i.e. inside a schema definition) or per-parameter (i.e. inside an operation parameter definition)?

I prefer the later.

If we allow arrayAssignmentType on per-property basis, then we can end up with different properties using different array assignment types. That feels too complicated to me.

?q[hello]=world
# arrayAssignmentType: numericAccessor
&q[foo][0]=blerg
&q[foo][1]=blah
# arrayAssignmentType: emptyAccessor
&q[bar][]=1
&q[bar][]=2
&q[bar][]=3

Anyhow, if there is a need for this extra complexity, then perhaps we can allow arrayAssignmentType at three levels?

  • global
  • specific to an operation parameter
  • specific to a property in a schema

The part important for me is that property-level or parameter-level setting should overwrite any globally-defined setting.

@darrelmiller

This comment has been minimized.

Copy link
Member

@darrelmiller darrelmiller commented Aug 2, 2019

Thanks everyone for the input here. This is exactly the type of information we needed, but didn't have when were originally defining the deepObject style.

I am a little concerned about the potential complexity for tooling to support a range of different formats, especially now we are talking about defining it at different levels and allowing overloading. I wonder if this is unnecessary complexity.

There are two major use-cases for being able to define how an object is serialized in the query string.

  1. In client tooling where it is necessary to construct a URL so an API can be called.
  2. In server side tooling to parse a URL and reconstruct a parameter value as an object.

If we go down the path of creating configuration parameters that reflect how a particular library/language/platform does the serialization then we are leaking implementation details through the OpenAPI description. This is somewhat defeating what OpenAPI is trying to do.

There doesn't seem to be a big a variation in formats as I once believed and I wonder if there is value in OpenAPI just picking the most compatible format and completely specifying it. From what I can tell it would be possible for us to add wording to the specification that would allow us to say deepObject follows these rules:
A parameter named q that has a value that looks like this,

{
    hello: "world",
    foo: [
      { value: "blerg" },
      { value: "blah" }
    ],
    bar: [1, 2, 3]
  }

would serialize as

? q[hello]=world&q[foo][0][value]=blerg&q[foo][1][value]=blah&q[bar][0]=1&q[bar][1]=2&q[bar][2]=3

This assumes that all numeric keys are references to elements of an array and all textual keys are references to elements of a map. The keys could be optionally quoted to allow keys for maps to be numbers.

The use of the empty indexer seems problematic to me because URL query parameters usually do not have an order that is semantically significant but array values usually are ordered.

If we assume most frameworks support both the numeric accessor and the empty accessor, then limiting deepObject to just using the numeric accessor should provide reasonable compatibility. It does appear this would be a problem for Rails. Some kind of OpenAPI specific parser would be needed to support the deserialization process. However, whichever format we pick, one or more platforms are going to have to do work to support it.

I do think that having a few platforms need to build libraries to conform to a consistent approach is a better long term outcome than creating a parameterization system that effectively describes the platform. If we did that we might as well create styles for rubyDeepObject, phpDeepObject, nodeDeepObject

I do however recognize there is a need to retroactively describe an API that has selected a non-standard serialization format. There is a solution to that. Instead of using the style parameter which is designed to be quick an easy way to identify a common serialization style, you can use content and a media type.

parameters:
  - name: q
    content:
      application/vnd.ruby.deepObject: {}

The purpose of media types is to describe semantics, syntax and serialization of data going over the wire. Normally, they are used for request and response bodies. However, OpenAPI allows defining a content object for parameters also. This also allows defining a schema if it is necessary to identify if a particular property is an object or an array when the syntax in the URL does not disambiguate.

In the past I was not a fan of OpenAPI picking a winner when it comes to the serialization format. However, considering the impact of the alternative approach, I have to wonder if it is the right thing to do. How would people feel if OpenAPI explicitly described a format for deepObject and required using content and a custom media type to describe parameters that don't match that standard?

@everlast240

This comment has been minimized.

Copy link

@everlast240 everlast240 commented Aug 2, 2019

Hey guys,
I've been trying different things for some time now, but can't get the standard JSONAPI sparse fieldsets params to work with swagger-ui...
JSONAPI specifies a way to fetch only required attributes of the included resources. E.g.:

fields[resource1]=attrib1,attrib2,attrib3&fields[resource2]=attrib1
// example:
fields[person]=first_name,last_name,age&fields[address]=city,full_address,latitude,longitude

This all works for my endpoints, but no matter how I format the payload I enter in SwaggerUI and/or setup the swagger JSON, I can't get it to work...
For example, using the endpoint(s) works like that:

curl -X GET "http://localhost:5000/person/39?include=address,address.country&fields[person]=first_name,last_name,age&fields[address]=city,full_address,latitude,longitude"

Tried different swagger JSON setups (e.g.with deepObject etc), and payloads in the swagger-ui text field (e.g.: {person: "first_name,last_name,age", address: "city,full_address,latitude,longitude"}), but no success for now...
Part of the complication is that the object / hash keys (model / resource name) are dynamic: they depend on the included resources (in the compound document) and I can't statically describe the schema definition.

Any ideas? Help is appreciated!
PS: Using Grape (https://github.com/ruby-grape/grape/) and the Netfix fast_jsonapi (https://github.com/Netflix/fast_jsonapi).

@br3nt

This comment has been minimized.

Copy link

@br3nt br3nt commented Aug 3, 2019

@darrelmiller, picking one format rather than a more versatile solution will do more harm than good.

It will alienate the frameworks that don't support whatever format is chosen. Using the example of numeric array accessors...

ASP.NET apps would be excluded as they use what I described as direct or traditional style array assignment. ASP.NETs qs format is more similar to:

?q.bar=1&q.bar=2&q.bar=3

JS frameworks which use the qs library, like ExpressJS and LoopBack, will also be excluded. As per the qs documents:

Any array members with an index of greater than 20 will instead be converted to an object with the index as the key. This is needed to handle cases when someone sent, for example, a[999999999] and it will take significant time to iterate over this huge array.

E.g:

?foo[21]=bar
{"foo":{"21":"bar"}}

However, qs supports both the direct or traditional style and the emptyAccessor. E.g:

?foo=bar&foo=baz
{"foo":["bar","baz"]}

?foo[]=bar&foo[]=baz
{"foo":["bar","baz"]}

Would tooling implements be expected to parse foo[21] as an array or a map? Is that fair? Why stop at 20? It's kind of arbitrary. At least the empty array accessor [] is consistent. It's always an array which gets populated based on the order of the query string.

PHP apps would be fine as PHP arrays work basically like hashes/maps.

Picking one option over another tells frameworks that their decisions are right or wrong. I don't think that's the job of OpenAPI Specifications. A better place to decide on standardization would be in a W3C proposal where the community could come to a consensus together.

Your concern about additional complexity is fair enough. However, I think using only content media types sweeps the problem under the rug a little. Small frameworks would be disadvantaged by this decision.

I think its reasonable to expect some minimum support from OpenAPI Specification for the array accessor styles of foo[]=a, foo[0]=a and foo=a&foo=b and it's not unreasonable to expect tooling implementers to support these three styles.

I think it would also be reasonable to assume that most (if not all) frameworks would offer consistent array accessor parsing functionality. As such a global option specifying array accessor styles would be sufficient to tell tooling which to use (rather than on a per-property or per-path basis which I agree is way too complex).

I don't think the above could be considered as leaking framework functionality into the OpenAPI-Specification.

Anything more complex than the above, then I also agree that content media types would be a good solution.

@ewrayjohnson

This comment has been minimized.

Copy link

@ewrayjohnson ewrayjohnson commented Aug 5, 2019

My apologies for not responding for while. This may seem to be to simplistic considering the complexity of other comments, but what if content (each qs param value) be subject to JSON.parse/JSON.stringify subject to URL/URI decoding/encoding as needed with whitespace removal.

For example:
{
"name": "ivan",
"birthdate": {
"gte": "1970-01-01"
},
"numbers": [5, 10]
}

can have whitespace removed as

{"name":"ivan","birthdate":{"gte":"1970-01-01"},"numbers":[5,10]}

encoded as

?param1=%7B%22name%22%3A%22ivan%22%2C%22birthdate%22%3A%7B%22gte%22%3A%221970-01-01%22%7D%2C%22numbers%22%3A%5B5%2C10%5D%7D

It may be possible that only certain characters need to be encoded (e.g. ampersand as %26)

@louisl

This comment has been minimized.

Copy link

@louisl louisl commented Aug 5, 2019

param1=%7B%22name%22%3A%22ivan%22%2C%22birthdate%22%3A%7B%22gte%22%3A%221970-01-01%22%7D%2C%22numbers%22%3A%5B5%2C10%5D%7D

That's pretty much the same as defining param1 as a string then decoding and objectifying the string which is how I'm getting around the issue for now.

The problem with that approach is there's no way in the spec to tell what a valid object/array would be for param1 or that param1 should even be an object/array.

@bajtos

This comment has been minimized.

Copy link
Author

@bajtos bajtos commented Aug 19, 2019

param1=%7B%22name%22%3A%22ivan%22%2C%22birthdate%22%3A%7B%22gte%22%3A%221970-01-01%22%7D%2C%22numbers%22%3A%5B5%2C10%5D%7D

In LoopBack, we support both style. When the parameter is described as array or object with style: deepObject, then:

  • First, we let qs to parse [] notation into an object or an array. If this returns an object value then we are done.
  • Otherwise if the query provides a string value, then we deserialize the value from JSON.

See https://github.com/strongloop/loopback-next/blob/d1eef59928fe8afb56e8bfce3d04e1e3a656ae2b/packages/rest/src/coercion/coerce-parameter.ts#L169-L200

The problem with that approach is there's no way in the spec to tell what a valid object/array would be for param1 or that param1 should even be an object/array.

Yes, the spec does not provide any means how to express this style. However, I can imagine adding e.g. a new style variant to describe JSON-serialized values.

For example:

   in: query
   name: filter
   schema:
     type: object
   style: json
   description: Options for filtering the results
   required: false
@louisl

This comment has been minimized.

Copy link

@louisl louisl commented Aug 19, 2019

Yes, the spec does not provide any means how to express this style. However, I can imagine adding e.g. a new style variant to describe JSON-serialized values.

For example:

   in: query
   name: filter
   schema:
     type: object
   style: json
   description: Options for filtering the results
   required: false

Something like that would be useful, I wonder if it could be expanded to somehow use a schema for validity of the JSON object?

@hkosova

This comment has been minimized.

Copy link
Contributor

@hkosova hkosova commented Aug 21, 2019

@ewrayjohnson

what if content (each qs param value) be subject to JSON.parse/JSON.stringify subject to URL/URI decoding/encoding as needed with whitespace removal.
For example:

{
  "name": "ivan",
  "birthdate": {
    "gte": "1970-01-01"
  },
  "numbers": [5, 10]
}

can have whitespace removed as

{"name":"ivan","birthdate":{"gte":"1970-01-01"},"numbers":[5,10]}

encoded as

?param1=%7B%22name%22%3A%22ivan%22%2C%22birthdate%22%3A%7B%22gte%22%3A%221970-01-01%22%7D%2C%22numbers%22%3A%5B5%2C10%5D%7D

I believe this particular use case is already covered by the content keyword instead of schema/style/explode.

parameters:
  - in: query
    name: param1
    content:
      application/json:
        schema:
          type: object
          properties:
            name:
              type: string
              example: ivan
          ...
@chargio

This comment has been minimized.

Copy link

@chargio chargio commented Nov 13, 2019

This would be using this feature extensively. We have many use cases where the user needs this feature

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.