Extra Fields

[richmolj]

The spec has great support for sparse fieldsets. I'd also like the opposite - rendering an attribute only when it is specifically requested.

Why might you want to do this? Well, some fields are computationally expensive, or only make sense in a specific context. For instance a Person might have a net_worth attribute that requires a number of calculations adding up assets, factoring in debt, etc.

The alternative is to make fields like this a relationship, and fetched via include. I believe this often adds additional awkwardness to the client and makes the API less expressive at best and misleading at worst.

I propose an extra_fields parameter with the same signature as fields, used to only render an attribute when it is specifically requested:

/people?extra_fields[people]=net_worth

This is currently implemented in my jsonapi_suite project.

[beauby]

How about making use of the fields query parameter for this, by introducing some kind of "relative" prefix (namely +/-)?

Example:

/people?fields[people]=+net_worth,-name

The idea is based on having a "default" fieldset, which can either be modified by +/-ing some fields, or fully specified with the current syntax.

[richmolj]

@beauby either approach sounds good to me, as long as we can get it through.

[dgeb]

The suggestion by @beauby is the one we considered initially. We pulled back on including this in 1.0 but deliberately left room for its inclusion later.

One design constraint to consider is that the spec will probably never include multi-word params to allow for API-specific params and to avoid taking a stand on separator characters.

The obvious caveat to include with the usage of + and/or - prefixes is that they could not be combined with unprefixed field names for a given type.

[ethanresnick]

+1 to @beauby's approach

[richmolj]

👍

So would you guys accept a PR with this @beauby's approach? Think we are all in agreement here.

[ethanresnick]

Hey, sorry, I'm realizing I may have spoken too soon.

I think that, if we do this, @beauby's approach is definitely the best. I also think we should definitely make explicit that, when no fields parameter is present, servers can return a default set of fields that omits any which are computationally expensive or rarely used.

What I'm not sure about is whether it's good for clients to be able to say "I want the default fieldset ± x fields"... What are the advantages of that over having clients just list their required fields explicitly? One advantage, obviously, is less typing, but anything else? I ask because, in addition to adding complexity (to the spec and implementations), supporting these requests also deprives the server of info about exactly which fields it clients are using. Of course, requests with no fields param also deprive the server of this information. Still, I think we should encourage requests that provide this information whenever possible. Why? Because knowing how many clients are relying on a given field can be a key tool for an API to use when deciding to a remove a field. Basically, the API will deprecate the field, then wait for usage to fall enough before removing it. If that field is in the 'default set', though, many responses will include that field (all that don't explicitly exclude it), even though the client doesn't need it, making actual usage of the field harder to gauge accurately.

[richmolj]

One advantage, obviously, is less typing, but anything else?

I think the caching picture is a little easier. /people?fields=+net_worth is more easily cached than the various combinations of 20 attributes.

supporting these requests also deprives the server of info about exactly which fields it clients are using...knowing how many clients are relying on a given field can be a key tool for an API to use when deciding to a remove a field

I do this in my jsorm client:

class Person extends ApplicationRecord {
  // ... code ...
  name: attr()
  age: attr()
  // etc
}

Since my models contain the whitelist of attributes, I can send this to the server in the headers (similarly, I have the client send an application name, so I can see which consumers are using which fields). Now, you may disagree with this pattern yourself, and so passing the whole field list might be best for you...but requiring the full field list opts me in to a pattern I don't want or need.

[ethanresnick]

I think the caching picture is a little easier. /people?fields=+net_worth is more easily cached than the various combinations of 20 attributes.

I think you're right that a request like /people?fields=+net_worth is probably more likely to get a cache than an exact fields combination. But just introducing this extra fields feature increases the total number of cacheable resources, and creates two different ways to request the same set of fields, which ends up driving down the cache hit rate for people who list the fields they need explicitly. In the end, I'm not sure it's a big win.

I can send this [list of fields I'm using] to the server in the headers (similarly, I have the client send an application name, so I can see which consumers are using which fields).

I really don't get the advantage of this... 1) it requires us to specify some alternate mechanism for the client to use to send the server the list of fields its actually using (i.e., the header you mention), when we already have the fields param; and 2) clients have to affirmatively start using that mechanism and, when some don't (which seems guaranteed in a public API), it further skews the usage stats the server sees.

I'm curious what @dgeb thinks but, from these arguments alone, I'm leaning toward this being an anti-pattern.

[richmolj]

which ends up driving down the cache hit rate for people who list the fields they need explicitly

This is my point - the solution is going to vary from use case to use case. My API that needs "extra fields" has two consumers, neither of which are specifying fieldsets. I would like to be able to optimize for my use case instead of being dropped into a lowest-common-denominator.

Re: doing this in the client/header, maybe a little clarification is needed. I was trying to say that this shouldn't be the concern of the the spec at all. If I want to determine the usage of my API, there are a variety of ways to solve this problem without opting everyone in to a pattern many don't currently use. Adding +/- fields would be completely optional, always requiring fields is not.

For all the above reasons, I could absolutely see this as an extension (ideally "recommended extension") once those are possible.

[ethanresnick]

Ok, cool. I'm fine with this as an extension. I just think that, if we put it in the base spec, it would be implemented and used by many people without a lot of thought about the costs, and end up doing more harm than good

[jgornick]

Glad I came across this issue. Similar conversation here: http://discuss.jsonapi.org/t/must-i-return-all-resource-object-attributes-without-a-sparse-fieldset-specified/486

I had a similar approach too using + and - to include or exclude fields from the default attribute set. There's also a concept of specifying different types of "views" for the returned attribute set.

Glad to see this might be turning into an extension.

[ahx]

We also have a use case for "extra fields". Did you folks get any more insights about what syntax you think would be better (extra_fields[type]=… vs fields[type]=+…)?

[dgeb]

I stand by my comment above: #1176 (comment)

I don't have any major concerns about just including this in the base spec.

@gabesullice any opinions?

[Doqnach]

As long as the documentation clearly indicates the + MUST be URL encoded.

[ahx]

As long as the documentation clearly indicates the + MUST be URL encoded.

I assume that clients will get this wrong all the time. Which would cause problems. It's also easy to get wrong for implementors, because "+" is interpreted as a space.

To illustrate this for example with using Ruby's Rack::Request#params:

GET /?fields[articles]=+body

would be interpreted as

{"fields"=>{"articles"=>" body"}}

[freddrake]

The specification has been clarified to indicate that the default fields sent for a type need not be the complete set of fields for the type (see Sparse Fieldsets).

Any desire for specifying a field set relative to the default field set (instead of in absolute terms) should be handled as a separate issue. (Frankly, I don't see the value, since a field need not be required in the default set.)

I think this issue should be closed.

[freddrake]

@gabesullice ?

[gabesullice]

@dgeb and I discussed this and agreed that at this time we don't believe it makes sense to add a new sparse fieldset mechanism at this time.

[hjoukl]

IIUC the downside being that if I want to get a field not in the default set I'll always need to explicitly mention each and every field from the default set:
"... If a client requests a restricted set of fields for a given resource type, an endpoint MUST NOT include additional fields in resource objects of that type in its response. ..." (https://jsonapi.org/format/#fetching-sparse-fieldsets)

I now wonder how fields plays generally with e.g. an OpenAPI description of an API: If I want to support the fields query parameter, wouldn't I then need to make all resource object attributes optional, even the ones I'd like to document as mandatory/required?

Is there any recommended way to document the fields that are usually the required response data set so that the API consumer can rely on these, unless she explicitly selects a restricted set? In other word, discriminate the response data model by "given fields" vs "no fields"...