-
Notifications
You must be signed in to change notification settings - Fork 874
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RFC: Exluding fields with Sparse Fieldsets #1632
Comments
Thanks a lot for this proposal and the clear structure in which it is provided. Not being able to request all fields of a resource without knowing them upfront has been raised as a limitation of the JSON:API specification often. Mainly motivated by discovery. Often in context of questions if server may response with a sparse fieldset by default if Requesting specific fields using an exclude lists seems reasonable as well. I'm not aware of any requests for such a feature though. We should be careful to not specify something without actual use case. Otherwise it may just not be implemented by most libraries. I find the syntax Some examples to illustrate that idea: Request all fields of a resource GET ?fields[post]=* Request all fields except GET ?fields[post]=*,-text Request that server excludes GET ?fields[post]=-text If adding to base specification, it must be optional. Otherwise it would be a breaking change for any server-side implementation. That means a server may support sparse fieldsets but may not support An alternative may be adding the feature using an extension. Haven't fully thought through that one yet. |
I have doubts about the use of the So basically, maybe we can do something like this: Request all default fields (same as not sending the GET ?fields[post]=* Request all default fields and also GET ?fields[post]=*,field_18,field_19,field_20 Regarding the exclusion of certain fields, I'm not sure if we should provide a way to do it, as maybe the use of the With this approach we still don't have a way to discover all available fields, but I'm not sure if the |
We too need this feature. However, I find the Suggestion: Current specification specifies Exclude All Except Given fields. Similarly, we can have a Include All Except Given by using a |
Thank you all for your feedback so far. Let me add a few details to explain the intent behind the proposal. Given the current spec, omitting the
as @jelhan pointed out, I also think that
I guess we can all agree on the fact that the stability (or idempotent result) of the structure of a requested resource object must be guaranteed by a well maintained, documented and versioned API: The list of fields returned with a resource object should only change across major versions, if the However, there are use cases where the client needs the server to explicitly agree on the list of fields that should be returned with the resource object.
is a contract for including no fields in the resource object, i.e. send the resource object with an empty
include the specified fields SyntaxNow, by allowing the client to exclusively use a wildcard instead of a definition of the list of fields, the server MUST agree on the following: Given the current version of the server API, include ALL fields in the resource object that could also be requested individually by the client. Example: If the resource object exposes the fields
MUST return the fields The ASTERISK is used here as as a representative for the ALL-semantics, since this character is already a known and accepted character for this purpose in informatics, so developers are given known tools at hand. For example, SELECT * FROM table requests a result sets containing all the fields from a table in sql. If this syntax exists for including ALL fields of a resource object, we could expand on this by allowing to submit a list of fields which denotes the list of fields that should be EXCLUDEd from the resource object:
(I do not provide a formal definition for WORD, but I'm sure you get the idea. In short:
Given the above example, where the resource object exposes the fields
reads as
Marking fields as excludedI tempered around with marking fields as excluded, but I thought that the syntax becomes too ambiguous. @jelhan thought about introducing a hyphen
whereas @jugaadi proposed to introduce the "inversion" of the
(note the hyphen Both examples provide the same meaning, just with a different syntax: EXCLUDE Both examples make constructs like this possible - and, given my own experience - more likely to happen:
and:
The question here is: what's given precedence, if any? Of course, edge cases like these can be clearly stated as invalid and require the server to respond with a
are also possible and should be responded with a Drawbacks of the Wildcard-characterAcross versions, a contract between the client and the API regarding ALL fields given a wildcard character is not stable. Given the fact that fieldsets may change across API versions anyway, I think this is negligible. For discussionThe following points could also be up for discussion:
|
Thanks for the proposal @ThorstenSuckow and for the discussion everyone! I can appreciate the benefit of allowing fields to be specified that are relative to a default fieldset in addition to allowing only a fixed set of fields. It seems important that relative fields not be mixed with fixed fields to simplify processing rules and clarify intention. I've considered a syntax to handle this over the years. The cleanest approach that I can arrive at is to require that For example, if
However, mixing of relative and fixed fields would be invalid:
Does anyone have concerns with this slight modification to the proposal? |
Hey @dgeb, thanks for the response. I think some kind of indicator to request ALL (i.e. a disjunction of default & optional) fields of a resource object would remove some of the opaqueness of the current specification. Other than that I think your modification fits the use case and requirement very well as it provides a clear definition of how a client could add more granularity to its requests: A sparse fieldset MUST either contain only fixed fields, or only relative fields. FIXED FIELDSETS denote the list of fields that should exclusively be returned with the response object and are already specified with Sparse Fieldsets. RELATIVE FIELDSETS denote the list of fields which are optionally included with or excluded from the resource object, based on a default fieldset defined by the implementing API, and as such available with its documentation (see addendum to this comment). Fields which should be optionally included MUST be prefixed with If a relative fieldset is specified and the server cannot include a requested field, the server MUST respond with a If a relative fieldset is specified and the server fails to identify an excluded field, the server MAY respond with a AddendumI think this would even more justify an endpoint for discovering the default fields of a resource object, e.g.
NotesDoes the current specification regarding an empty No fields returned:
Fields returned relative to the default fieldset:
The intention is clear. Is the language for transposing this concept clear enough? |
One of my concerns with this is that, for consistency, such a wildcard would also be allowed to be specified on its own to request default fields, which would make explicit what was formerly implicit. Clients would then be tempted to specify Also, I'm not sure that For these reasons, I'd just as soon leave this additional complexity out of the proposal.
I am uncomfortable adding these requirements for relative fieldsets that are not imposed upon fixed fieldsets. There are times when a set of fields may be requested but are not available to a particular user, particularly due to authorization concerns. The current approach allows servers to omit some requested fields without forcing the client to have full knowledge of what fields are authorized prior to making a request.
That seems like a reasonable use of the description document referenced by
There is certainly some nuance here. I am trying to be accepting of the general proposal, since it's come up many times, without breaking backwards compat in any way. |
If this turn out to be a breaking issue, can we defer this from 1.1 now and consider in future? |
I think there are different understanding of what
The default fieldset could be a subset of all fields a resource has. So both could have very different meaning. This discussion seems to assume that the default fieldset us a fixed one. That is not guaranteed by the spec. If client does not request specific fields explicitly a server may response with any fieldset. It could even pick the fields to be included randomly. Any specification of a relative fieldset can only specify that specific fields must be included ( Looking at the many different proposals each having its own trade-offs, I think it would be best to experiment in user-land using an extension first before adding anything to the base specification. |
I think this is a reasonable solution. And with v1.1 so close, it is probably the only practical one for now. |
The idea behind the wildcard
Yes, default and optional :)
That is a valid point and I haven't thought about this.
Thanks for pointing this one out! |
If you have already an idea for the syntax being used, I'm eager to adopt this since I'm currently in the process of rewriting an API that is in the need of fields being excluded from requests. If there's anything I can help with or contribute to the project/docs, let me know! |
Extensions, which will be included in upcoming v1.1 version of the spec, allows to extend base spec at user-land. You do not need to wait for any editor of the base spec to write an extension. You could write, publish and extension yourself. The same applies to profiles. Actually that's the main idea behind it: Unblock consumers to extend (extensions) and further specify (profiles) JSON:API without editors of the base spec being a bottleneck. I'm happy to pair on public API and review a draft for such an extension. Feel free to send me a direct message on Twitter if you like. |
Hey @jelhan, I have collected the feedback from this discussion and created a draft for this feature over at https://github.com/ThorstenSuckow/relfield:
Let me know what you think! |
Hey all! I have updated the specifications with the Providing full compatibility with the fieldset's base specification, the extension should now cover most of the use cases if excluding/adding fields is required by a client. If prefixes are omitted with the The current draft can be found here: https://github.com/ThorstenSuckow/relfield Up for discussion: The draft allows for mixing This means, given the current draft:
must yield a
would be allowed. |
@auvipy |
Hey there, given some feedback I have updated the draft for this extension. The changes include:
Thanks @jelhan for your feedback and contributions so far ! |
JSON:API RFC: Exluding fields with Sparse Fieldsets
Introduction
This RFC proposes an addition to the syntax used with Sparse Fieldsets, namely to add the asterisk character (U+002A ASTERISK, "*") to the
fields[TYPE]
parameter, allowing forand
Proposal
The following is defined in the JSON:API specification (V1.0, also upcoming V1.1) by the time of writing this RFC:
and
furthermore:
While the specifications gives the client a clear indication of how to define the fields included with the resource object in the response, it is not possible to fall back to a set of "default fields" that exclude a specific set of fields at the same time.
The only option to fall back to a predefined list of fields is by omitting the
fields[TYPE]
parameter. The implementing API will then take care of computing the fields to return; this may also lead to no fields being returned at all, as per specification.Drawbacks of the current specification
The current specification makes it hard to maintain large set of fields which are part of larger resource objects, specially DTOs/value objects that aggregate data without providing the option to query related resources for data.
Take for example a fictional document represented by a resource object that has a list of 20 fields,
field_1 .. field_20
. The computing of the values forfield_18 .. field_20
are costly and require complex operations that produce more load on the server: The client uses these fields only on some occasions, e.g. in master/detail views: A lean representation of the resource object in a grid vs. the resource object loaded in its entirety for the detail view. Instead of designing the API to use specifics of the represented entities and provide different resource locations for the resource object (Object A containingfield_1 .. field_ 17
, Object B containingfield_1 .. field_20
) , sparse fieldsets are used.The current specifications require the following syntax to make sure
field_18
,field_19
andfield_20
are NOT included with the resource object in its response:The same could be achieved with the asterisk character (U+002A ASTERISK, "*") character by inversing the list of included fields, and providing a syntax for excluding fields:
Changes that affect the current specification
This changes the specification of the JSON:API in its current version to the following:
BEGIN
The value of the
fields
parameter MUST be a comma-separated (U+002C COMMA, ",") list that refers to the name(s) of the fields to be returned. An empty value indicates that no fields should be returned.If the value of the
fields
parameter is not a comma-separated list of fields, and it is not empty, it MUST be an asterisk (U+002A ASTERISK, "*"), or a comma-separated list of fields starting with an asterisk (U+002A ASTERISK, "*"). The asterisk MUST be treated as if the client had specified ALL fields of the requested resource object, optionally followed by the list of fields that MUST NOT be included with the resource object in its response.END
The following lets the server decide which fields to return with the resource object as per current specification:
The following extends the current specification by requesting ALL available fields of the resource object by using a wildcard with
fields[TYPE]
The following extends the current specification to return the resource object with all available fields, excluding the
title
field, which MUST NOT be included in the resource object in its response:Additions
According to https://www.rfc-editor.org/rfc/rfc3986#appendix-A, it should be safe to use the asterisk character unencoded with the query string.
The text was updated successfully, but these errors were encountered: