sparse fieldsets filtering support

[wwwdata]

if the appropriate query parameters are available. The api goes through all
the attributes and only includes the ones from the query parameter. For
example: ?fields[posts]=title,content

This is pretty ugly because of our internal data structure, that hopefully will be better sometime, see #202

It's already working but I now noticed I forgot to implement it for slices of data when a FindAll query get's called for example.

TODO:

• More Tests
• Support for data slices

This will resolve #183

[sharpner]

neat. What happens if I use an invalid field? The documentation does not specify behaviour here...

[wwwdata]

good point. I really don't know in the current implementation, nothing happens but we could also return an error json

[sharpner]

Since the documentation says f a client requests a restricted set of fields, an endpoint MUST NOT include additional fields in the response. wouldn't it be logical that the result would be empty if there was an invalid field? since it must not contain more.... ^^"

[sharpner]

lets see http://discuss.jsonapi.org/t/fetching-with-sparse-fieldsets-what-about-invalid-fields-in-request/164

[wwwdata]

hm, let's see what they respond to it i am curious 😄

[sharpner]

Nothing helpful. Wanna go with an error?

[wwwdata]

ok, i will check all query parameters and if something is wrong return a 400, bad request error with all fields that are not in the marshalled structs. I like this better anyways

[sharpner]

me too - thank you!

[interlock]

So using this PR on a fork right now and it works awesome. We do some security filtering on our resources. If you own the resource you can see more fields than if you do not. This presents a problem for us, since a FindAll/FindPaginated response, in some cases, can have resources owned by the requestor and not owned by the requestor.

I probably sound like an interface nut, but besides the fields[mode]=id,... would it be possible for implement something that would allow us to indicate which fields to specifically filter for each resource in a slice?

[wwwdata]

hm, yes @interlock that makes sense. We can add another method to the resource structs that just returns an []string with all the field names that should be in the result like it's done via the query parameters. Theoretically you could also set these parameter into the request inside the api2go resource methods, but that is pretty dirty.

So we need to do it this way: If there are fields specified via the new method and there are fields specified by the user via the query parameter, we return the intersection of both of them, right? If the user requests a field that either isn't there, or is forbidden for him, because it was filtered out with the new method, we will return an error and pretend that this field never existed, because for that specific user, the api2go resource looks different with lesser fields.

[sharpner]

Sounds good to me!

[interlock]

That helps skipping editing the Request.Url.RawQuery we are currently doing. Will that implementation request which fields to show for each marshalled resource on the same request? This is really only important for FindAll and FindPaginated, perhaps relationships as well... but we're not there yet.

We can stash what we need to make that work in the api2go.Request, so that would be a preferred param to a function that does that 👍

[wwwdata]

hm you are right. So a method on the struct that will be marshalled is not very helpful because you define the fields in a CRUD method like FindAll or FindPaginated.

What other options do we have? I guess we could also put it in the Responder. But that would break the api.

@interlock you suggest to do a function that writes this in the Request right? So add a method here: https://godoc.org/github.com/manyminds/api2go#Request to set the fields?
This method could be named FilterFields(type string, fields []string) for example and can also be called for related structs if needed. It will then extend or initially set the query parameters to filter fields.

[interlock]

That would simplify what we currently do, but does not solve filtering a model down based on our white_list of fields.

Use case

A public API request to find users in your area: /user/filter[locality]=Vancouver&page[limit]=100&page[offset]=0

We have a white_list that only returns some information from each users profile. Protecting their phone number and email for example. If the requestors user resource appeared in the list, that record should actually contain all the information in that record. In effect, based on the ownership of the resource, a different white_list should be applied for fields we return.

[wwwdata]

hm, that is a different thing. Do I understand this correctly that the following can happen:

• I call the FindAll action
• I get 10 Records of the same type
• Based on something like my current user id, I get 9 out of 10 records with filtered fields, and 1 records contains more fields than the others?

This is more than a normal filter fields functionality described in the jsonapi spec, which intended to minimize the json payload and therefore speed up the request (as far as I understood the feature).

We now want to define on a per-struct (record) basis which fields should be contained in the json.

If that is the case, I would fall back to my initial suggestion, to define another method on the struct that get's marshalled, so in your use case the User struct. You can then implement your business logic like you want and all that matters is, that this function returns the correct list of fields that need to be in the json. Does that make sense?

[wwwdata]

Another quick fix: Why don't you just define 2 structs. 1 Complete struct with all the fields, and another struct with a reduced fieldset. I guess it should work to also pass a different struct in the Response. Our marshaler eats everything. That would be a lot easier and cleaner in your code I guess, I'm just not 100% sure if that works. For this FindAll scenario, you would have to return a []interface{} which these 2 different struct types..

edit: Ok, that Idea pretty much sucks if you have a lot of relationships etc etc etc ... can this be solved with a composition of a basic struct + more fields?

[interlock]

Actually lets say that white_list case is moot for now. Discussion with the team has determined we will be a little more strict with our filtering based on that filters the request had. Solves that problem for us and simplifies our security layer 😄

[wwwdata]

ok, for this PR i will just implement another check and if there is an invalid field(s) in the query string, output an error(s).

[sharpner]

👍
let's keep it the way we always did, one thing at a time :-)

[sharpner]

shouldn't the code namespaced with api2go? We can not know if a user already has a INVALID_FIELD_QUERY_PARAM :p