Allow dynamic specification of fields to return in REST API

[arthanson]

NetBox version

v3.7.2

Feature type

Change to existing functionality

Proposed functionality

Allow dynamically specifying which fields to return in REST APIs, something like ?fields=a,b,c or ?exclude=e,f. Similar to GraphQL how you can only return the fields you specifically need. There are several projects that do this - they are at different levels of maintainership so might make sense to just roll our own:

• https://stackoverflow.com/questions/27935558/dynamically-exclude-or-include-a-field-in-django-rest-framework-serializer
• https://github.com/rsinger86/drf-flex-fields
• https://github.com/csdenboer/drf-dynamic-serializers
• https://github.com/AltSchool/dynamic-rest
• https://github.com/imbokov/drf_dynamics
• https://timmytango.com/notes/excluding-fields-in-drf-serializers/

These offer differing levels of functionality:

• specify which fields to include: ?fields=
• specify which fields to exclude: ?exclude= we probably don't need this
• allow specifying field inclusion on FK references via double-underscore reference: ?fields='front_port__name,... we probably don't need this

Use case

This has several advantages:

• Allow getting rid of brief API
• more flexible and performant as can only return fields you specifically want
• Works better / more flexible for the API for dropdown queries where the existing brief APIs don't quite fit

Database changes

None

External dependencies

Potential to use external library, but we may want to just roll our own.

[jeremystretch]

Tentatively tagging this for v4.0 as it represents a major enhancement to the REST API. There's also probably some overlap with #13283.

[jeremystretch]

I've spent most of the day tackling this, and I'm happy to report that it went smoother than expected. Not only can serializer fields be dynamically toggled, we also obviate the need to explicitly prefetch fields on querysets.

For example, in the FR branch, GET /api/dcim/sites/ will return a complete representation of each site using 7 queries, whereas a request for GET /api/dcim/sites/?fields=id,name,status,region will populate only the four requested fields using only 2 queries.

GET /api/dcim/sites/?fields=id,name,status,region

{
    "count": 24,
    "next": null,
    "previous": null,
    "results": [
        {
            "id": 24,
            "name": "Butler Communications",
            "status": {
                "value": "active",
                "label": "Active"
            },
            "region": {
                "id": 40,
                "url": "http://netbox:8000/api/dcim/regions/40/",
                "display": "North Carolina",
                "name": "North Carolina",
                "slug": "us-nc",
                "_depth": 2
            }
        },
        {
            "id": 2,
            "name": "DM-Akron",
            "status": {
                "value": "active",
                "label": "Active"
            },
            "region": {
                "id": 51,
                "url": "http://netbox:8000/api/dcim/regions/51/",
                "display": "Ohio",
                "name": "Ohio",
                "slug": "us-oh",
                "_depth": 2
            }
        },
    ...

As mentioned, this largely removes the need for "brief" mode (?brief=true), however we'll need to consider how best to convey the minimal representation of an object. But that will need to be a separate discussion; this work does not make any changes to brief mode beyond some prefetch optimizations (namely removing the need to declare brief_prefetch_fields).

specify which fields to exclude: ?exclude= we probably don't need this

I haven't bothered to include support for excluding fields, as I agree it's probably of very limited utility, though it remains possible.

allow specifying field inclusion on FK references via double-underscore reference

I've limited this to first order fields on a serializer for now. While we could feasibly extend this, we should figure out the long-term plan for nested serializers first. IMO this would should be considered out of scope for this FR.