[Feature] - Add iregex filter for devices model and return custom_fields as labels

[oijkn]

Hi, below is the Netbox API return from /api/dcim/devices/?name=router-01.example.com and I would like to know if you could implement in your API a filter on the model or model slug (using the model__iregex) that will return all devices matching this model as well as an option to return all custom_fields as labels, please.

{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "id": 8,
            "url": "https://netbox.example.com/api/dcim/devices/8/",
            "display": "router-01.example.com",
            "name": "router-01.example.com",
            "device_type": {
                "id": 61,
                "url": "https://netbox.example.com/api/dcim/device-types/61/",
                "display": "Dlink xSeries",
                "manufacturer": {
                    "id": 12,
                    "url": "https://netbox.example.com/api/dcim/manufacturers/12/",
                    "display": "DLINK",
                    "name": "DLINK",
                    "slug": "dlink"
                },
                "model": "Dlink xSeries",
                "slug": "dlink-xseries"
            },
            "device_role": {
                "id": 5,
                "url": "https://netbox.example.com/api/dcim/device-roles/5/",
                "display": "ROUTER",
                "name": "ROUTER",
                "slug": "ROUTER"
            },
            "tenant": {
                "id": 1,
                "url": "https://netbox.example.com/api/tenancy/tenants/1/",
                "display": "HOME",
                "name": "HOME",
                "slug": "home"
            },
            "platform": null,
            "serial": "01234567890",
            "asset_tag": null,
            "site": {
                "id": 1,
                "url": "https://netbox.example.com/api/dcim/sites/1/",
                "display": "Malakoff",
                "name": "Paris",
                "slug": "paris"
            },
            "location": null,
            "rack": {
                "id": 8,
                "url": "https://netbox.example.com/api/dcim/racks/8/",
                "display": "RACK 11",
                "name": "RACK 11"
            },
            "position": 6,
            "face": {
                "value": "front",
                "label": "Front"
            },
            "parent_device": null,
            "status": {
                "value": "active",
                "label": "Active"
            },
            "airflow": {
                "value": "side-to-rear",
                "label": "Side to rear"
            },
            "primary_ip": null,
            "primary_ip4": null,
            "primary_ip6": null,
            "cluster": null,
            "virtual_chassis": null,
            "vc_position": null,
            "vc_priority": null,
            "comments": "",
            "local_context_data": null,
            "tags": [],
            "custom_fields": {
                "IP": "10.1.1.10",
                "snmp_communaute": "read",
                "snmp_communaute_alternate": null,
                "snmp_version": "2c",
                "snmp_port": "161",
                "ping_frequence": "60",
                "snmp_frequence": "60",
                "client": null,
                "exploitant": "Operator",
                "partenaire": null
            },
            "config_context": {},
            "created": "2022-07-12T12:21:22.461170Z",
            "last_updated": "2022-07-12T12:23:18.688284Z"
        }
    ]
}

Thank you in advance.

[oijkn]

Here is what I tried :

What works, but hard-coded

class DeviceViewSet(NetBoxModelViewSet):  # pylint: disable=too-many-ancestors
    queryset = Device.objects.prefetch_related(
        "device_type__manufacturer",
        "device_role",
        "tenant",
        "platform",
        "site",
        "location",
        "rack",
        "parent_bay",
        "virtual_chassis__master",
        "primary_ip4__nat_outside",
        "primary_ip6__nat_outside",
        "tags",
    ).filter(
        Q(device_type__model__iregex=r'^Dlink.*$')
    )

    filterset_class = DeviceFilterSet
    serializer_class = PrometheusDeviceSerializer
    pagination_class = None

What does not work, calling the url api/plugins/prometheus-sd/devices?model=Dlink

class DeviceViewSet(NetBoxModelViewSet):  # pylint: disable=too-many-ancestors
    queryset = Device.objects.prefetch_related(
        "device_type__manufacturer",
        "device_role",
        "tenant",
        "platform",
        "site",
        "location",
        "rack",
        "parent_bay",
        "virtual_chassis__master",
        "primary_ip4__nat_outside",
        "primary_ip6__nat_outside",
        "tags",
    )

    def get_queryset(self):
        queryset = super().get_queryset()
        device_type__model = self.request.query_params.get("model", None)
        if device_type__model is not None:
            queryset = queryset.filter(
                Q(device_type__model__iregex=r'^{}.*$'.format(device_type__model))
            )
        return queryset

    filterset_class = DeviceFilterSet
    serializer_class = PrometheusDeviceSerializer
    pagination_class = None

HTTP 400 Bad Request
Allow: GET, POST, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "model": [
        "Select a valid choice. Dlink is not one of the available choices."
    ]
}

EDIT:

I think I found the solution, the problem comes from the line :

device_type__model = self.request.query_params.get("model", None)

I renamed the param model to device_model and it works :

# netbox_prometheus_sd/api/views.py

from django.db.models import Q

class DeviceViewSet(NetBoxModelViewSet):  # pylint: disable=too-many-ancestors
    queryset = Device.objects.prefetch_related(
        "device_type__manufacturer",
        "device_role",
        "tenant",
        "platform",
        "site",
        "location",
        "rack",
        "parent_bay",
        "virtual_chassis__master",
        "primary_ip4__nat_outside",
        "primary_ip6__nat_outside",
        "tags",
    )

    def get_queryset(self):
        queryset = super().get_queryset()
        request = self.get_serializer_context()['request']
        device_model = request.query_params.get("deviceModel", None)
        if device_model is not None:
            queryset = queryset.filter(
                Q(device_type__model__iregex=device_model)
            )
        return queryset

    filterset_class = DeviceFilterSet
    serializer_class = PrometheusDeviceSerializer
    pagination_class = None

[oijkn]

I'm having trouble displaying the custom_fields, I have the sensation that the object is not present.

Here is what I tried but without success :

# netbox_prometheus_sd/api/serializers.py

        if hasattr(obj, "custom_fields") and obj.custom_fields is not None and len(obj.custom_fields.all()):
            labels["custom_fields"] = ",".join(
                [
                    f"{cf.name}={cf.value}"
                    for cf in obj.custom_fields.all()
                    if cf.value is not None
                ]
            )

EDIT:

By persevering, I found the solution that I share below:

# netbox_prometheus_sd/api/serializers.py

        if hasattr(obj, "custom_field_data") and obj.custom_field_data is not None:
            for key, value in obj.custom_field_data.items():
                if value is not None:
                    labels["custom_field_" + key.lower()] = value

@FlxPeters are you interested by a PR ?

[FlxPeters]

Custom fields have been merged to main branch. Please provide feedback if the solution meets your requirements.

[AlexDaichendt]

Hey!
any ETA when the custom_fields will hit PyPI?
Thank you for maintaining this plugin, Felix :)

[FlxPeters]

I just released a pre-release on PyPi: https://pypi.org/project/netbox-plugin-prometheus-sd/0.6.0rc1/
Please provide feedback if this fits your needs or if we have to adjust the labels.

[AlexDaichendt]

Awesome, thank you :) I'll tinker around with it for a couple days and report back!

[MrXermon]

I just wanted to make a issue/PR to include the custom fields and checked-out the pre-release which you mentioned above. Seems to work at-least for my use-case.

[AlexDaichendt]

Works great for me. So far I did not encounter any limitations

[streaming-pete]

I did a quick test with the git repo and it worked pretty good. Thanks I'd been wanting to do this for a while.
I have noticed that the netbox plugin version and the pypi app version don't match. 0.5.0 from PyPi seems to display as 0.4 in netbox plugins. took me a while to work out what was going on.

[candlerb]

It gets a bit awkward where custom fields have data type "multiselect". They are currently returned as a not-quite JSON list:

      "__meta_netbox_custom_field_snmp_module": "['if_mib', 'ubiquiti_unifi']",

("not-quite JSON" because it uses single quotes rather than double quotes).

This is rather inconvenient when it comes to the new multi-module support in SNMP exporter (v0.24.0+), which requires a plain comma-separated list like /snmp?target=X.X.X.X&module=if_mib,ubiquiti_unifi

AFAICT there's no global search-replace in Prometheus relabelling, so this was the best I could come up with:

      - source_labels: [__meta_netbox_custom_field_snmp_module]
        target_label: __param_module
      # Ugh: multiselect is of form ['foo','bar'] and we need foo,bar. There is no gsub.
      - source_labels: [__param_module]
        regex: "\\['(.*)'\\]"
        target_label: __param_module
      - source_labels: [__param_module]
        regex: "(.*)', *'(.*)"
        replacement: "$1,$2"
        target_label: __param_module
      - source_labels: [__param_module]
        regex: "(.*)', *'(.*)"
        replacement: "$1,$2"
        target_label: __param_module
      - source_labels: [__param_module]
        regex: "(.*)', *'(.*)"
        replacement: "$1,$2"
        target_label: __param_module

(which works for a maximum of 4 selected values). That's pretty ugly.

I'm not sure of the best solution here. Automatically converting a list value into a comma-joined string would be the easy thing for this particular case, but I guess there might be other cases where people would want something different. (I considered that a plain value could contain a comma, but this doesn't apply in a Netbox custom field select or multiselect, because the list of allowed values in the custom field definition is itself comma-separated)

---

EDIT: Since Netbox 3.6 has added Custom Field choice sets, comma is no longer a special field (although colon is)

For fields with a comma you could do backslash escaping ("\" -> "\\", "," -> "\,") or URL-style escaping ("%" -> "%25", "," -> "%2c"), or CSV formatting (wrap field with double quotes, and replace one double-quote with two - but only if the field contains a comma)