Design a mechanism to parse Personal Data Fields (PDF) from API responses for Cardholders using pydantic

[devraj]

Gallagher's REST APIs return configurable fields (per instance of the server) called Personal Data Fields (PDFs). These are additional fields that are relevant to the organisation and are not common. These key of the fields are always prefixed with an @ symbol e.g @Student ID in an example from their docs:

{
  "href": "https://localhost:8904/api/cardholders/325",
  "id": "325",
  "firstName": "Algernon",
  "lastName": "Boothroyd",
  "shortName": "Q",
  "description": "Quartermaster",
  "authorised": true,
  "lastSuccessfulAccessTime": "2004-11-18T19:21:52Z",
  "lastSuccessfulAccessZone": {
    "href": "https://localhost:8904/api/access_zones/333",
    "name": "Twilight zone"
  },
  "serverDisplayName": "ruatoria.satellite.int",
  "division": {
    "href": "https://localhost:8904/api/divisions/2"
  },
  "@Student ID": "8904640"
}

We are using pyndatic schemas to parse the API response. The aim would be to use event based methods to dynamically parse these and make them available as a dictionary:

cardholder.pdfs["StudentId"]

for the API user to access these fields in a Pythonic manner.

Note that the response has a field called personalDataDefinitions which contains references to each one of the PDF definitions.

[devraj]

I would really like a very pyndatic solution to this, and hence have started a discussion on the pydantic Github project.

Post a resolution this should be merged in our documentation.

[devraj]

For the purposes of the Gallagher API, it should be noted that

https://commandcentre-api-au.security.gallagher.cloud/api/personal_data_fields/

returns a valid set of Personal Data Fields for the command centre we are communicating with. I propose that we call and memoize #14 the response and use this as part of the validation process to ensure that unexpected keys don't pass through the validation process.

[devraj]

Doing a bit of research I found the Extras option in pyndatic which allows for fields other than the ones defined in the model to be populated. You can allow extras by modifying the configuration and can validate the model by providing a custom validator:

from pydantic import BaseModel, Field, root_validator, ValidationError

class ExtendedModel(BaseModel):
    name: str = Field(..., description="The name of the entity")
    value: int = Field(..., description="The value associated with the entity")

    @root_validator(pre=True)
    def validate_extras(cls, values):
        known_fields = {"name", "value"}

        # Validate all extra fields, or apply specific rules
        for key in values:
            if key not in known_fields:
                # Example: ensuring extra fields start with "extra_"
                if not key.startswith("extra_"):
                    raise ValidationError(f"Extra field '{key}' is not allowed.")

        return values

    class Config:
        # Allow extra fields to be passed
        extra = Extra.allow

example usage:

# Example usage
try:
    model = ExtendedModel(name="example", value=42, extra_field="some extra data")
    print(model.dict())
except ValidationError as e:
    print(e)

Another approach would be store a dictionary of items and then provide a getter function to fetch values using a key:

from pydantic import BaseModel, Field, PrivateAttr

class DynamicModel(BaseModel):
    name: str = Field(..., description="The name of the entity")
    value: int = Field(..., description="The value associated with the entity")

    # Store additional fields in a private dictionary
    _extra: dict = PrivateAttr(default={})

    def __init__(self, **kwargs):
        # Separate known fields from extra fields
        known_fields = {field: kwargs.pop(field) for field in self.__fields__ if field in kwargs}
        extra_fields = kwargs

        # Set known fields using the base constructor
        super().__init__(**known_fields)

        # Store extra fields
        self._extra = extra_fields

    def get_extra(self, key):
        return self._extra.get(key)

# Example usage
data = {
    "name": "example",
    "value": 42,
    "additional_field": "some extra data"
}

model = DynamicModel(**data)
print(model.name)  # Outputs: example
print(model.value)  # Outputs: 42
print(model.get_extra("additional_field"))  # Outputs: some extra data

This could be more appropriate for our use case at the keys would be dynamically discovered from the API and are configurable by the administrator of the system.

Note: we should post our approach on the pyndatic discussion for others to benefit from.

[devraj]

Having implemented a few underlying features ahead of finishing this off, I think we should tackle this in the following order:

• Parse and test the personalDataDefinitions property that defines the details of the PDF Fields
• Outline a formal syntax of accessing PDF fields (possibly object.pdf.attribute_name)
• Implement custom getter that is least error prone
• Documentation on the design and how a user accesses these variables.

[devraj]

First a note that the examples I posted last month are specific to the way Pydantic v1 works.

Second I realised that the personalDataFields send back a value field which contain the same value as it appears on the top level dictionary i.e:

personalDataDefinitions.@Key.value maps to object.@Key

See a peculiar structure where the list of personalDataDefinitions has an object, which has a dynamic key and the value the dynamic key leads to is an object with the meta data and the value of the field for that cardholder.

Initially I was going down a path of implementing a cache of the personalDataFields which is unnecessary as the cardholder detail response will always return what we need. Hence it makes sense to parse the personalDataDefinitions (see commit) and then make the top level keys accessible as aliases.