AdvancedConnectorTutorial

Tutorial: Implementing custom attributes for the YAML database connection

The YamlEngine defined in the Basic Connector Tutorial can retrieve complete items from a YAML file, assuming the structure of objects in the file matches exactly the structure of the expected DatabaseItem. But what if we want to load a custom DatabaseItem from an already existing YAML file that does not match the expected Item specification, e.g. some config files used by other application?

As a first simple example, we will define a YamlAttribute that allows us to fetch a value from a field at a different path from the current one in the DatabaseItem model. For example, we want the login field of the DatabaseItem to be retrieved from the name attribute of the object in the YAML file. To do so, we define a custom YamlItem as follows

database_item = YamlItem(
    model={
        "login": YamlAttribute(["name"])
    }
    )

A custom DatabaseItem attribute must support the same request operations as the ItemMapper. Indeed, requests built by attribute will be executed as any other request by the database connection. Requests performed by the ItemMapper are called "base" request. Each attribute of the model can either modify the base request, or return new requests that will be handled independently by the database connection. Both approaches are valid, and depends on the requests structure, performances, and the logic required by each attribute.

Here is the detail of each DatabaseAttribute operation:

• search_request(base_request, _id): build a new request or update the base_request to retrieve the value of the attribute.
• load(base_response, attribute_response): returns the value of the attribute. attribute_response is the response to the request built by search_request, if any.
• create_request(base_request, value): build a new request or update the base_request to create the attribute with the specified value.
• delete_request(base_request, _id): build a new request or update the base_request to delete the attribute in the item associated to _id
• update_request(base_request, _id, value): build a new request or update the base request to update the attribute to value in the item associated to _id.

The following examples will demonstrate how to implement the YamlAttribute using two different methods: modifying the base request or creating new requests.

Method 1: altering the base request

This is the recommended method for best performance, when requested attributes can easily be added to the base request (for example in the SELECT field of an SQL request or in the list of attributes of an LDAP SEARCH).

# backo/yaml/item.py

from ..database.item import ItemMapper, DatabaseItem, DatabaseAttribute

...


class YamlAttribute(DatabaseAttribute):
    def __init__(self, path):
        super().__init__()
        """
        :param attribute: Name of the field to load from the Yaml file
        """
        self.path = path

    def search_request(self, base_request, _id):
        # Nothing to do, the requested field will already be included in the
        # response of the base_request, that include the complete object
        # associated to _id
        pass

    def load(self, base_response, _attribute_response):
        # _attribute_response is None, because no request was returned by
        # search_request. But the field can be retrieved from the response of
        # the base request
        value = find(base_response.value, self.path)
        # The value must be deleted from the response so this field is not
        # included in the generated JSON-like dict
        delete(base_response.value, self.path)
        return value

    def create_request(self, base_request: YamlCreateRequest, value):
        # Adds the value of the attribute to the base create request
        update(base_request.value, self.path, value)
        # Removes the replaced path from the original user item
        delete(base_request.value, self.attribute_path)

    def update_request(self, base_request, _id, value):
        # Adds the value of the attribute to the base create request
        update(base_request.value, self.path, value)
        # Removes the replaced path from the original user item
        delete(base_request.value, self.attribute_path)

    def delete_request(self, base_request, _id):
        # Nothing to do, the complete item will be deleted by the base request
        pass

The custom YamlAttribute can now be used as follows:

# examples/yaml/attributes.py

from backo.yaml.engine import YamlEngine
from backo.yaml.item import YamlItem, YamlAttribute

if __name__ == "__main__":
    yaml_engine = YamlEngine(
        "test.yaml", database_item=YamlItem(model={"login": YamlAttribute(["name"])})
    )
    item_id = yaml_engine.create(
        {"login": "pipo", "gid": 12, "description": "Example user"}
    )
    print(f"Item {item_id} created.")
    print(yaml_engine.search(item_id))
    print()

    yaml_engine.save(
        item_id, {"login": "molo", "gid": 13, "description": "Updated user"}
    )
    print(f"Item {item_id} updated.")
    print(yaml_engine.search(item_id))
    print()

    yaml_engine.delete(item_id)
    print(f"Item {item_id} deleted.")

$ uv run examples/yaml/attributes.py
Item c9b4149e-d1ba-466b-b574-474fa4aeaec2 created.
{'description': 'Example user', 'gid': 12, 'login': 'pipo', '_id': 'c9b4149e-d1ba-466b-b574-474fa4aeaec2'}

Item c9b4149e-d1ba-466b-b574-474fa4aeaec2 updated.
{'description': 'Updated user', 'gid': 13, 'login': 'molo', '_id': 'c9b4149e-d1ba-466b-b574-474fa4aeaec2'}

Item c9b4149e-d1ba-466b-b574-474fa4aeaec2 deleted.

Notice how the login field is included in engine.search responses, but not name. More however, if you create some items with the engine, the database file looks as follows:

e8642beb-12f8-477d-b93d-bfa0eee1b964:
  description: Example user
  gid: 12
  name: pipo
ef7f4920-78cc-4a07-97e8-ea64e4eeb3fa:
  description: Updated user
  gid: 13
  name: molo

In the database, name is included but not login.

Method 2: creating requests for each attribute

This is the recommended method for complex attributes for which the base request cannot be enough. For example, look for an attribute in an other YAML file, in an other SQL table, or in an other LDAP entry.

class YamlAttribute(DatabaseAttribute):
    def __init__(self, path):
        """
        :param attribute: Name of the field to load from the Yaml file
        """
        self.path = path

    def search_request(self, base_request, _id):
        # Request the attribute at path, relative the base search
        return YamlSearchRequest(base_request.path + self.path)

    def load(self, base_response, attribute_response):
        # The value must be deleted from the response so this field is not
        # included in the generated JSON-like dict
        delete(base_response.value, self.path)
        # Returns the value of the attribute obtained from the attribute search request
        return attribute_response.value

    def create_request(self, base_request: YamlCreateRequest, value):
        # Removes the replaced path from the original user item
        delete(base_request.value, self.attribute_path)
        # Creates the attribute at path, relative the base search
        return YamlCreateRequest(base_request.path + self.path, value)

    def update_request(self, base_request, _id, value):
        # Removes the replaced path from the original user item
        delete(base_request.value, self.attribute_path)
        # Updates the attribute at path, relative the base search
        return YamlUpdateRequest(base_request.path + self.path, value)

    def delete_request(self, base_request, _id):
        # Deletes the attribute at path, relative the base search
        # This is actually not necessary as the complete object is deleted by
        # the base request
        return YamlDeleteRequest(base_request.path + self.path)

The same example as before can be used without any modification, and the result is the same:

$ uv run examples/yaml/attributes.py
Item 49025c47-f509-4e16-82d3-5d40b76b3aca created.
{'description': 'Example user', 'gid': 12, 'login': 'pipo', '_id': '49025c47-f509-4e16-82d3-5d40b76b3aca'}

Item 49025c47-f509-4e16-82d3-5d40b76b3aca updated.
{'description': 'Updated user', 'gid': 13, 'login': 'molo', '_id': '49025c47-f509-4e16-82d3-5d40b76b3aca'}

Item 49025c47-f509-4e16-82d3-5d40b76b3aca deleted.