AdvancedConnectorTutorial

Tutorial: Implementing custom attributes for the YAML connector

The YamlEngine defined in the Basic Connector Tutorial can retrieve complete items from a YAML file, assuming the structure of objects in the file matches exactly the structure of the expected DatabaseItem. But what if we want to load a custom DatabaseItem from an already existing YAML file that does not match the expected JSON-like structure specification, e.g. some config files used by other application?

To do so, we define a YamlAttribute that allows us to fetch a value from a field at a different path from the current one in the DatabaseItem model. For example, to retrieve the login field from the name field of the YAML file:

database_item = YamlItem(
    model={
        "login": YamlAttribute(["name"])
    }
    )

A custom DatabaseAttribute attribute must support the operations similar to the ItemMapper. Indeed, requests built by attributes will be executed as any other request by the database connection. Requests performed by the ItemMapper are called base request. Each attribute of the model can either modify the base request, or return new requests that will be handled independently by the database connection. Both approaches are valid, and depends on the requests structure, performances, and the logic required by each attribute.

Here is the detail of each DatabaseAttribute operation:

search_request(base_request, _id): build a new request or update the base_request to retrieve the value of the attribute.
load(base_response, attribute_response): returns the value of the attribute. attribute_response is the response to the request built by search_request, if any.
create_request(base_request, value): build a new request or update the base_request to create the attribute with the specified value.
delete_request(base_request, _id): build a new request or update the base_request to delete the attribute in the item associated to _id
update_request(base_request, _id, value): build a new request or update the base request to update the attribute to value in the item associated to _id.

The following examples will demonstrate how to implement the YamlAttribute using two different methods: modifying the base request or creating new requests.

Base initialization vs attributes

Using attributes to specify the model raises issues about how to manage base initialization. There might be 3 situations:

Only base initialization is performed. This means all data retrieved from the base request will be included in generated JSON-like structures.
Attribute only initialization. Only fields explicitly specified in the model are included in JSON-like structures.
Do both. Use custom attributes (e.g. to rename fields), and use base initialization for unspecified attributes.

Since item initialization is performed by the item mapper, it is the entry point where to implement each scenario. The first has been implemented in the Basic connector tutorial.

The second scenario can be implemented modifying the item mapper as follows:

class MapByKey(ItemMapper):
    ...

    def load(self, init_value, _base_search_response):
        return init_value

    ...

    def load_items(self, init_factory, base_request_response: YamlSelectResponse):
        return [
            (_id, init_factory()) for _id in base_request_response.values.keys()
        ]

The init_value is an empty structure corresponding to the type of model, i.e. [] or {}. The value returned by load() must always be of the same type as the init_value.

The init_factory is a callable that return init values as defined before, and can be used to initialize multiple items.

Finally, the third scenario can be implemented as follows:

class MapByKey(ItemMapper):
    def __init__(self, generate_id=uuid4, empty_init=False):
        self.generate_id = generate_id
        self.empty_init = empty_init

    ...
 
    def load(self, init_value, base_search_response):
        if self.empty_init:
            return init_value
        return base_search_response.value

    ...

    def load_items(self, init_factory, base_request_response: YamlSelectResponse):
        if self.empty_init:
            return [
                (_id, init_factory()) for _id in base_request_response.values.keys()
            ]
        return list(base_request_response.values.items())

In this case, the user specifies if extra values should be considered using the empty_init parameter. Supporting this features requires a bit of extra logic at the attribute level, as it will be illustrated in the next example.

Init attribute set up

First, we define the attribute to receive as parameter the path to the field to retrieve in the YAML file. This path is relative to the root of the item retrieved by the item mapper.

# yaml/attribute.py

from connecto.database.item import DatabaseAttribute

...


class YamlAttribute(DatabaseAttribute):
    def __init__(self, path):
        super().__init__()
        self.path = path

Then we allow path to be None, so that YamlAttribute() means the value is located in the YAML file at the same path that the attribute in the model. To do so, it's possible to override the set_attribute_path method of the DatabaseAttibute that is called by the database engine to store the path to the attribute with the model in the attribute_path Python attribute.

from connecto.attribute import DatabaseAttribute
from connecto.utils.nested_data_path import equal_path


class YamlAttribute(DatabaseAttribute):
    def __init__(self, path=None):
        super().__init__()
        self.path = path

        # True if and only if the value of the attribute is at a path that is
        # different from the path of the attribute within the model.
        self.path_to_self = False 

    def set_attribute_path(self, attribute_path):
        super().set_attribute_path(attribute_path)

        if self.path is None:
            # No path was specified by the user, so use attribute path as path.
            self.path = self.attribute_path
            self.path_to_self = True
        elif equal_path(self.attribute_path, self.path):
            # A path was specified but it's actually the same as the path of the
            # attribute in the model
            self.path_to_self = True

Implementing operations

Once the attribute is set up, it's required to implement operations that can be performed on it.

There is two methods to do so:

The attribute modifies the base request
The attribute build it's own requests

Notice the two methods are not exclusive in the generic case, and each attribute is allowed to do both.

Method 1: altering the base request

This is the recommended method for best performance, when requested attributes can easily be added to the base request (for example in the SELECT field of an SQL request or in the list of attributes of an LDAP SEARCH).

If no base initialization is considered, the implementation of the attribute can be as simple as:

# yaml/attribute.py

from connecto.attribute import DatabaseAttribute
from connecto.utils.nested_data_path import find, update, delete, equal_path

...


class YamlAttribute(DatabaseAttribute):
    ...
        
    def search_request(self, base_request, _id):
        # Nothing to do, the requested field will already be included in the
        # response of the base_request, that include the complete object
        # associated to _id
        pass

    def load(self, base_response, _attribute_response):
        # _attribute_response is None, because no request was returned by
        # search_request. But the field can be retrieved from the response of
        # the base request
        value = find(base_response.value, self.path)

        return value

    def create_request(self, base_request, value):
        # Adds the value of the attribute to the base create request
        update(base_request.value, self.path, value)

    def update_request(self, base_request, _id, value):
        # Adds the value of the attribute to the base create request
        update(base_request.value, self.path, value)

    def delete_request(self, base_request, _id):
        # Nothing to do, the complete item will be deleted by the base request
        pass

    def select_request(self, base_request, item_filter):
        # Nothing to do, the requested field will already be included in the
        # response of the base_request, that include the complete object
        # associated to each item
        pass

If it's also required to support base initialization, a few extra logic is required to properly handle attributes. For example, if a name field is defined in the YAML file and the model is {"login": YamlAttribute(["name"])}, this probably means that we don't want to include the original name field in the JSON-like structure, even if we want to include fields not specified in the model. We can use the previously defined _path_to_self attribute to replace fields when required, as illustrated in the next implementation:

# yaml/attribute.py

from connecto.attribute import DatabaseAttribute
from connecto.utils.nested_data_path import find, update, delete, equal_path

...


class YamlAttribute(DatabaseAttribute):
    ...
        
    def search_request(self, base_request, _id):
        # Nothing to do, the requested field will already be included in the
        # response of the base_request, that include the complete object
        # associated to _id
        pass

    def load(self, base_response, _attribute_response):
        # _attribute_response is None, because no request was returned by
        # search_request. But the field can be retrieved from the response of
        # the base request
        value = find(base_response.value, self.path)

        if not self.path_to_self:
            # The value must be deleted from the response so this field is not
            # included in the generated JSON-like structure
            delete(base_response.value, self.path)
        return value

    def create_request(self, base_request, value):
        if not self.path_to_self:
            # Removes the replaced path from the original user item
            delete(base_request.value, self.attribute_path)

        # Adds the value of the attribute to the base create request
        update(base_request.value, self.path, value)

    def update_request(self, base_request, _id, value):
        if not self.path_to_self:
            # Removes the replaced path from the original user item
            delete(base_request.value, self.attribute_path)

        # Adds the value of the attribute to the base create request
        update(base_request.value, self.path, value)

    def delete_request(self, base_request, _id):
        # Nothing to do, the complete item will be deleted by the base request
        pass

    def select_request(self, base_request, item_filter):
        # Nothing to do, the requested field will already be included in the
        # response of the base_request, that include the complete object
        # associated to each item
        pass

Notice that the base initialization issue is not specific to YAML. The exact same issue indeed arises to implement an SQL item mapper that might return values of all columns by default, or an LDAP item mapper that might return values of all attributes of an entry by default.

The custom YamlAttribute can now be used as follows with the last item mapper implementation:

from .yaml.engine import YamlEngine
from .yaml.item import YamlItem
from .yaml.attribute import YamlAttribute

if __name__ == "__main__":
    yaml_engine = YamlEngine(
        "test.yaml", database_item=YamlItem(model={"login": YamlAttribute(["name"])})
    )
    item_id = yaml_engine.create(
        {"login": "pipo", "gid": 12, "description": "Example user"}
    )
    print(f"Item {item_id} created.")
    print(yaml_engine.search(item_id))
    print()

    yaml_engine.save(
        item_id, {"login": "molo", "gid": 13, "description": "Updated user"}
    )
    print(f"Item {item_id} updated.")
    print(yaml_engine.search(item_id))
    print()

    yaml_engine.create(
        {"login": "bato", "gid": 10, "description": "New user"}
    )
    print(f"Selected items:")
    print(yaml_engine.select())
    print()

    yaml_engine.delete(item_id)
    print(f"Item {item_id} deleted.")

$ uv run examples/yaml/attributes.py
Item 93628a44-ae87-4655-bc60-d06a6ff2c4c2 created.
('93628a44-ae87-4655-bc60-d06a6ff2c4c2', {'description': 'Example user', 'gid': 12, 'login': 'pipo'})

Item 93628a44-ae87-4655-bc60-d06a6ff2c4c2 updated.
('93628a44-ae87-4655-bc60-d06a6ff2c4c2', {'description': 'Updated user', 'gid': 13, 'login': 'molo'})

Selected items:
[('93628a44-ae87-4655-bc60-d06a6ff2c4c2', {'description': 'Updated user', 'gid': 13, 'login': 'molo'}), ('f175070c-1ca9-4388-
af13-ebf6e1989c90', {'description': 'New user', 'gid': 10, 'login': 'bato'})]

Item 93628a44-ae87-4655-bc60-d06a6ff2c4c2 deleted.

Notice how the login field is included in search and select responses, but not name. Moreover, if you create some items with the engine, the database file looks as follows:

e8642beb-12f8-477d-b93d-bfa0eee1b964:
  description: Example user
  gid: 12
  name: pipo
ef7f4920-78cc-4a07-97e8-ea64e4eeb3fa:
  description: Updated user
  gid: 13
  name: molo

In the database, name is included but not login.

Method 2: creating requests for each attribute

This is the recommended method for complex attributes for which the base request cannot be enough. For example, look for an attribute in an other YAML file, in an other SQL table, or in an other LDAP entry.

class YamlAttribute(DatabaseAttribute):
    ...

    def search_request(self, base_request, _id):
        # Request the attribute at path, relative the base search
        return YamlSearchRequest(base_request.path + self.path)

    def load(self, base_response, attribute_response):
        # The value must be deleted from the response so this field is not
        # included in the generated JSON-like dict
        delete(base_response.value, self.path)
        # Returns the value of the attribute obtained from the attribute search
        # request
        return attribute_response.value

    def create_request(self, base_request: YamlCreateRequest, value):
        # Removes the replaced path from the original user item
        delete(base_request.value, self.attribute_path)
        # Creates the attribute at path, relative the base search
        return YamlCreateRequest(base_request.path + self.path, value)

    def update_request(self, base_request, _id, value):
        # Removes the replaced path from the original user item
        delete(base_request.value, self.attribute_path)
        # Updates the attribute at path, relative the base search
        return YamlUpdateRequest(base_request.path + self.path, value)

    def delete_request(self, base_request, _id):
        # Deletes the attribute at path, relative the base search. This is
        # actually not necessary as the complete object is deleted by the base
        # request
        return YamlDeleteRequest(base_request.path + self.path)

    def select_request(self, base_request, item_filter):
        # Selects the attribute at self.path for all items
        return YamlSelectRequest(self.path)

Finally, to properly support this use case, it's require to adapt the select_response of the item mapper to appropriatly pass attribute response to each selected item:

# yaml/mapper.py

class MapByKey(ItemMapper):
    ...

    def select_response(
        self,
        _id,
        base_request_response: YamlSelectResponse,
        attribute_responses: YamlSelectResponse,
    ):
        # Base response for the item associated to _id
        base_response = YamlSearchResponse(base_request_response.values[_id])

        # Response for the attribute of the item associated to _id
        item_attribute_response = None
        if attribute_responses is not None:
            item_attribute_response = YamlSearchResponse(
                attribute_responses.values[_id]
            )
        return base_response, item_attribute_response

Tip

Do not forget to support the case where attributes_responses is None, if no request was required by the attribute.

The same examples as before can be used without any modification, and the result is the same:

$ uv run examples/yaml/attributes.py
Item 3683a400-180c-4ee3-a5b5-e86c31fa24c3 created.
('3683a400-180c-4ee3-a5b5-e86c31fa24c3', {'description': 'Example user', 'gid': 12, 'login': 'pipo'})

Item 3683a400-180c-4ee3-a5b5-e86c31fa24c3 updated.
('3683a400-180c-4ee3-a5b5-e86c31fa24c3', {'description': 'Updated user', 'gid': 13, 'login': 'molo'})

Selected items:
[('3683a400-180c-4ee3-a5b5-e86c31fa24c3', {'description': 'Updated user', 'gid': 13, 'login': 'molo'}), ('4e1570e4-32e2-48f8-
a4f8-582f47e4cb26', {'description': 'New user', 'gid': 10, 'login': 'bato'})]

Item 3683a400-180c-4ee3-a5b5-e86c31fa24c3 deleted.

Navigation Menu

AdvancedConnectorTutorial

Tutorial: Implementing custom attributes for the YAML connector

Base initialization vs attributes

Init attribute set up

Implementing operations

Method 1: altering the base request

Method 2: creating requests for each attribute

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally