WorkingPrinciple

Working principle of connecto

Even if most tasks (including the implementation of a new connector) should not require a deep understanding of how connecto works internally, here we provided some insights about working principles of connecto. This might be useful to understand how connecto build and execute requests and how responses are given back to connectors.

flowchart TB;
    Database@{ shape: cyl, label: "Database" };
    Database --- DatabaseConnection;

    DatabaseEngine -- execute requests --> DatabaseConnection;
    DatabaseItem -- requests --> DatabaseEngine;
    DatabaseEngine -- responses --> DatabaseItem;
    DatabaseItem -- load --> JSONstruct[JSON-like structure];

Loading

Given a DatabaseItem, the DatabaseEngine workflow consists in three steps:

1. ask the DatabaseItem for requests required to perform each operation (search, create, update, delete, select)
2. perform those requests against the real database backend using the DatabaseConnection
3. provide the responses to the DatabaseItem.load() method so it can build a JSON-like representation of the Item when required (e.g. for search, select).

This concept allows to easily implement DatabaseAttributes that can be used to define complex models, because each attribute only needs to implement the logic required to manage itself, independently of the encapsulating model.

New DatabaseConnections are also easy to implement because each operation can be implemented independently (no need to implement create, delete or update if you want a read-only connection). Moreover, the DatabaseConnection only focuses on executing each operation from requests, but it is not concerned about models and attributes: it is the responsibility of each attribute to build valid requests.

The DatabaseItem builds requests using the following procedure:

1. build the base request required by the item_mapper to perform the operation
2. increment the base request or build additional requests as required to perform the operation on each attribute of the model.

The set of requests build by an item can be as complex as required.

For example, given the following model:

{
  "field1": attribute_1,
  "field2": attribute_2,
  "nested_field": {
    "field3": [attribute_3, attribute_4]
  }
}

the following set of requests will be built for a search operation, where base_request=item_mapper.search_request(<id>) is the request built by the item_mapper:

{
  "field1": attribute_1.search_request(base_request),
  "field2": attribute_2.search_request(base_request),
  "nested_field": {
    "field3": [attribute_3.search_request(base_request), attribute_4.search_request(base_request)]
  }
}

Each attribute is allowed to either modify the base_request, build a new request or do nothing. For example, LdapAttribute("name") will add "name" to the LDAP attributes of the base LDAP search query. ReverseRef will return a search queries to build the filter used to find referenced items.

Then the DatabaseEngine will perform requests using the DatabaseConnection.execute_search method to produce a base_response and attributes_responses organised as follows:

{
  "field1": response_1,
  "field2": response_2,
  "nested_field": {
    "field3": [response_3, response_4]
  }
}

All responses are then passed to DatabaseItem.load so that each field is loaded as follows to generate the final JSON-like dict:

{
  "field1": attribute_1.load(base_response, response_1),
  "field2": attribute_2.load(base_response, response_2),
  "nested_field": {
    "field3": [attribute_3.load(base_response, response_3), attribute_4.load(base_response, response_4)]
  }
}

Each attribute is even allowed to build a structured set of request. Considering the item:

{
    "data": data_attribute
}

the following requests might be built:

{
    "data": [request0, {"req1": request1, "req2": request2}]
}

Each request will be executed by the engine so that the loaded item is:

{
    "data": attribute1.load(
            base_response,
            [response0, {"req1": response1, "req2": response2}]
        )
}

Notice that in most cases, the base item_mapper request is enough, and database attributes will look for values in the base response. The ability to modify the base request allows to perform a single fined-tuned request, while still allowing additional requests for complex database attributes. This allows the DatabaseEngine to be very efficient, in spite of an high level of abstraction.

The fact that each attribute receives only the response to its own request allows to easily build modular attributes with only a few lines of codes, because each attribute does not need to consider the rest of the model to handle each operation.

Implementation examples for YAML are provided in the Basic Connector Tutorial and Advanced Connector Tutorial.