ConnectorIntroduction

Introduction to the Database Engine

Once an Item is defined in Backo, it can receive JSON-like dicts from DatabaseEngines to initialize Items. Backo then exposes Items through a generated CRUD API, allowing users to safely and easily perform standard operations on them.

graph LR;
    user[End user]
    user <-- API --> Backo;
    Backo <-- JSON-like dict --> DatabaseEngine;

Loading

However, the DatabaseEngine can be connected to a database that returns data that can be very far from the expected JSON dict, for example LDAP trees or SQL tables.

Moreover, if the DatabaseEngine should work from an existing database, it is very likely that expected attributes of the backo Item does not match the existing database fields. Actually, the ability to translate complex databases to an easily understandable JSON based CRUD application is a main interest of Backo.

It is not even guaranteed that the JSON-like dict representing the item can be retrieved from a single database request. It might be required to perform several requests to build the final Backo item, especially to handle Ref, RefList and so on. You might also define a single Backo item that is an aggregation of fields from several data sources.

The purpose of the DatabaseEngine is to orchestrate database requests to build the expected JSON-like dict, allowing users to handle complex database schemes with a very simple low-code syntax.

For developers, only a few specific features must be implemented to allow the DatabaseEngine to support a new database (e.g. LDAP, SQL, key value stores...).

Backo Items vs Database Items

Here is an example of a Backo Item:

user = Item(schema={
        "name": String(),
        "surname": String(),
        "group": Ref(collection="groups", field="$.users", required=True),
    })

It defines how users will be exposed to end users through the generated API, ensure type verification for updates, safe deletion of referenced items and so on, but it does not define how the name, surname, and addr fields should be retrieved from the database. The Item is voluntarily designed to be independent of the actual database in which it is stored.

All of the database specific mechanics must then be specified in a dedicated DatabaseItem.

To summarize:

• Backo uses Items as a specification of how to expose an API to the final users
• The DatabaseEngine uses DatabaseItems as a specification of how to load Items from a database

A DatabaseItem is a generic structure that might look similar to the corresponding Item, but does not represent the same information. It is based on two structures:

• item_mapper: specifies how base objects are built from the external database.
• model: an optional nested dict/list structure mapping fields to custom database attributes. Each value describes exactly how to retrieve the value that will be associated to the key in the finally produced JSON-like dict. The nature of attributes can be very specific to the database currently considered.

The following examples will illustrate various usage and implementation.

Examples

A basic YAML example

This is a trivial example, that is used by default by the YamlItem.

user_database = YamlItem(
        item_mapper=MapByKey(),
        model={}
    )

The Yaml file (passed as parameter to the YamlConnection, not represented here) is expected to represent an object with ids of Items as keys and Items themselves as values, for example:

item1:
  foo: bar
  name: "Item 1"
item2:
  foo: baz
  name: "Item 2"

Using MapByKey() as the item_mapper tells the database engine to use Yaml keys as _ids to perform search, creation, update, or any other operation.

The model is empty because no custom attribute is needed: the generate JSON-like dict will exactly match the structure of item objects.

See the Basic Connector Tutorial for an example implementation.

A YAML example with extra features

user_database = YamlItem(
        item_mapper=MapByKey(),
        model={
          "login"= YamlAttribute(["name"]),
        }
    )

This example is the same as before, but the YamlAttribute specifies that the value associated to the key name must be used as login in the built Item. The base Yaml object is still used as a base for the returned value.

See the Advanced Connector Tutorial for an example implementation.

A simple LDAP example

user_database = LdapItem(
        item_mapper=MapByDN(),
        model={
            "name": Attribute("uid"),
            "description": Attribute("description")
        }
    )

With this example, the LdapItem will load() the JSON-like dict by loading values of name and description fields from the uid and description attributes of the LDAP entry obtained from the result of the search for the DN associated to the item, that is used as _id.

A complex LDAP example

The following example illustrates how the DatabaseEngine can handle complex database schemes that can be very different from the expected JSON-like dict, contrary to the previous Yaml example.

Warning

This is work in progress. The API might change in the future, but it illustrates target LDAP features.

group_database = LdapItem(
    item_mapper=MapByAttribute("gidNumber", base_dn="ou=groups,cn=example,cn=org"),
    model={
        "name": Attribute("name"),
        "admin": LdapRef(
                attribute="owner",
                referenced_item=LdapItem(
                    item_mapper=MapByDN()
                    )
                ),
        "members": LdapRefList(
                attribute="member",
                referenced_item=LdapItem(
                    item_mapper=MapByDN()
                    )
                )
        }
        )

user_database = LdapItem(
        item_mapper=MapByDN(),
        model={
            "name": Attribute("uid"),
            "surname": Attribute("sn"),
            "groups": ReverseRefList(
                referenced_item=LdapItem(
                    item_mapper=MapByAttribute("gidNumber", base_dn="ou=groups,cn=example,cn=org"),
                    model=Attribute("member")
                    )
                reverse=DNInAttributeList("member")
            )
        }
    )

The group_database is defined as an LDAP item mapped by gid within the ou=groups,cn=example,cn=org. This means that gid will be used as _id, and the LdapConnection will look for the item with gidNumber == _id among the entries in ou=groups,cn=example,cn=org.

The admin attribute is a Ref to an other item that is mapped by DN. This means the value of admin in the final JSON-like dict is the DN of the referenced item. The referenced_item is a DatabaseItem, so it should be understood as any other DatabaseItem that could be treated outside a Ref. The attribute="owner" parameter states that the _id of the referenced item (the DN in this case) should be looked for in the owner attribute of the group.

The members RefList is similar, except that members is a list of DNs stored in member attributes of the group entry.

On the other hand, the user_database also uses MapByDN. This will allow Backo to properly handle admin and members Refs if required, because admin and members will be properly associated to the corresponding _id of user_database in generated JSON-like dicts.

The groups attribute is a bit more complicated, because it is a reference to an other entry and there is no corresponding attribute in the LDAP entry associated to the user. But we still want to include group _ids in the generated JSON-like dict, so Backo can later user it as a full featured Ref. In the database, we now the groups of the user can be retrieved from LDAP entries associated to groups: that's why the referenced_item is an LdapItem similar to group_database. Actually, group_database could be used directly as the referenced_item. The only difference is that the referenced_item is only required to include attributes used for the reverse search. In this case, the reverse search process looks for the DN of the user entry within the LDAP member list of each group entry to build the list of groups associated to the user. Because the referenced_item is mapped by attribute "gidNumber", the gid of each group will be included in the final groups list in the generated JSON-like dict.

The DatabaseEngine is responsible to build and execute all the extra LDAP requests required to build the list of groups, but it is transparent from the end user point of view: only a description of expected fields and where to find them is provided, but the execution flow is completely managed by the DatabaseEngine.

As you might have guessed, because referenced_item is actually a DatabaseItem, it is possible to reference items coming from any type of database, only specifying the specific connection that will be used by the attribute. This means the LdapItem can reference items coming from a YAML file, an SQL database or anything else. This will be detailed in a future example.

Also note that the DatabaseEngine only handles the representation of the reference in the field itself: it is not possible to create, update or delete a remote object from a reference. If an item is added to a database ref list, the DatabaseEngine only performs requests to update the list of reference for the current item in the database, if required (e.g. adding a new member=<DN> attribute in the group entry), but it does not ensure the update of the reference in the remote item. This logical part is indeed handled by backo itself, once all Refs and RefLists have been properly initialized with valid _ids by the DatabaseEngine.

How the DatabaseEngine works

flowchart TB;
    Database@{ shape: cyl, label: "Database" };
    Database --- DatabaseConnection;

    DatabaseEngine -- execute requests --> DatabaseConnection;
    DatabaseItem -- requests --> DatabaseEngine;
    DatabaseEngine -- responses --> DatabaseItem;
    DatabaseItem -- load --> JSONdict[JSON dict];

Loading

Given a DatabaseItem, the DatabaseEngine workflow consists in three steps:

1. ask the DatabaseItem for requests required to perform the operation (search, create, update, delete, select)
2. perform those requests against the real database backend using the DatabaseConnection
3. provide the responses to the DatabaseItem.load() method so it can build a JSON-like dict representation of the Item when required (e.g. for search, select).

The DatabaseItem builds requests using the following procedure:

1. build the base request required by the item_mapper to perform the operation
2. increment the base request or build additional requests as required to perform the operation on each attribute of the model.

While the item_mapper will provide a single request mapped to a single response instance, requests required by the model can be as complex as required. The associated responses will be stored in a structure that mirrors model.

For example, given the following model:

{
  "field1": attribute_1,
  "field2": attribute_2,
  "nested_field": {
    "field3": [attribute_3, attribute_4]
  }
}

the following set of requests will be built for a search operation, where base_request=item_mapper.search_request(<id>) is the request built by the item_mapper:

{
  "field1": attribute_1.search_request(base_request),
  "field2": attribute_2.search_request(base_request),
  "nested_field": {
    "field3": [attribute_3.search_request(base_request), attribute_4.search_request(base_request)]
  }
}

Each attribute is allowed to either modify the base_request, build a new request or do nothing. For example, the LDAP Attribute("name") will add "name" to the LDAP attributes of the base LDAP search query. ReverseRef will return a new search query for the set of referenced items so they can be associated to the item when the ReverseRef attribute is loaded.

Then the DatabaseEngine will perform requests using the DatabaseConnection.execute_search method to produce a base_response and attributes_responses organised as follows:

{
  "field1": response_1,
  "field2": response_2,
  "nested_field": {
    "field3": [response_3, response_4]
  }
}

All responses are then passed to DatabaseItem.load so that each field is loaded as follows to generate the final JSON-like dict:

{
  "field1": attribute_1.load(base_response, response_1),
  "field2": attribute_2.load(base_response, response_2),
  "nested_field": {
    "field3": [attribute_3.load(base_response, response_3), attribute_4.load(base_response, response_4)]
  }
}

Each attribute is even allowed to build a structured set of request. Considering the item:

{
    "data": data_attribute
}

the following requests might be built:

{
    "data": [request0, {"req1": request1, "req2": request2}]
}

Each request will be executed by the engine so that the loaded item is:

{
    "data": attribute1.load(
            base_response,
            [response0, {"req1": response1, "req2": response2}]
        )
}

Notice that in most cases, the base item_mapper request is enough, and database attributes will look for values in the base response. The ability to modify the base request allows to perform a single fined-tuned request, while still allowing additional requests for complex database attributes. This allows the DatabaseEngine to be very efficient, in spite of an high level of abstraction.

The fact that each attribute received only the response to its own request allows to easily build modular attributes with only a few lines of codes. New connectors are also easy to build because each operation can be implemented independently (no need to implement create, delete or update if you want a read-only connection). Moreover, the connector only focuses on executing each operation from requests, but it is not concerned about models and attributes: it is the responsibility of each attribute to build valid requests.

Implementation examples for YAML are provided in the Basic Connector Tutorial and Advanced Connector Tutorial.