Introduction

Introduction to connecto

Web and CRUD APIs are usually based on JSON to exchange data with the backend. It is then the responsibility of the backend to interpret requests and manage data persistence appropriatly according to the type of database it is connected to.

This task can be cumbersome, because the database scheme and API might be very far from the expected JSON exposed to the user. It also implies a lot of set up code that is not directly linked to the project features themselves, and to redesign how data will be committed to the database at each new project, with a lot of useless set up code.

connecto is a proposal for a low-code API that allows you to precisely describe how to build a JSON-like representation of your data from an existing database. Based on this description, connecto will efficiently handle the logic required to search, select, delete and update items in the database. All the complexity of real database operations is completely abstracted in generic search(), select(), delete() and save() methods. This allow the user to only focus on the necessary code that is required to run the application.

graph LR;
    user[End user]
    user <-- JSON --> Backend;
    Backend <-- JSON-like structure --> connecto;
    connecto <--> database

Loading

connecto is also designed to allow developers to easily implement new connectors to their own database, as explained in the Basic connector tutorial.

Use case examples

In most cases, a trivial set up might be enough. Here is a full featured example for a YAML database.

from connecto.yaml import YamlEngine

if __name__ == "__main__":
    yaml_engine = YamlEngine("test.yaml")

    _id = yaml_engine.create(
        {"name": "pipo", "gid": 12, "description": "Example user"}
    )

    print(f"Item {item_id} created.")

    _id, item = yaml_engine.search(_id))
    print(item)

    item["name"] = "molo"
    item["description"] = "Updated user"

    yaml_engine.save(item_id, item)

    print(f"Item {item_id} updated.")

    print("Selected items:")
    print(yaml_engine.select())
    print()

    yaml_engine.delete(_id)

Item 1db54689-ab2b-4542-b989-fc5e7bee2752 created.
{'gid': 12, 'name': 'pipo', 'description': 'Example user'}

Selected items:
[('1db54689-ab2b-4542-b989-fc5e7bee2752', {'gid': 12, 'name': 'molo', 'description': 'Updated user'})]

Item 1db54689-ab2b-4542-b989-fc5e7bee2752 updated.
('1db54689-ab2b-4542-b989-fc5e7bee2752', {'gid': 12, 'name': 'molo', 'description': 'Updated user'})

In this example, the YamlEngine("test.yaml") is enough to initialized a functionnal database engine that support all operations. Since no model is specified, it is assumed the structure of items in the YAML file matches the expected JSON-like structure. A custom model can be specified to control where each field is located.

from connecto.yaml import YamlEngine, YamlItem, YamlAttribute

if __name__ == "__main__":
    yaml_engine = YamlEngine(
            "test.yaml",
            YamlItem(
                model={
                    "login": YamlAttribute(["name"]),
                    "info": YamlAttribute(["description"])
                    }
                )
            )

    _id = yaml_engine.create(
        {"login": "pipo", "gid": 12, "info": "Example user"}
    )

    print(f"Item {_id} created.")

    _id, item = yaml_engine.search(_id)
    print(item)
    print()

    item["login"] = "molo"
    item["info"] = "Updated user"

    yaml_engine.save(_id, item)

    print(f"Item {_id} updated.")
    print(yaml_engine.search(_id))

    print("Selected items:")
    print(yaml_engine.select())
    print()

    yaml_engine.delete(_id)

Item 7699f489-3bf6-4887-b6ed-ae30d1a33512 created.
{'gid': 12, 'info': 'Example user', 'login': 'pipo'}

Selected items:
[('7699f489-3bf6-4887-b6ed-ae30d1a33512', {'gid': 12, 'info': 'Updated user', 'login': 'molo'})]

Item 7699f489-3bf6-4887-b6ed-ae30d1a33512 updated.
('7699f489-3bf6-4887-b6ed-ae30d1a33512', {'gid': 12, 'info': 'Updated user', 'login': 'molo'})

The name and description fields have been replaced by login and info in the exposed JSON-like structure, even if the underlying database still works with name and description (because in this example, this is the existing database scheme):

$ cat test.yaml
c67a397e-d4b6-4b65-b1d5-6a84e36d3806:
  description: Updated user
  gid: 12
  name: molo

A simple LDAP example

Warning

This is work in progress. The API might change in the future, but it illustrates target LDAP features.

from connecto.ldap import LdapEngine, LdapItem, MapByDN, LdapAttribute

from ldap3 import Server, Connection
server = Server('my_server')
conn = Connection(server, 'my_user', 'my_password', ...)

user_database = LdapEngine(
    conn,
    LdapItem(
        item_mapper=MapByDN(),
        model={
            "name": LdapAttribute("uid"),
            "description": LdapAttribute("description")
        }
    )
)

With this example, the LdapItem will load() the JSON-like dict by loading values of name and description fields from the uid and description attributes of the LDAP entry obtained from the result of the search for the DN associated to the item, that is used as _id.

A complex LDAP example

The following example illustrates how the DatabaseEngine can handle complex database schemes that can be very different from the expected JSON-like dict, contrary to the previous Yaml example.

Warning

This is work in progress. The API might change in the future, but it illustrates target LDAP features.

from ldap3 import Server, Connection
from connecto.ldap import LdapEngine, LdapItem, MapByAttribute, MapByDN

server = Server('my_server')
conn = Connection(server, 'my_user', 'my_password', ...)

group_engine = LdapEngine(
    conn,
    LdapItem(
        item_mapper=MapByAttribute("gidNumber", base_dn="ou=groups,cn=example,cn=org"),
        model={
            "name": LdapAttribute(),
            "admin": LdapAttribute(),
            "members": LdapAttribute("member")
            }
    )
)

user_engine = LdapEngine(
        conn,
        LdapItem(
            item_mapper=MapByDN(),
            model={
                "name": LdapAttribute("uid"),
                "surname": LdapAttribute("sn"),
                "groups": ReverseRefList(
                    select_engine=group_engine
                    item_filter=("$.members", Operator.CONTAINS, LdapDN())
                )
            }
        )
    )

The group_item is defined as an LDAP item mapped by gidNumber within the ou=groups,cn=example,cn=org. This means that gidNumber will be used as _id, and connecto will look for the item with gidNumber == _id among the entries in ou=groups,cn=example,cn=org.

Specifying LdapAttribute() without arguments states that the LDAP attribute has the same name as the current field in the model.

The admin attribute is a reference to an user, but it's treated as any other attribute because an user DN is expected in this field in the LDAP database. Since the user database uses MapByDN(), the value of this field can be used to easily resolve references.

The members is similarly of list of references represented in the database as a list of user DNs in the member attribute. The LDAP backend will automatically load the members field as a list.

The user_item uses MapByDN() so the DN of each entry will be considered as ID.

The groups field is based on a generic ReverseRefList attribute. It is useful if we want to fill the groups field of each user with a list of group reference even if no corresponding attribute exist in the LDAP database. In this example, the ReverseRefList will use the provided select_engine to select items associated to the reference. Since MapByAttribute("gidNumber", ...) is used as item mapper for the select engine, the groups field will be filled with gidNumbers of corresponding groups.

The item_filter is a generic filter that is used to find items associated to the user. It is applied on the model of the select_engine. Here it is applied to the $.members field. The filter is itself built on connecto attributes that will be resolved relative to the encapsulating item (not the referenced item). In this example LdapDN() is resolved to the DN of the user for which the reference must be built. The ReverseRefList can then be interpreted as "gidNumbers of groups where the list of members contain the DN of the user".

connecto is responsible to build and execute all the LDAP requests required to build each attribute. The user only focuses

However, connecto only handles the representation of the reference in the field itself: it is not possible to create, update or delete a remote object from a reference. Even if connecto can be used to add an item to the members list in the database, it does not ensure the list of groups will be updated in referenced items. Such logic can be handled by backo using Refs and RefLists that can be initialized from JSON-like structures produced by connecto.

What's next

If you want to implement a new connector, it might be useful to learn more about Working principles of connecto, even if the Basic tutorial might be enough.