ReferencesTutorial

Managing refs

Backo uses Refs and RefLists to dynamically manage references between Items, based on the internal Backo _ids of each item. This allows to transparently retrieve referenced item, and to automatically update remote fields in the reference item (e.g. if you modify the group of an user, the group is also modified to remove the user from the users list).

In this context, the purpose of the DatabaseEngine is only to load _ids of each referenced item so they can be loaded in Refs and RefLists. It only manages the content of each attribute representing references, but do not manage referenced items themselves.

The basic use case

This means in most cases, an attribute representing a reference in the DatabaseItem is a regular attribute that just need to load a value as any other attribute.

Here is an example:

# examples/yaml/ref.py

from backo.yaml.engine import YamlEngine
from backo.yaml.item import YamlItem, YamlAttribute

if __name__ == "__main__":
    yaml_group_engine = YamlEngine(
        "groups.yaml", database_item=YamlItem(
            model={
                "gid": YamlAttribute(["gid"]),
                "users": YamlAttribute(["users"])
                }
            )
    )
    yaml_user_engine = YamlEngine(
        "users.yaml", database_item=YamlItem(
            model={
                "group": YamlAttribute(["group"])
                }
            )
    )

    group_id = yaml_group_engine.create(
        {"gid": 12, "users": []}
    )

    user_id = yaml_user_engine.create(
            {"group": group_id}
            )
    group = yaml_group_engine.search(group_id)
    group["users"].append(user_id)
    yaml_group_engine.save(group_id, group)

    print(f"Group: {yaml_group_engine.search(group_id)}")
    print(f"User: {yaml_user_engine.search(user_id)}")

$ uv run examples/yaml/ref.py
Group: {'_id': '1f4cda00-f9d6-42ae-83f0-7b0dbbcd6aed', 'gid': 12, 'users': ['5b3795ac-11eb-4cc6-83af-7c0cda648e0f']}
User: {'group': '1f4cda00-f9d6-42ae-83f0-7b0dbbcd6aed', '_id': '5b3795ac-11eb-4cc6-83af-7c0cda648e0f'}

The previous YamlItems could then be bound to Backo Items defined as:

group = Item(schema={
    "gid": Int(),
    "users": RefList(collection="users", field="$.group")
    })

user = Item(schema={
    "group": Ref(collection="groups", field="$.users")
    })

Ref and RefList will transparently work with group and user _ids to manage references. However, they do not consider how _ids are actually mapped to the database, has this is the purpose of the DatabaseItem.

The only requirement on defined DatabaseItems for Backo references to work is that the values of reference attributes in the database can be used to load referenced items.

For example, let's consider the group field of users in the database is not the UUID YAML key used by the default key mapper, but the gid of the group. Then it's only required to change the item_mapper of the group DatabaseItem:

...

if __name__ == "__main__":
    yaml_group_engine = YamlEngine(
        "groups.yaml", database_item=YamlItem(
            item_mapper=MapByAttribute("gid"),
            model={
                "gid": YamlAttribute(["gid"]),
                "users": YamlAttribute(["users"])
                }
            )
    )
    
    ...

Since the gid attribute will now be used as the _id of groups, it is also what will be used to feed the group field of users. This will be handled transparently by Backo, as backo only considers generic _ids provided by the database engine.

$ uv run examples/yaml/ref.py
Group: {'_id': 12, 'gid': 12, 'users': ['5b3795ac-11eb-4cc6-83af-7c0cda648e0f']}
User: {'group': 12, '_id': '5b3795ac-11eb-4cc6-83af-7c0cda648e0f'}

The database file will look as follows:

# groups.yaml
- gid: 12
  users: ['5b3795ac-11eb-4cc6-83af-7c0cda648e0f']

# users.yaml

5b3795ac-11eb-4cc6-83af-7c0cda648e0f:
  group: 12

Handling one-way references

Backo references are always two-way (explicit one-to-one or one-to-many), and this is happy. This means any Ref or RefList must have its counterpart defined in the corresponding field.

However, some evil database does not enforce this (e.g. LDAP or some SQL scheme), what prevents the definition of references as in the previous example.

A classical example is how groups can be represented in an LDAP database using a groupOfNames. An entry for such a group typically looks like:

dn: cn=users,ou=groups,dc=example,dc=org
objectClass: top
objectClass: groupOfNames
cn: users
member: uid=user1,ou=people,dc=example,dc=org
member: uid=user2,ou=people,dc=example,dc=org

Considering the current DatabaseEngine features, it will be very easy to build an LdapItem that can load the DNs of users in a members list. For example:

group_database_item=LdapItem(
    # Map by DN by default
    model={
        "name": LdapAttribute("cn"),
        "members": LdapAttribute("member")
    }
)

The database engine could then produce JSON-like dicts for this database item that can be bound to the following Backo item:

group_item = Item(schema={
    "name": String(),
    "members": RefList(collection="users", field="$.groups")
    })

Then you expect the Backo user item to be defined like this to complete the two-way reference:

user_item = Item(schema={
    "groups": RefList(collection="groups", field="$.members")
})

You now need to define a DatabaseItem to load group _ids in the groups field. Here things become complicated, because your infamous LDAP schema does not provide any attribute in the entry of users indicating in which group it is included. But you still need it for Backo to work, and you want it because you want to easily retrieve and update groups of users using user["groups"].

Such one-way reference can actually occur in many existing database scheme. In order to easily solve it, especially from the end user point of view, we are going to use the DatabaseEngine features to build requests required to found the list of groups associated to each user.

To be continued...