Moving towards a 3-tier layer architecture?

[aldbr]

Current state

In the backend, we have 2 layers:

• the routers, presenting the API to the users, and containing the business logic
• the dbs, also containing some business logic and dealing with the dbs through an ORM

Context:

For now it's fine, but in the near future we are going to include:

• Agents: as they are hosted in the backend, they could be allowed to bypass routers and directly call the source code to avoid going through the network if it's not necessary
• Similarly, we will have services depending on other services, such as the production manager depending on the transformation manager depending on the job manager. To go faster, they could be allowed to bypass networking and call directly the source code.

Problem

In the good old Dirac, it seems like we implicitly made the choice to merge the business logic with the data access layer, which creates tremendous DB interfaces, not always easy to follow. Also, as this is - it seems to me at least - implicit, we sometimes find some logic in both the Services and the DB interfaces (we can even see that in the only service we imported to DiracX).

Potential solution

In order to have a clear separation of concerns and a clear organization of the code, given our requirements, I would go towards a new layer, so that we would have:

• Presentation layer: diracx-routers and FastAPI, that would be in charge of the API and the presentation of the information returned (e.g. HTTP Exceptions)
• Business Logic layer: diracx-business (?) that would simply contain Dirac business logic (potentially allowing us to move from whatever framework we find in the future in an "easier" way). This layer would call the DAL to store information.
• Data Access layer: diracx-dbs, which would only contain operations with the DBs and nothing else.

flowchart LR
    dbs["diracx-dbs (sqlalchemy/os)"] -- uses --> core["diracx-core (domain)"]
    business["diracx-business (Dirac)"] -- uses --> dbs & core
    services["diracx-services (FastAPI)"] -- uses --> business & core
    services -- generates --> openapi["OpenAPI"]
    tasks["diracx-tasks (celery?)"] -- uses  --> business & core
    client["diracx-client (autorest)"] -- consumes --> openapi
    client -- uses --> core
    api["diracx-api"] -- uses --> client
    cli["diracx-cli (typer)"] -- uses --> api & client
     dbs:::Pine
     business:::Pine
     services:::Pine
     openapi:::Rose
     tasks:::Pine
     client:::Sky
     api:::Sky
     cli:::Sky
    classDef Rose stroke-width:1px, stroke-dasharray:none, stroke:#FF5978, fill:#FFDFE5, color:#8E2236
    classDef Sky stroke-width:1px, stroke-dasharray:none, stroke:#374D7C, fill:#E2EBFF, color:#374D7C
    classDef Pine stroke-width:1px, stroke-dasharray:none, stroke:#254336, fill:#27654A, color:#FFFFFF

Loading

What do you think?

[aldbr]

After discussing with @chaen and @chrisburr, we decided that for now, the business logic could live in the diracx-dbs package to avoid over-designing diracx, but might be subject to changes in next releases.

We could have a structure such as:

- dal
  - sql/
    - job/
      - db.py
      - schema.py  
  - os/
    - job_parameters.py 
- bl/
  - jobs/
    - query.py
    - sandboxes.py
    - submission.py 

The BL currently contained in diracx-routers could be moved in diracx-dbs/bl, and diracx-routers would then act as a presentation layer.

[fstagni]

So, this is not an issue... (and can be closed).

[aldbr]

I will try to apply that structure and open a PR and then if we are all on the same page, we can close this issue I think.