Backend Architecture

Hybrid approach

Monolithical web server with background jobs and external services:

flowchart LR
    Frontend --> Controllers

    subgraph backend[Backend]
        subgraph Web[Web Server]
            Controllers
            Auth[Auth Service]
            Users[Users Service]
            Projects[Projects Service]
            Assets[Assets Service]
            Storage[Storage Service]

            Auth --> AuthImpl[Auth Implementation]
            Users --> UsersImpl[Users Implementation]
            Projects --> ProjectsImpl[Projects Implementation]
            Assets --> AssetsImpl[Assets Implementation]
            Assets --> Storage
            Storage --> FileStorageImpl[File Storage Implementation]
            Storage --> BlobStorageImpl[Blob Storage Implementation]
            Storage --> EtcImpl[...]

            AuthImpl --> UtilitiesDB[(Utilities DB)]
            UsersImpl --> UtilitiesDB
            ProjectsImpl --> UtilitiesDB
            AssetsImpl --> UtilitiesDB

            FileStorageImpl --> FileStorage[(File Storage)]

            BlobStorageImpl --> BlobStorage[(Blob Storage)]

            Controllers --> Auth
            Controllers --> Users
            Controllers --> Projects
            Controllers --> Assets
        end

        subgraph BackgroundServices[Background Services]
            DiscoverJob[BPS Discovery Batch Job]
            SimulationJob[Simulation Batch Job]
            WTAJob[Waiting Time Analysis Batch Job]

            Controllers ---> DiscoverJob
            Controllers ---> SimulationJob
            Controllers ---> WTAJob
        end
    end

    subgraph ExternalServices[External Services]
        Notifications[Notifications Service]
        Discovery[BPS Discovery Service]
        Simulation[Simulation Service]
        WTA[WTA Service]
        
        DiscoverJob ---> Discovery
        DiscoverJob ---> Assets
        DiscoverJob ---> Notifications

        SimulationJob ---> Simulation
        SimulationJob ---> Assets
        SimulationJob ---> Notifications

        WTAJob ---> WTA
        WTAJob ---> Assets
        WTAJob --> Notifications
    end

Loading

Event-driven approach using microservices

flowchart LR
    Frontend --> Controllers

    subgraph Web[Web Server]
        Controllers
    end

    Controllers --> Auth
    Controllers --> Users
    Controllers --> Projects
    Controllers --> Assets

    subgraph Broker[Message Topics]
        BPSDiscoveryRequestTopic[[BPS Discovery Request Topic]]
        BPSDiscoveryResultTopic[[BPS Discovery Result Topic]]
        NotificationRequestTopic[[Notification Request Topic]]

        Controllers -- publish ---> BPSDiscoveryRequestTopic
    end

    subgraph Processors[Topic Processors]
        subgraph Group
            BPSDiscoveryRequestProcessor1[[BPS Discovery Request Processor 1]]
            BPSDiscoveryRequestProcessor2[[BPS Discovery Request Processor 2]]
        end
        BPSDiscoveryRequestProcessor1 -- subscribes --> BPSDiscoveryRequestTopic
        BPSDiscoveryRequestProcessor1 -- publishes --> BPSDiscoveryResultTopic
        BPSDiscoveryRequestProcessor2 -- subscribes --> BPSDiscoveryRequestTopic
        BPSDiscoveryRequestProcessor2 -- publishes --> BPSDiscoveryResultTopic
        
        BPSDiscoveryResultProcessor -- subscribes --> BPSDiscoveryResultTopic
        BPSDiscoveryResultProcessor -- publishes --> NotificationRequestTopic

        NotificationProcessor -- subscribes --> NotificationRequestTopic
    end

    BPSDiscoveryRequestProcessor1 -- uses ---> Simod
    BPSDiscoveryRequestProcessor1 -- uses ---> Assets
    BPSDiscoveryRequestProcessor2 -- uses ---> Simod
    BPSDiscoveryRequestProcessor2 -- uses ---> Assets
    BPSDiscoveryResultProcessor -- uses ---> Assets
    NotificationProcessor -- uses ---> Notifications

    Assets -- uses --> Storage

Loading

• Control the number of concurrently running batch processes
• Distribute batch processes across multiple machines (e.g., BPS discovery on several machines, simulation on several machines, etc.)
• Should each processor call an external service or should it use the underlying tool directly?

P2P communication: New BPS Discovery Request

This is the point-to-point communication approach (vs. distributed messaging). Gateway serves as (a) a load balancer and (b) service discovery. We omit auth requests on the diagram to simplify it.

flowchart LR
    subgraph node1[pix.cloud.ut.ee]
        webserver
        gateway
        assets
    end

    subgraph node2[simod1.cloud.ut.ee]
        simod-http1
    end

    subgraph node3[simod2.cloud.ut.ee]
        simod-http2
    end

    frontend -- 1. new job --> gateway
    gateway -- 1. new job --> webserver
    webserver -- 2. get assets --> gateway
    gateway -- 2. get assets --> assets
    webserver -- 3. new job --> gateway -- 3. new job --> simod-http1 & simod-http2
    simod-http1 -- 4. job done webhook ---> gateway -- 4. job done webhook --> webserver
    simod-http2 -- 4. job done webhook ---> gateway
    webserver -- 5. get results --> gateway
    gateway -- 5. get results --> simod-http1 & simod-http2
    webserver -- 6. save results --> gateway
    gateway -- 6. save results --> assets

Loading

Distributed messaging: New BPS Discovery Request

flowchart LR
    subgraph node1[pix.cloud.ut.ee]
        webserver
        gateway
        assets
        kafka
    end

    subgraph node2[simod1.cloud.ut.ee]
        simod-worker1
    end

    subgraph node3[simod2.cloud.ut.ee]
        simod-worker2
    end

    frontend -- 1. new job --> gateway -- 1. new job --> webserver -- 1. new job --> kafka

    kafka -- 2. new job --> simod-worker1
    kafka -- 2. new job --> simod-worker2

    simod-worker1 -- 3. save results --> gateway -- 3. save results --> assets
    simod-worker2 -- 3. save results --> gateway -- 3. save results --> assets

    simod-worker1 -- 4. job result --> kafka
    simod-worker2 -- 4. job result --> kafka

Loading