Skip to content

Backend Solution

Jump to bottom
Ihar Suvorau edited this page Sep 14, 2023 · 6 revisions

Architecture

Hybrid approach

Monolithical web server with background jobs and external services:

flowchart LR
    Frontend --> Controllers

    subgraph backend[Backend]
        subgraph Web[Web Server]
            Controllers
            Auth[Auth Service]
            Users[Users Service]
            Projects[Projects Service]
            Assets[Assets Service]
            Storage[Storage Service]

            Auth --> AuthImpl[Auth Implementation]
            Users --> UsersImpl[Users Implementation]
            Projects --> ProjectsImpl[Projects Implementation]
            Assets --> AssetsImpl[Assets Implementation]
            Assets --> Storage
            Storage --> FileStorageImpl[File Storage Implementation]
            Storage --> BlobStorageImpl[Blob Storage Implementation]
            Storage --> EtcImpl[...]

            AuthImpl --> UtilitiesDB[(Utilities DB)]
            UsersImpl --> UtilitiesDB
            ProjectsImpl --> UtilitiesDB
            AssetsImpl --> UtilitiesDB

            FileStorageImpl --> FileStorage[(File Storage)]

            BlobStorageImpl --> BlobStorage[(Blob Storage)]

            Controllers --> Auth
            Controllers --> Users
            Controllers --> Projects
            Controllers --> Assets
        end

        subgraph BackgroundServices[Background Services]
            DiscoverJob[BPS Discovery Batch Job]
            SimulationJob[Simulation Batch Job]
            WTAJob[Waiting Time Analysis Batch Job]

            Controllers ---> DiscoverJob
            Controllers ---> SimulationJob
            Controllers ---> WTAJob
        end
    end

    subgraph ExternalServices[External Services]
        Notifications[Notifications Service]
        Discovery[BPS Discovery Service]
        Simulation[Simulation Service]
        WTA[WTA Service]
        
        DiscoverJob ---> Discovery
        DiscoverJob ---> Assets
        DiscoverJob ---> Notifications

        SimulationJob ---> Simulation
        SimulationJob ---> Assets
        SimulationJob ---> Notifications

        WTAJob ---> WTA
        WTAJob ---> Assets
        WTAJob --> Notifications
    end
Loading

Event-driven approach using microservices

flowchart LR
    Frontend --> Controllers

    subgraph Web[Web Server]
        Controllers
    end

    Controllers --> Auth
    Controllers --> Users
    Controllers --> Projects
    Controllers --> Assets

    subgraph Broker[Message Topics]
        BPSDiscoveryRequestTopic[[BPS Discovery Request Topic]]
        BPSDiscoveryResultTopic[[BPS Discovery Result Topic]]
        NotificationRequestTopic[[Notification Request Topic]]

        Controllers -- publish ---> BPSDiscoveryRequestTopic
    end

    subgraph Processors[Topic Processors]
        subgraph Group
            BPSDiscoveryRequestProcessor1[[BPS Discovery Request Processor 1]]
            BPSDiscoveryRequestProcessor2[[BPS Discovery Request Processor 2]]
        end
        BPSDiscoveryRequestProcessor1 -- subscribes --> BPSDiscoveryRequestTopic
        BPSDiscoveryRequestProcessor1 -- publishes --> BPSDiscoveryResultTopic
        BPSDiscoveryRequestProcessor2 -- subscribes --> BPSDiscoveryRequestTopic
        BPSDiscoveryRequestProcessor2 -- publishes --> BPSDiscoveryResultTopic
        
        BPSDiscoveryResultProcessor -- subscribes --> BPSDiscoveryResultTopic
        BPSDiscoveryResultProcessor -- publishes --> NotificationRequestTopic

        NotificationProcessor -- subscribes --> NotificationRequestTopic
    end

    BPSDiscoveryRequestProcessor1 -- uses ---> Simod
    BPSDiscoveryRequestProcessor1 -- uses ---> Assets
    BPSDiscoveryRequestProcessor2 -- uses ---> Simod
    BPSDiscoveryRequestProcessor2 -- uses ---> Assets
    BPSDiscoveryResultProcessor -- uses ---> Assets
    NotificationProcessor -- uses ---> Notifications

    Assets -- uses --> Storage
Loading
  • Control the number of concurrently running batch processes
  • Distribute batch processes across multiple machines (e.g., BPS discovery on several machines, simulation on several machines, etc.)
  • Should each processor call an external service or should it use the underlying tool directly?

P2P communication: New BPS Discovery Request

This is the point-to-point communication approach (vs. distributed messaging). Gateway serves as (a) a load balancer and (b) service discovery. We omit auth requests on the diagram to simplify it.

flowchart LR
    subgraph node1[pix.cloud.ut.ee]
        webserver
        gateway
        assets
    end

    subgraph node2[simod1.cloud.ut.ee]
        simod-http1
    end

    subgraph node3[simod2.cloud.ut.ee]
        simod-http2
    end

    frontend -- 1. new job --> gateway
    gateway -- 1. new job --> webserver
    webserver -- 2. get assets --> gateway
    gateway -- 2. get assets --> assets
    webserver -- 3. new job --> gateway -- 3. new job --> simod-http1 & simod-http2
    simod-http1 -- 4. job done webhook ---> gateway -- 4. job done webhook --> webserver
    simod-http2 -- 4. job done webhook ---> gateway
    webserver -- 5. get results --> gateway
    gateway -- 5. get results --> simod-http1 & simod-http2
    webserver -- 6. save results --> gateway
    gateway -- 6. save results --> assets
Loading

Distributed messaging: New BPS Discovery Request

flowchart LR
    subgraph node1[pix.cloud.ut.ee]
        webserver
        gateway
        assets
        kafka
    end

    subgraph node2[simod1.cloud.ut.ee]
        simod-worker1
    end

    subgraph node3[simod2.cloud.ut.ee]
        simod-worker2
    end

    frontend -- 1. new job --> gateway -- 1. new job --> webserver -- 1. new job --> kafka

    kafka -- 2. new job --> simod-worker1
    kafka -- 2. new job --> simod-worker2

    simod-worker1 -- 3. save results --> gateway -- 3. save results --> assets
    simod-worker2 -- 3. save results --> gateway -- 3. save results --> assets

    simod-worker1 -- 4. job result --> kafka
    simod-worker2 -- 4. job result --> kafka
Loading

Data Model

Data models for services:

  • ProcessingRequestsService
  • UsersService
  • ProjectsService
  • AssetsService
  • NotificationsService
erDiagram
    ProcessingRequest {
        string id
        enum type
        enum status
        string userID
        string projectID
        string[] inputAssetsIDs
        string[] outputAssetsIDs
        string message
        string startTime
        string endTime
    }

    User {
        string id
        string firstName
        string lastName
        string email
        string password
        string[] projectsIDs
        datetine creationTime
        datetime modificationTime
        datetime deletionTime
        datetime lastLoginTime
    }

    Project {
        string id
        string name
        string description
        string[] userIDs
        string[] assetsIDs
        string[] processingRequestsIDs
        datetime creationTime
        datetime modificationTime
        datetime deletionTime
    }

    Asset {
        string id
        string name
        string description
        enum type
        string projectID
        string[] processingRequestsIDs
        url location
        datetime creationTime
        datetime modificationTime
        datetime deletionTime
    }

    Notification {
        string id
        string userID
        string message
        enum status
        datetime creationTime
    }

    User }|--o{ Project : own

    User }|--o{ Notification : receive
    
    Project ||--o{ Asset : owns

    User ||--o{ ProcessingRequest : initiates
    ProcessingRequest ||--o{ Asset : "uses and produces"
    ProcessingRequest }o--|| Project : "has"
Loading

Use case 1: Waiting Time Analysis

WTA workflow is the first core use case we want to cover. It engages all our current core services: WTA, simulation, BPS discovery.

Workflow

flowchart LR
    UploadLog[Upload a log] --> RunWTA[Run WTA]
    RunWTA --> ExamineWTAReport[Examine WTA report]
    RunWTA --> DiscoverBPSModel[Discover BPS model]
    DiscoverBPSModel --> SimulateBPSModel[Simulate BPS model]
    SimulateBPSModel --> ExamineSimulationReport[Examine simulation report]
    ExamineSimulationReport --> ModifySimulationScenario[Modify simulation scenario]
    ModifySimulationScenario --> SimulateBPSModel
Loading

File upload sequence diagram

sequenceDiagram
    actor User
    participant Frontend
    participant Backend
    participant Assets
    participant Projects
    participant WaitingTimeAnalysis
    participant Simulation
    participant BPSDiscovery
    participant Notifications

    User ->> Frontend: upload event log
    Frontend ->> Backend: upload event log
    Backend ->> Projects: get project
    Projects -->> Backend: project
    Backend ->> Assets: upload event log
    Assets -->> Backend: event log uploaded
    Backend -->> Frontend: event log uploaded
    Frontend -->> User: event log uploaded
Loading

Analysis sequence diagram

sequenceDiagram
    actor User
    participant Frontend
    participant Backend
    participant Assets
    participant Projects
    participant WaitingTimeAnalysis
    participant Simulation
    participant BPSDiscovery
    participant Notifications

    User ->> Frontend: run WTA on event log
    Frontend ->> Backend: run WTA on event log
    Backend -->> Frontend: WTA accepted
    Frontend -->> User: WTA accepted
    Backend ->> Assets : get event log
    Assets -->> Backend: event log
    Backend ->> WaitingTimeAnalysis: run analysis on event log
    activate WaitingTimeAnalysis
    WaitingTimeAnalysis -->> Backend: analysis finished
    deactivate WaitingTimeAnalysis
    Backend ->> Assets: save analysis results
    Assets -->> Backend: analysis results saved
    Backend -->> Frontend: analysis finished
    Backend ->> Notifications: send notification
    activate Notifications
    Notifications -->> Backend: notification sent
    deactivate Notifications
Loading

Clone this wiki locally