Sequential and Concurrent Simulation Modes

This page describes the sequential and concurrent modes used in the wall construction simulation process, detailing their execution models, synchronization mechanisms, and the rationale behind multiple concurrency implementations. It explains how threading and multiprocessing are used to simulate real-world crew coordination while showcasing different approaches to task distribution, inter-process communication (IPC), and logging consistency across execution modes.

Table of Contents

1. Overview

2. Performance Considerations
   2.1 Sequential Mode: The Most Efficient Approach
   2.2 Overhead in Threading and Multiprocessing
     2.2.1 Constraints on Task Decomposition and Synchronization
     2.2.2 Differences in Progress Tracking
     2.2.3 Threading Overhead
     2.2.4 Multiprocessing Overhead
   2.3 Purpose of Concurrent Execution

3. Concurrency Configuration
   3.1 Available Concurrency Modes
   3.2 Environment Variables Affecting Concurrency

4. Concurrency Implementation
   4.1 Wall Construction
     4.1.1 Selecting the Simulation Type
     4.1.2 Managing Simulation Execution
     4.1.3 Sequential Mode - Vectorized Computation and Progress Tracking
     4.1.4 Handling Abort Signals from Celery

   4.2 Concurrency Utilities Module Structure

   4.3 Shared Logic for Threading and Multiprocessing
     4.3.1 Dynamic attribute resolution
     4.3.2 Task queue initialization
     4.3.3 Log extraction
     4.3.4 Centralized logging
     4.3.5 Log file management

   4.4 Threading-Based Execution
     4.4.1 Implementing Multi-Threaded Simulation
     4.4.2 Thread Coordination and Crew Synchronization

   4.5 Multiprocessing-Based Execution
     4.5.1 Managing Multi-Process Simulation
     4.5.2 Inter-Process Communication and Crew Synchronization
     4.5.3 Multiprocessing Result handling

1. Overview

In The Wall API, the wall construction simulation operates in:

• Sequential Mode: The fastest and most efficient mode, optimized through vectorized calculations, with minimal synchronization overhead and efficient batch processing.

• Concurrent Mode: Uses threading or multiprocessing to simulate multi-worker execution, showcasing task distribution, process coordination, and real-time progress tracking.

Both modes support abort signal handling, allowing simulations to be gracefully terminated when necessary.

---

2. Performance Considerations

The performance of the wall construction simulation is primarily influenced by synchronization requirements and the inherent structure of the task. While concurrent execution modes exist, they serve a specific role in demonstrating inter-process coordination and task distribution, rather than optimizing computation.

2.1 Sequential Mode: The Most Efficient Approach

Sequential execution remains the most efficient method for this simulation due to:

• Vectorized calculations: The simulation relies on NumPy for optimized numerical operations, reducing the need for explicit iteration.

• Efficient progress tracking: Instead of tracking every intermediate step, sequential mode stores only essential simulation data at key milestones, minimizing function calls and memory allocations.

• No inter-process synchronization overhead: Unlike concurrent modes, sequential execution avoids locks, inter-thread signaling, and inter-process communication (IPC).

Return to Top

2.2 Overhead in Threading and Multiprocessing

🗒️ Concurrency in the simulation primarily exists to showcase inter-process communication (IPC) and synchronization mechanisms, rather than to improve raw performance.

2.2.1 Constraints on Task Decomposition and Synchronization

• The main reason the concurrent modes do not achieve the same efficiency as the sequential mode is that the task cannot be effectively decomposed into independent subtasks due to its need for strict synchronization. The simulation models real-world construction crews working in tandem, where progress tracking and coordination introduce significant IPC overhead.

• Additionally, each iteration of the task has a low computational cost, meaning that CPU resources remain underutilized, and the primary bottleneck is the overhead of managing inter-process communication rather than raw processing power.

Return to Top

2.2.2 Differences in Progress Tracking

A key distinction between sequential and concurrent modes is how progress is tracked and logged:

• Sequential Mode: Uses batch processing for progress updates, reducing function calls and minimizing synchronization overhead. Only the essential simulation data is stored at key milestones.

• Concurrent Mode: Requires granular event logging, tracking real-time construction progress events such as wall section completions and crew relieving. Unlike the sequential approach, where updates happen in batches, concurrent execution logs detailed progress dynamically to maintain an accurate representation of task coordination.

Return to Top

2.2.3 Threading Overhead

• Synchronization overhead: Thread coordination relies on synchronization primitives like threading.Condition and threading.Event, which introduce waiting times as threads synchronize execution.

• Context switching: Frequent thread context switches introduce CPU overhead, making execution less efficient compared to the streamlined sequential approach.

Return to Top

2.2.4 Multiprocessing Overhead

• Inter-Process Communication (IPC) latency: Synchronization across processes relies on shared data structures (e.g., multiprocessing.Manager()), adding latency.

• Serialization and Deserialization: Data exchanged between processes must be serialized and deserialized, leading to additional processing delays.

• Process creation costs: Unlike threads, processes require more memory and startup time, adding initialization overhead.

Return to Top

2.3 Purpose of Concurrent Execution

The concurrent execution modes exist to:

• Demonstrate inter-process communication (IPC) - Showcasing how multiple workers coordinate tasks.

• Provide real-time progress logging - Capturing step-by-step construction details in log files.

• Showcase different synchronization techniques - Implementing multiple concurrency models (threading.Condition, threading.Event, multiprocessing.Event, Manager().Event, and Manager().Condition) to highlight various synchronization mechanisms.

✔ The existence of multiple concurrency simulation modes is primarily for demonstration purposes, showcasing familiarity with different synchronization techniques and execution models rather than addressing distinct performance needs.

Return to Top

---

3. Concurrency Configuration

3.1 Available Concurrency Modes

The concurrency mode for simulations is controlled through the CONCURRENT_SIMULATION_MODE environment variable, which determines whether the simulation executes in threaded, or multiprocessing mode.

Mode	Description
threading_v1	Uses Python threads with threading.Condition for worker synchronization.
threading_v2	Uses Python threads with threading.Event, providing a different synchronization approach.
multiprocessing_v1	Utilizes multiple processes and multiprocessing.Event for synchronization.
multiprocessing_v2	Uses multiple processes, but relies on multiprocessing.Manager().Event for IPC and coordination.
multiprocessing_v3	Implements multi-process execution, utilizing multiprocessing.Manager().Condition as an alternative synchronization mechanism.

Return to Top

3.2 Environment Variables Affecting Concurrency

Several environment variables impact the concurrency settings and execution behavior:

Variable	Description
CONCURRENT_SIMULATION_MODE	Specifies the concurrency mode (one of the modes listed above).
CPU_THREADS	Number of available CPU threads, used for multiprocessing.
MAX_CONCURRENT_NUM_CREWS_MULTIPROCESSING	Maximum crews allowed in multiprocessing before switching to sequential mode.
MAX_CONCURRENT_NUM_CREWS_THREADING	Maximum crews allowed in threading before switching to sequential mode.
MAX_SECTIONS_COUNT_CONCURRENT_MULTIPROCESSING	Maximum sections processed concurrently in multiprocessing before switching to sequential mode.
MAX_SECTIONS_COUNT_CONCURRENT_THREADING	Similar to above, but for threading.
SECTION_COMPLETION_GRACE_PERIOD_MULTIPROCESSING	Wait time before marking a section as complete in multiprocessing.
SECTION_COMPLETION_GRACE_PERIOD_THREADING	Similar to above, but for threading.
VERBOSE_MULTIPROCESSING_LOGGING	Enables detailed multiprocessing logs for debugging, (adding some overhead).

✔ These settings ensure that concurrent execution remains efficient while preventing unnecessary overhead from excessive synchronization.

💡 The simulation mode (sequential or concurrent) is determined dynamically based on the num_crews query parameter.

💡 For a detailed description of the environment variables affecting concurrency, refer to the Concurrency and Performance section of the Environment Variables Index page.

Return to Top

---

4. Concurrency Implementation

The concurrency utilities provide the infrastructure for managing execution across different simulation modes.

4.1 Wall Construction

The wall_construction.py module orchestrates the simulation process, selecting the appropriate simulation type, managing progress tracking, and handling external abort signals.

4.1.1 Selecting the Simulation Type

The simulation type (sequential or concurrent) is determined in wall_construction.manage_num_crews(), which evaluates the number of crews (num_crews) and total sections (sections count) against predefined thresholds before the simulation begins.

The decision process follows these steps:

1. The simulation type is set to sequential when any of the following conditions are met:

   • num_crews is not provided, meaning the system assumes an unlimited number of crews and distributes work in a straightforward manner.
   • num_crews is explicitly set to 0, indicating no parallel workers are assigned.
   • num_crews is equal to or greater than the sections count, meaning there are enough crews to work on all sections simultaneously, eliminating the need for coordination overhead.
   • If the CONCURRENT_SIMULATION_MODE is set to threading (threading_v1 or threading_v2), the system falls back to sequential execution when:
     • The num_crews exceeds the threshold allowed for threading (MAX_CONCURRENT_NUM_CREWS_THREADING), leading to excessive thread management overhead.
     • The sections count exceeds the threading concurrency limit (MAX_SECTIONS_COUNT_CONCURRENT_THREADING), making the threading mode inefficient.
   • If the CONCURRENT_SIMULATION_MODE is set to multiprocessing (multiprocessing_v1, v2, or v3), the system reverts to sequential execution when:
     • The num_crews exceeds the threshold allowed for multiprocessing (MAX_CONCURRENT_NUM_CREWS_MULTIPROCESSING), causing high inter-process communication (IPC) overhead.
     • The sections count exceeds the multiprocessing concurrency threshold (MAX_SECTIONS_COUNT_CONCURRENT_MULTIPROCESSING), making process management too expensive.

2. If none of the above conditions are met, the simulation type is set to concurrent, meaning either threading or multiprocessing will be used based on the predefined system configuration (CONCURRENT_SIMULATION_MODE).

✔ While all simulations can run in sequential mode for maximum efficiency, concurrency is enabled in smaller configurations (fewer crews, fewer sections) to demonstrate multi-worker execution, progress tracking, and task coordination.

✔ For larger configurations, the system defaults to sequential execution to maintain efficiency and avoid excessive synchronization costs.

Return to Top

4.1.2 Managing Simulation Execution

Once the simulation type is selected, wall_construction.calc_wall_profile_data() initiates the execution process. This function serves as the entry point for wall construction simulations, handling the following cases:

• Sequential Simulation: Runs wall_construction.calc_wall_profile_data_sequential(), utilizing vectorized NumPy computations for maximum efficiency.
• Threading-Based Simulation: Delegates task execution to ThreadingWallBuilder from threading_utils.py, enabling multi-threaded processing.
• Multiprocessing-Based Simulation: Spawns multiple worker processes using MultiprocessingWallBuilder from multiprocessing_utils.py, distributing workloads across separate processes.

✔ The execution flow is designed to maintain accuracy while accommodating different simulation types. Sequential execution minimizes overhead, while concurrent modes introduce explicit logging and controlled synchronization, providing clear visibility into multi-worker execution and task coordination.

Return to Top

4.1.3 Sequential Mode - Vectorized Computation and Progress Tracking

In sequential mode, the system executes wall_construction.calc_wall_profile_data_sequential(), which is optimized using NumPy-based vectorized computations. This function utilizes:

• Batch updates via base_concurrency_utils.update_wall_profile_data_batch().
• Direct memory operations, avoiding redundant function calls for efficiency.
• Minimal synchronization overhead, as it operates in a single-threaded manner.

✔ Sequential execution is the preferred approach for larger wall configurations because it avoids inter-process communication overhead. It is also optimized for operations like wall profile updates and batch computations, which are not CPU-intensive in this simulation, ensuring maximum efficiency.

Return to Top

4.1.4 Handling Abort Signals from Celery

The wall construction simulation includes an abort signal handling mechanism to ensure that active simulations can be terminated in a controlled manner. This is implemented via a dedicated listener thread, which continuously monitors whether the Celery task that initiated the simulation has been aborted.

• The core functionality is managed through:

  • wall_construction.start_abort_signal_listener_thread() - Spawns a separate listener thread that periodically checks if the Celery task has been aborted.
  • Abort flags for different concurrency modes:
    • celery_task_aborted - Used to signal an abort request for threading-based simulations.
    • celery_task_aborted_mprcss - Used for multiprocessing-based simulations.

• If the listener thread detects an abort signal:

  • The abort flags (celery_task_aborted, celery_task_aborted_mprcss) are set to True, signaling all active simulations to terminate.
  • In each simulation mode the abort flags are checked at key points in the simulation loop and if an abort signal is detected, the execution is stopped. The handling of the abort signal is implemented separately in:
    • Sequential Mode → wall_construction.calc_wall_profile_data_sequential()
    • Threading Mode → threading_utils.process_sections()
    • Multiprocessing Mode → multiprocessing_utils.process_sections()

✔ This mechanism ensures that task cancellation is handled cleanly, preventing unnecessary computations while maintaining proper system state.

✔ Unlike Celery's basic .revoke() method, which forcibly stops tasks and may leave them in an inconsistent state, the abort signal mechanism ensures controlled termination, preserving data integrity and synchronization.

💡 For more details, refer to the Abort signals section in the Celery Asynchronous Task Processing page.

Return to Top

---

4.2 Concurrency Utilities Module Structure

• The concurrency utilities provide a unified framework that supports both threading and multiprocessing execution modes.
• The the_wall_api/utils/concurrency_utils folder contains core modules responsible for managing execution across the concurrency modes.

Module	Description
├── utils/
├── concurrency_utils/
├── base_concurrency_utils.py	Shared logic for threading and multiprocessing execution.
├── threading_utils.py	Implements multi-threaded execution.
└── multiprocessing_utils.py	Manages multi-process execution.

Return to Top

---

4.3 Shared Logic for Threading and Multiprocessing

The base_concurrency_utils.py module provides common functionality for both threading and multiprocessing execution modes, ensuring a unified approach to simulation management.

Key responsibilities of the base_concurrency_utils.BaseWallBuilder class include:

4.3.1 Dynamic attribute resolution

• Implements attribute delegation (via __getattr__()), allowing BaseWallBuilder to forward both method calls and attribute lookups to the underlying wall_construction instance.

✔ This design choice ensures that threading and multiprocessing modes retain full access to core simulation logic while avoiding duplicate attributes, method definitions, and redundant boiler-plate forwarding code in concurrency-specific classes.

Return to Top

4.3.2 Task queue initialization

• The BaseWallBuilder.init_sections_queue() method prepares the queue structure used to manage wall section assignments across crews. Each entry in the queue represents a compound wall section identifier + starting height (profile_id, section_id, height), which is later distributed to crews for construction.

✔ This ensures controlled section distribution and consistency across concurrency modes.

Return to Top

4.3.3 Log extraction

• Generates real-time progress messages for section updates, completions, and crew changes while ensuring that the final log stream is consistently formatted and written to the build simulation log file.

✔ This ensures a unified logging approach across concurrency modes.

Return to Top

4.3.4 Centralized logging

• The BaseWallBuilder.setup_logger()method establishes a shared logging framework that prevents conflicts when multiple concurrent simulations are running.

  • Threading mode: Uses a shared logger instance with thread-safe writes, ensuring that logs from different threads do not overlap or become interleaved.
  • Multiprocessing mode: Uses process-aware log handlers to prevent log corruption and ensure safe writes from multiple processes.
    • Non-Manager implementation (multiprocessing_v1): Uses a queue handler (QueueHandler), where worker processes enqueue log records into a shared result_queue. These logs are sequentially processed by a dedicated result_handler thread, which runs as a separate thread in the main process and stores logs in an in-memory StringIO buffer.
    • Manager-based implementation (multiprocessing_v2, multiprocessing_v3): Also uses a queue handler , but logs are processed in real time by a dedicated queue listener (QueueListener). Like result_handler, the queue listener retrieves log messages from a shared result_queue and writes them to a StringIO buffer.

Return to Top

4.3.5 Log file management

• Ensures structured, process-safe logging while preserving the synchronized simulation progress.

  • Filename format:
    <timestamp>_<wall_config_hash>_<num_crews>_<random_token>.log

    • <timestamp> → YYYY_MM_DD_HH_MM_SS_uS (e.g., 2025_02_17_14_30_45_123456).
    • <wall_config_hash> → Unique identifier for the wall configuration.
    • <num_crews> → Number of crews in the simulation.
    • <random_token> → Random 8-character hex string for uniqueness.

  • File location:
    Logs are stored in logs/build_simulations/, created automatically if missing.

  • Buffered logging:
    Build simulations progress is stored in-memory (StringIO) and dumped into the log file unformatted at the simulation’s end, preserving the synchronized event sequence.

✔ This mechanism ensures thread/process safety, preventing issues such as overlapping log entries, race conditions, log corruption, and inconsistent formatting in concurrent simulations.

✔ Using StringIO for buffered logging minimizes I/O overhead by keeping logs in memory during execution.

Return to Top

---

4.4 Threading-Based Execution

4.4.1 Implementing Multi-Threaded Simulation

The threading_utils.py module manages the multi-threaded simulation through the ThreadingWallBuilder class.

• Key characteristics of the multi-threaded simulation:

  • Thread pool management - A ThreadPoolExecutor dynamically assigns worker threads to process wall sections in parallel while handling task scheduling efficiently.
  • Shared memory access - Unlike multiprocessing, all threads operate within the same memory space, avoiding the need for inter-process communication (IPC) overhead.
  • Task queueing - Wall sections are retrieved and processed by available threads from a centralized sections queue, ensuring controlled execution flow.

Return to Top

4.4.2 Thread Coordination and Crew Synchronization

• Synchronization mechanisms:

  • threading.Condition (threading_v1) - Provides a structured synchronization model with a built-in locking context, allowing threads to both wait and notify others within the same block.
  • threading.Event (threading_v2) - Uses a shared event flag to signal threads when they should proceed. While more lightweight, it requires separate logic for notifying and waiting.

• Crew coordination:

  • Utilizes the Condition and Event synchronization primitives to simulate real-world end-of-day crew alignment by ensuring all threads complete their tasks before transitioning to the next phase (day).
  • Handles crew relieving, ensuring that workers are dismissed when there are no more sections to process.

• Abort signal handling:

  • Worker threads continuously check the abort flag (celery_task_aborted) to detect termination requests and exit gracefully when an abort signal is received.

✔ This approach maintains structured execution flow, ensuring that crews complete their assignments in a controlled and realistic manner.

Return to Top

---

4.5 Multiprocessing-Based Execution

4.5.1 Managing Multi-Process Simulation

The multiprocessing_utils.py module provides the concurrency framework for multi-process simulation through the MultiprocessingWallBuilder class.

Key characteristics of the multi-process simulation:

• Process-based parallelism - Unlike threading, multiprocessing creates separate memory spaces for each worker, avoiding Global Interpreter Lock (GIL) limitations.

• Inter-process task distribution - Wall sections are assigned to worker processes via a shared sections queue, ensuring even distribution of workloads.

• Static method design - The MultiprocessingWallBuilder functions implementing the core synchronization logic are defined as @staticmethod to avoid IPC serialization issues. Since Python's multiprocessing relies on pickle for passing objects between processes, instance methods — including their self reference — cannot be pickled if they contain non-serializable attributes like locks or queues.

• Manager vs. non-Manager implementations - Two approaches exist for managing shared state in the multi-process simulation:

  • Non-Manager (multiprocessing_v1) - Uses native synchronization primitives (multiprocessing.Value, multiprocessing.Event, multiprocessing.Queue), reducing serialization overhead.
  • Manager (multiprocessing_v2, multiprocessing_v3) - Uses multiprocessing.Manager() to create shared objects, simplifying access to complex data structures at the cost of additional IPC overhead.

• Process lifecycle management:

  • Worker processes are spawned using explicit Process objects for multiprocessing_v1, ensuring direct process control without additional IPC overhead.
  • When a Manager is required (multiprocessing_v2, multiprocessing_v3), ProcessPoolExecutor is used to handle worker process execution, as it integrates with Manager-based shared objects while abstracting manual process management.

✔ These design choices address serialization constraints in multiprocessing, ensuring efficient parallel execution and structured task management.

Return to Top

4.5.2 Inter-Process Communication and Crew Synchronization

• Synchronization mechanisms:

  • multiprocessing.Event (multiprocessing_v1) - Provides a basic signaling mechanism where worker processes check a shared event flag to determine if they should proceed.
  • multiprocessing.Manager().Event (multiprocessing_v2) - A Manager-managed event object enabling shared state synchronization across processes. Requires separate logic for notifying and waiting.
  • multiprocessing.Manager().Condition (multiprocessing_v3) - Provides a more structured mechanism for worker coordination, allowing processes to wait and notify in a controlled manner, similar to threading.Condition.

• Crew coordination:

  • Simulates real-world crew synchronization by ensuring that all worker processes complete their assigned sections before transitioning to the next phase (day).
  • Handles crew relieving, ensuring that processes are terminated only when there are no remaining unprocessed sections.

• Inter-Process Communication (IPC):

  • Worker processes operate in separate memory spaces, requiring explicit data exchange via multiprocessing queues and Manager objects.
  • multiprocessing.Queue is used in multiprocessing_v1, enabling direct process-to-process communication with minimal serialization overhead.
  • multiprocessing.Manager().Queue is used in multiprocessing_v2/v3, providing a centralized queue managed by a separate Manager process, at the cost of additional IPC latency.

• Abort signal handling:

  • Worker processes continuously check the abort flag (celery_task_aborted_mprcss) to detect termination requests and exit cleanly without leaving incomplete tasks, preventing potential data inconsistencies.

✔ This approach maintains structured execution flow in a multi-process environment, handling IPC constraints while ensuring realistic crew synchronization and safe termination of worker processes.

Return to Top

4.5.3 Multiprocessing Result handling

• Non-Manager Implementation (multiprocessing_v1)

  • Logging:

    • Uses a queue handler (QueueHandler), where worker processes enqueue log records into a shared result_queue.
    • A dedicated result_handler thread, runs in the main process, retrieves logs sequentially from the same result_queue and stores them in an in-memory StringIO buffer.

  • Wall Data Aggregation:

    • Worker processes push simulation results into the shared result_queue.
    • The result_handler retrieves progress data and updates the wall_profile_data dictionary, tracking simulation progress in the main process.

• Manager-Based Implementation (multiprocessing_v2, multiprocessing_v3)

  • Logging:

    • Also uses a queue handler, but logs are processed by a dedicated queue listener (QueueListener).
    • Log records are continuously pulled from a shared result_queue and stored in an in-memory StringIO buffer, ensuring non-blocking logging behavior.

  • Wall Data Aggregation:

    • Worker processes push simulation results into an additional shared result_queue_with_manager, different from the one used for logging.
    • The result_handler retrieves progress data and updates wall_profile_data in the main process, handling data aggregation separately from logging.

• Termination: Once all worker processes finish, a termination signal (None) is placed in all result queue instances, prompting the handler thread to exit.

✔ This approach ensures structured result collection and prevents race conditions in logging.

---

Next:   Test Suite
Back to: Celery - Asynchronous Task Processing

Return to Top
DTS Index
Home Page

---