Redis In Memory Cache and Celery Message Broker

This page provides an overview of how Redis is used in The Wall API for in-memory caching and as a Celery message broker. It details Redis configuration, caching policies, and its integration with Celery for task processing.

Table of Contents

1. Overview
2. Redis Configuration
3. Redis Databases and Their Roles
4. In-Memory Cache
   4.1 Wall Configuration Caching Behavior
   4.2 Redis Cache Keys Construction Logic
5. Celery Message Broker

1. Overview

• Redis is used for both caching and task queuing within the application.
• It operates as an in-memory cache to improve API performance by reducing redundant database queries.
• It functions as a Celery message broker, handling distributed task execution and queue management.

2. Redis Configuration

• Redis is configured in:

  settings.py

  
  

  REDIS_CACHE_TRANSIENT_DATA_TIMEOUT = int(os.getenv('REDIS_CACHE_TRANSIENT_DATA_TIMEOUT', 60 * 60 * 24 * 7))
  
  CACHES = {
      'default': {
          'BACKEND': 'django_redis.cache.RedisCache',
          'LOCATION': REDIS_URL,
          'OPTIONS': {
              'CLIENT_CLASS': 'django_redis.client.DefaultClient',
              'IGNORE_EXCEPTIONS': True,
              'SOCKET_CONNECT_TIMEOUT': REDIS_SOCKET_CONNECT_TIMEOUT,
              'SOCKET_TIMEOUT': REDIS_SOCKET_TIMEOUT,
          },
          'TIMEOUT': None
      }
  }

  env_utils.py

  
  

  if PROJECT_MODE == 'prod_v1':
      REDIS_PASSWORD = os.getenv('REDIS_PASSWORD', '')
      REDIS_URL = REDIS_URL.replace('REDIS_PASSWORD', REDIS_PASSWORD)
  elif PROJECT_MODE == 'prod_v2':
      REDIS_PASSWORD = open(os.environ['REDIS_PASSWORD_FILE']).read().strip()
      REDIS_URL = REDIS_URL.replace('REDIS_PASSWORD', REDIS_PASSWORD)
  
  CELERY_BROKER_URL = REDIS_URL.replace('REDIS_DB_NUMBER', REDIS_DB_NUMBER_CELERY)
  
  if not ACTIVE_TESTING:
      REDIS_URL = REDIS_URL.replace('REDIS_DB_NUMBER', REDIS_DB_NUMBER)
  else:
      REDIS_URL = REDIS_URL.replace('REDIS_DB_NUMBER', '1')

  
  

• The values of these settings are loaded from .env files stored in the config/envs/ folder.

  Variable	Description
  REDIS_CACHE_TRANSIENT_DATA_TIMEOUT	Redis TTL (Time To Live) for transient data
  REDIS_DB_NUMBER	In-memory caching database number
  REDIS_DB_NUMBER_CELERY	Celery database number
  REDIS_PASSWORD	Redis password
  REDIS_SOCKET_CONNECT_TIMEOUT	Connection timeout in seconds
  REDIS_SOCKET_TIMEOUT	Redis request timeout
  REDIS_URL	Redis connection URL

  🗒️ In the Production v2 environment, REDIS_PASSWORD is stored in Docker secrets instead of .env files.

  💡 config/envs/ folder breakdown → Environment Configuration Files Folder Structure.

  💡 For a detailed description of the Redis environment variables, refer to the Redis Caching and Message Queue Configuration section of the Environment Variables Index page.
  
  

• Redis configuration file

  Additionally, Redis server settings are managed in config/envs/common/redis.conf to optimize memory management and persistence:

  Setting	Value	Reasoning
  maxmemory	512mb	✔ Restricts Redis to a maximum of 512MB of RAM, preventing excessive memory usage.
  maxmemory-policy	volatile-lru	✔ Evicts only the least recently used (LRU) keys that have an expiration (TTL), ensuring Celery task data persists while only temporary cache is removed.
  appendonly	yes	✔ Enables Append-Only File (AOF) logging, ensuring data is persisted and recoverable after a crash or restart.
  auto-aof-rewrite-percentage	100	✔ Triggers an AOF rewrite when the file doubles in size, reducing disk space usage while maintaining data integrity.
  auto-aof-rewrite-min-size	64mb	✔ Prevents frequent AOF rewrites by ensuring a minimum file size of 64MB before triggering a rewrite, improving performance.

Return to Top

---

3. Redis Databases and Their Roles

To efficiently manage different types of data, the application uses multiple Redis databases, separating application caching, test suite data, and Celery task management.

Redis DB	Purpose	Details
DB 0	Main application data cache	Stores only TTL-based cache keys for frequently accessed API data.
DB 1	Test suite cache	Used for testing and cleared automatically after each run.
DB 2	Celery broker	Handles Celery task queuing, results, and error tracking.

• DB 0 - Main Cache:

  ✔ Improves response times by caching frequently requested API data.
  ✔ TTL-based eviction (maxmemory-policy=volatile-lru) automatically removes expired cache keys, preventing stale data accumulation.

• DB 1 - Test Suite Cache:

  ✔ Ensures test data is isolated from production.
  ✔ Is automatically cleared after test runs to maintain a clean and predictable test state.

• DB 2 - Celery Broker:

  ✔ Allows task state tracking and result retrieval by temporarily storing task metadata in Redis.
  ✔ Persists the global error counter (unknown_errors_counter) to track unknown errors reliably.

Return to Top

---

4. In-Memory Cache

4.1 Wall Configuration Caching Behavior

The In-Memory Caching is used for cost and usage requests to improve response times and reduce database load.

• Caching Strategy

  • Controlled by MAX_SECTIONS_COUNT_FULL_RANGE_CACHING.

  • When a wall configuration has fewer sections than this threshold, all simulation results for different num_crews values are precomputed and stored in the database. Redis caching is not used.

  • If the section count exceeds this threshold, results are computed on demand and cached in the DB and Redis only for the requested num_crews value.

  ✔ Redis caching is not used for full-range simulations, as the additional overhead slows down response times without significant benefits.

• Dynamic Cache Population:

  • On a cost or usage request, the system first checks Redis.
  • If the data is not cached, it is retrieved from the database and then stored in Redis for subsequent requests.
  • If Redis data is evicted, it is automatically repopulated on the next database retrieval.

  ✔ Dynamic cache population prevents redundant database queries by keeping frequently accessed data in Redis. This ensures fast retrieval while also maintains consistency with the database on subsequent requests.

• Deferred caching:

  • When a Wall instance is created, a database transaction is executed first to store the related objects network.
  • The system acquires a PostgreSQL advisory lock to prevent concurrent processes from modifying the same Wall object while it’s being cached.
  • Only after the transaction is successfully committed is the computed Wall data written to Redis.
  • If the database transaction fails or is rolled back, the cache is never updated, preventing Redis from containing outdated or invalid data.

  ✔ This guarantees that Redis always reflects fully processed and committed Wall cost and usage results.

Return to Top

4.2 Redis Cache Keys Construction Logic

• Profiles Endpoints

  • Keys are generated following a predefined format in storage_utils.py. The cache key structure follows the unique constraint of the ORM model of the cached data.
  • The keys include the wall_config_hash, num_crews, profile_id, and other relevant parameters, mirroring the rules that enforce uniqueness in the database.
  • The same logic is reused to build unique keys for the PostgreSQL advisory lock.

  ✔ Cache keys are constructed deterministically, allowing identical requests to reuse the same cache entry, preventing redundant storage and computation.

• Global Error Counter

  • The unknown_errors_counter is stored as a persistent Redis key in DB2, tracking unknown errors throughout the system.
  • The key is incremented atomically, preventing race conditions when multiple processes log errors simultaneously.

  ✔ Unlike transient cache entries, this counter is not subject to TTL eviction, ensuring historical error tracking remains intact.

  💡 For a detailed breakdown of the error logging, refer to the Error Logging section of the Error Handling page.

Return to Top

---

5. Celery Message Broker

Redis serves as the Celery message broker, handling asynchronous task execution by queuing, distributing, and tracking tasks across multiple workers.

• Celery workers pull tasks from Redis, ensuring distributed execution without blocking the main application.
• Redis DB2 is used to store task metadata and results temporarily.
• Celery Beat schedules periodic tasks, storing scheduled jobs in Redis.

💡 For detailed information on Celery’s role in the system, refer to the Celery - Asynchronous Task Processing page.

---

Next:   Celery - Asynchronous Task Processing
Back to: PostgreSQL - Database Configuration

Return to Top
DTS Index
Home Page

---