diff --git a/docs/book/v1/overview.md b/docs/book/v1/overview.md index efa2f8e..5e5d06a 100644 --- a/docs/book/v1/overview.md +++ b/docs/book/v1/overview.md @@ -1,6 +1,11 @@ # Overview -> Dotkernel component used to queue tasks to be processed asynchronously based on [netglue/laminas-messenger](https://github.com/netglue/laminas-messenger) +> [Dotkernel Queue](https://github.com/dotkernel/dot-queue) is a component based on [**Symfony Messenger**](https://github.com/symfony/messenger) that is used to queue asynchronous tasks. +[netglue/laminas-messenger](https://github.com/netglue/laminas-messenger) is a layer that turns Symfony Messenger into a middleware compatible with Mezzio/Laminas applications. + +Some everyday **operations are time-consuming and resource-intensive**, so it's best if they run on separate machines, decoupled from the regular request-response cycle. +**Asynchronous execution** performed by background workers ensures that these operations won't overload the main platform. +It allows the main platform to return a response and remain responsive for new requests, while tasks with long execution times are scheduled to run later. ![Queue process](https://docs.dotkernel.org/img/queue/schema.png) diff --git a/docs/book/v1/what-is-queue.md b/docs/book/v1/what-is-queue.md new file mode 100644 index 0000000..3d2ddc1 --- /dev/null +++ b/docs/book/v1/what-is-queue.md @@ -0,0 +1,40 @@ +# Tasks Delegated to the Queue + +Normally, the tasks delegated to the queue: + +- Take extended periods of time to execute and may be interrupted by PHP limitations (like the PHP `max_execution_time` parameter). +- Are external which introduces delays from authentication, awaiting responses from potentially multiple interrogations, or may even fail completely because the server is offline. +- Are not part of the PHP response (sending emails, processing videos). + +Here are some example tasks meant for the queue: + +- **Data Processing** like big data analytics, scientific simulations, or mathematical computations. +- **File Handling & Media Processing** like video and image processing, or compression/decompression of large files. +- **Networking** like uploading/downloading large files. +- **Database Operations** like imports/exports or migrations. +- **System & Infrastructure Tasks** like OS updates, software compilation, CI pipelines. + +The long execution times are caused by: + +- Data size - gigabytes, terabytes etc. +- Complex algorithms. +- Hardware limitations - CPU speed, memory, storage, network bandwidth. +- External dependencies - waiting on APIs or human input. + +## How the Queue Works + +- The queue system has an active **daemon** that listens for TPC connections on a specific port and **stores incoming messages** into Redis. +This method supports a **large number of requests per second** without overloading. +- Operations are then **scheduled for execution** when resources are available. +The order of the execution uses the **FIFO** (First-In, First-Out) method where the oldest request is processed first, followed by newer requests. + +## Main Features + +- **Logging** - Ensures maintainability and allows debugging. +- **Security** - The **firewall** allows requests only from whitelisted IPs for fast response times. +- **Retry Mechanism** - Retries failing tasks a certain number of times before removing a task from the queue. +- **Reporting** - The logs enable developers to investigate various metrics via console commands like: + - Queue length - How many jobs are in the to-do list. + - Processing time per job. + - Error rates - How many messages failed. + - Throughput - Jobs/sec processed. diff --git a/mkdocs.yml b/mkdocs.yml index 6e043e3..f3aa499 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -9,6 +9,7 @@ nav: - Home: index.md - v1: - Overview: v1/overview.md + - "Understanding the Queue": v1/what-is-queue.md - Server Setup: v1/server-setup.md - Installation: v1/installation.md - Control Commands: v1/control-commands.md