questdb · tris0laris · May 28, 2025 · Mar 6, 2025 · Mar 10, 2025 · Mar 10, 2025
diff --git a/documentation/concept/interval-scan.md b/documentation/concept/interval-scan.md
@@ -42,7 +42,7 @@ returned from a sub-query is not guaranteed to be in timestamp order.
 <Screenshot
   alt="Interval scan."
   height={433}
-  src="images/blog/2023-04-25/interval_scan.svg"
+  src="images/blog/2023-04-25/intervalScan.webp"
   width={650}
 />
 

diff --git a/documentation/concept/partitions.md b/documentation/concept/partitions.md
@@ -18,8 +18,8 @@ import Screenshot from "@theme/Screenshot"
 <Screenshot
   alt="Diagram of data column files and how they are partitioned to form a table"
   height={373}
-  src="images/docs/concepts/partitionModel.svg"
-  width={745}
+  src="images/docs/concepts/partitionModel.webp"
+  width={700}
   forceTheme="dark"
 />
 

diff --git a/documentation/concept/storage-model.md b/documentation/concept/storage-model.md
@@ -25,7 +25,7 @@ import Screenshot from "@theme/Screenshot"
 <Screenshot
   alt="Architecture of the file storing a column"
   height={435}
-  src="images/docs/concepts/columnUpdate.svg"
+  src="images/docs/concepts/columnUpdate.webp"
   width={745}
 />
 
@@ -39,7 +39,7 @@ mapped memory page, where the required value is read from.
 <Screenshot
   alt="Diagram showing how the data from a column file is mapped to the memory"
   height={447}
-  src="images/docs/concepts/columnRead.svg"
+  src="images/docs/concepts/columnRead.webp"
   width={745}
 />
 
@@ -96,10 +96,7 @@ either randomly (via queries) or incrementally (as a data queue). QuestDB
 provides a variety of reader implementations.
 
 <Screenshot
-  alt="Architecture of the storage model with column files, readers/writers and the mapped memory"
-  height={596}
-  src="images/docs/concepts/storageSummarized.svg"
-  width={745}
+  src="images/docs/concepts/storageSummary.webp"
 />
 
 <span />
diff --git a/documentation/guides/active-directory-pingfederate.md b/documentation/guides/active-directory-pingfederate.md
@@ -1,5 +1,5 @@
 ---
-title: Active Directory (PingFederate) guide
+title: Active Directory (PingFederate)
 description: ""
 ---
 

diff --git a/documentation/guides/architecture/data-ingestion-engine.md b/documentation/guides/architecture/data-ingestion-engine.md
@@ -0,0 +1,110 @@
+---
+title: Data Ingestion Engine
+slug: data-ingestion
+description: The QuestDB Data Ingestion Engine supports bulk and streaming ingestion. It writes data to a row-based write-ahead log (WAL) and then converts it into a columnar format. In QuestDB Enterprise, the WAL segments ship to object storage for replication.
+---
+
+
+## Data ingestion & write path
+
+The QuestDB Data Ingestion Engine supports bulk and streaming ingestion. It writes data to a row-based write-ahead log
+(WAL) and then converts it into a columnar format. In QuestDB Enterprise, the WAL segments ship to object storage for replication.
+
+### Bulk ingestion
+
+- **CSV ingestion:**
+  QuestDB offers a CSV ingestion endpoint via the [REST API](/docs/reference/api/rest/) and web console.
+   A specialized COPY command uses [io_uring](/blog/2022/09/12/importing-300k-rows-with-io-uring) on
+   fast drives to speed up ingestion.
+
+### Real-time streaming
+
+- **High-frequency writes:**
+  The streaming ingestion path handles millions of rows per second with non-blocking I/O.
+
+- **Durability:**
+  As seen above, the system writes data to a row-based write-ahead log (WAL) and then converts it
+  into column-based files for efficient reads. At the expense of performance, `sync` mode can be
+  enabled on commit for extra durability.
+
+- **Concurrent writes:**
+  Multiple connections writing to the same table create parallel WAL files that the engine
+  later consolidates into columnar storage.
+
+```text
+
+Contents of the `db` folder, showing multiple pending WAL files,
+and the binary columnar data.
+
+├── db
+│   ├── Table
+│   │   │
+│   │   ├── Partition 1
+│   │   │   ├── _archive
+│   │   │   ├── column1.d
+│   │   │   ├── column2.d
+│   │   │   ├── column2.k
+│   │   │   └── ...
+│   │   ├── Partition 2
+│   │   │   ├── _archive
+│   │   │   ├── column1.d
+│   │   │   ├── column2.d
+│   │   │   ├── column2.k
+│   │   │   └── ...
+│   │   ├── txn_seq
+│   │   │   ├── _meta
+│   │   │   ├── _txnlog
+│   │   │   └── _wal_index.d
+│   │   ├── wal1
+│   │   │   └── 0
+│   │   │       ├── _meta
+│   │   │       ├── _event
+│   │   │       ├── column1.d
+│   │   │       ├── column2.d
+│   │   │       └── ...
+│   │   ├── wal2
+│   │   │   └── 0
+│   │   │   │   ├── _meta
+│   │   │   │   ├── _event
+│   │   │   │   ├── column1.d
+│   │   │   │   ├── column2.d
+│   │   │   │   └── ...
+│   │   │   └── 1
+│   │   │       ├── _meta
+│   │   │       ├── _event
+│   │   │       ├── column1.d
+│   │   │       ├── column2.d
+│   │   │       └── ...
+│   │   │
+│   │   ├── _meta
+│   │   ├── _txn
+│   │   └── _cv
+
+```
+
+### Ingestion via ILP protocol
+
+- **Native ILP integration:**
+  QuestDB supports the [Influx Line Protocol](/docs/reference/api/ilp/overview/)
+   (ILP) for high-speed data ingestion.
+
+- **Extensions to ILP:**
+  QuestDB extends ILP to support different timestamp units and the array data type.
+
+- **Minimal parsing overhead:**
+  The ILP parser quickly maps incoming data to internal structures.
+
+- **Parallel ingestion:**
+  The ILP path uses off-heap buffers and direct memory management to bypass JVM heap  allocation.
+
+- **Protocol versatility:**
+  In addition to ILP, QuestDB also supports ingestion via [REST](/docs/reference/sql/overview/#rest-http-api)
+  and [PostgreSQL wire](/docs/reference/sql/overview/#postgresql) protocols.
+
+
+## Next Steps
+
+- Back to the [QuestDB Architecture](/docs/guides/architecture/questdb-architecture) overview
+- [QuestDB GitHub Repository](https://github.com/questdb/questdb)
+- [QuestDB Documentation](/docs)
+
diff --git a/documentation/guides/architecture/memory-management.md b/documentation/guides/architecture/memory-management.md
@@ -0,0 +1,48 @@
+---
+title: Memory Management
+slug: memory-management
+description: QuestDB leverages both memory mapping and explicit memory management techniques, and integrates native code for performance-critical tasks.
+---
+
+
+## Memory management and native integration
+
+QuestDB leverages both memory mapping and explicit memory management techniques, and integrates native code for performance-critical tasks.
+
+### Memory-mapped files
+
+- **Direct OS integration:**
+  Memory-mapped files let QuestDB use the operating system's page cache. This reduces explicit
+  I/O calls and speeds up sequential reads.
+
+- **Sequential access:**
+  When data partitions by incremental timestamp, memory mapping ensures that reads are
+  sequential and efficient.
+
+### Direct memory management and native integration
+
+- **Off-heap memory usage:**
+  QuestDB allocates direct memory via memory mapping and low-level APIs (such as Unsafe) to
+  bypass the JVM garbage collector. This reduces latency spikes and garbage collection delays.
+
+- **Hotpath efficiency:**
+  The system pre-allocates and reuses memory in critical code paths, avoiding dynamic allocation
+  on the hotpath.
+
+- **Native code integration:**
+  QuestDB uses native libraries written in C++ and Rust for performance-critical tasks. These
+  native components share off-heap buffers with Java via JNI.
+  - **Zero-copy interoperability:**
+    Sharing memory between Java and native code minimizes data copying and reduces latency.
+  - **Hybrid architecture:**
+    This integration lets QuestDB use Java for rapid development and C++/Rust for low-level,
+    high-performance routines.
+
+## Next Steps
+
+- Back to the [QuestDB Architecture](/docs/guides/architecture/questdb-architecture) overview
+- [QuestDB GitHub Repository](https://github.com/questdb/questdb)
+- [QuestDB Documentation](/docs)
+
+
+
diff --git a/documentation/guides/architecture/networking-layer.md b/documentation/guides/architecture/networking-layer.md
@@ -0,0 +1,54 @@
+---
+title: Networking Layer
+slug: networking-layer
+description: The system exposes RESTful APIs and implements ILP and PostgreSQL wire protocols so that existing tools and drivers work out-of-the-box. It also offers a health and metrics endpoint.
+---
+
+
+## Networking layer
+
+QuestDB exposes several network interfaces and protocols to allow different client applications to interact with the database
+
+### InfluxDB Line protocol (ILP) over HTTP or TCP
+
+The [Influx Line Protocol](/docs/reference/api/ilp/overview/) allows for very high throughput of incoming data. It supports
+both HTTP (recommended) or TCP. QuestDB provides official clients in seven different programming languages, as well as
+integrations with third-party tools like Apache Kafka, Apache Flink, or Telegraf. Any ILP-compatible library can be used
+for ingesting data into QuestDB over HTTP.
+
+The default port number for ILP over HTTP is `9000`, and for ILP over TCP is `9009`.
+
+### PostgreSQL wire protocol
+
+QuestDB exposes a [PostgreSQL wire](/docs/reference/sql/overview/#postgresql) protocol, which can be used to send SQL
+statements both for data definition or for data manipulation. When used for data ingestion, throughput is noticeably
+lower than using the ILP protocol.
+
+QuestDB implements the wire protocol, allowing many third-party libraries to query QuestDB directly. Some client libraries
+might be incompatible if they rely heavily on PostgreSQL metadata, as QuestDB implements only a subset of it. For an
+overview of some key differences on QuestDB schema design, please visit our
+[Schema Design Essentials](/docs/guides/schema-design-essentials/) guide.
+
+The default port number for the pg-wire interface is `8812`.
+
+### HTTP Rest API
+
+QuestDB [REST API](/docs/reference/sql/overview/#rest-http-api) can be used to issue SQL statements over HTTP. It also
+exposes and endpoint for importing CSV files, and for exporting tables and query results.
+
+The default port number for the REST API is `9000`.
+
+### Minimal HTTP server for health-check and metrics
+
+QuestDB exposes an HTTP interface for monitoring. Please see the [Observability](/docs/guides/architecture/observability) section
+for more information.
+
+The default port number for the minimal HTTP server is `9003`.
+
+## Next Steps
+
+- Back to the [QuestDB Architecture](/docs/guides/architecture/questdb-architecture) overview
+- [QuestDB GitHub Repository](https://github.com/questdb/questdb)
+- [QuestDB Documentation](/docs)
+
+
diff --git a/documentation/guides/architecture/observability.md b/documentation/guides/architecture/observability.md
@@ -0,0 +1,43 @@
+---
+title: Observability
+slug: observability
+description: QuestDB provides real-time metrics, a health check endpoint, and logging to monitor performance and simplify troubleshooting.
+---
+
+
+## Observability & diagnostics
+
+QuestDB provides real-time metrics, a health check endpoint, and logging to monitor performance and simplify troubleshooting.
+
+- **Metrics:**
+  QuestDB exposes detailed [metrics in Prometheus format](/docs/operations/logging-metrics/#metrics), including query
+  statistics, memory usage, and I/O details.
+
+- **Health check:**
+  A [minimal HTTP server](/docs/operations/logging-metrics/#minimal-http-server) monitors system health.
+
+- **Metadata tables:**
+  The engine provides [metadata tables](/docs/reference/function/meta/) to query
+   table status, partition status, query execution, and latency.
+
+- **Extensive logging:**
+  [Logging](/docs/operations/logging-metrics/) covers SQL parsing, execution, background processing, and runtime exceptions. The framework minimizes performance impact.
+
+- **Real-time metric dashboards:**
+  The web console lets you create dashboards that display per-table metrics.
+
+<Screenshot
+  alt="Metric dashboard at the QuestDB Console"
+  title="Metric dashboard at the QuestDB Console"
+  height={447}
+  src="images/guides/questdb-internals/telemetry.webp"
+  width={745}
+/>
+
+
+## Next Steps
+
+- Back to the [QuestDB Architecture](/docs/guides/architecture/questdb-architecture) overview
+- [QuestDB GitHub Repository](https://github.com/questdb/questdb)
+- [QuestDB Documentation](/docs)
+