diff --git a/documentation/concept/interval-scan.md b/documentation/concept/interval-scan.md index 0b559fb63..7be3aab45 100644 --- a/documentation/concept/interval-scan.md +++ b/documentation/concept/interval-scan.md @@ -42,7 +42,7 @@ returned from a sub-query is not guaranteed to be in timestamp order. diff --git a/documentation/concept/partitions.md b/documentation/concept/partitions.md index 252ad60a7..4662d4b00 100644 --- a/documentation/concept/partitions.md +++ b/documentation/concept/partitions.md @@ -18,8 +18,8 @@ import Screenshot from "@theme/Screenshot" diff --git a/documentation/concept/storage-model.md b/documentation/concept/storage-model.md index 1697c9680..3b50b4529 100644 --- a/documentation/concept/storage-model.md +++ b/documentation/concept/storage-model.md @@ -25,7 +25,7 @@ import Screenshot from "@theme/Screenshot" @@ -39,7 +39,7 @@ mapped memory page, where the required value is read from. @@ -96,10 +96,7 @@ either randomly (via queries) or incrementally (as a data queue). QuestDB provides a variety of reader implementations. diff --git a/documentation/guides/active-directory-pingfederate.md b/documentation/guides/active-directory-pingfederate.md index 5018ddcea..d9dcb8644 100644 --- a/documentation/guides/active-directory-pingfederate.md +++ b/documentation/guides/active-directory-pingfederate.md @@ -1,5 +1,5 @@ --- -title: Active Directory (PingFederate) guide +title: Active Directory (PingFederate) description: "" --- diff --git a/documentation/guides/architecture/data-ingestion-engine.md b/documentation/guides/architecture/data-ingestion-engine.md new file mode 100644 index 000000000..1eb497ac6 --- /dev/null +++ b/documentation/guides/architecture/data-ingestion-engine.md @@ -0,0 +1,110 @@ +--- +title: Data Ingestion Engine +slug: data-ingestion +description: The QuestDB Data Ingestion Engine supports bulk and streaming ingestion. It writes data to a row-based write-ahead log (WAL) and then converts it into a columnar format. In QuestDB Enterprise, the WAL segments ship to object storage for replication. +--- + + +## Data ingestion & write path + +The QuestDB Data Ingestion Engine supports bulk and streaming ingestion. It writes data to a row-based write-ahead log +(WAL) and then converts it into a columnar format. In QuestDB Enterprise, the WAL segments ship to object storage for replication. + +### Bulk ingestion + +- **CSV ingestion:** + QuestDB offers a CSV ingestion endpoint via the [REST API](/docs/reference/api/rest/) and web console. + A specialized COPY command uses [io_uring](/blog/2022/09/12/importing-300k-rows-with-io-uring) on + fast drives to speed up ingestion. + +### Real-time streaming + +- **High-frequency writes:** + The streaming ingestion path handles millions of rows per second with non-blocking I/O. + +- **Durability:** + As seen above, the system writes data to a row-based write-ahead log (WAL) and then converts it + into column-based files for efficient reads. At the expense of performance, `sync` mode can be + enabled on commit for extra durability. + +- **Concurrent writes:** + Multiple connections writing to the same table create parallel WAL files that the engine + later consolidates into columnar storage. + +```text + +Contents of the `db` folder, showing multiple pending WAL files, +and the binary columnar data. + +├── db +│ ├── Table +│ │ │ +│ │ ├── Partition 1 +│ │ │ ├── _archive +│ │ │ ├── column1.d +│ │ │ ├── column2.d +│ │ │ ├── column2.k +│ │ │ └── ... +│ │ ├── Partition 2 +│ │ │ ├── _archive +│ │ │ ├── column1.d +│ │ │ ├── column2.d +│ │ │ ├── column2.k +│ │ │ └── ... +│ │ ├── txn_seq +│ │ │ ├── _meta +│ │ │ ├── _txnlog +│ │ │ └── _wal_index.d +│ │ ├── wal1 +│ │ │ └── 0 +│ │ │ ├── _meta +│ │ │ ├── _event +│ │ │ ├── column1.d +│ │ │ ├── column2.d +│ │ │ └── ... +│ │ ├── wal2 +│ │ │ └── 0 +│ │ │ │ ├── _meta +│ │ │ │ ├── _event +│ │ │ │ ├── column1.d +│ │ │ │ ├── column2.d +│ │ │ │ └── ... +│ │ │ └── 1 +│ │ │ ├── _meta +│ │ │ ├── _event +│ │ │ ├── column1.d +│ │ │ ├── column2.d +│ │ │ └── ... +│ │ │ +│ │ ├── _meta +│ │ ├── _txn +│ │ └── _cv + +``` + +### Ingestion via ILP protocol + +- **Native ILP integration:** + QuestDB supports the [Influx Line Protocol](/docs/reference/api/ilp/overview/) + (ILP) for high-speed data ingestion. + +- **Extensions to ILP:** + QuestDB extends ILP to support different timestamp units and the array data type. + +- **Minimal parsing overhead:** + The ILP parser quickly maps incoming data to internal structures. + +- **Parallel ingestion:** + The ILP path uses off-heap buffers and direct memory management to bypass JVM heap allocation. + +- **Protocol versatility:** + In addition to ILP, QuestDB also supports ingestion via [REST](/docs/reference/sql/overview/#rest-http-api) + and [PostgreSQL wire](/docs/reference/sql/overview/#postgresql) protocols. + + +## Next Steps + +- Back to the [QuestDB Architecture](/docs/guides/architecture/questdb-architecture) overview +- [QuestDB GitHub Repository](https://github.com/questdb/questdb) +- [QuestDB Documentation](/docs) + diff --git a/documentation/guides/architecture/memory-management.md b/documentation/guides/architecture/memory-management.md new file mode 100644 index 000000000..50dda1c65 --- /dev/null +++ b/documentation/guides/architecture/memory-management.md @@ -0,0 +1,48 @@ +--- +title: Memory Management +slug: memory-management +description: QuestDB leverages both memory mapping and explicit memory management techniques, and integrates native code for performance-critical tasks. +--- + + +## Memory management and native integration + +QuestDB leverages both memory mapping and explicit memory management techniques, and integrates native code for performance-critical tasks. + +### Memory-mapped files + +- **Direct OS integration:** + Memory-mapped files let QuestDB use the operating system's page cache. This reduces explicit + I/O calls and speeds up sequential reads. + +- **Sequential access:** + When data partitions by incremental timestamp, memory mapping ensures that reads are + sequential and efficient. + +### Direct memory management and native integration + +- **Off-heap memory usage:** + QuestDB allocates direct memory via memory mapping and low-level APIs (such as Unsafe) to + bypass the JVM garbage collector. This reduces latency spikes and garbage collection delays. + +- **Hotpath efficiency:** + The system pre-allocates and reuses memory in critical code paths, avoiding dynamic allocation + on the hotpath. + +- **Native code integration:** + QuestDB uses native libraries written in C++ and Rust for performance-critical tasks. These + native components share off-heap buffers with Java via JNI. + - **Zero-copy interoperability:** + Sharing memory between Java and native code minimizes data copying and reduces latency. + - **Hybrid architecture:** + This integration lets QuestDB use Java for rapid development and C++/Rust for low-level, + high-performance routines. + +## Next Steps + +- Back to the [QuestDB Architecture](/docs/guides/architecture/questdb-architecture) overview +- [QuestDB GitHub Repository](https://github.com/questdb/questdb) +- [QuestDB Documentation](/docs) + + + diff --git a/documentation/guides/architecture/networking-layer.md b/documentation/guides/architecture/networking-layer.md new file mode 100644 index 000000000..4e8b400e0 --- /dev/null +++ b/documentation/guides/architecture/networking-layer.md @@ -0,0 +1,54 @@ +--- +title: Networking Layer +slug: networking-layer +description: The system exposes RESTful APIs and implements ILP and PostgreSQL wire protocols so that existing tools and drivers work out-of-the-box. It also offers a health and metrics endpoint. +--- + + +## Networking layer + +QuestDB exposes several network interfaces and protocols to allow different client applications to interact with the database + +### InfluxDB Line protocol (ILP) over HTTP or TCP + +The [Influx Line Protocol](/docs/reference/api/ilp/overview/) allows for very high throughput of incoming data. It supports +both HTTP (recommended) or TCP. QuestDB provides official clients in seven different programming languages, as well as +integrations with third-party tools like Apache Kafka, Apache Flink, or Telegraf. Any ILP-compatible library can be used +for ingesting data into QuestDB over HTTP. + +The default port number for ILP over HTTP is `9000`, and for ILP over TCP is `9009`. + +### PostgreSQL wire protocol + +QuestDB exposes a [PostgreSQL wire](/docs/reference/sql/overview/#postgresql) protocol, which can be used to send SQL +statements both for data definition or for data manipulation. When used for data ingestion, throughput is noticeably +lower than using the ILP protocol. + +QuestDB implements the wire protocol, allowing many third-party libraries to query QuestDB directly. Some client libraries +might be incompatible if they rely heavily on PostgreSQL metadata, as QuestDB implements only a subset of it. For an +overview of some key differences on QuestDB schema design, please visit our +[Schema Design Essentials](/docs/guides/schema-design-essentials/) guide. + +The default port number for the pg-wire interface is `8812`. + +### HTTP Rest API + +QuestDB [REST API](/docs/reference/sql/overview/#rest-http-api) can be used to issue SQL statements over HTTP. It also +exposes and endpoint for importing CSV files, and for exporting tables and query results. + +The default port number for the REST API is `9000`. + +### Minimal HTTP server for health-check and metrics + +QuestDB exposes an HTTP interface for monitoring. Please see the [Observability](/docs/guides/architecture/observability) section +for more information. + +The default port number for the minimal HTTP server is `9003`. + +## Next Steps + +- Back to the [QuestDB Architecture](/docs/guides/architecture/questdb-architecture) overview +- [QuestDB GitHub Repository](https://github.com/questdb/questdb) +- [QuestDB Documentation](/docs) + + diff --git a/documentation/guides/architecture/observability.md b/documentation/guides/architecture/observability.md new file mode 100644 index 000000000..067032737 --- /dev/null +++ b/documentation/guides/architecture/observability.md @@ -0,0 +1,43 @@ +--- +title: Observability +slug: observability +description: QuestDB provides real-time metrics, a health check endpoint, and logging to monitor performance and simplify troubleshooting. +--- + + +## Observability & diagnostics + +QuestDB provides real-time metrics, a health check endpoint, and logging to monitor performance and simplify troubleshooting. + +- **Metrics:** + QuestDB exposes detailed [metrics in Prometheus format](/docs/operations/logging-metrics/#metrics), including query + statistics, memory usage, and I/O details. + +- **Health check:** + A [minimal HTTP server](/docs/operations/logging-metrics/#minimal-http-server) monitors system health. + +- **Metadata tables:** + The engine provides [metadata tables](/docs/reference/function/meta/) to query + table status, partition status, query execution, and latency. + +- **Extensive logging:** + [Logging](/docs/operations/logging-metrics/) covers SQL parsing, execution, background processing, and runtime exceptions. The framework minimizes performance impact. + +- **Real-time metric dashboards:** + The web console lets you create dashboards that display per-table metrics. + + + + +## Next Steps + +- Back to the [QuestDB Architecture](/docs/guides/architecture/questdb-architecture) overview +- [QuestDB GitHub Repository](https://github.com/questdb/questdb) +- [QuestDB Documentation](/docs) + diff --git a/documentation/guides/architecture/overview.md b/documentation/guides/architecture/overview.md new file mode 100644 index 000000000..9f134ac2e --- /dev/null +++ b/documentation/guides/architecture/overview.md @@ -0,0 +1,84 @@ +--- +title: Overview +slug: questdb-architecture +description: A deep technical dive into the internal architecture, storage engine, query processing, and native integrations of QuestDB. +--- + +QuestDB offers high-speed ingestion and low-latency analytics on time-series data. + + + + +This document explains QuestDB's internal architecture. + +## Key components + +QuestDB is comprised of several key components: + +- **[Storage engine](/docs/guides/architecture/storage-engine):** + Uses a column-oriented design to ensure high I/O performance and low latency. + +- **[Memory management and native integration](/docs/guides/architecture/memory-management):** + The system leverages both memory mapping and explicit memory management techniques,integrating native code for performance-critical tasks. + +- **[Query engine](/docs/guides/architecture/query-engine):** + A custom SQL parser, a just-in-time (JIT) compiler, and a vectorized execution engine process + data in table page frames for better CPU use. + +- **[Time-series optimizations](/docs/guides/architecture/time-series-optimizations):** + QuestDB is specifically designed for time-series, and it provides several optimizations, such as a designated timestamp, sequential reads, materialized views, and in-memory processing. + +- **[Data ingestion engine](/docs/guides/architecture/data-ingestion):** + TSupports both bulk and streaming ingestion. It writes data to a row-based write-ahead + log (WAL) and then converts it into a columnar format. In QuestDB Enterprise, the WAL segments + are shipped to object storage for replication. + +- **[Networking layer](/docs/guides/architecture/networking-layer):** + The system exposes RESTful APIs and implements ILP and PostgreSQL wire protocols to ensure compatibility with existing tools and drivers. It also offers a health and metrics endpoint. + +- **[Replication layer](/docs/guides/architecture/replication-layer):** + QuestDB Enterprise supports horizontal scalability for reads via read replicas, and for + writes via multi-primary configurations. + +- **[Security](/docs/guides/architecture/security):** + QuestDB implements enterprise-grade security with TLS, single sign-on (SSO), and role-based access control with + fine-grained granularity. + +- **[Observability](/docs/guides/architecture/observability):** + QuestDB exposes real-time metrics, health checks, and structured logs to monitor performance and streamline diagnostics. + +- **[Web console](/docs/guides/architecture/web-console):** + The engine includes a web console for running SQL statements, bulk loading CSV files, and displaying monitoring dashboards. QuestDB Enterprise supports single sign-on (SSO) in the web console. + + + + +## Design patterns & best practices throughout the codebase + +- **Immutable data structures:** + The system favors immutability to avoid concurrency issues and simplify state + management. + +- **Modular architecture:** + Each component (eg., storage, query processing, ingestion, etc.) has well-defined interfaces that enhance maintainability and decouple functionality. + +- **Factory & builder patterns:** + These patterns are used to centralize construction logic for complex objects such as SQL execution plans and storage buffers. + +- **Lazy initialization:** + Resource-intensive components initialize only when needed to reduce startup overhead. + +- **Rigorous testing & benchmarks:** + [Unit tests, integration tests](https://github.com/questdb/questdb/tree/master/core/src/test), + and performance benchmarks ensure that new enhancements do not compromise + reliability or performance. + +## Further reading & resources + +- [QuestDB GitHub Repository](https://github.com/questdb/questdb) +- [QuestDB Documentation](/docs) diff --git a/documentation/guides/architecture/query-engine.md b/documentation/guides/architecture/query-engine.md new file mode 100644 index 000000000..ab9e8d247 --- /dev/null +++ b/documentation/guides/architecture/query-engine.md @@ -0,0 +1,77 @@ +--- +title: Query Engine +slug: query-engine +description: The QuestDB Query Engine includes A custom SQL parser, a just-in-time (JIT) compiler, and a vectorized execution engine to process data in table page frames for better CPU use. + + +--- + +## Query engine + +The QuestDB Query Engine includes A custom SQL parser, a just-in-time (JIT) compiler, and a vectorized execution engine +to process data in table page frames for better CPU use. + + +### SQL parsing & optimization + +- **Custom SQL parser:** + The parser supports QuestDB's SQL dialect and time-series extensions. It converts SQL queries + into an optimized abstract syntax tree (AST). + +- **Compilation pipeline:** + The engine compiles SQL into an execution plan through stages that push down predicates and + rewrite queries to remove unnecessary operations. + +- **Optimization techniques:** + The planner applies rule-based rewrites and simple cost estimations to choose efficient + execution paths. + +- **Columnar reads:** + Table columns are randomly accessible. Columns with fixed size data types are read by translating + the record number into a file offset by a simple bit shift. The offset in the column file is then + translated into an offset in a lazily mapped memory page, where the required value is read from. + + + +### Execution model + +- **Operator pipeline:** + The execution plan runs as a series of operators (filters, joins, aggregators) in a tightly + integrated pipeline. + + + +- **JIT compilation and Vectorized processing:** + Queries with a `WHERE` clause [compile](/docs/concept/jit-compiler) critical parts of the execution plan to native machine code (SIMD AVX-2 instructions) just in time. Vectorised instructions apply + the same operation to many data elements simultaneously. This maximizes CPU cache use and reduces overhead. + +- **Multi-threaded execution:** + On top of the JIT, QuestDB tries to execute as many queries as possible in a multi-threaded, + multi-core fashion. Some queries, for example those involving an index, are executed on a single + thread. Other queries, like those involving `GROUP BY` and `SAMPLE BY`, execute a pipeline with some single-threaded stages and some multi-threaded stages to avoid slow downs when groups are unbalanced. + +- **Worker pools:** QuestDB allows to configure different pools for specialized functions, like +parsing incoming data, applying WAL file changes, handling PostgreSQL-Wire protocol, or responding to HTTP connections. By default, most tasks are handled by a shared worker pool. + +- **Query plan caching:** + The system caches query plans for reuse within the same connection. (Query results are not + cached.) + +- **Column data caching:** + Data pages read from disk are kept in system memory. Sufficient memory prevents frequent disk + reads. + +## Next Steps + +- Back to the [QuestDB Architecture](/docs/guides/architecture/questdb-architecture) overview +- [QuestDB GitHub Repository](https://github.com/questdb/questdb) +- [QuestDB Documentation](/docs) diff --git a/documentation/guides/architecture/replication-layer.md b/documentation/guides/architecture/replication-layer.md new file mode 100644 index 000000000..877b1bf33 --- /dev/null +++ b/documentation/guides/architecture/replication-layer.md @@ -0,0 +1,60 @@ +--- +title: Replication Layer +slug: replication-layer +description: QuestDB Enterprise supports horizontal scalability for reads with read replicas, and for writes with multi-primary. +--- + + +## Replication + +QuestDB Enterprise offers primary-replica replication with eventual consistency. Replication in QuestDB operates by +designating a single "primary" database that uploads Write Ahead Log (WAL) files to a remote object store. These files +can be downloaded and applied by any number of "replica" instances, either continuously or at a later time. + +Full details of replication can be found at the [replication concepts](/docs/concept/replication/) +page. + +### Type of instances on a replicated QuestDB cluster + +#### Primary instances + +Primary instances offer the same features as stand-alone instances, supporting both reads and writes. To keep the +overhead of replication to a minimum, primary instances only ship WAL segments to the designated object store. There +is no communication between the primary and the replicas. Metadata is also replicated to the object store to avoid any +inconsistencies between different instances in the cluster. + +A typical replicated deployment will have at least one primary and one replica, but it is also possible to set up a +primary to replicate changes to the object store, even if there are no read replicas configured. The replicated data +could be used for point-in-time recovery. + +For demanding scenarios, QuestDB Enterprise allows [multi-primary ingestion](/docs/operations/multi-primary-ingestion/), +which allows both increasing the write throughput, and enabling high availabilty. In order to enable multi-primary ingestion, +a `Distributed Sequencer` instance will need to be created. + + +#### Read Replicas + +Read Replicas are designed to distribute data and load across multiple locations and instances. Each replica can have +different hardware configuration, to allow for optimizations depending on the usage. + +Replicas can be local, cross-zonal, or cross-regional. As long as the replicas can read metadata and WAL segments from +the object store, they will catch up with the data ingested by the primaries. + +At the moment of writing this guide, read replicas will replicate from all tables and partitions from the primaries. + +#### Distributed Sequencer + +When multi-primary ingestion is configured, it is necessary to coordinate writes, to ensure transactions are consistent +and conflict-free. + +The sequencer coordinates transactions by assigning monotonic transaction numbers to writes. The distributed sequencer +uses [FoundationDB](https://www.foundationdb.org/) as the backend for storing transaction metadata and enabling synchronization +across primaries. + + +## Next Steps + +- Back to the [QuestDB Architecture](/docs/guides/architecture/questdb-architecture) overview +- [QuestDB GitHub Repository](https://github.com/questdb/questdb) +- [QuestDB Documentation](/docs) + diff --git a/documentation/guides/architecture/security.md b/documentation/guides/architecture/security.md new file mode 100644 index 000000000..cca2c95ee --- /dev/null +++ b/documentation/guides/architecture/security.md @@ -0,0 +1,39 @@ +--- +title: Security +slug: security +description: QuestDB implements enterprise-grade security with TLS, single-sign-on, and role-based access control with fine-grained granularity. +--- + + +## Security + + QuestDB implements enterprise-grade security with TLS, single-sign-on, and role-based access control with + fine-grained granularity. + +- **Built-in admin and read-only users:** + QuestDB includes built-in admin and read-only users for the pgwire protocol and HTTP endpoints using HTTP Basic Auth. + +- **HTTP basic authentication:** + You can enable HTTP Basic Authentication for the HTTP API, web console, and pgwire + protocol. Health-check and metrics endpoints can be configured independently. + +- **Token-based authentication:** + QuestDB Enterprise offers HTTP and JWT token authentication. QuestDB Open Source + supports token authentication for ILP over TCP. + +- **TLS on all protocols:** + QuestDB Enterprise supports TLS on all protocols and endpoints. + +- **Single sign-on:** + QuestDB Enterprise supports SSO via OIDC with Active Directory, EntraID, or OAuth2. + +- **Role-based access control:** + Enterprise users can create user groups and assign service accounts and users. + Grants [can be configured](/docs/operations/rbac/) individually or at the + group level with fine granularity, including column-level access. + +## Next Steps + +- Back to the [QuestDB Architecture](/docs/guides/architecture/questdb-architecture) overview +- [QuestDB GitHub Repository](https://github.com/questdb/questdb) +- [QuestDB Documentation](/docs) diff --git a/documentation/guides/architecture/storage-engine.md b/documentation/guides/architecture/storage-engine.md new file mode 100644 index 000000000..201a3af03 --- /dev/null +++ b/documentation/guides/architecture/storage-engine.md @@ -0,0 +1,80 @@ +--- +title: Storage Engine +slug: storage-engine +description: The QuestDB Storage Engine uses a column-oriented design to ensure high I/O performance and low latency. +--- + + +## Storage engine + +The QuestDB Storage Engine implements a row-based write path for maximum ingestion throughput, and a column-based +read path for maximum query performance. + + +### Parallel Write-Ahead-Log + +- **Two-phase writes**: All changes to data are recorded in a Write-Ahead-Log (WAL) before they +are written to the database files. This means that in case of a system crash or power failure, the database can recover to a consistent state by replaying the log entries. + +- **Commit and write separation**: By decoupling the transaction commit from the disk write process, +a WAL improves the performance of write-intensive workloads, as it allows for sequential disk writes +which are generally faster than random ones. + +- **Per-table WAL**: WAL files are separated per table, and also per active connection, allowing for +concurrent data ingestion, modifications, and schema changes without locking the entire table. + +- **WAL Consistency**: QuestDB implements a component called "Sequencer", which ensures that data +appears consistent to all readers, even during ongoing write operations. + + +- **TableWriter**: Changes stored in the WAL, is stored in columnat format by the TableWriter, which +can handle and resolve out-of-order data writes, and enables deduplication. Column files use an +[append model](/docs/concept/storage-model/). + + + + +### Data Deduplication + +When enabled, [data deduplication](https://questdb.com/docs/concept/deduplication/) works on all the data inserted into +the table and replaces matching rows with the new versions. Only new rows that do no match existing data will be inserted. + +Generally, if the data have mostly unique timestamps across all the rows, the performance impact of deduplication is low. +Conversely, the most demanding data pattern occurs when there are many rows with the same timestamp that need to be +deduplicated on additional columns. + + +### Column-oriented storage + +- **Data layout:** + The system stores each table as separate files per column. Fixed-size data types use one file + per column, while variable-size data types (such as `VARCHAR` or `STRING`) use two files per column. + + + + +- **CPU optimization:** + Columnar storage improves CPU use during vectorized operations, which speeds up + aggregations and computations. + +- **Compression:** + Uniform data types allow efficient compression that reduces disk space and speeds up reads + when [ZFS compression](/docs/guides/compression-zfs/) is enabled. Parquet files generated + by QuestDB use native compression. + +## Next Steps + +- Back to the [QuestDB Architecture](/docs/guides/architecture/questdb-architecture) overview +- [QuestDB GitHub Repository](https://github.com/questdb/questdb) +- [QuestDB Documentation](/docs) + diff --git a/documentation/guides/architecture/time-series-optimizations.md b/documentation/guides/architecture/time-series-optimizations.md new file mode 100644 index 000000000..49f6065f1 --- /dev/null +++ b/documentation/guides/architecture/time-series-optimizations.md @@ -0,0 +1,118 @@ +--- +title: Time Series Optimizations +slug: time-series-optimizations +description: QuestDB is specifically designed for time-series, and it provides several optimizations, like a designated timestamp, sequential reads, materialized-views, or in-memory processing. +--- + + +## Time-series optimizations + +QuestDB is specifically designed for time-series, and it provides several optimizations, like a designated timestamp, +sequential reads, materialized-views, or in-memory processing. + +### Designated timestamp + +- **Timestamp sorting:** + Data stores in order of incremental timestamp. Since ingestion is usually + chronological, the system uses a fast append-only strategy, except for updates and out-of-order data. + +- **Rapid interval queries and sequential reads:** + Sorted data lets the system quickly locate the start and end of data files, which speeds + up [interval queries](/docs/concept/interval-scan/). When data is accessed by increasing timestamp, + reads are sequential for each column file, which makes I/O very efficient. + + + + +- **Out-of-order data:** + When data arrives out of order, QuestDB [rearranges it](/docs/concept/partitions/#splitting-and-squashing-time-partitions) to maintain timestamp order. The + engine splits partitions to minimize [write amplification](/docs/operations/capacity-planning/#write-amplification) and compacts them in the background. + + +### Data partitioning and sequential reads + +- **Partitioning by time:** + Data [partitions by timestamp](/docs/concept/partitions/) with hourly, daily, weekly, monthly, or yearly resolution. + + + + + + +- **Partition pruning:** + The design lets the engine skip partitions that fall outside query filters. Combined with + incremental timestamp sorting, this reduces latency. + + +- **Lifecycle policies:** + The system can delete partitions manually or automatically via TTL. It also supports + detaching or attaching partitions using SQL commands. + +### Materialized views + + - [Materialize views](https://questdb.com/docs/concept/mat-views/) are auto-refreshing tables storing the pre-computed results of a query. Unlike regular views, which + compute their results at query time, materialized views persist their data to disk, making them particularly efficient + for expensive aggregate queries that are run frequently. + + - QuestDB supports materialized views for `SAMPLE BY` queries, including those joining with other tables. + + - Materialized sampled intervals are automatically refreshed whenever the base table receivews new or updated rows. + + - Materialized views can be chained, with the output of one being the input of another one, and support TTLs for lifecycle management. + + +### In-memory processing + +- **Caching:** + The engine uses the OS cache to access recent and frequently accessed data in memory, reducing + disk reads. + +- **Off-heap buffers:** + Off-heap memory, managed via memory mapping and direct allocation, avoids garbage + collection overhead. + +- **Optimized in-memory handling:** + Apart from using CPU-level optimizations such as SIMD, QuestDB uses specialized hash tables (all of them with open + addressing and linear probing), and implements algorithms for reducing the memory + footprint of many operations. + +- **Custom memory layout for different data types:** + Specialized data types, like `Symbol`, `VARCHAR`, or `UUID`, are designed to use minimal disk and memory. For example, + char sequences shorter than 9 bytes are fully inlined within our `VARCHAR` header and do not occupy any additional data space. + + ```text + Internal Representation of the VARCHAR data type + +Varchar header (column file): ++------------+-------------------+-------------------+ +| 32 bits | 48 bits | 48 bits | +| len + flags| prefix | offset | ++------------+-------------------+-------------------+ + │ ++------------------------------------+ points to +│ +▼ +Varchar data (column file): ++---+---+---+---+---+---+---+---+---+---+---+ +| H | e | l | l | o | | w | o | r | l | d | ++---+---+---+---+---+---+---+---+---+---+---+ + +``` + +## Next Steps + +- Back to the [QuestDB Architecture](/docs/guides/architecture/questdb-architecture) overview +- [QuestDB GitHub Repository](https://github.com/questdb/questdb) +- [QuestDB Documentation](/docs) + diff --git a/documentation/guides/architecture/web-console.md b/documentation/guides/architecture/web-console.md new file mode 100644 index 000000000..0622249bc --- /dev/null +++ b/documentation/guides/architecture/web-console.md @@ -0,0 +1,33 @@ +--- +title: Web Console +slug: web-console +description: The engine includes a web console to run SQL statements, bulk load CSV files, and show + monitoring dashboards. QuestDB Enterprise supports single sign-on (SSO) in the web console. +--- + + +## Web Console + +The [QuestDB Web console](/docs/web-console/) is ideal for interactive exploration, and implements: + +- **SQL Editor** featuring tabs and SQL autocompletion. +- **Table and Materialized View** explorer. +- **CSV import** manager. +- **Single-sign-on**, when enabled for enterprise users. +- **Basic Auth**, when enabled for both OSS and Enterprise users. +- **Real-time metric dashboards**, to track transactions throughput, latency, and write amplification. + + + +## Next Steps + +- Back to the [QuestDB Architecture](/docs/guides/architecture/questdb-architecture) overview +- [QuestDB GitHub Repository](https://github.com/questdb/questdb) +- [QuestDB Documentation](/docs) + + + diff --git a/documentation/guides/microsoft-entraid-oidc.md b/documentation/guides/microsoft-entraid-oidc.md index 10107862e..b873b4b18 100644 --- a/documentation/guides/microsoft-entraid-oidc.md +++ b/documentation/guides/microsoft-entraid-oidc.md @@ -1,5 +1,5 @@ --- -title: Microsoft EntraID OIDC guide +title: Microsoft EntraID OIDC description: "QuestDB Enterprise guide to demonstrate Microsoft EntraID OpenID Connect." --- diff --git a/documentation/operations/rbac.md b/documentation/operations/rbac.md index d41ad904e..ea60828f1 100644 --- a/documentation/operations/rbac.md +++ b/documentation/operations/rbac.md @@ -63,6 +63,8 @@ this ability. The below diagram is an example of a QuestDB access control system. Inherited permissions are shown in grey colour: + + - - -
Interval forward scan
Interval forward scan
Table
Table
designated timestamp
designated timestamp
Partition
Partition
Interval
Interval
Partition
Partition
Interval
Interval
Interval
Interval
Interval backward scan
Interval backward scanText is not SVG - cannot display \ No newline at end of file diff --git a/static/images/docs/concepts/columnRead.svg b/static/images/docs/concepts/columnRead.svg deleted file mode 100644 index ec813ac7d..000000000 --- a/static/images/docs/concepts/columnRead.svg +++ /dev/null @@ -1 +0,0 @@ -data column filememory pageRead value at row NPage + Offset \ No newline at end of file diff --git a/static/images/docs/concepts/columnRead.webp b/static/images/docs/concepts/columnRead.webp new file mode 100644 index 000000000..f0a3335e4 Binary files /dev/null and b/static/images/docs/concepts/columnRead.webp differ diff --git a/static/images/docs/concepts/columnUpdate.svg b/static/images/docs/concepts/columnUpdate.svg deleted file mode 100644 index a5f1179f1..000000000 --- a/static/images/docs/concepts/columnUpdate.svg +++ /dev/null @@ -1 +0,0 @@ -column filecopy value mov [0x1234], xSliding area of file mapped to RAMMoves down once exhaustedIncrement file offset \ No newline at end of file diff --git a/static/images/docs/concepts/columnUpdate.webp b/static/images/docs/concepts/columnUpdate.webp new file mode 100644 index 000000000..25d92ae27 Binary files /dev/null and b/static/images/docs/concepts/columnUpdate.webp differ diff --git a/static/images/docs/concepts/partitionModel.svg b/static/images/docs/concepts/partitionModel.svg deleted file mode 100644 index 9396bbcce..000000000 --- a/static/images/docs/concepts/partitionModel.svg +++ /dev/null @@ -1 +0,0 @@ -table directorycolumn filesTimecolumn filescolumn filesData fileTime partitionTime partitionTime partition \ No newline at end of file diff --git a/static/images/docs/concepts/partitionModel.webp b/static/images/docs/concepts/partitionModel.webp new file mode 100644 index 000000000..e648fbbdf Binary files /dev/null and b/static/images/docs/concepts/partitionModel.webp differ diff --git a/static/images/docs/concepts/storageSummarized.svg b/static/images/docs/concepts/storageSummarized.svg deleted file mode 100644 index 197283b6f..000000000 --- a/static/images/docs/concepts/storageSummarized.svg +++ /dev/null @@ -1 +0,0 @@ -table directorytx filemetadatapartition directorycolumn filecolumn filecolumn fileCommitted data, available to readSliding memory mapped page in write only modecommit automatically updates read record countAPPEND positionReader 1Reader 2Reader 3WriterSingle WriterMultiple Readers \ No newline at end of file diff --git a/static/images/docs/concepts/storageSummary.webp b/static/images/docs/concepts/storageSummary.webp new file mode 100644 index 000000000..b144fd2e9 Binary files /dev/null and b/static/images/docs/concepts/storageSummary.webp differ diff --git a/static/images/guides/questdb-internals/columnRead.webp b/static/images/guides/questdb-internals/columnRead.webp new file mode 100644 index 000000000..f0a3335e4 Binary files /dev/null and b/static/images/guides/questdb-internals/columnRead.webp differ diff --git a/static/images/guides/questdb-internals/columnarStorage.webp b/static/images/guides/questdb-internals/columnarStorage.webp new file mode 100644 index 000000000..68b4d9c8d Binary files /dev/null and b/static/images/guides/questdb-internals/columnarStorage.webp differ diff --git a/static/images/guides/questdb-internals/intervalScan.webp b/static/images/guides/questdb-internals/intervalScan.webp new file mode 100644 index 000000000..f47ca0267 Binary files /dev/null and b/static/images/guides/questdb-internals/intervalScan.webp differ diff --git a/static/images/guides/questdb-internals/partitionModel.webp b/static/images/guides/questdb-internals/partitionModel.webp new file mode 100644 index 000000000..e648fbbdf Binary files /dev/null and b/static/images/guides/questdb-internals/partitionModel.webp differ diff --git a/static/images/guides/questdb-internals/query_plan.webp b/static/images/guides/questdb-internals/query_plan.webp new file mode 100644 index 000000000..7a1715082 Binary files /dev/null and b/static/images/guides/questdb-internals/query_plan.webp differ diff --git a/static/images/guides/questdb-internals/questdb-high-level-architecture.svg b/static/images/guides/questdb-internals/questdb-high-level-architecture.svg new file mode 100644 index 000000000..a137f35b4 --- /dev/null +++ b/static/images/guides/questdb-internals/questdb-high-level-architecture.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/static/images/guides/questdb-internals/telemetry.webp b/static/images/guides/questdb-internals/telemetry.webp new file mode 100644 index 000000000..f89b31af6 Binary files /dev/null and b/static/images/guides/questdb-internals/telemetry.webp differ diff --git a/static/images/guides/questdb-internals/walData.webp b/static/images/guides/questdb-internals/walData.webp new file mode 100644 index 000000000..2b8aff951 Binary files /dev/null and b/static/images/guides/questdb-internals/walData.webp differ diff --git a/static/images/guides/questdb-internals/web-console.webp b/static/images/guides/questdb-internals/web-console.webp new file mode 100644 index 000000000..6bd39d916 Binary files /dev/null and b/static/images/guides/questdb-internals/web-console.webp differ