diff --git a/docs/api/stream/delete.md b/docs/api/stream/delete.md index 186e6161..94f60eb4 100644 --- a/docs/api/stream/delete.md +++ b/docs/api/stream/delete.md @@ -3,46 +3,322 @@ description: >- Delete OpenObserve streams via API. Deletion is async and handled by the compactor. Configure auto-deletion with data retention environment settings. --- -# Delete stream +## Delete stream +OpenObserve provides multiple deletion strategies to manage your data lifecycle: immediate complete stream deletion, targeted time-range deletion with job tracking, and automatic retention-based cleanup. -Endpoint: `DELETE /api/{organization}/streams/{stream}?type={StreamType}` +## Overview +The Delete Stream API allows you to: -## Request +- Delete an entire stream and all its data +- Delete data within a specific time period with job tracking +- Monitor deletion job progress across clusters +- Manage cached query results +All deletion operations are asynchronous and processed by the Compactor service. -- type: logs / metrics / traces +## Base URL +`https://example.remote.dev/` +Replace `example.remote.dev` with your actual OpenObserve instance URL. - default is `logs`. +## Content type +All requests and responses use JSON format. +``` +Content-Type: application/json +``` +## Endpoints + +### Delete entire stream +Delete a complete stream and all associated data. -## Response +#### Request +**Method**: `DELETE`
+**Path**: `/api/{org_id}/streams/{stream_name}?type=logs&delete_all=true`
+**Parameters**: +| Name | Type | Location | Required | Description | +|------|------|----------|----------|-------------| +| org_id | string | path | Yes | Organization identifier | +| stream_name | string | path | Yes | Name of the stream to delete | +| type | string | query | Yes | Stream type: `logs`, `metrics`, or `traces` | +| delete_all | boolean | path | Yes | Delete all related resources like alerts and dashboards | +#### Request example +```bash +curl -X 'DELETE' \ + 'https://example.remote.dev/api/default/streams/pii_test?type=logs&delete_all=true' \ + -H 'accept: application/json' +``` + +#### Response +**Status Code:** `200 OK` ```json { - "code": 200, - "message": "stream deleted" + "code": 200, + "message": "stream deleted" } ``` -The data delete is an asynchronous operation. it will delete by `Compactor`. +#### Response fields +| Field | Type | Description | +|-------|------|-------------| +| code | integer | HTTP status code | +| message | string | Confirmation message | + +#### Status codes +| Code | Meaning | +|------|---------| +| 200 | Stream deleted successfully | +| 400 | Invalid parameters | +| 404 | Stream not found | +| 500 | Internal server error | + +#### Behavior +Deletion is asynchronous and does not happen immediately: + +1. When you call this API, the deletion request is marked in the system. +2. The API responds immediately, you do not wait for actual deletion. +3. A background service called Compactor checks for pending deletions every 10 minutes. +4. When Compactor runs, it starts deleting your stream. This can take anywhere from seconds to several minutes depending on how much data the stream contains. +5. In the worst-case scenario (if you request deletion just before Compactor runs), the entire process could take up to 30 minutes total. +6. You do not need to wait. The deletion happens in the background. You can check the stream status later to confirm it has been deleted. + +!!! note "Notes" + + - This operation cannot be undone. + - Data is deleted from both the `file_list` table and object store. + - No job tracking is available for this endpoint + +!!! note "Environment variables" + - You can change the `compactor` run interval: `ZO_COMPACT_INTERVAL=600`. Unit is second. default is `10 minutes`. + - You can configure data life cycle to auto delete old data: `ZO_COMPACT_DATA_RETENTION_DAYS=30`. The system will auto delete the data after `30` days. Note that the value must be greater than `0`. -> it will execute by an interval `10 minutes` as default. So maybe the data will delete after 30 minutes. You don't need to wait it done, you can confirm the delete result hours later. +### Delete stream data by time range +Delete stream data within a specific time period with job tracking. -You can change the `compactor` run interval by an environment: +#### Request +**Method:** `DELETE` +
+**Path:** `/api/{org_id}/streams/{stream_name}/data_by_time_range?start=&end=` +#### Parameters +| Parameter | Type | Location | Description | +|-----------|------|----------|-------------| +| `org_id` | string | Path | Organization identifier | +| `stream_name` | string | Path | Name of the stream | +| `start` | long | path | Start timestamp in microseconds (UTC). Inclusive. | +| `end` | long | path | End timestamp in microseconds (UTC). Inclusive. | +#### Request example +```bash +curl -X DELETE \ + 'https://example.remote.dev/api/default/streams/test_stream/data_by_time_range?start=1748736000000000&end=1751241600000000' ``` -ZO_COMPACT_INTERVAL=600 +#### Response +**Status Code:** `200 OK` +```json +{ + "id": "30ernyKEEMznL8KIXEaZhmDYRR9" +} ``` +#### Response fields +| Field | Type | Description | +|-------|------|-------------| +| `id` | string | Unique job ID for tracking deletion progress | +#### Status codes +| Code | Meaning | +|------|---------| +| 200 | Deletion job created successfully | +| 400 | Invalid parameters (For example, invalid timestamp format) | +| 404 | Stream not found | +#### Behavior +- Initiates a compaction delete job. +- Returns a job ID that can be used to track progress. +- Deletes data from: -Unit is second. default is `10 minutes`. + - `file_list` table + - Object store (for example, S3) +- Granularity: + - **Logs:** Data is deleted every hour. + - **Traces:** Data is deleted daily. +--- -## Data lifecycle +### Get delete job status +Check the status of a time-range deletion job. -You can configure data life cycle to auto delete old data. it is a awesome feature. +#### Request +**Method:** `GET` +
+**Path:** `/api/{org_id}/streams/{stream_name}/data_by_time_range/status/{id}` -To enable this feature, you just need to add an environment: +#### Parameters +| Parameter | Type | Location | Description | +|-----------|------|----------|-------------| +| `org_id` | string | Path | Organization identifier | +| `stream_name` | string | Path | Name of the stream | +| `id` | string | Path | Job ID returned from deletion request | +#### Request example +```bash +curl -X GET \ + 'https://example.remote.dev/api/default/streams/test_stream/data_by_time_range/status/30ernyKEEMznL8KIXEaZhmDYRR9' ``` -ZO_COMPACT_DATA_RETENTION_DAYS=30 + +#### Response: Completed +**Status Code:** `200 OK` +```json +{ + "id": "30f080gLbU4i21VpY2O3YzwrKDH", + "status": "Completed", + "metadata": [ + { + "cluster": "dev3", + "region": "us-test-3", + "id": "30f080gLbU4i21VpY2O3YzwrKDH", + "key": "default/logs/delete_d3/2025-07-27T04:00:00Z,2025-07-28T04:00:00Z", + "created_at": 1754003156467113, + "ended_at": 1754003356516415, + "status": "Completed" + }, + { + "cluster": "dev4", + "region": "us-test-4", + "id": "30f080gLbU4i21VpY2O3YzwrKDH", + "key": "default/logs/delete_d3/2025-07-27T04:00:00Z,2025-07-28T04:00:00Z", + "created_at": 1754003156467113, + "ended_at": 1754003326523177, + "status": "Completed" + } + ] +} ``` +#### Response: Pending +**Status Code:** `200 OK` +```json +{ + "id": "30f080gLbU4i21VpY2O3YzwrKDH", + "status": "Pending", + "metadata": [ + { + "cluster": "dev3", + "region": "us-test-3", + "id": "30f080gLbU4i21VpY2O3YzwrKDH", + "key": "default/logs/delete_d3/2025-07-27T04:00:00Z,2025-07-28T04:00:00Z", + "created_at": 1754003156467113, + "ended_at": 0, + "status": "Pending" + }, + { + "cluster": "dev4", + "region": "us-test-4", + "id": "30f080gLbU4i21VpY2O3YzwrKDH", + "key": "default/logs/delete_d3/2025-07-27T04:00:00Z,2025-07-28T04:00:00Z", + "created_at": 1754003156467113, + "ended_at": 0, + "status": "Pending" + } + ] +} +``` +#### Response: With Errors +**Status Code:** `200 OK` +```json +{ + "id": "30fCWBSNWwTWnRJE0weFfDIc3zz", + "status": "Pending", + "metadata": [ + { + "cluster": "dev4", + "region": "us-test-4", + "id": "30fCWBSNWwTWnRJE0weFfDIc3zz", + "key": "default/logs/delete_d4/2025-07-21T14:00:00Z,2025-07-22T00:00:00Z", + "created_at": 1754009269552227, + "ended_at": 1754009558553845, + "status": "Completed" + } + ], + "errors": [ + { + "cluster": "dev3", + "error": "Error getting delete job status from cluster node: Status { code: Internal, message: \"Database error: DbError# SeaORMError# job not found\", metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\", \"date\": \"Fri, 01 Aug 2025 00:58:01 GMT\", \"content-length\": \"0\"} }, source: None }", + "region": "us-test-3" + } + ] +} +``` +#### Response fields +| Field | Type | Description | +|-------|------|-------------| +| `id` | string | Job identifier | +| `status` | string | Overall job status: `Completed` or `Pending` | +| `metadata` | array | Array of per-cluster deletion details | +| `metadata[].cluster` | string | Cluster identifier | +| `metadata[].region` | string | Region/zone identifier | +| `metadata[].id` | string | Job ID | +| `metadata[].key` | string | Database key for the deletion operation | +| `metadata[].created_at` | long | Job creation timestamp in microseconds | +| `metadata[].ended_at` | long | Job completion timestamp in microseconds (0 if still pending) | +| `metadata[].status` | string | Individual cluster deletion status | +| `errors` | array | Array of errors from specific clusters (if any) | +| `errors[].cluster` | string | Cluster where error occurred | +| `errors[].region` | string | Region identifier | +| `errors[].error` | string | Error message | + +#### Status Codes +| Code | Meaning | +|------|---------| +| 200 | Status retrieved successfully | +| 404 | Job ID not found | + +#### Behavior +- Returns current status of deletion job +- Shows progress across all clusters in distributed setup +- Shows error details if any cluster encountered failures +- Status of `Pending` means deletion is still in progress +- Status of `Completed` means all clusters finished deletion +--- + +### Delete cache results +Delete cached query results for a stream. +#### Request +**Method:** `DELETE` +
+**Path:** `/api/{org_id}/streams/{stream_name}/cache/status/results?type=&ts=` + +### Parameters +| Parameter | Type | Location | Description | +|-----------|------|----------|-------------| +| `org_id` | string | path | Organization identifier | +| `stream_name` | string | Path | Stream name (use `_all` to delete cache for all streams) | +| `type` | string | path | Stream type: `logs`, `metrics`, or `traces` | +| `ts` | long | path | Timestamp threshold in microseconds. Deletes cache from start up to this timestamp. Retains cache from timestamp onwards. | + +#### Request example +```bash +curl -X DELETE \ + 'https://example.remote.dev/api/default/streams/test_stream/_all/cache/results?type=logs&ts=1753849800000' +``` + +### Response +**Status Code:** `200 OK` +```json +{ + "code": 200, + "message": "cache deleted" +} +``` + +### Response Fields +| Field | Type | Description | +|-------|------|-------------| +| `code` | integer | HTTP status code | +| `message` | string | Confirmation message | + +### Status Codes +| Code | Meaning | +|------|---------| +| 200 | Cache deleted successfully | +| 400 | Invalid parameters | +| 404 | Stream not found | -The value have to greater than `0`, and it is means how many days you want to keep data. it will auto delete the data older than `N` days. +### Behavior +- Accepts `ts` (timestamp) query parameter in microseconds +- Deletes cache from `cache_start` up to the given `ts` +- Retains cache from `ts` onwards \ No newline at end of file diff --git a/docs/environment-variables.md b/docs/environment-variables.md index 03a0ca0d..b85b192c 100644 --- a/docs/environment-variables.md +++ b/docs/environment-variables.md @@ -15,10 +15,7 @@ OpenObserve is configured using the following environment variables. | ZO_LOCAL_MODE | true | If local mode is set to true, OpenObserve becomes single node deployment.If it is set to false, it indicates cluster mode deployment which supports multiple nodes with different roles. For local mode one needs to configure SQLite DB, for cluster mode one needs to configure PostgreSQL (recommended) or MySQL. | | ZO_LOCAL_MODE_STORAGE | disk | Applicable only for local mode. By default, local disk is used as storage. OpenObserve supports both disk and S3 in local mode. | | ZO_NODE_ROLE | all | Node role assignment. Possible values are ingester, querier, router, compactor, alertmanager, and all. A single node can have multiple roles by specifying them as a comma-separated list. For example, compactor, alertmanager. | -| ZO_NODE_ROLE_GROUP | "" | Each query-processing node can be assigned to a specific group using ZO_NODE_ROLE_GROUP.
-**interactive**: Handles queries triggered directly by users through the UI.
-**background**: Handles automated or scheduled queries, such as alerts and reports.
-**empty string** (default): Handles all query types.
+| ZO_NODE_ROLE_GROUP | "" | Each query-processing node can be assigned to a specific group using ZO_NODE_ROLE_GROUP.
- **interactive**: Handles queries triggered directly by users through the UI.
- **background**: Handles automated or scheduled queries, such as alerts and reports.
- **empty string** (default): Handles all query types.
In high-load environments, alerts or reports might run large, resource-intensive queries. By assigning dedicated groups, administrators can prevent such queries from blocking or slowing down real-time user searches. | | ZO_NODE_HEARTBEAT_TTL | 30 | Time-to-live (TTL) for node heartbeats in seconds. | | ZO_INSTANCE_NAME | - | In the cluster mode, each node has a instance name. Default is instance hostname. | @@ -57,10 +54,12 @@ In high-load environments, alerts or reports might run large, resource-intensive | Environment Variable | Default Value | Description | |---------------------|---------------|-------------| | ZO_COLS_PER_RECORD_LIMIT | 1000 | Maximum number of fields allowed per record during ingestion. Records with more fields than this limit are discarded. | +| ZO_ENTRY_PER_SCHEMA_VERSION_ENABLED | true | Enables version tracking of entries per schema to support versioned reads and analytics. | | ZO_WIDENING_SCHEMA_EVOLUTION | true | If set to false, user can add new columns to the ingested data, but changes to existing column data types are not supported. | | ZO_SKIP_SCHEMA_VALIDATION | false | By default, every ingested record is validated against the schema. If the schema is fixed, validation can be skipped to double ingestion performance. | | ZO_ALLOW_USER_DEFINED_SCHEMAS | false | When set to true, allows users to define user-defined schemas for a stream. | | ZO_SKIP_FORMAT_STREAM_NAME | false | When set to true, it skips formatting stream name while ingestion. | +| ZO_INGEST_INFER_SCHEMA_PER_REQUEST | true | Infers and updates the schema automatically for each ingestion request. | | ZO_CONCATENATED_SCHEMA_FIELD_NAME | _all | Field name where all fields, after applying user-defined schema rules and flattening, are stored as a single field. The default value is _all. For example, if the processed data is {a:5, b:"abc"}, an additional field _all is added as {a:5, b:"abc", _all:"{a:5,b:"abc"}"}. This field is used for full-text search, allowing match_all queries to run across all data instead of being limited to a single field. | | ZO_INGEST_ALLOWED_UPTO | 5 | Discards events older than the specified number of hours. By default, OpenObserve accepts data only if it is not older than 5 hours from the current ingestion time. | | ZO_INGEST_ALLOWED_IN_FUTURE | 24 | Discards events dated beyond the specified number of future hours. By default, OpenObserve accepts data only if it is not timestamped more than 24 hours into the future. | @@ -88,11 +87,7 @@ In high-load environments, alerts or reports might run large, resource-intensive | ZO_MEM_TABLE_MAX_SIZE | 0 | Total size limit of all memtables. Multiple memtables exist for different organizations and stream types. Each memtable cannot exceed ZO_MAX_FILE_SIZE_IN_MEMORY, and the combined size cannot exceed this limit. If exceeded, the system returns a MemoryTableOverflowError to prevent out-of-memory conditions. Default is 50 percent of total memory. | | ZO_MEM_PERSIST_INTERVAL | 5 | Interval in seconds at which immutable memtables are persisted from memory to disk. Default is 5 seconds. | | ZO_FEATURE_SHARED_MEMTABLE_ENABLED | false | When set to true, it turns on the shared memtable feature and several organizations can use the same in-memory table instead of each organization creating its own. This helps reduce memory use when many organizations send data at the same time. It also works with older non-shared write-ahead log (WAL) files. | -| ZO_MEM_TABLE_BUCKET_NUM | 1 | This setting controls how many in-memory tables OpenObserve creates, and works differently depending on whether shared memtable is enabled or disabled.
-**When ZO_FEATURE_SHARED_MEMTABLE_ENABLED is true (shared memtable enabled)**: -
-OpenObserve creates the specified number of shared in-memory tables that all organizations use together.
**If the number is higher**: OpenObserve creates more shared tables. Each table holds data from fewer organizations. This can make data writing faster because each table handles less data. However, it also uses more memory.
-**If the number is lower**: OpenObserve creates fewer shared tables. Each table holds data from more organizations. This saves memory but can make data writing slightly slower when many organizations send data at the same time. +| ZO_MEM_TABLE_BUCKET_NUM | 1 | This setting controls how many in-memory tables OpenObserve creates, and works differently depending on whether shared memtable is enabled or disabled.
**When ZO_FEATURE_SHARED_MEMTABLE_ENABLED is true (shared memtable enabled)**: OpenObserve creates the specified number of shared in-memory tables that all organizations use together.
**If the number is higher**: OpenObserve creates more shared tables. Each table holds data from fewer organizations. This can make data writing faster because each table handles less data. However, it also uses more memory.
**If the number is lower**: OpenObserve creates fewer shared tables. Each table holds data from more organizations. This saves memory but can make data writing slightly slower when many organizations send data at the same time.
**When ZO_FEATURE_SHARED_MEMTABLE_ENABLED is false (shared memtable disabled)**:
@@ -108,6 +103,7 @@ This is particularly useful when you have only one organization, as creating mul | ZO_FEATURE_INDEX_EXTRA_FIELDS | - | Automatically enables global secondary indexing on the specified fields if they exist in the ingested log data. Example: field1,field2 | | ZO_FEATURE_QUERY_PARTITION_STRATEGY | file_num | Query partition strategy. Possible values are file_num, file_size, file_hash. | | ZO_ENABLE_INVERTED_INDEX | true | Enables inverted index creation. | +| ZO_FEATURE_QUERY_REMOVE_FILTER_WITH_INDEX | true | Optimizes query execution by removing redundant filters when an index fully covers the query condition. When enabled, OpenObserve returns results directly from the inverted index without performing position verification. This improves query performance but can also expand the result set compared to running the query with filter verification. | ## Compaction and Data Retention | Environment Variable | Default Value | Description | @@ -115,6 +111,7 @@ This is particularly useful when you have only one organization, as creating mul | ZO_COMPACT_ENABLED | true | Enables compact for small files. | | ZO_COMPACT_INTERVAL | 60 | The interval at which job compacts small files into larger files. default is 60s, unit: second | | ZO_COMPACT_MAX_FILE_SIZE | 256 | Max file size for a single compacted file, after compaction all files will be below this value. Default is 256MB, unit: MB | +| ZO_IGNORE_FILE_RETENTION_BY_STREAM | false | Ignores stream-level file retention settings and applies the global retention policy. | | ZO_COMPACT_DATA_RETENTION_DAYS | 3650 | Data retention days, default is 10 years. Minimal 3. eg: 30, it means will auto delete the data older than 30 days. You also can set data retention for stream in the UI. | | ZO_COMPACT_SYNC_TO_DB_INTERVAL | 1800 | The interval time in seconds after which compactor sync cache to db is run. | | ZO_COMPACT_DELETE_FILES_DELAY_HOURS | 2 | The number of hours to delay to delete the pending deleted files by compactor. Value can not be less than 1. | @@ -332,7 +329,14 @@ This is particularly useful when you have only one organization, as creating mul | ZO_CLI_USER_COOKIE | - | Cookie value used by the CLI user. | | ZO_COOKIE_MAX_AGE | 2592000 | Cookie max age in seconds. | | ZO_EXT_AUTH_SALT | openobserve | Salt used for external authentication. | -| O2_ACTION_SERVER_TOKEN | - | Token used by the action server. | + + + +## Action +| Environment Variable | Default Value | Description | +| ------------------------- | ------------- | -------------------------------------- | +| O2_ACTION_ENABLED | true | Enables the Actions feature for running real-time or scheduled automation tasks. | +| O2_ACTION_SERVER_TOKEN | - | Token used by Action Server for authentication during action execution. Previously, O2_SCRIPT_SERVER_TOKEN. | ## NATS | Environment Variable | Default Value | Description | @@ -352,6 +356,10 @@ This is particularly useful when you have only one organization, as creating mul | ZO_NATS_QUEUE_MAX_SIZE | 2048 | Maximum queue size in megabytes. | | ZO_NATS_KV_WATCH_MODULES | 2048 | Defines which internal modules use the NATS Key-Value Watcher instead of the default NATS Queue for event synchronization. Add one or more module prefixes separated by commas, such as /nodes/ or /user_sessions/. When left empty, all modules use the default NATS Queue mechanism. | | ZO_NATS_EVENT_STORAGE | memory | Controls how NATS JetStream stores event data. Use memory for high-speed, in-memory event storage or file for durable, disk-based storage that persists across restarts.
Performance Benchmark Results:
• File Storage: 10,965 ops/sec (10.71 MB/s throughput, ~911 µs mean latency)
• Memory Storage: 16,957 ops/sec (16.56 MB/s throughput, ~589 µs mean latency)
Memory storage offers ~55 percent higher throughput and lower latency, while file storage ensures durability. | +| ZO_NATS_V211_SUPPORT | false | Enables support for NATS version 2.11 TTL markers for node health detection. When set to true, NATS generates purge events when a node fails to send a keepalive message within the TTL period (configured by `ZO_NODE_HEARTBEAT_TTL`, default `30` seconds). This allows other nodes to immediately detect and remove the crashed node from their cache, preventing query timeouts and connection errors that would otherwise occur while nodes wait to detect the failure.
+When set to false, nodes rely on slower failure detection mechanisms and continue attempting to communicate with crashed nodes, resulting in timeout or connection errors.
+**Note**: This feature requires NATS version 2.11 or later. Setting this to true with earlier NATS versions will cause configuration errors. The default is false to maintain compatibility with NATS versions prior to 2.11. | + ## S3 and Object Storage @@ -618,7 +626,18 @@ This is particularly useful when you have only one organization, as creating mul | ZO_FEATURE_QUERY_QUEUE_ENABLED | true | Enterprise edition must not enable this. In the open source edition, when enabled, the system processes only one search request at a time. | | ZO_FEATURE_QUERY_INFER_SCHEMA | false | Deprecated. Not used by the code. Will be removed. | | ZO_FEATURE_DISTINCT_EXTRA_FIELDS | - | Reserved for future use. Contact the maintainers for guidance. | - +| ZO_HISTOGRAM_ENABLED | true | Enables histogram-based query aggregation and visual display for dashboards and log views. | +| ZO_METRICS_CACHE_ENABLED | true | Enables metrics cache to store and reuse results for repeated metrics queries. | +| ZO_SHOW_STREAM_DATES_DOCS_NUM | true | Displays the date range and document count for streams in the user interface. | +| ZO_SKIP_FORMAT_BULK_STREAM_NAME | false | Keeps the original stream name format when ingesting through bulk APIs. | +| ZO_METRICS_CACHE_MAX_ENTRIES | 10000 | Maximum number of entries stored in the metrics cache. | +| ZO_PARQUET_MAX_ROW_GROUP_SIZE | 0 | Maximum row group size used when writing Parquet files. Zero uses system default. | +| ZO_QUERY_DEFAULT_LIMIT | 1000 | Default row limit applied when a SQL query does not specify a LIMIT clause. | +| ZO_QUERY_INGESTER_TIMEOUT | 0 | Timeout duration (in seconds) for ingester-side query execution. Zero disables timeout. | +| ZO_USAGE_REPORTING_THREAD_NUM | 0 |Number of threads used for usage reporting operations. | +| ZO_USER_DEFINED_SCHEMA_MAX_FIELDS | 600 | Maximum number of fields allowed in a user-defined schema. | +| ZO_UI_ENABLED | true | Enables or disables the OpenObserve web user interface. | +| ZO_PRINT_KEY_EVENT | false | Enables printing of key-level events in logs for debugging. | --- diff --git a/docs/images/dashboard-refresh-query.png b/docs/images/dashboard-refresh-query.png new file mode 100644 index 00000000..f108f707 Binary files /dev/null and b/docs/images/dashboard-refresh-query.png differ diff --git a/docs/images/logs-refresh-query.png b/docs/images/logs-refresh-query.png new file mode 100644 index 00000000..f3e61794 Binary files /dev/null and b/docs/images/logs-refresh-query.png differ diff --git a/docs/images/not-equal.png b/docs/images/not-equal.png new file mode 100644 index 00000000..0939fd58 Binary files /dev/null and b/docs/images/not-equal.png differ diff --git a/docs/images/not-in-function.png b/docs/images/not-in-function.png new file mode 100644 index 00000000..8356b470 Binary files /dev/null and b/docs/images/not-in-function.png differ diff --git a/docs/operator-guide/systemd.md b/docs/operator-guide/systemd.md index 69c04fc0..a79f0c55 100644 --- a/docs/operator-guide/systemd.md +++ b/docs/operator-guide/systemd.md @@ -12,8 +12,8 @@ Install OpenObserve as a system service use systemd. `/etc/openobserve.env` ```toml -ZO_ROOT_USER_EMAIL = "root@example.com" -ZO_ROOT_USER_PASSWORD = "Complexpass#123" +ZO_ROOT_USER_EMAIL = "example@example.com" +ZO_ROOT_USER_PASSWORD = "Example@123" ZO_DATA_DIR = "/data/openobserve" ``` diff --git a/docs/sql-functions/.pages b/docs/sql-functions/.pages index 0571231a..ffb8d4b8 100644 --- a/docs/sql-functions/.pages +++ b/docs/sql-functions/.pages @@ -2,6 +2,7 @@ nav: - SQL Functions Overview: index.md - Full-Text Search Functions: full-text-search.md +- Secondary Index Functions: secondary-index.md - Array Functions: array.md - Aggregate Functions: aggregate.md - Approximate Aggregate Functions: approximate-aggregate \ No newline at end of file diff --git a/docs/sql-functions/full-text-search.md b/docs/sql-functions/full-text-search.md index 289747ab..58b7d009 100644 --- a/docs/sql-functions/full-text-search.md +++ b/docs/sql-functions/full-text-search.md @@ -1,6 +1,6 @@ --- title: Full-Text Search Functions in OpenObserve -description: This page describes the full-text search functions supported in OpenObserve, including their syntax, behavior, and examples. Functions such as str_match, str_match_ignore_case, match_all, re_match, and re_not_match allow users to filter logs based on exact string matches, case-insensitive searches, keyword searches across multiple fields, and pattern-based filtering using regular expressions. The guide also explains the role of inverted indexing and how to enable it for enhanced search coverage. Sample queries and output visuals are provided to help users apply these functions effectively in log analysis. +description: This page describes the full-text search functions supported in OpenObserve, including their syntax, behavior, and examples. --- The full-text search functions allow you to filter records based on keyword or pattern matches within one or more fields.
This page lists the full-text search functions supported in OpenObserve, along with their usage formats, descriptions, and examples. diff --git a/docs/sql-functions/secondary-index.md b/docs/sql-functions/secondary-index.md new file mode 100644 index 00000000..0576d984 --- /dev/null +++ b/docs/sql-functions/secondary-index.md @@ -0,0 +1,43 @@ +--- +title: Secondary Index Functions in OpenObserve +description: This page describes the secondary index functions supported in OpenObserve. +--- +Secondary index functions allow you to filter and optimize queries using fields that are configured as secondary indexes in stream settings. + +## `!=` (Not Equal) +**Syntax**: `field_name != 'value'`
+**Description**: + +- Filters records where the specified indexed field does not match the given value. +- The comparison is case sensitive for string fields. +- When the environment variable `ZO_FEATURE_QUERY_NOT_FILTER_WITH_INDEX` is set to `true`, OpenObserve uses the secondary index to exclude the specified value before scanning data. +
+ +**Example**: +```sql +SELECT * +FROM "default" +WHERE k8s_namespace_name != 'ingress-nginx'; +``` +![not-equal](../images/not-equal.png) + +This query returns logs from the `default` stream where the `k8s_namespace_name` field is any value other than `ingress-nginx`. Note that the `k8s_namespace_name` is configured as a secondary index and the environment variable `ZO_FEATURE_QUERY_NOT_FILTER_WITH_INDEX` is set to `true`. + +## NOT IN +**Syntax**: `field_name NOT IN ('value1', 'value2')`
+**Description**: + +- Filters records where the specified indexed field does not match any value in the provided list. +- The comparison is case sensitive for string fields. +- When the environment variable `ZO_FEATURE_QUERY_NOT_FILTER_WITH_INDEX` is set to `true`, OpenObserve uses the secondary index to exclude all listed values before scanning data. +
+ +**Example**: +```sql +SELECT * +FROM "default" +WHERE k8s_namespace_name NOT IN ('ingress-nginx', 'httptester'); +``` +![not-in-function](../images/not-in-function.png) + +This query returns logs from the `default` stream where `k8s_namespace_name` is not any of the listed namespaces. Note that the `k8s_namespace_name` field configured as a secondary indexed field and the environment variable `ZO_FEATURE_QUERY_NOT_FILTER_WITH_INDEX` is set to `true` \ No newline at end of file diff --git a/docs/user-guide/logs/.pages b/docs/user-guide/logs/.pages index c0357db4..61c9b46c 100644 --- a/docs/user-guide/logs/.pages +++ b/docs/user-guide/logs/.pages @@ -1,6 +1,7 @@ nav: - Logs Overview: index.md - Logs in OpenObserve: logs.md + - Refresh Cache Run Query: refresh-cache-run-query.md - Search Around: search-around.md - Quick Mode and Interesting Fields: quickmode.md - Explain and Analyze Query: explain-analyze-query.md diff --git a/docs/user-guide/logs/explain-analyze-query.md b/docs/user-guide/logs/explain-analyze-query.md index f98c915c..b7ec9bde 100644 --- a/docs/user-guide/logs/explain-analyze-query.md +++ b/docs/user-guide/logs/explain-analyze-query.md @@ -84,7 +84,7 @@ The execution tree displays operators from top to bottom on your screen, but you --- -## Explain Query +## Explain query The Explain Query option shows how OpenObserve interprets and executes a SQL query. It displays two views: @@ -117,19 +117,16 @@ The Physical Plan shows how OpenObserve executes your query, including the speci !!! note "Common operators you will see:" - - **DataSourceExec**: Reads data from storage - - **RemoteScanExec**: Reads data from distributed partitions or remote nodes - - **FilterExec**: Applies filtering operations - - **ProjectionExec**: Handles column selection and expression computation - - **AggregateExec**: Performs aggregation operations - - May show `mode=Partial` or `mode=FinalPartitioned` - - **RepartitionExec**: Redistributes data across partitions - - May show `Hash([column], N)` or `RoundRobinBatch(N)` - - **CoalesceBatchesExec**: Combines data batches - - **SortExec**: Sorts data - - May show `TopK(fetch=N)` for optimized sorting - - **SortPreservingMergeExec**: Merges sorted data streams - - **CooperativeExec**: Coordinates distributed execution + - **DataSourceExec**: Reads data from storage. + - **RemoteScanExec**: Reads data from distributed partitions or remote nodes. + - **FilterExec**: Applies filtering operations. + - **ProjectionExec**: Handles column selection and expression computation. + - **AggregateExec**: Performs aggregation operations. May show `mode=Partial` or `mode=FinalPartitioned`. + - **RepartitionExec**: Redistributes data across partitions. May show `Hash([column], N)` or `RoundRobinBatch(N)`. + - **CoalesceBatchesExec**: Combines data batches. + - **SortExec**: Sorts data. May show `TopK(fetch=N)` for optimized sorting. + - **SortPreservingMergeExec**: Merges sorted data streams. + - **CooperativeExec**: Coordinates distributed execution. --- diff --git a/docs/user-guide/logs/refresh-cache-run-query.md b/docs/user-guide/logs/refresh-cache-run-query.md new file mode 100644 index 00000000..e04ed6d9 --- /dev/null +++ b/docs/user-guide/logs/refresh-cache-run-query.md @@ -0,0 +1,83 @@ +# Refresh Cache and Run Query + +## Overview + +The Refresh Cache and Run Query option allows users to manually clear existing cached results for a query and execute it again using the most recent data. This feature ensures query accuracy in scenarios where cached results may be outdated due to delayed ingestion or cache corruption. + +--- + +## Why this feature exists + +When OpenObserve executes a query, it caches the results on disk to speed up future query execution. Any repeated execution of the same query for the same time range retrieves data directly from this cache. However, in some cases the cached data may not reflect the latest state of the system. Common scenarios include: + +* Delayed ingestion: New logs arrive after the cache has already been created, so the cached result misses this late data. +* Cache corruption or invalid cache: System or storage issues may cause incomplete or corrupted cache entries. + +In such cases, users can use Refresh Cache and Run Query to invalidate the old cache and rebuild it with updated results. + +--- + +!!! note "Where to find it" + It is available in both **Logs** and **Dashboards**. + + **Logs page**: + ![alt text](../../images/logs-refresh-query.png) + + 1. Navigate to **Logs** in the left navigation panel. + 2. Select the required stream. + 3. Define your query in the query editor. + 4. Select your time range. + 5. Click the dropdown attached to the **Run query** button. + 6. Select **Refresh Cache and Run Query**. + + This clears the cache for the selected query range and reloads results using the latest ingested data. + + **Dashboard panels**: + ![dashboard-refresh-query](../../images/dashboard-refresh-query.png) + + 1. Navigate to **Dashboards** from the left navigation panel. + 2. Open the folder containing your dashboard. + 3. Select the desired dashboard. + 4. Locate the panel for which you want to refresh the cache. + 5. Open the panel configuration dropdown and select **Refresh Cache and Reload**. + + This removes any cached data for the query in that panel and re-executes the underlying query to fetch updated results. + +--- + +!!! note "Who should use this feature" + + The feature is primarily designed for system administrators or advanced users who are aware of delayed ingestion or caching anomalies within their clusters or regions. They can use this control to refresh cached data for affected time windows. + + Regular users typically do not need to use this option unless instructed by an administrator. + +--- + +## How it works + +When you select Refresh Cache and Run Query, the following operations occur: + +1. OpenObserve identifies the cache entry corresponding to the specific query, start time, and end time. +2. The system deletes this cache entry from disk. +3. The query runs again against the source data, bypassing the cache. +4. The query results are written back to the cache with the updated values. + +This process ensures that the next time the same query is executed for the same time range, the result will come from the refreshed cache. + +--- + +## Example scenario + +Assume you run a query for the last three hours and find that the result count at 9:00 a.m. is 3,173. The result is now cached. Even if you rerun the same query multiple times, it continues to return 3,173 from the cache. + +Later, additional data for that time period arrives due to ingestion delay, and the actual count should now be 3,373. Since the system continues to read from cache, you will still see 3,173 until you refresh it. + +When you select Refresh Cache and Run Query: + +- The existing cache for that query and time range (with 3,173) is deleted. +- The query runs again directly from the raw data. +- The new result 3,373 replaces the old cache entry. + +You now see the updated result, and future runs use this refreshed cache. + +--- \ No newline at end of file diff --git a/docs/user-guide/performance/.pages b/docs/user-guide/performance/.pages index d3c5e4df..1962275e 100644 --- a/docs/user-guide/performance/.pages +++ b/docs/user-guide/performance/.pages @@ -1,6 +1,7 @@ nav: - Performance Overview: index.md - Download Manager: download-manager.md + - Result Cache: result-cache.md - Monitor Download Queue Size and Disk Cache Metrics: monitor-download-queue-size-and-disk-cache-metrics.md - Configure Disk Cache Eviction Strategy: disk-cache-strategy.md - Tantivy Index: tantivy-index.md diff --git a/docs/user-guide/performance/result-cache.md b/docs/user-guide/performance/result-cache.md new file mode 100644 index 00000000..5f3c94cf --- /dev/null +++ b/docs/user-guide/performance/result-cache.md @@ -0,0 +1,51 @@ +## Overview +Previously, when a query was executed, the system used gRPC calls to check whether other nodes in the cluster already had cached results for that query. This required distributed coordination across all nodes.
+OpenObserve now implements a local result caching system that improves query performance by storing query results on the same node where the query is executed. The caching system relies on **consistent hashing**, which ensures that identical queries are always routed to the same node. This deterministic routing enables each node to maintain and reuse its own cached results without requiring cross-node communication.
+The local cache reduces network overhead, improves response times, and ensures stable query performance across large deployments. + +## How result cache works + +### Query routing and cache locality +OpenObserve uses consistent hashing to route each query to a specific node. +Consistent hashing is a distributed hashing technique that assigns queries to nodes based on a hash value generated from the query text, time range, and other parameters. This guarantees that the same query always lands on the same node. +Because of this routing behavior: + +- Each node stores and retrieves its own cache. +- There is no need to query other nodes for cached data. +- Cache lookups are faster and more predictable. + +## Cache operations +The result cache is stored locally on each node and managed through a read-write mechanism that maintains accuracy and performance. + +### Write logic +When query results are written to the cache, the following checks are applied to maintain data quality: + +1. Remove incomplete histogram records: Any record that does not contain complete data for the time interval is discarded. +2. Exclude recent data: Records newer than `ZO_CACHE_DELAY_SECS` (default is `300`s) are not cached. This prevents incomplete or in-progress data from being stored. +3. Skip invalid cache files: Empty cache files or files covering less than `300` seconds of data are not saved. +This write logic ensures that only stable and complete query results are cached. + +### Read logic +When a query is executed, the node performs the following steps: + +1. Finds all cache files belonging to the same query. +2. Selects the most appropriate file based on time range and histogram alignment. +3. If multiple cache files are allowed (which is configured using `ZO_USE_MULTIPLE_RESULT_CACHE`), retrieves all matching files and merges the results. +4. Generates a new query time range based on existing cache results. + + - The system checks which parts of the requested time range already exist in cache and which do not. + - It executes a new query only for the uncached portion of the time range. + - The newly retrieved data is then merged with the cached data. + - The merged dataset is filtered to include records within the full query time range, then sorted and deduplicated before returning the output.
+ Example:
+ If a user queries data for 9 AM to 11 AM and cache files are available for 9 AM to 10 AM, OpenObserve will query only the missing 10 AM to 11 AM data. It then merges this new result with the cached 9 AM to 10 AM data and returns the combined dataset to the user. + The resulting dataset is then returned as the query output. + +## Cache invalidation +Cache invalidation ensures that query results remain accurate when underlying data changes.
+In OpenObserve, cache retrieval is handled locally by each node, but cache invalidation must still be coordinated across all nodes to maintain consistency. This is done using a dedicated background mechanism that relies on the `delete_result_cache` RPC. The `delete_result_cache` RPC is a remote procedure call (RPC) endpoint that allows nodes in the OpenObserve cluster to coordinate cache invalidation. + +## Configuration +| Variable | Default | Description | +| ------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| ZO_CACHE_DELAY_SECS | 300 | Defines the delay window in seconds before recent query results are eligible for caching. This prevents caching of incomplete data that may still be changing. | \ No newline at end of file diff --git a/docs/user-guide/streams/.pages b/docs/user-guide/streams/.pages index 60604675..48f944c2 100644 --- a/docs/user-guide/streams/.pages +++ b/docs/user-guide/streams/.pages @@ -7,4 +7,6 @@ nav: - Summary Streams: summary-streams.md - Field and Index Types in Streams: fields-and-index-in-streams.md - Query Recommendations Stream: query-recommendations.md + - Distinct Values: distinct-values.md + \ No newline at end of file diff --git a/docs/user-guide/streams/distinct-values.md b/docs/user-guide/streams/distinct-values.md new file mode 100644 index 00000000..e69de29b