From 1e159506c00382cd40a60f58abe74828ba6a3f5d Mon Sep 17 00:00:00 2001 From: BohuTANG Date: Wed, 2 Jul 2025 10:46:49 +0800 Subject: [PATCH] aduit trail --- docs/en/guides/56-security/audit-trail.md | 108 ++++++++++++++++++ docs/en/guides/56-security/index.md | 1 + .../access-history.md | 15 ++- .../32-system-history-tables/index.md | 81 ++++++++----- .../32-system-history-tables/log-history.md | 17 ++- .../32-system-history-tables/login-history.md | 15 ++- .../profile-history.md | 16 ++- .../32-system-history-tables/query-history.md | 15 ++- 8 files changed, 235 insertions(+), 33 deletions(-) create mode 100644 docs/en/guides/56-security/audit-trail.md diff --git a/docs/en/guides/56-security/audit-trail.md b/docs/en/guides/56-security/audit-trail.md new file mode 100644 index 0000000000..39e72922fb --- /dev/null +++ b/docs/en/guides/56-security/audit-trail.md @@ -0,0 +1,108 @@ +--- +title: Audit Trail +--- + +import EEFeature from '@site/src/components/EEFeature'; + + + +Databend system history tables automatically capture detailed records of database activities, providing a complete audit trail for compliance and security monitoring. + +Allows the auditing of the user: +- **Query execution** - Complete SQL execution audit trail (`query_history`) +- **Data access** - Database object access and modifications (`access_history`) +- **Authentication** - Login attempts and session tracking (`login_history`) + +## Available Audit Tables + +Databend provides five system history tables that capture different aspects of database activity: + +| Table | Purpose | Key Use Cases | +|-------|---------|---------------| +| [query_history](/sql/sql-reference/system-history-tables/query-history) | Complete SQL execution audit trail | Performance monitoring, security auditing, compliance reporting | +| [access_history](/sql/sql-reference/system-history-tables/access-history) | Database object access and modifications | Data lineage tracking, compliance auditing, change management | +| [login_history](/sql/sql-reference/system-history-tables/login-history) | Authentication attempts and sessions | Security monitoring, failed login detection, access pattern analysis | + +## Audit Use Cases & Examples + +### Security Monitoring + +**Monitor Failed Login Attempts** + +Track authentication failures to identify potential security threats and unauthorized access attempts. + +```sql +-- Check for failed login attempts (security audit) +SELECT event_time, user_name, client_ip, error_message +FROM system_history.login_history +WHERE event_type = 'LoginFailed' +ORDER BY event_time DESC; +``` + +Example output: +``` +event_time: 2025-06-03 06:07:32.512021 +user_name: root1 +client_ip: 127.0.0.1:62050 +error_message: UnknownUser. Code: 2201, Text = User 'root1'@'%' does not exist. +``` + +### Compliance Reporting + +**Track Database Schema Changes** + +Monitor DDL operations for compliance and change management requirements. + +```sql +-- Audit DDL operations (compliance tracking) +SELECT query_id, query_start, user_name, object_modified_by_ddl +FROM system_history.access_history +WHERE object_modified_by_ddl != '[]' +ORDER BY query_start DESC; +``` + +Example for `CREATE TABLE` operation: +``` +query_id: c2c1c7be-cee4-4868-a28e-8862b122c365 +query_start: 2025-06-12 03:31:19.042128 +user_name: root +object_modified_by_ddl: [{"object_domain":"Table","object_name":"default.default.t","operation_type":"Create"}] +``` + +**Audit Data Access Patterns** + +Track who accessed what data and when for compliance and data governance. + +```sql +-- Track data access for compliance +SELECT query_id, query_start, user_name, base_objects_accessed +FROM system_history.access_history +WHERE base_objects_accessed != '[]' +ORDER BY query_start DESC; +``` + +### Operational Monitoring + +**Complete Query Execution Audit** + +Maintain comprehensive records of all SQL operations with user and timing information. + +```sql +-- Complete query audit with user and timing information +SELECT query_id, sql_user, query_text, query_start_time, query_duration_ms, client_address +FROM system_history.query_history +WHERE event_date >= TODAY() - INTERVAL 7 DAY +ORDER BY query_start_time DESC; +``` + +Example output: +``` +query_id: 4e1f50a9-bce2-45cc-86e4-c7a36b9b8d43 +sql_user: root +query_text: SELECT * FROM t +query_start_time: 2025-06-12 03:31:35.041725 +query_duration_ms: 94 +client_address: 127.0.0.1 +``` + +For detailed information about each audit table and their specific fields, see the [System History Tables](/sql/sql-reference/system-history-tables/) reference documentation. diff --git a/docs/en/guides/56-security/index.md b/docs/en/guides/56-security/index.md index 1fef1209f7..040c7dd596 100644 --- a/docs/en/guides/56-security/index.md +++ b/docs/en/guides/56-security/index.md @@ -7,6 +7,7 @@ Databend offers **enterprise-grade security and reliability features** that safe | Security Feature | Purpose | When to Use | |-----------------|---------|------------| | [**Access Control**](/guides/security/access-control) | Manage user permissions | When you need to control data access with role-based security and object ownership | +| [**Audit Trail**](audit-trail.md) | Track database activities | When you need comprehensive audit trails for security monitoring, compliance, and performance analysis | | [**Network Policy**](/guides/security/network-policy) | Restrict network access | When you want to limit connections to specific IP ranges even with valid credentials | | [**Password Policy**](/guides/security/password-policy) | Set password requirements | When you need to enforce password complexity, rotation, and account lockout rules | | [**Masking Policy**](/guides/security/masking-policy) | Hide sensitive data | When you need to protect confidential data while still allowing authorized access | diff --git a/docs/en/sql-reference/00-sql-reference/32-system-history-tables/access-history.md b/docs/en/sql-reference/00-sql-reference/32-system-history-tables/access-history.md index fa39ea0691..236e3553e7 100644 --- a/docs/en/sql-reference/00-sql-reference/32-system-history-tables/access-history.md +++ b/docs/en/sql-reference/00-sql-reference/32-system-history-tables/access-history.md @@ -2,7 +2,20 @@ title: system_history.access_history --- -This table provides detailed logging of objects accessed and modified by each query, including tables, columns, and stages, as part of the query metadata. It provides structured information about DDL and DML operations to enhance auditing. +import FunctionDescription from '@site/src/components/FunctionDescription'; + + + +import EEFeature from '@site/src/components/EEFeature'; + + + +**Data lineage and access control audit** - Tracks all database objects (tables, columns, stages) accessed or modified by queries. Essential for: + +- **Data Lineage**: Understand data flow and dependencies across your database +- **Compliance Reporting**: Track who accessed sensitive data and when +- **Change Management**: Monitor DDL operations and schema modifications +- **Security Analysis**: Identify unusual access patterns or unauthorized data access ## Fields diff --git a/docs/en/sql-reference/00-sql-reference/32-system-history-tables/index.md b/docs/en/sql-reference/00-sql-reference/32-system-history-tables/index.md index 26b292e0ca..8b4340d72f 100644 --- a/docs/en/sql-reference/00-sql-reference/32-system-history-tables/index.md +++ b/docs/en/sql-reference/00-sql-reference/32-system-history-tables/index.md @@ -2,57 +2,84 @@ title: System History Tables --- -import FunctionDescription from '@site/src/components/FunctionDescription'; +import EEFeature from '@site/src/components/EEFeature'; - + # System History Tables -System history tables store persistent data in the `system_history` schema for auditing, troubleshooting, and compliance purposes. They track query execution, user logins, and system logs that can be queried using standard SQL. +Databend's system history tables provide **Data Governance** capabilities by automatically tracking database activities for compliance, security monitoring, and performance analysis. -## Available System History Tables +## Available Tables -| Table | Description | -|-----------------------------------------------------|-----------------------------------------------------------------| -| [system_history.log_history](log-history.md) | Stores raw log entries from various system components. | -| [system_history.query_history](query-history.md) | Stores structured details of query execution. | -| [system_history.profile_history](profile-history.md)| Stores detailed query execution profiles and statistics. | -| [system_history.login_history](login-history.md) | Records information about user login events. | -| [system_history.access_history](access-history.md) | Stores information about query access events. | +| Table | Purpose | Key Use Cases | +|-------|---------|---------------| +| [query_history](query-history.md) | Complete SQL execution audit trail | Performance analysis, compliance tracking, usage monitoring | +| [access_history](access-history.md) | Data access and modification logs | Data lineage, compliance reporting, change management | +| [login_history](login-history.md) | User authentication tracking | Security auditing, failed login monitoring, access pattern analysis | +| [profile_history](profile-history.md) | Detailed query execution profiles | Performance optimization, resource planning, bottleneck identification | +| [log_history](log-history.md) | Raw system logs and events | System troubleshooting, error analysis, operational monitoring | -## Enabling System History Tables +## Configuration -> **Note:** In **Databend Cloud**, system history tables are automatically enabled and ready to use without any configuration needed. The following section applies only to **self-hosted Databend**. +### Databend Cloud +✅ **Automatically enabled** - All system history tables are ready to use without any configuration. -In self-hosted Databend, system history tables are disabled by default. To enable them, configure the `[log.history]` section in your `databend-query.toml` file. +### Self-Hosted Databend -Configuration Example: +
+📝 **Manual configuration required** - Click to expand configuration details + +#### Minimal Configuration +To enable system history tables, you must configure all 5 tables in your `databend-query.toml`: ```toml [log.history] -# Enable history tables on = true -level = "INFO" -# Configure retention policies for each table +# All 5 tables must be configured to enable history logging +# retention is optional (default: 168 hours = 7 days) [[log.history.tables]] -table_name = "log_history" -retention = 168 # 7 days (in hours) +table_name = "query_history" +retention = 168 # Optional: 7 days (default) [[log.history.tables]] -table_name = "query_history" -retention = 168 +table_name = "login_history" +retention = 168 # Optional: 7 days (default) + +[[log.history.tables]] +table_name = "access_history" +retention = 168 # Optional: 7 days (default) [[log.history.tables]] table_name = "profile_history" -retention = 168 +retention = 168 # Optional: 7 days (default) [[log.history.tables]] -table_name = "login_history" -retention = 168 +table_name = "log_history" +retention = 168 # Optional: 7 days (default) +``` + +#### Custom Storage (Optional) +By default, history tables use your main database storage. To use separate S3 storage: + +```toml +[log.history] +on = true + +[log.history.storage] +type = "s3" + +[log.history.storage.s3] +bucket = "your-history-bucket" +root = "history_tables" +endpoint_url = "https://s3.amazonaws.com" +access_key_id = "your-access-key" +secret_access_key = "your-secret-key" ``` -> **Note:** The `log_history` table is enabled by default when history logging is turned on. The `level` configuration determines the number of log entries stored in the log_history table. A more detailed level will result in more entries. +> ⚠️ **Note:** When changing storage configuration, existing history tables will be dropped and recreated. +
-For more details about configuration options, see [Query Configuration: [log.history] Section](/guides/deploy/references/node-config/query-config#loghistory-section). +For complete configuration options, see [Query Configuration: [log.history] Section](/guides/deploy/references/node-config/query-config#loghistory-section). diff --git a/docs/en/sql-reference/00-sql-reference/32-system-history-tables/log-history.md b/docs/en/sql-reference/00-sql-reference/32-system-history-tables/log-history.md index dd9a29365e..611aa350bb 100644 --- a/docs/en/sql-reference/00-sql-reference/32-system-history-tables/log-history.md +++ b/docs/en/sql-reference/00-sql-reference/32-system-history-tables/log-history.md @@ -2,9 +2,22 @@ title: system_history.log_history --- -Stores raw log entries ingested from various nodes. This table acts as the primary source for subsequent log transformations. +import FunctionDescription from '@site/src/components/FunctionDescription'; -All the other log tables are derived from this table, the difference is that other log tables will do some transformations to make the data more structured. + + +import EEFeature from '@site/src/components/EEFeature'; + + + +**System operations audit trail** - Raw log repository from all Databend nodes and components. Foundation for operational intelligence: + +- **System Monitoring**: Track system health, performance, and resource usage +- **Troubleshooting**: Debug issues with detailed error logs and system events +- **Operational Analytics**: Analyze system behavior patterns and trends +- **Root Cause Analysis**: Investigate system failures and performance bottlenecks + +> **Note:** This table contains raw log data that feeds into other specialized history tables. Other tables provide structured, query-specific views of this data. ## Fields diff --git a/docs/en/sql-reference/00-sql-reference/32-system-history-tables/login-history.md b/docs/en/sql-reference/00-sql-reference/32-system-history-tables/login-history.md index 55f901fac9..692c2f1758 100644 --- a/docs/en/sql-reference/00-sql-reference/32-system-history-tables/login-history.md +++ b/docs/en/sql-reference/00-sql-reference/32-system-history-tables/login-history.md @@ -2,7 +2,20 @@ title: system_history.login_history --- -Records all login attempts in the system, including successful and failed login attempts. This table is useful for auditing user access and troubleshooting authentication issues. +import FunctionDescription from '@site/src/components/FunctionDescription'; + + + +import EEFeature from '@site/src/components/EEFeature'; + + + +**Authentication security audit** - Comprehensive logging of all user login attempts (successful and failed). Critical for: + +- **Security Monitoring**: Detect brute force attacks and unauthorized access attempts +- **Compliance Auditing**: Track user authentication for regulatory requirements +- **Access Pattern Analysis**: Monitor when and how users access the system +- **Incident Investigation**: Investigate security incidents and authentication issues ## Fields diff --git a/docs/en/sql-reference/00-sql-reference/32-system-history-tables/profile-history.md b/docs/en/sql-reference/00-sql-reference/32-system-history-tables/profile-history.md index 9feaef869e..19237d4797 100644 --- a/docs/en/sql-reference/00-sql-reference/32-system-history-tables/profile-history.md +++ b/docs/en/sql-reference/00-sql-reference/32-system-history-tables/profile-history.md @@ -1,7 +1,21 @@ --- title: system_history.profile_history --- -Stores detailed execution profiles for SQL queries in Databend. Each entry provides performance metrics and execution statistics, allowing users to analyze and optimize query performance. + +import FunctionDescription from '@site/src/components/FunctionDescription'; + + + +import EEFeature from '@site/src/components/EEFeature'; + + + +**Query performance deep-dive analytics** - Detailed execution profiles and statistics for every SQL query. Essential for: + +- **Performance Optimization**: Identify bottlenecks and optimize slow queries +- **Resource Planning**: Understand memory, CPU, and I/O usage patterns +- **Execution Analysis**: Analyze query plans and execution statistics +- **Capacity Management**: Monitor resource consumption trends over time ## Fields diff --git a/docs/en/sql-reference/00-sql-reference/32-system-history-tables/query-history.md b/docs/en/sql-reference/00-sql-reference/32-system-history-tables/query-history.md index ecb093a485..46986f7980 100644 --- a/docs/en/sql-reference/00-sql-reference/32-system-history-tables/query-history.md +++ b/docs/en/sql-reference/00-sql-reference/32-system-history-tables/query-history.md @@ -2,7 +2,20 @@ title: system_history.query_history --- -Records detailed logs of all SQL query executions in Databend. For each query, two entries are generated: one when the query starts and another when it finishes. This table is valuable for monitoring query activity, auditing user actions, and analyzing performance metrics. +import FunctionDescription from '@site/src/components/FunctionDescription'; + + + +import EEFeature from '@site/src/components/EEFeature'; + + + +**Complete SQL execution audit trail** - Records comprehensive details of all SQL queries executed in Databend. Each query generates two entries (start and finish), providing complete visibility into: + +- **Performance Analysis**: Query duration, resource usage, and optimization opportunities +- **Security Auditing**: Who executed what queries, when, and from where +- **Compliance Tracking**: Complete audit trail for regulatory requirements +- **Usage Monitoring**: Database activity patterns and user behavior analysis ## Fields