Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 108 additions & 0 deletions docs/en/guides/56-security/audit-trail.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
---
title: Audit Trail
---

import EEFeature from '@site/src/components/EEFeature';

<EEFeature featureName='AUDIT TRAIL'/>

Databend system history tables automatically capture detailed records of database activities, providing a complete audit trail for compliance and security monitoring.

Allows the auditing of the user:
- **Query execution** - Complete SQL execution audit trail (`query_history`)
- **Data access** - Database object access and modifications (`access_history`)
- **Authentication** - Login attempts and session tracking (`login_history`)

## Available Audit Tables

Databend provides five system history tables that capture different aspects of database activity:

| Table | Purpose | Key Use Cases |
|-------|---------|---------------|
| [query_history](/sql/sql-reference/system-history-tables/query-history) | Complete SQL execution audit trail | Performance monitoring, security auditing, compliance reporting |
| [access_history](/sql/sql-reference/system-history-tables/access-history) | Database object access and modifications | Data lineage tracking, compliance auditing, change management |
| [login_history](/sql/sql-reference/system-history-tables/login-history) | Authentication attempts and sessions | Security monitoring, failed login detection, access pattern analysis |

## Audit Use Cases & Examples

### Security Monitoring

**Monitor Failed Login Attempts**

Track authentication failures to identify potential security threats and unauthorized access attempts.

```sql
-- Check for failed login attempts (security audit)
SELECT event_time, user_name, client_ip, error_message
FROM system_history.login_history
WHERE event_type = 'LoginFailed'
ORDER BY event_time DESC;
```

Example output:
```
event_time: 2025-06-03 06:07:32.512021
user_name: root1
client_ip: 127.0.0.1:62050
error_message: UnknownUser. Code: 2201, Text = User 'root1'@'%' does not exist.
```

### Compliance Reporting

**Track Database Schema Changes**

Monitor DDL operations for compliance and change management requirements.

```sql
-- Audit DDL operations (compliance tracking)
SELECT query_id, query_start, user_name, object_modified_by_ddl
FROM system_history.access_history
WHERE object_modified_by_ddl != '[]'
ORDER BY query_start DESC;
```

Example for `CREATE TABLE` operation:
```
query_id: c2c1c7be-cee4-4868-a28e-8862b122c365
query_start: 2025-06-12 03:31:19.042128
user_name: root
object_modified_by_ddl: [{"object_domain":"Table","object_name":"default.default.t","operation_type":"Create"}]
```

**Audit Data Access Patterns**

Track who accessed what data and when for compliance and data governance.

```sql
-- Track data access for compliance
SELECT query_id, query_start, user_name, base_objects_accessed
FROM system_history.access_history
WHERE base_objects_accessed != '[]'
ORDER BY query_start DESC;
```

### Operational Monitoring

**Complete Query Execution Audit**

Maintain comprehensive records of all SQL operations with user and timing information.

```sql
-- Complete query audit with user and timing information
SELECT query_id, sql_user, query_text, query_start_time, query_duration_ms, client_address
FROM system_history.query_history
WHERE event_date >= TODAY() - INTERVAL 7 DAY
ORDER BY query_start_time DESC;
```

Example output:
```
query_id: 4e1f50a9-bce2-45cc-86e4-c7a36b9b8d43
sql_user: root
query_text: SELECT * FROM t
query_start_time: 2025-06-12 03:31:35.041725
query_duration_ms: 94
client_address: 127.0.0.1
```

For detailed information about each audit table and their specific fields, see the [System History Tables](/sql/sql-reference/system-history-tables/) reference documentation.
1 change: 1 addition & 0 deletions docs/en/guides/56-security/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ Databend offers **enterprise-grade security and reliability features** that safe
| Security Feature | Purpose | When to Use |
|-----------------|---------|------------|
| [**Access Control**](/guides/security/access-control) | Manage user permissions | When you need to control data access with role-based security and object ownership |
| [**Audit Trail**](audit-trail.md) | Track database activities | When you need comprehensive audit trails for security monitoring, compliance, and performance analysis |
| [**Network Policy**](/guides/security/network-policy) | Restrict network access | When you want to limit connections to specific IP ranges even with valid credentials |
| [**Password Policy**](/guides/security/password-policy) | Set password requirements | When you need to enforce password complexity, rotation, and account lockout rules |
| [**Masking Policy**](/guides/security/masking-policy) | Hide sensitive data | When you need to protect confidential data while still allowing authorized access |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,20 @@
title: system_history.access_history
---

This table provides detailed logging of objects accessed and modified by each query, including tables, columns, and stages, as part of the query metadata. It provides structured information about DDL and DML operations to enhance auditing.
import FunctionDescription from '@site/src/components/FunctionDescription';

<FunctionDescription description="Introduced or updated: v1.2.764"/>

import EEFeature from '@site/src/components/EEFeature';

<EEFeature featureName='ACCESS HISTORY'/>

**Data lineage and access control audit** - Tracks all database objects (tables, columns, stages) accessed or modified by queries. Essential for:

- **Data Lineage**: Understand data flow and dependencies across your database
- **Compliance Reporting**: Track who accessed sensitive data and when
- **Change Management**: Monitor DDL operations and schema modifications
- **Security Analysis**: Identify unusual access patterns or unauthorized data access


## Fields
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,57 +2,84 @@
title: System History Tables
---

import FunctionDescription from '@site/src/components/FunctionDescription';
import EEFeature from '@site/src/components/EEFeature';

<FunctionDescription description="Introduced or updated: v1.2.752"/>
<EEFeature featureName='SYSTEM HISTORY'/>

# System History Tables

System history tables store persistent data in the `system_history` schema for auditing, troubleshooting, and compliance purposes. They track query execution, user logins, and system logs that can be queried using standard SQL.
Databend's system history tables provide **Data Governance** capabilities by automatically tracking database activities for compliance, security monitoring, and performance analysis.

## Available System History Tables
## Available Tables

| Table | Description |
|-----------------------------------------------------|-----------------------------------------------------------------|
| [system_history.log_history](log-history.md) | Stores raw log entries from various system components. |
| [system_history.query_history](query-history.md) | Stores structured details of query execution. |
| [system_history.profile_history](profile-history.md)| Stores detailed query execution profiles and statistics. |
| [system_history.login_history](login-history.md) | Records information about user login events. |
| [system_history.access_history](access-history.md) | Stores information about query access events. |
| Table | Purpose | Key Use Cases |
|-------|---------|---------------|
| [query_history](query-history.md) | Complete SQL execution audit trail | Performance analysis, compliance tracking, usage monitoring |
| [access_history](access-history.md) | Data access and modification logs | Data lineage, compliance reporting, change management |
| [login_history](login-history.md) | User authentication tracking | Security auditing, failed login monitoring, access pattern analysis |
| [profile_history](profile-history.md) | Detailed query execution profiles | Performance optimization, resource planning, bottleneck identification |
| [log_history](log-history.md) | Raw system logs and events | System troubleshooting, error analysis, operational monitoring |

## Enabling System History Tables
## Configuration

> **Note:** In **Databend Cloud**, system history tables are automatically enabled and ready to use without any configuration needed. The following section applies only to **self-hosted Databend**.
### Databend Cloud
✅ **Automatically enabled** - All system history tables are ready to use without any configuration.

In self-hosted Databend, system history tables are disabled by default. To enable them, configure the `[log.history]` section in your `databend-query.toml` file.
### Self-Hosted Databend

Configuration Example:
<details>
<summary>📝 **Manual configuration required** - Click to expand configuration details</summary>

#### Minimal Configuration
To enable system history tables, you must configure all 5 tables in your `databend-query.toml`:

```toml
[log.history]
# Enable history tables
on = true
level = "INFO"

# Configure retention policies for each table
# All 5 tables must be configured to enable history logging
# retention is optional (default: 168 hours = 7 days)
[[log.history.tables]]
table_name = "log_history"
retention = 168 # 7 days (in hours)
table_name = "query_history"
retention = 168 # Optional: 7 days (default)

[[log.history.tables]]
table_name = "query_history"
retention = 168
table_name = "login_history"
retention = 168 # Optional: 7 days (default)

[[log.history.tables]]
table_name = "access_history"
retention = 168 # Optional: 7 days (default)

[[log.history.tables]]
table_name = "profile_history"
retention = 168
retention = 168 # Optional: 7 days (default)

[[log.history.tables]]
table_name = "login_history"
retention = 168
table_name = "log_history"
retention = 168 # Optional: 7 days (default)
```

#### Custom Storage (Optional)
By default, history tables use your main database storage. To use separate S3 storage:

```toml
[log.history]
on = true

[log.history.storage]
type = "s3"

[log.history.storage.s3]
bucket = "your-history-bucket"
root = "history_tables"
endpoint_url = "https://s3.amazonaws.com"
access_key_id = "your-access-key"
secret_access_key = "your-secret-key"
```

> **Note:** The `log_history` table is enabled by default when history logging is turned on. The `level` configuration determines the number of log entries stored in the log_history table. A more detailed level will result in more entries.
> ⚠️ **Note:** When changing storage configuration, existing history tables will be dropped and recreated.

</details>

For more details about configuration options, see [Query Configuration: [log.history] Section](/guides/deploy/references/node-config/query-config#loghistory-section).
For complete configuration options, see [Query Configuration: [log.history] Section](/guides/deploy/references/node-config/query-config#loghistory-section).
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,22 @@
title: system_history.log_history
---

Stores raw log entries ingested from various nodes. This table acts as the primary source for subsequent log transformations.
import FunctionDescription from '@site/src/components/FunctionDescription';

All the other log tables are derived from this table, the difference is that other log tables will do some transformations to make the data more structured.
<FunctionDescription description="Introduced or updated: v1.2.764"/>

import EEFeature from '@site/src/components/EEFeature';

<EEFeature featureName='LOG HISTORY'/>

**System operations audit trail** - Raw log repository from all Databend nodes and components. Foundation for operational intelligence:

- **System Monitoring**: Track system health, performance, and resource usage
- **Troubleshooting**: Debug issues with detailed error logs and system events
- **Operational Analytics**: Analyze system behavior patterns and trends
- **Root Cause Analysis**: Investigate system failures and performance bottlenecks

> **Note:** This table contains raw log data that feeds into other specialized history tables. Other tables provide structured, query-specific views of this data.

## Fields

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,20 @@
title: system_history.login_history
---

Records all login attempts in the system, including successful and failed login attempts. This table is useful for auditing user access and troubleshooting authentication issues.
import FunctionDescription from '@site/src/components/FunctionDescription';

<FunctionDescription description="Introduced or updated: v1.2.764"/>

import EEFeature from '@site/src/components/EEFeature';

<EEFeature featureName='LOGIN HISTORY'/>

**Authentication security audit** - Comprehensive logging of all user login attempts (successful and failed). Critical for:

- **Security Monitoring**: Detect brute force attacks and unauthorized access attempts
- **Compliance Auditing**: Track user authentication for regulatory requirements
- **Access Pattern Analysis**: Monitor when and how users access the system
- **Incident Investigation**: Investigate security incidents and authentication issues


## Fields
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,21 @@
---
title: system_history.profile_history
---
Stores detailed execution profiles for SQL queries in Databend. Each entry provides performance metrics and execution statistics, allowing users to analyze and optimize query performance.

import FunctionDescription from '@site/src/components/FunctionDescription';

<FunctionDescription description="Introduced or updated: v1.2.764"/>

import EEFeature from '@site/src/components/EEFeature';

<EEFeature featureName='PROFILE HISTORY'/>

**Query performance deep-dive analytics** - Detailed execution profiles and statistics for every SQL query. Essential for:

- **Performance Optimization**: Identify bottlenecks and optimize slow queries
- **Resource Planning**: Understand memory, CPU, and I/O usage patterns
- **Execution Analysis**: Analyze query plans and execution statistics
- **Capacity Management**: Monitor resource consumption trends over time

## Fields

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,20 @@
title: system_history.query_history
---

Records detailed logs of all SQL query executions in Databend. For each query, two entries are generated: one when the query starts and another when it finishes. This table is valuable for monitoring query activity, auditing user actions, and analyzing performance metrics.
import FunctionDescription from '@site/src/components/FunctionDescription';

<FunctionDescription description="Introduced or updated: v1.2.764"/>

import EEFeature from '@site/src/components/EEFeature';

<EEFeature featureName='QUERY HISTORY'/>

**Complete SQL execution audit trail** - Records comprehensive details of all SQL queries executed in Databend. Each query generates two entries (start and finish), providing complete visibility into:

- **Performance Analysis**: Query duration, resource usage, and optimization opportunities
- **Security Auditing**: Who executed what queries, when, and from where
- **Compliance Tracking**: Complete audit trail for regulatory requirements
- **Usage Monitoring**: Database activity patterns and user behavior analysis

## Fields

Expand Down
Loading