Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 52 additions & 23 deletions docs/en/docs/ecosystem/audit-plugin.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,21 +5,21 @@
}
---

<!--
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
with the License. You may obtain a copy of the License at

  http://www.apache.org/licenses/LICENSE-2.0
http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Expand All @@ -34,30 +34,47 @@ This plugin can periodically import the FE audit log into the specified Doris cl

### FE Configuration

The audit log plug-in framework is disabled by default in Doris. You need to add `plugin_enable = true `to the FE configuration file to enable plugin framework.
The audit log plug-in framework is enabled by default in Doris and is controlled by the FE configuration `plugin_enable`

### AuditLoader Configuration

The configuration of the auditloader plugin is located in `$ {DORIS}/fe_plugins/auditloader/src/main/assembly/`.
1. Download the Audit Loader plugin

Open `plugin.conf` for configuration. See the comments of the configuration items.
The Audit Loader plug-in is provided by default in the Doris distribution. After downloading the Doris installation package through [DOWNLOAD](https://doris.apache.org/download), decompress it and enter the directory, you can find the auditloader.zip file in the extensionsaudit_loader subdirectory.

<version since="1.2.0"></version>
Audit log plugin supports importing slow query logs into a separate slow table since version 1.2, `doris_slow_log_tbl__`, which is closed by default. In the plugin configuration file, add `enable_slow_log = true` to enable the function. And you could modify 'qe_slow_log_ms' item in FE configuration file to change slow query threshold.
2. Unzip the installation package

### Compile
```shell
unzip auditloader.zip
```

After executing `sh build_plugin.sh` in the Doris code directory, you will get the `auditloader.zip` file in the `fe_plugins/output` directory.
Unzip and generate the following files:

### Deployment
* auditloader.jar: plug-in code package.
* plugin.properties: plugin properties file.
* plugin.conf: plugin configuration file.

You can place this file on an http download server or copy(or unzip) it to the specified directory of all FEs. Here we use the latter.
3. Modify plugin.conf

### Installation
The following configurations are available for modification:

After deployment is complete, and before installing the plugin, you need to create the audit log database and tables previously specified in `plugin.conf`. If `enable_slow_log` is set true, the slow table `doris_slow_log_tbl__` needs to be created, with the same schema as `doris_audit_log_tbl__`. The database and table creation statement is as follows:
* frontend_host_port: FE node IP address and HTTP port in the format <fe_ip>:<fe_http_port>. The default value is 127.0.0.1:8030.
* database: Audit log database name.
* audit_log_table: Audit log table name.
* slow_log_table: Slow query log table name.
* enable_slow_log: Whether to enable the slow query log import function. The default value is false.
* user: Cluster username. The user must have INSERT permission on the corresponding table.
* password: Cluster user password.

```
### 创建库表

In Doris, you need to create the library and table of the audit log. The table structure is as follows:

If you need to enable the slow query log import function, you need to create an additional slow table `doris_slow_log_tbl__`, whose table structure is consistent with `doris_audit_log_tbl__`.

Among them, the `dynamic_partition` attribute selects the number of days for audit log retention according to your own needs.

```sql
create database doris_audit_db__;

create table doris_audit_db__.doris_audit_log_tbl__
Expand All @@ -81,7 +98,7 @@ create table doris_audit_db__.doris_audit_log_tbl__
sql_hash varchar(48) comment "Hash value for this query",
sql_digest varchar(48) comment "Sql digest for this query",
peak_memory_bytes bigint comment "Peak memory bytes used on all backends of this query",
stmt string comment "The original statement, trimed if longer than 2G "
stmt string comment "The original statement, trimed if longer than 2G"
) engine=OLAP
duplicate key(query_id, `time`, client_ip)
partition by range(`time`) ()
Expand Down Expand Up @@ -117,7 +134,7 @@ create table doris_audit_db__.doris_slow_log_tbl__
sql_hash varchar(48) comment "Hash value for this query",
sql_digest varchar(48) comment "Sql digest for this query",
peak_memory_bytes bigint comment "Peak memory bytes used on all backends of this query",
stmt string comment "The original statement, trimed if longer than 2G"
stmt string comment "The original statement, trimed if longer than 2G "
) engine=OLAP
duplicate key(query_id, `time`, client_ip)
partition by range(`time`) ()
Expand All @@ -137,8 +154,20 @@ properties(
>
> In the above table structure: stmt string, this can only be used in 0.15 and later versions, in previous versions, the field type used varchar

The `dynamic_partition` attribute selects the number of days to keep the audit log based on your needs.
### Deployment

You can place the packaged auditloader.zip on an http server, or copy `auditloader.zip` to the same specified directory in all FEs.

### Installation

通过以下语句安装 Audit Loader 插件:

```sql
INSTALL PLUGIN FROM [source] [PROPERTIES ("key"="value", ...)]
```

Detailed command reference: [INSTALL-PLUGIN.md](../sql-manual/sql-reference/Database-Administration-Statements/INSTALL-PLUGIN)

After that, connect to Doris and use the `INSTALL PLUGIN` command to complete the installation. After successful installation, you can see the installed plug-ins through `SHOW PLUGINS`, and the status is `INSTALLED`.
After successful installation, you can see the installed plug-ins through `SHOW PLUGINS`, and the status is `INSTALLED`.

Upon completion, the plug-in will continuously import audit date into this table at specified intervals.
After completion, the plugin will continuously insert audit logs into this table at specified intervals.
63 changes: 49 additions & 14 deletions docs/zh-CN/docs/ecosystem/audit-plugin.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,30 +34,53 @@ Doris 的审计日志插件是在 FE 的插件框架基础上开发的。是一

### FE 配置

审计日志插件框架在 Doris 中是默认关闭的,需要在FE的配置文件中,增加`plugin_enable = true`启用plugin框架
审计日志插件框架在 Doris 中是默认开启的的,由 FE 的配置 `plugin_enable` 控制

### AuditLoader 配置

auditloader plugin的配置位于`${DORIS}/fe_plugins/auditloader/src/main/assembly/`.
1. 下载 Audit Loader 插件

打开 `plugin.conf` 进行配置。配置项说明参见注释
Audit Loader 插件在 Doris 的发行版中默认提供,通过 [DOWNLOAD](https://doris.apache.org/zh-CN/download) 下载 Doris 安装包解压并进入目录后即可在 extensions/audit_loader 子目录下找到 auditloader.zip 文件

<version since="1.2.0"></version>
从 1.2 版本开始,审计日志插件支持将慢查询日志导入到单独的慢表 `doris_slow_log_tbl__` 中,Doris 中默认关闭,在审计日志的配置文件中,增加 `enable_slow_log = true`,开启该功能。并且可以在 FE 配置文件中修改 `qe_slow_log_ms` 项来修改慢查询阈值。
2. 解压安装包

### 编译
```shell
unzip auditloader.zip
```

在 Doris 代码目录下执行 `sh build_plugin.sh` 后,会在 `fe_plugins/output` 目录下得到 `auditloader.zip` 文件。
解压生成以下文件:

### 部署
* auditloader.jar:插件代码包。
* plugin.properties:插件属性文件。
* plugin.conf:插件配置文件。

您可以将这个文件放置在一个 http 服务器上,或者拷贝`auditloader.zip`(或者解压`auditloader.zip`)到所有 FE 的指定目录下。这里我们使用后者。
3. 修改 plugin.conf

### 安装
以下配置可供修改:

部署完成后,安装插件前,需要创建之前在 `plugin.conf` 中指定的审计日志数据库和表。若开启了慢查询日志导入功能,需要创建慢表 `doris_slow_log_tbl__`,其表结构与 `doris_audit_log_tbl__` 一致。其中建库与建表语句如下:
* frontend_host_port:FE 节点 IP 地址和 HTTP 端口,格式为 <fe_ip>:<fe_http_port>。 默认值为 127.0.0.1:8030。
* database:审计日志库名。
* audit_log_table:审计日志表名。
* slow_log_table:慢查询日志表名。
* enable_slow_log:是否开启慢查询日志导入功能。默认值为 false。
* user:集群用户名。该用户必须具有对应表的 INSERT 权限。
* password:集群用户密码。

```
4. 重新打包 Audit Loader 插件

```shell
zip -r -q -m auditloader.zip auditloader.jar plugin.properties plugin.conf
```

### 创建库表

在 Doris 中,需要创建审计日志的库和表,表结构如下:

若需开启慢查询日志导入功能,还需要额外创建慢表 `doris_slow_log_tbl__`,其表结构与 `doris_audit_log_tbl__` 一致。

其中 `dynamic_partition` 属性根据自己的需要,选择审计日志保留的天数。

```sql
create database doris_audit_db__;

create table doris_audit_db__.doris_audit_log_tbl__
Expand Down Expand Up @@ -137,8 +160,20 @@ properties(
>
> 上面表结构中:stmt string ,这个只能在0.15及之后版本中使用,之前版本,字段类型使用varchar

其中 `dynamic_partition` 属性根据自己的需要,选择审计日志保留的天数。
### 部署

您可以将 打包好的 auditloader.zip 放置在一个 http 服务器上,或者拷贝`auditloader.zip` 到所有 FE 的相同指定目录下。

### 安装

通过以下语句安装 Audit Loader 插件:

```sql
INSTALL PLUGIN FROM [source] [PROPERTIES ("key"="value", ...)]
```

详细命令参考:[INSTALL-PLUGIN.md](../sql-manual/sql-reference/Database-Administration-Statements/INSTALL-PLUGIN)

之后,连接到 Doris 后使用 `INSTALL PLUGIN` 命令完成安装。安装成功后,可以通过 `SHOW PLUGINS` 看到已经安装的插件,并且状态为 `INSTALLED`。
安装成功后,可以通过 `SHOW PLUGINS` 看到已经安装的插件,并且状态为 `INSTALLED`。

完成后,插件会不断的以指定的时间间隔将审计日志插入到这个表中。