Skip to content

Commit

Permalink
Merge pull request #502 from ApsaraDB/POLARDB_11_DEV
Browse files Browse the repository at this point in the history 
merge: 20240414
  • Loading branch information
@mrdrivingduck
mrdrivingduck committed Apr 14, 2024
2 parents 49d141c + 7de734f commit 301f650
Show file tree
Hide file tree
Showing 20 changed files with 401 additions and 63 deletions.
1 change: 1 addition & 0 deletions docs/.vuepress/config.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ export default defineUserConfig({
},
}),
mdEnhancePlugin({
katex: true,
footnote: true,
}),
registerComponentsPlugin({
Expand Down
16 changes: 8 additions & 8 deletions docs/contributing/contributing-polardb-docs.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ PolarDB for PostgreSQL 的文档使用 [VuePress 2](https://v2.vuepress.vuejs.or

## 本地文档开发

若您发现文档中存在内容或格式错误，或者您希望能够贡献新文档，那么您需要在本地安装并配置文档开发环境。本项目的文档是一个 Node.js 工程，以 [Yarn](https://yarnpkg.com/) 作为软件包管理器。[Node.js®](https://nodejs.org/en/) 是一个基于 Chrome V8 引擎的 JavaScript 运行时环境。
若您发现文档中存在内容或格式错误，或者您希望能够贡献新文档，那么您需要在本地安装并配置文档开发环境。本项目的文档是一个 Node.js 工程，以 [pnpm](https://pnpm.io/) 作为软件包管理器。[Node.js®](https://nodejs.org/en/) 是一个基于 Chrome V8 引擎的 JavaScript 运行时环境。

### Node 环境准备

Expand Down Expand Up @@ -40,27 +40,27 @@ node -v
npm -v
```

使用 `npm` 全局安装软件包管理器 `yarn`：
使用 `npm` 全局安装软件包管理器 `pnpm`：

```bash:no-line-numbers
npm install -g yarn
yarn -v
npm install -g pnpm
pnpm -v
```

### 文档依赖安装

在 PolarDB for PostgreSQL 工程的根目录下运行以下命令，`yarn` 将会根据 `package.json` 安装所有依赖：
在 PolarDB for PostgreSQL 工程的根目录下运行以下命令，`pnpm` 将会根据 `package.json` 安装所有依赖：

```bash:no-line-numbers
yarn
pnpm install
```

### 运行文档开发服务器

在 PolarDB for PostgreSQL 工程的根目录下运行以下命令：

```bash:no-line-numbers
yarn docs:dev
pnpm run docs:dev
```

文档开发服务器将运行于 `http://localhost:8080/PolarDB-for-PostgreSQL/`，打开浏览器即可访问。对 Markdown 文件作出修改后，可以在网页上实时查看变化。
Expand Down Expand Up @@ -94,7 +94,7 @@ PolarDB for PostgreSQL 的文档资源位于工程根目录的 `docs/` 目录下

`.vuepress/` 目录下包含文档工程的全局配置信息：

- `config.js`：文档配置
- `config.ts`：文档配置
- `configs/`：文档配置模块（导航栏 / 侧边栏、英文 / 中文等配置）
- `public/`：公共静态资源
- `styles/`：文档主题默认样式覆盖
Expand Down
118 changes: 118 additions & 0 deletions docs/contributing/trouble-issuing.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
polar_stat_env
--------------------------------------------------------------------
{ +
"Role": "Primary", +
"CPU": { +
"Architecture": "x86_64", +
"Model Name": "Intel(R) Xeon(R) Platinum 8369B CPU @ 2.70GHz",+
Expand All @@ -36,3 +37,120 @@
}
(1 row)
```

通过 ePQ 功能可以直接获取整个集群中所有计算节点的硬件配置信息：

```sql:no-line-numbers
=> CREATE EXTENSION polar_stat_env;
=> SET polar_enable_px TO ON;
=> SET polar_px_use_master TO ON;
=> SET polar_px_use_standby TO ON;
=> SELECT * FROM polar_global_function('polar_stat_env');
polar_stat_env
---------------------------------------------------------------------
{ +
"Role": "Standby", +
"CPU": { +
"Architecture": "x86_64", +
"Model Name": "Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz",+
"CPU Cores": "104", +
"CPU Thread Per Cores": "2", +
"CPU Core Per Socket": "26", +
"NUMA Nodes": "2", +
"L1d cache": "32K", +
"L1i cache": "32K", +
"L2 cache": "1024K", +
"L3 cache": "36608K" +
}, +
"Memory": { +
"Memory Total (GB)": "754", +
"HugePage Size (MB)": "2", +
"HugePage Total Size (GB)": "42" +
}, +
"OS Params": { +
"OS": "5.10.134-16.1.al8.x86_64", +
"Swappiness(1-100)": "0", +
"Vfs Cache Pressure(0-1000)": "500", +
"Min Free KBytes(KB)": "20971520" +
} +
}
{ +
"Role": "Replica", +
"CPU": { +
"Architecture": "x86_64", +
"Model Name": "Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz",+
"CPU Cores": "104", +
"CPU Thread Per Cores": "2", +
"CPU Core Per Socket": "26", +
"NUMA Nodes": "2", +
"L1d cache": "32K", +
"L1i cache": "32K", +
"L2 cache": "1024K", +
"L3 cache": "36608K" +
}, +
"Memory": { +
"Memory Total (GB)": "754", +
"HugePage Size (MB)": "2", +
"HugePage Total Size (GB)": "42" +
}, +
"OS Params": { +
"OS": "5.10.134-16.1.al8.x86_64", +
"Swappiness(1-100)": "0", +
"Vfs Cache Pressure(0-1000)": "500", +
"Min Free KBytes(KB)": "20971520" +
} +
}
{ +
"Role": "Primary", +
"CPU": { +
"Architecture": "x86_64", +
"Model Name": "Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz",+
"CPU Cores": "104", +
"CPU Thread Per Cores": "2", +
"CPU Core Per Socket": "26", +
"NUMA Nodes": "2", +
"L1d cache": "32K", +
"L1i cache": "32K", +
"L2 cache": "1024K", +
"L3 cache": "36608K" +
}, +
"Memory": { +
"Memory Total (GB)": "754", +
"HugePage Size (MB)": "2", +
"HugePage Total Size (GB)": "42" +
}, +
"OS Params": { +
"OS": "5.10.134-16.1.al8.x86_64", +
"Swappiness(1-100)": "0", +
"Vfs Cache Pressure(0-1000)": "500", +
"Min Free KBytes(KB)": "20971520" +
} +
}
{ +
"Role": "Replica", +
"CPU": { +
"Architecture": "x86_64", +
"Model Name": "Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz",+
"CPU Cores": "104", +
"CPU Thread Per Cores": "2", +
"CPU Core Per Socket": "26", +
"NUMA Nodes": "2", +
"L1d cache": "32K", +
"L1i cache": "32K", +
"L2 cache": "1024K", +
"L3 cache": "36608K" +
}, +
"Memory": { +
"Memory Total (GB)": "754", +
"HugePage Size (MB)": "2", +
"HugePage Total Size (GB)": "42" +
}, +
"OS Params": { +
"OS": "5.10.134-16.1.al8.x86_64", +
"Swappiness(1-100)": "0", +
"Vfs Cache Pressure(0-1000)": "500", +
"Min Free KBytes(KB)": "20971520" +
} +
}
(4 rows)
```
2 changes: 1 addition & 1 deletion docs/theory/buffer-management.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ To apply the WAL records of a page up to a specified LSN, each read-only node ma

For a specific page, more changes mean more LSNs and a longer period of time required to apply WAL records. To minimize the number of WAL records that need to be applied for each page, PolarDB provides consistent LSNs.

After all changes that are made up to the consistent LSN of a page are written to the shared storage, the page is persistently stored. The primary node sends the write LSN and consistent LSN of the page to each read-only node, and each read-only node sends the apply LSN of the page to the primary node. The read-only nodes do not need to apply the WAL records that are generated before the consistent LSN of the page. Therefore, all LSNs that are smaller than the consistent LSN can be removed from the LogIndex of the page. This reduces the number of WAL records that the read-only nodes need to apply. This also reduces the storage space that is occupied by LogIndex records.
After all changes that are made up to the consistent LSN of a page are written to the shared storage, the page is persistently stored. The primary node sends the write LSN and consistent LSN of the page to each read-only node, and each read-only node sends the apply LSN of the page and the min used LSN of the page to the primary node. The read-only nodes do not need to apply the WAL records that are generated before the consistent LSN of the page while reading it from shared storage. But the read-only nodes may still need to apply the WAL records that are generated before the consistent LSN of the page while replaying outdated page in buffer pool. Therefore, all LSNs that are smaller than the consistent LSN and the min used LSN can be removed from the LogIndex of the page. This reduces the number of WAL records that the read-only nodes need to apply. This also reduces the storage space that is occupied by LogIndex records.

### Flush Lists

Expand Down
3 changes: 2 additions & 1 deletion docs/theory/logindex.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ After the data in an Inactive LogIndex Memtable is flushed to the disk, the LogI

![image.png](../imgs/58_LogIndex_10.png)

All modified data pages recorded in WAL logs before the LSN of consistent data are persisted to the shared storage based on the information described in [Buffer Management](./buffer-management.md). The LSN of consistent data is the LSN before which data is consistent between the primary node and read-only nodes. Read-only nodes do not need to replay WAL logs generated before the LSN of consistent data. In this case, the WAL logs for the LSNs that are smaller than the LSN of consistent data can be cleared from LogIndex Tables. This way, the primary node can truncate LogIndex Tables that are no longer used in the storage. This enables more efficient log replay for read-only nodes and reduces the space occupied by LogIndex Tables.
All modified data pages recorded in WAL logs before the consistent LSN are persisted to the shared storage based on the information described in [Buffer Management](./buffer-management.md). The primary node sends the write LSN and consistent LSN to each read-only node, and each read-only node sends the apply LSN and the min used LSN to the primary node. In this case, the WAL logs whose LSNs are smaller than the consistent LSN and the min used LSN can be cleared from LogIndex Tables. This way, the primary node can truncate LogIndex Tables that are no longer used in the storage. This enables more efficient log replay for read-only nodes and reduces the space occupied by LogIndex Tables.

## Log replay

Expand All @@ -90,6 +90,7 @@ For scenarios in which LogIndex Tables are used, the startup processes of read-o

- The background replay process replays WAL logs in the sequence of WAL logs. The process retrieves modified pages from LogIndex Memtables and LogIndex Tables based on the LSN of a page that you want to replay. If a page exists in a buffer pool, the page is replayed. Otherwise, the page is skipped. The background replay process replays WAL logs generated for the next LSN of a page in a buffer pool in the sequence of LSNs. This prevents a large number of LSNs for a single page that you want to replay from being accumulated.
- The backend process replays only the pages it must access. If the backend process must access a page that does not exist in a buffer pool, the process reads this page from the shared storage, writes the page to a buffer pool, and replays this page. If the page exists in a buffer pool and is marked as an outdated page, the process replays the most recent WAL logs of this page. The backend process retrieves the LSNs of the page from LogIndex Memtables and LogIndex Tables based on the value of PageTag. After the process retrieves the LSNs, the process generates the LSNs for the page in sequence. Then, the process reads the complete WAL logs from the shared storage based on the generated LSNs to replay the page.
- According to the above two points, we can know that both the background replay process and the backend process will use LogIndex information to apply WAL logs on some pages. Therefore, the min used LSN of current RO node is defined as the minimum LSN of WALs which being applying by the background replay process and all backend processes. The RO node sends the current min used LSN to the primary node which would use this LSN to truncate those no longer used LogIndex Tables.

![image.png](../imgs/59_LogIndex_11.png)

Expand Down
16 changes: 8 additions & 8 deletions docs/zh/contributing/contributing-polardb-docs.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ PolarDB for PostgreSQL 的文档使用 [VuePress 2](https://v2.vuepress.vuejs.or

## 本地文档开发

若您发现文档中存在内容或格式错误，或者您希望能够贡献新文档，那么您需要在本地安装并配置文档开发环境。本项目的文档是一个 Node.js 工程，以 [Yarn](https://www.yarnpkg.cn/) 作为软件包管理器。[Node.js®](https://nodejs.org/zh-cn/) 是一个基于 Chrome V8 引擎的 JavaScript 运行时环境。
若您发现文档中存在内容或格式错误，或者您希望能够贡献新文档，那么您需要在本地安装并配置文档开发环境。本项目的文档是一个 Node.js 工程，以 [pnpm](https://pnpm.io/) 作为软件包管理器。[Node.js®](https://nodejs.org/zh-cn/) 是一个基于 Chrome V8 引擎的 JavaScript 运行时环境。

### Node 环境准备

Expand Down Expand Up @@ -36,27 +36,27 @@ node -v
npm -v
```

使用 `npm` 全局安装软件包管理器 `yarn`：
使用 `npm` 全局安装软件包管理器 `pnpm`：

```bash:no-line-numbers
npm install -g yarn
yarn -v
npm install -g pnpm
pnpm -v
```

### 文档依赖安装

在 PolarDB for PostgreSQL 工程的根目录下运行以下命令，`yarn` 将会根据 `package.json` 安装所有依赖：
在 PolarDB for PostgreSQL 工程的根目录下运行以下命令，`pnpm` 将会根据 `package.json` 安装所有依赖：

```bash:no-line-numbers
yarn
pnpm install
```

### 运行文档开发服务器

在 PolarDB for PostgreSQL 工程的根目录下运行以下命令：

```bash:no-line-numbers
yarn docs:dev
pnpm run docs:dev
```

文档开发服务器将运行于 `http://localhost:8080/PolarDB-for-PostgreSQL/`，打开浏览器即可访问。对 Markdown 文件作出修改后，可以在网页上实时查看变化。
Expand Down Expand Up @@ -90,7 +90,7 @@ PolarDB for PostgreSQL 的文档资源位于工程根目录的 `docs/` 目录下

`.vuepress/` 目录下包含文档工程的全局配置信息：

- `config.js`：文档配置
- `config.ts`：文档配置
- `configs/`：文档配置模块（导航栏 / 侧边栏、英文 / 中文等配置）
- `public/`：公共静态资源
- `styles/`：文档主题默认样式覆盖
Expand Down
Loading

0 comments on commit 301f650

Please sign in to comment.