add more explanation about tiflash encryption (#12497)

encryption-at-rest.md

@@ -170,6 +170,127 @@ When it is enabled (by default), the data format of encryption metadata is unrec
 
 To avoid the error above, you can first set `security.encryption.enable-file-dictionary-log` to `false` and start TiKV with v4.0.9 or later. Once TiKV starts successfully, the data format of encryption metadata is downgraded to the version recognizable to earlier TiKV versions. At this point, you can then downgrade your TiKV cluster to an earlier version.
 
+## TiFlash encryption at rest
+
+### Overview
+
+The encryption algorithm currently supported by TiFlash is consistent with that supported by TiKV, including AES128, AES192, AES256, and SM4 (only in v6.4.0 and later versions), in CTR mode. TiFlash also uses envelope encryption. Therefore, two types of keys are used in TiFlash when encryption is enabled.
+
+* Master key. The master key is provided by user and is used to encrypt the data keys TiFlash generates. Management of master key is external to TiFlash.
+* Data key. The data key is generated by TiFlash and is the key actually used to encrypt data.
+
+The same master key can be shared by multiple instances of TiFlash, and can also be shared among TiFlash and TiKV. The recommended way to provide a master key in production is via AWS KMS. Alternatively, if using custom key is desired, supplying the master key via file is also supported. The specific method to generate master key and the format of the master key are the same as TiKV.
+
+TiFlash uses the current data key to encrypt all data placed on the disk, including data files, Schmea files, and temporary data files generated during calculations. Data keys are automatically rotated by TiFlash every week by default, and the period is configurable. On key rotation, TiFlash does not rewrite all existing files to replace the key, but background compaction tasks are expected to rewrite old data into new data files, with the most recent data key, if the cluster gets constant write workload. TiFlash keeps track of the key and encryption method used to encrypt each of the files and use the information to decrypt the content on reads.
+
+### Key creation
+
+To create a key on AWS, refer to the steps to create a key for TiKV.
+
+### Configure encryption
+
+To enable encryption, you can add the encryption section in the `tiflash-learner.toml` configuration file:
+
+```
+[security.encryption]
+data-encryption-method = "aes128-ctr"
+data-key-rotation-period = "168h" # 7 days
+```
+
+Alternatively, add the following contents in the TiUP cluster template:
+
+```
+server_configs:
+  tiflash-learner:
+    security.encryption.data-encryption-method: "aes128-ctr"
+    security.encryption.data-key-rotation-period: "168h" # 7 days
+```
+
+Possible values for `data-encryption-method` are "aes128-ctr", "aes192-ctr", "aes256-ctr", "sm4-ctr" (only in v6.4.0 and later versions) and "plaintext". The default value is "plaintext", which means encryption is not turned on. `data-key-rotation-period` defines how often TiFlash rotates the data key. Encryption can be turned on for a fresh TiFlash cluster, or an existing TiFlash cluster, though only data written after encryption is enabled is guaranteed to be encrypted. To disable encryption, remove `data-encryption-method` in the configuration file, or reset it to "plaintext", and restart TiFlash. To change encryption method, update `data-encryption-method` in the configuration file and restart TiFlash. To change the encryption algorithm, replace `data-encryption-method` with a supported encryption algorithm and then restart TiFlash. After the replacement, as new data is written in, the encryption file generated by the previous encryption algorithm is gradually rewritten to a file generated by the new encryption algorithm.
+
+The master key has to be specified if encryption is enabled (that is,`data-encryption-method` is not "plaintext"). To specify an AWS KMS CMK as the master key, add the `encryption.master-key` section after the `encryption` section in the `tiflash-learner.toml` configuration file:
+
+```
+[security.encryption.master-key]
+type = "kms"
+key-id = "0987dcba-09fe-87dc-65ba-ab0987654321"
+region = "us-west-2"
+endpoint = "https://kms.us-west-2.amazonaws.com"
+```
+
+Alternatively, add the following contents in the TiUP cluster template:
+
+```
+server_configs:
+  tiflash-learner:
+    security.encryption.master-key.type: "kms"
+    security.encryption.master-key.key-id: "0987dcba-09fe-87dc-65ba-ab0987654321"
+    security.encryption.master-key.region: "us-west-2"
+    security.encryption.master-key.endpoint: "https://kms.us-west-2.amazonaws.com"
+```
+
+The meanings of the preceding configuration items are the same as those of TiKV.
+
+To specify a master key that is stored in a file, add the following configuration in the `tiflash-learner.toml` configuration file:
+
+```
+[security.encryption.master-key]
+type = "file"
+path = "/path/to/key/file"
+```
+
+Alternatively, add the following contents in the TiUP cluster template:
+
+```
+server_configs:
+  tiflash-learner:
+    security.encryption.master-key.type: "file"
+    security.encryption.master-key.path: "/path/to/key/file"
+```
+
+The meanings of the preceding configuration items and the content format of the key file are the same as those of TiKV.
+
+### Rotate the master key
+
+To rotate the master key of TiFlash, follow the steps to rotate the master key of TiKV. Currently, TiFlash does not support online master key rotation, either. Therefore, you need to restart TiFlash to make the rotation effective. It is recommended to do a rolling restart to a running TiFlash cluster serving online query.
+
+To rotate the KMS CMK, add the following contents in the `tiflash-learner.toml` configuration file:
+
+```
+[security.encryption.master-key]
+type = "kms"
+key-id = "50a0c603-1c6f-11e6-bb9e-3fadde80ce75"
+region = "us-west-2"
+
+[security.encryption.previous-master-key]
+type = "kms"
+key-id = "0987dcba-09fe-87dc-65ba-ab0987654321"
+region = "us-west-2"
+```
+
+Alternatively, add the following contents in the TiUP cluster template:
+
+```
+server_configs:
+  tiflash-learner:
+    security.encryption.master-key.type: "kms"
+    security.encryption.master-key.key-id: "50a0c603-1c6f-11e6-bb9e-3fadde80ce75"
+    security.encryption.master-key.region: "us-west-2"
+    security.encryption.previous-master-key.type: "kms"
+    security.encryption.previous-master-key.key-id: "0987dcba-09fe-87dc-65ba-ab0987654321"
+    security.encryption.previous-master-key.region: "us-west-2"
+```
+
+### Monitoring and debugging
+
+To monitor encryption at rest, if you deploy TiFlash with Grafana, you can look at the **Encryption** panel in the **TiFlash-Proxy-Details** dashboard. The meaning of monitoring items is the same as that of TiKV.
+
+For debugging, since TiFlash reuses TiKV's logic for managing encrypted metadata, the `tikv-ctl` command can be used to dump encryption metadata such as encryption method and data key ID used to encryption the file, as well as list of data keys. This operation can expose sensitive data and is therefore not recommended in production. Refer to [TiKV Control](/tikv-control.md#dump-encryption-metadata) for more details.
+
+### Compatibility between TiKV versions
+
+TiFlash also optimizes encrypted metadata operations in v4.0.9, and its compatibility requirements are the same as those of TiKV. For details, see [Compatibility between TiKV versions](#compatibility-between-tikv-versions).
+
 ## BR S3 server-side encryption
 
 To enable S3 server-side encryption when backup to S3 using BR, pass `--s3.sse` argument and set value to "aws:kms". S3 will use its own KMS key for encryption. Example: