add more explanation about tiflash encryption

encryption-at-rest.md

@@ -170,6 +170,127 @@ When it is enabled (by default), the data format of encryption metadata is unrec
 
 To avoid the error above, you can first set `security.encryption.enable-file-dictionary-log` to `false` and start TiKV with v4.0.9 or later. Once TiKV starts successfully, the data format of encryption metadata is downgraded to the version recognizable to earlier TiKV versions. At this point, you can then downgrade your TiKV cluster to an earlier version.
 
+## TiFlash encryption at rest
+
+### Overview
+
+The encryption algorithm currently supported by TiFlash is consistent with that of TiKV, including AES128, AES192, AES256, or SM4 (only in v6.4.0 and later versions), in CTR mode. TiFlash also uses envelope encryption. As a result, two types of keys are used in TiFlash when encryption is enabled.
+
+* Master key. The master key is provided by user and is used to encrypt the data keys TiFlash generates. Management of master key is external to TiFlash.
+* Data key. The data key is generated by TiFlash and is the key actually used to encrypt data.
+
+The same master key can be shared by multiple instances of TiFlash, and can also be shared among TiFlash and TiKV. The recommended way to provide a master key in production is via AWS KMS. Alternatively, if using custom key is desired, supplying the master key via file is also supported. The specific method to generate master key and the format of the master key are the same as TiKV.
+
+TiFlash uses the current data key to encrypt all data placed on disk, including data files, Schmea files, and temporary data files generated during calculations. Data keys are automatically rotated by TiKV every week by default, but the period is configurable. On key rotation, TiFlash does not rewrite all existing files to replace the key, but background compaction tasks are expected to rewrite old data into new data files, with the most recent data key, if the cluster gets constant write workload. TiFlash keeps track of the key and encryption method used to encrypt each of the files and use the information to decrypt the content on reads.
+
+### Key creation
+
+To create a key on AWS, please refer to the steps to create a key for TiKV.
+
+### Configure encryption
+
+To enable encryption, you can add the encryption section in the tiflash-learner.toml configuration file:
+
+```
+[security.encryption]
+data-encryption-method = "aes128-ctr"
+data-key-rotation-period = "168h" # 7 days
+```
+
+Or add the following contents in tiup cluster template:
+
+```
+server_configs:
+  tiflash-learner:
+    security.encryption.data-encryption-method: "aes128-ctr"
+    security.encryption.data-key-rotation-period: "168h" # 7 days
+```
+
+Possible values for `data-encryption-method` are "aes128-ctr", "aes192-ctr", "aes256-ctr", "sm4-ctr" (only in v6.4.0 and later versions) and "plaintext". The default value is "plaintext", which means encryption is not turned on. `data-key-rotation-period` defines how often TiFlash rotates the data key. Encryption can be turned on for a fresh TiFlash cluster, or an existing TiFlash cluster, though only data written after encryption is enabled is guaranteed to be encrypted. To disable encryption, remove `data-encryption-method` in the configuration file, or reset it to "plaintext", and restart TiFlash. To change encryption method, update `data-encryption-method` in the configuration file and restart TiFlash. To change the encryption algorithm, replace `data-encryption-method` with a supported encryption algorithm and then restart TiFlash. After the replacement, as new data is written in, the encryption file generated by the previous encryption algorithm is gradually rewritten to a file generated by the new encryption algorithm.
+
+The master key has to be specified if encryption is enabled (that is,`data-encryption-method` is not "plaintext"). To specify a AWS KMS CMK as master key, add the `encryption.master-key` section after the `encryption` section in the tiflash-learner.toml config file:
+
+```
+[security.encryption.master-key]
+type = "kms"
+key-id = "0987dcba-09fe-87dc-65ba-ab0987654321"
+region = "us-west-2"
+endpoint = "https://kms.us-west-2.amazonaws.com"
+```
+
+Or add the following contents in tiup cluster template:
+
+```
+server_configs:
+  tiflash-learner:
+    security.encryption.master-key.type: "kms"
+    security.encryption.master-key.key-id: "0987dcba-09fe-87dc-65ba-ab0987654321"
+    security.encryption.master-key.region: "us-west-2"
+    security.encryption.master-key.endpoint: "https://kms.us-west-2.amazonaws.com"
+```
+
+The meanings of the above configuration items are the same as those of TiKV.
+
+To specify a master key that's stored in a file, add the following configuration in tiflash-learner.toml configuration file:
+
+```
+[security.encryption.master-key]
+type = "file"
+path = "/path/to/key/file"
+```
+
+Or add the following contents in tiup cluster template:
+
+```
+server_configs:
+  tiflash-learner:
+    security.encryption.master-key.type: "file"
+    security.encryption.master-key.path: "/path/to/key/file"
+```
+
+The meaning of the above configuration items and the content format of the key file are the same as those of TiKV.
+
+### Rotate the master key
+
+To rotate master key of TiFlash, follow the same steps as tikv. Currently online master key rotation is also not supported, so you need to restart TiFlash. It is advised to do a rolling restart to a running TiFlash cluster serving online query.
+
+To rotate the KMS CMK, add the following contents in tiflash-learner.toml configuration file:
+
+```
+[security.encryption.master-key]
+type = "kms"
+key-id = "50a0c603-1c6f-11e6-bb9e-3fadde80ce75"
+region = "us-west-2"
+
+[security.encryption.previous-master-key]
+type = "kms"
+key-id = "0987dcba-09fe-87dc-65ba-ab0987654321"
+region = "us-west-2"
+```
+
+Or add the following contents in tiup cluster template:
+
+```
+server_configs:
+  tiflash-learner:
+    security.encryption.master-key.type: "kms"
+    security.encryption.master-key.key-id: "50a0c603-1c6f-11e6-bb9e-3fadde80ce75"
+    security.encryption.master-key.region: "us-west-2"
+    security.encryption.previous-master-key.type: "kms"
+    security.encryption.previous-master-key.key-id: "0987dcba-09fe-87dc-65ba-ab0987654321"
+    security.encryption.previous-master-key.region: "us-west-2"
+```
+
+### Monitoring and debugging
+
+To monitor encryption at rest, if you deploy TiFlash with Grafana, you can look at the **Encryption** panel in the **TiFlash-Proxy-Details** dashboard. The meaning of monitoring items is the same as that of TiKV.
+
+For debugging, since TiFlash reuses TiKV's logic for managing encrypted metadata, the `tikv-ctl` command can be used to dump encryption metadata such as encryption method and data key id used to encryption the file, as well as list of data keys. Since the operation can expose sensitive data, it is not recommended to use in production. Please refer to [TiKV Control](/tikv-control.md#dump-encryption-metadata) document.
+
+### Compatibility between TiKV versions
+
+TiFlash also optimizes encrypted metadata operations in v4.0.9, and its compatibility requirements are the same as those of TiKV.
+
 ## BR S3 server-side encryption
 
 To enable S3 server-side encryption when backup to S3 using BR, pass `--s3.sse` argument and set value to "aws:kms". S3 will use its own KMS key for encryption. Example: