[FEATURE] Encrypt volume backup to remote backup store without in-cluster volume encryption

[innobead]

Is your feature request related to a problem? Please describe (👍 if you like this request)

For volume data encryption, Longhorn supports FS volume encryption (block volume encryption will be #4883 ), so it can do volume encryption in transit and at rest. When encrypting a volume and backing it up to the remote backup store, the data will be encrypted as well and this is how to achieve at-rest encryption. However, this at-rest backup encryption needs to rely on in-cluster volume encryption.

To make the encryption operation flexible and meet users' different encryption compliance requirements, supporting backup volume encryption to the remote backup store can be independent of in-cluster volume encryption.

Describe the solution you'd like

• Have a global setting or volume-specific setting to enable remote backup encryption. By default, it's disabled.
• Users can provide an encryption key via different ciphers like AES-256
• Users can use external KMS providers for their key source

Describe alternatives you've considered

Just rely on the server-side encryption of backup store, but it will be vendor-lockin

Additional context

cc @longhorn/dev

Related Tickets

#8453

[PatrickHuetter]

This feature would be awesome since it would be possible to use cheap third party s3 storage providers like https://www.idrive.com/object-storage-e2/ without having the fear of getting unauthorized data access to backups. Especially if you don't want to encrypt volumes locally.

[rebeccazzzz]

Hey team! Please add your planning poker estimate with Zenhub @chriscchien @ejweber @ChanYiLin @mantissahz @james-munson @PhanLe1010 @jillian-maroket @FrankYang0529 @roger-ryao @shuo-wu @yangchiu

[rebeccazzzz]

Please add your planning poker estimate with Zenhub @derekbit