utils,framework/db: Introduce new database encryption cipher based on AesGcmJce #7003

weizhouapache · 2022-12-20T12:21:35Z

Description

CloudStack has used StandardPBEStringEncryptor from jasypt for more than 10 years.

The encryptor use algorithm "PBEWithMD5AndDes", which is considered as Insecure, because it uses MD5 and DES which has only 56-bits key.

After investigation, we decided to replace it with an implementation of AES-GCM algorithm.

Main changes of this PR

Introduce new encryptor AeadBase64Encryptor, which is based on AesGcmJce
Improve cloudstack-setup-databases to set up cloudstack database with different encryptors
Improve cloudstack-migrate-database to migrate cloudstack properties and database with different management key, database key or encryptor version.
Improve cloudstack startup process so that both legacy and new encryptors work. Existing environments work well without any change.

More details on cwiki: https://cwiki.apache.org/confluence/display/CLOUDSTACK/New+database+encryption+cipher+-+AeadBase64Encryptor

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Screenshots (if appropriate):

How Has This Been Tested?

cloudstack-setup-databases to set up database with different parameters
cloudstack-migrate-databases to change keys and migrate databases.

… AesGcmJce including all changes until 2022-12-19

codecov · 2022-12-20T15:15:46Z

Codecov Report

Merging #7003 (3804850) into main (4d41b6b) will increase coverage by 6.32%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##               main    #7003       +/-   ##
=============================================
+ Coverage      6.56%   12.89%    +6.32%     
- Complexity     8441     9019      +578     
=============================================
  Files          4361     2711     -1650     
  Lines        370925   268077   -102848     
  Branches      47614    45304     -2310     
=============================================
+ Hits          24360    34566    +10206     
+ Misses       343630   229046   -114584     
- Partials       2935     4465     +1530

Impacted Files	Coverage Δ
...va/com/cloud/upgrade/dao/DatabaseAccessObject.java	`0.00% <0.00%> (-97.92%)`	⬇️
...ain/java/com/cloud/upgrade/dao/DbUpgradeUtils.java	`0.00% <0.00%> (-93.75%)`	⬇️
...java/com/cloud/upgrade/DatabaseUpgradeChecker.java	`0.00% <0.00%> (-41.25%)`	⬇️
.../com/cloud/storage/dao/StoragePoolTagsDaoImpl.java	`12.90% <0.00%> (-31.70%)`	⬇️
...va/com/cloud/upgrade/DatabaseVersionHierarchy.java	`56.81% <0.00%> (-29.12%)`	⬇️
...rce/wrapper/LibvirtResizeVolumeCommandWrapper.java	`49.50% <0.00%> (-28.63%)`	⬇️
...n/java/com/cloud/upgrade/dao/Upgrade490to4910.java	`0.00% <0.00%> (-26.93%)`	⬇️
...in/java/com/cloud/upgrade/dao/LegacyDbUpgrade.java	`0.00% <0.00%> (-25.00%)`	⬇️
...in/java/com/cloud/upgrade/dao/Upgrade444to450.java	`0.00% <0.00%> (-25.00%)`	⬇️
...in/java/com/cloud/upgrade/dao/Upgrade453to460.java	`0.00% <0.00%> (-25.00%)`	⬇️
... and 3946 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

engine/schema/src/main/java/com/cloud/upgrade/DatabaseUpgradeChecker.java

github-actions · 2022-12-23T11:01:49Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

weizhouapache · 2022-12-23T21:23:50Z

@blueorangutan package

blueorangutan · 2022-12-23T21:24:04Z

@weizhouapache a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-12-23T22:11:54Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 5082

weizhouapache · 2022-12-23T22:21:20Z

@blueorangutan test matrix

blueorangutan · 2022-12-23T22:22:03Z

@weizhouapache a Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2022-12-25T03:58:28Z

Trillian test result (tid-58)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43288 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/FR68-pr7003-t58-kvm-centos7.zip
Smoke tests completed. 105 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_upgrade_kubernetes_cluster	`Failure`	461.89	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	`Failure`	600.84	test_kubernetes_clusters.py

blueorangutan · 2022-12-25T04:36:45Z

Trillian test result (tid-59)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server 7
Total time taken: 45344 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/FR68-pr7003-t59-vmware-70u3.zip
Smoke tests completed. 106 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2023-02-01T01:26:41Z

Trillian test result (tid-117)
Environment: kvm-rocky8 (x2), Advanced Networking with Mgmt server r8
Total time taken: 51161 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/FR68-pr7003-t117-kvm-rocky8.zip
Smoke tests completed. 107 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2023-02-01T02:20:40Z

Trillian test result (tid-118)
Environment: kvm-rocky8 (x2), Advanced Networking with Mgmt server r8
Total time taken: 52300 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/FR68-pr7003-t118-kvm-rocky8.zip
Smoke tests completed. 107 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2023-02-01T03:18:24Z

Trillian test result (tid-6044)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 50363 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7003-t6044-vmware-67u3.zip
Smoke tests completed. 106 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_02_list_cpvm_vm	`Failure`	0.04	test_ssvm.py
test_04_cpvm_internals	`Failure`	0.04	test_ssvm.py
test_06_stop_cpvm	`Failure`	0.05	test_ssvm.py
test_07_reboot_ssvm	`Failure`	111.41	test_ssvm.py
test_08_reboot_cpvm	`Failure`	0.04	test_ssvm.py
test_10_reboot_cpvm_forced	`Failure`	0.04	test_ssvm.py
test_12_destroy_cpvm	`Error`	2302.45	test_ssvm.py

rohityadavcloud

LGTM - didn't test it but the code and CLI changes looks good to me.

rohityadavcloud · 2023-02-01T08:45:56Z

@weizhouapache should this have any schema change in the 4.17.2.0->4.18.0.0 sql upgrade path?

weizhouapache · 2023-02-01T09:05:07Z

@weizhouapache should this have any schema change in the 4.17.2.0->4.18.0.0 sql upgrade path?

@rohityadavcloud
yes, just pushed a commit to resize passphrase column
732e05e

no other schema change is required.

weizhouapache · 2023-02-01T09:05:21Z

@blueorangutan package

blueorangutan · 2023-02-01T09:06:03Z

@weizhouapache a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

weizhouapache · 2023-02-01T09:17:38Z

@blueorangutan package

blueorangutan · 2023-02-01T09:18:04Z

@weizhouapache a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-02-01T10:06:20Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 5467

weizhouapache · 2023-02-01T10:08:54Z

@blueorangutan test rocky8 kvm-rocky8

blueorangutan · 2023-02-01T10:10:03Z

@weizhouapache a Trillian-Jenkins test job (rocky8 mgmt + kvm-rocky8) has been kicked to run smoke tests

blueorangutan · 2023-02-01T10:10:59Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 5468

sonarcloud · 2023-02-01T10:19:56Z

SonarCloud Quality Gate failed.

0 Bugs
0 Vulnerabilities
0 Security Hotspots
22 Code Smells

12.5% Coverage
9.9% Duplication

weizhouapache · 2023-02-01T10:54:05Z

@blueorangutan test matrix

blueorangutan · 2023-02-01T10:55:03Z

@weizhouapache a Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2023-02-01T22:45:00Z

Trillian test result (tid-6064)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41099 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7003-t6064-kvm-centos7.zip
Smoke tests completed. 107 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2023-02-01T22:48:48Z

Trillian test result (tid-6062)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41507 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7003-t6062-xenserver-71.zip
Smoke tests completed. 107 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2023-02-02T00:06:59Z

Trillian test result (tid-6063)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 46138 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7003-t6063-vmware-67u3.zip
Smoke tests completed. 107 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2023-02-02T01:58:55Z

Trillian test result (tid-6061)
Environment: kvm-rocky8 (x2), Advanced Networking with Mgmt server r8
Total time taken: 55014 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7003-t6061-kvm-rocky8.zip
Smoke tests completed. 102 look OK, 5 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestRouterDnsService>:setup	`Error`	0.00	test_router_dnsservice.py
ContextSuite context=TestRemoteDiagnostics>:setup	`Error`	0.00	test_diagnostics.py
ContextSuite context=TestPrivateGwACL>:setup	`Error`	0.00	test_privategw_acl.py
ContextSuite context=TestDomainsServiceOfferings>:setup	`Error`	3.61	test_domain_service_offerings.py
test_01_template_usage	`Error`	11.51	test_usage.py
test_01_volume_usage	`Error`	187.50	test_usage.py

utils,framework/db: Introduce new database encryption cipher based on…

1f05d1b

… AesGcmJce including all changes until 2022-12-19

boring-cyborg bot added component:build component:database component:management-server component:packaging component:storage component:usage-server distro:centos distro:ubuntu labels Dec 20, 2022

Update: fix code smells

534c711

weizhouapache closed this Dec 20, 2022

weizhouapache reopened this Dec 20, 2022

DaanHoogland reviewed Dec 21, 2022

View reviewed changes

engine/schema/src/main/java/com/cloud/upgrade/DatabaseUpgradeChecker.java Outdated Show resolved Hide resolved

weizhouapache added 5 commits December 21, 2022 14:36

Update: fix code smells (part2)

1a3aae8

Update: revert FOOTER of cloudstack-migrate-databases to use \n

f1975c6

Update: update help message of cloudstack-setup-databases

3858b25

Update: fix code smells (part3)

b3d4770

Update: make changes as per suggestions

52f43ca

github-actions bot added the status:has-conflicts label Dec 23, 2022

Merge remote-tracking branch 'apache/main' into 4.18-new-cipher-aead

4754b73

github-actions bot removed the status:has-conflicts label Dec 23, 2022

rohityadavcloud approved these changes Feb 1, 2023

View reviewed changes

Update 7003: modify "passphrase" column to varchar(255)

3804850

weizhouapache force-pushed the 4.18-new-cipher-aead branch from 732e05e to 3804850 Compare February 1, 2023 09:17

apache deleted a comment from blueorangutan Feb 2, 2023

DaanHoogland merged commit 62e342c into main Feb 2, 2023

weizhouapache mentioned this pull request Mar 2, 2023

Move PassphraseVO to use String instead of byte[] to support Encrypt annotation #7302

Merged

12 tasks

DaanHoogland deleted the 4.18-new-cipher-aead branch March 14, 2023 14:22

weizhouapache mentioned this pull request Feb 9, 2024

Cloudstack is not FIPS compliant #6246

Open

sureshanaparti mentioned this pull request Jul 17, 2024

Update doc with new database encryptor apache/cloudstack-documentation#414

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

utils,framework/db: Introduce new database encryption cipher based on AesGcmJce #7003

utils,framework/db: Introduce new database encryption cipher based on AesGcmJce #7003

weizhouapache commented Dec 20, 2022

codecov bot commented Dec 20, 2022 •

edited

Loading

github-actions bot commented Dec 23, 2022

weizhouapache commented Dec 23, 2022

blueorangutan commented Dec 23, 2022

blueorangutan commented Dec 23, 2022

weizhouapache commented Dec 23, 2022

blueorangutan commented Dec 23, 2022

blueorangutan commented Dec 25, 2022

blueorangutan commented Dec 25, 2022

blueorangutan commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

rohityadavcloud left a comment •

edited

Loading

rohityadavcloud commented Feb 1, 2023

weizhouapache commented Feb 1, 2023

weizhouapache commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

weizhouapache commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

weizhouapache commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

sonarcloud bot commented Feb 1, 2023

weizhouapache commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

blueorangutan commented Feb 2, 2023

blueorangutan commented Feb 2, 2023

utils,framework/db: Introduce new database encryption cipher based on AesGcmJce #7003

utils,framework/db: Introduce new database encryption cipher based on AesGcmJce #7003

Conversation

weizhouapache commented Dec 20, 2022

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Screenshots (if appropriate):

How Has This Been Tested?

codecov bot commented Dec 20, 2022 • edited Loading

Codecov Report

github-actions bot commented Dec 23, 2022

weizhouapache commented Dec 23, 2022

blueorangutan commented Dec 23, 2022

blueorangutan commented Dec 23, 2022

weizhouapache commented Dec 23, 2022

blueorangutan commented Dec 23, 2022

blueorangutan commented Dec 25, 2022

blueorangutan commented Dec 25, 2022

blueorangutan commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

rohityadavcloud left a comment • edited Loading

Choose a reason for hiding this comment

rohityadavcloud commented Feb 1, 2023

weizhouapache commented Feb 1, 2023

weizhouapache commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

weizhouapache commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

weizhouapache commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

sonarcloud bot commented Feb 1, 2023

weizhouapache commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

blueorangutan commented Feb 1, 2023

blueorangutan commented Feb 2, 2023

blueorangutan commented Feb 2, 2023

codecov bot commented Dec 20, 2022 •

edited

Loading

rohityadavcloud left a comment •

edited

Loading