[BACKPORT 2.18][PLAT-10529][PLAT-10530][PLAT-11176] Adds support for …

…CA having cert chain in YBA trust's store

Summary:
Currently YBA assumes that the CA certs added to YBA trust store will be a single root
cert.
With this diff we enable the support for cert chain as well.
This was observed in fidelity environment where our migration V274 failed for the same reason.

Some minor other improvements/fixes -
  - Fix the deletion of CA certs from YBA's trust store. In case the deletion fails in the first attempt the `certContent` that stores the filePath starts storing `certContent` which causes the subsequent deletion attempt to fail - This diff fixes it.

[PLAT-11176][PLAT-11170] Pass Java PKCS TrustStore for play.ws.ssl connections

This diff fixes two issues -
  - **PLAT-11176**: Previously, we were only passing YBA's PEM trust store from the custom CA trust store for `play.ws.ssl` TLS handshakes. Consequently, when we attempted to upload multiple CA certificates to YBA's trust store, it resulted in SSL handshake failures for the previously uploaded certificates. With this update, we have included YBA's Java trust store as well.

  - **PLAT-11170**: There was an issue with deletion of CA cert from YBA's trust store. Specifically, when we had uploaded one certificate chain and another certificate that only contained the root of the previously uploaded certificate chain, the deletion of the latter was failing. This issue has been resolved in this diff.

Depends on - D29985, D29143
Original Commit -
yugabyte@863ae72
yugabyte@4c8978b

Test Plan:
**Case1**
  - Ran the migration with the fidelity postgres dump.
  - Ensured that the certs are correctly importerd in both YBA's PKCS12/PEM trust store.

**Case2**
  - Deployed a keycloak server (OIDC server) - [[ https://10.23.16.17:8443 | https://10.23.16.17/ ]] that supports custom certs.
  - Created a cert chain certificates (root -> intermediate -> client).
  - Deployed the above server with client certificate.
  - Added the root/intermediate certs in YBA's trust store.
  - Ensured authentication is successful.
  - Deleted the certs from YBA trust store.
  - Now ensured SSO login is broken.
  - Uploaded partial, i.e., root only cert to YBA trust store.
  - Ensured that SSO login is broken.

**Case3**
 - Verified crud for the custom CA trust store.

**Case4**
 - Added a cert chain with root (r1) & intermediate (i1) -> (cert1)
 - Added another cert chain with root(r1) & intermediate (i2) -> (cert2)
 - Ensured our PEM store contains 3 entries now.
 - Removed cert1 from the trust store.
 - Verified that r1 & i2 are present in the YBA's PEM store.
 - Added back cert1 in trust store.
 - Replaced cert1 with some other cert chain -> (cert3) [root (r2) & intermediate i3]
 - Verified that PEM trust store contain now 4 certs -> [r1, i2, r2, i3].
 - For PKCS12 store, we add/remove/delete based on the alias (cert name). So we don't need any special handling for that.

**Case5**
  - Ensured that the migration V274 is idempotent, i.e, the directory created are cleared in case the migration fails, so that we remains in the same state from YBA's perspective.

iTest pipeline
UT's

CA trust store related iTests

**PLAT-11170**
  - Uploaded the root cert to YBA's trust store.
  - Created a certificate chain using the root certificate mentioned above and also uploaded it.
  - Verified that deletion of cert uploaded in yugabyte#1 was successful.

**PLAT-11176**
  - Created HA setup with two standup portals.
  - Each portal is using it's own custom CA certs.
  - Uploaded both the cert chains to YBA's trust store.
  - Verified that the backup is successful on both the standby setups configured.

Reviewers: #yba-api-review, nbhatia, cwang, amalyshev

Reviewed By: amalyshev

Subscribers: yugaware

Tags: #jenkins-ready

Differential Revision: https://phorge.dev.yugabyte.com/D30055

managed/src/main/java/com/yugabyte/yw/common/WSClientRefresher.java

@@ -79,6 +79,7 @@ private WSClient newClient(String ybWsConfigPath) {
     if (!ybaStoreConfig.isEmpty() && customCAStoreManager.isEnabled()) {
       // Add JRE default cert paths as well in this case.
       ybaStoreConfig.add(customCAStoreManager.getJavaDefaultConfig());
+      ybaStoreConfig.addAll(customCAStoreManager.getYBAJavaKeyStoreConfig());
 
       Config customWsConfig =
           ConfigFactory.empty()
@@ -89,7 +90,7 @@ private WSClient newClient(String ybWsConfigPath) {
       ybWsOverrides = customWsConfig.getValue("play.ws");
     }
 
-    log.info(
+    log.debug(
         "Creating ws client with config override: {}",
         ybWsOverrides.render(ConfigRenderOptions.concise()));
 

managed/src/main/java/com/yugabyte/yw/common/certmgmt/CertificateHelper.java

@@ -767,11 +767,16 @@ public static void writeCertBundleToCertPath(List<X509Certificate> certs, String
 
   public static void writeCertBundleToCertPath(
       List<X509Certificate> certs, String certPath, boolean syncToDB) {
+    writeCertBundleToCertPath(certs, certPath, syncToDB, false);
+  }
+
+  public static void writeCertBundleToCertPath(
+      List<X509Certificate> certs, String certPath, boolean syncToDB, boolean append) {
     File certFile = new File(certPath);
     // Create directory to store the certFile.
     certFile.getParentFile().mkdirs();
     log.info("Dumping certs at path: {}", certPath);
-    try (JcaPEMWriter certWriter = new JcaPEMWriter(new FileWriter(certFile))) {
+    try (JcaPEMWriter certWriter = new JcaPEMWriter(new FileWriter(certFile, append))) {
       for (X509Certificate cert : certs) {
         log.info(getCertificateProperties(cert));
         certWriter.writeObject(cert);

managed/src/main/java/com/yugabyte/yw/common/certmgmt/castore/CustomCAStoreManager.java

@@ -258,6 +258,7 @@ public boolean deleteCA(UUID customerId, UUID certId, String storagePath) {
 
     String trustStoreHome = getTruststoreHome(storagePath);
     String certPath = getCustomCACertsPath(trustStoreHome, certId);
+
     boolean deleted = false;
     boolean suppressErrors = false;
     char[] truststorePassword = getTruststorePassword();
@@ -275,6 +276,9 @@ public boolean deleteCA(UUID customerId, UUID certId, String storagePath) {
     } catch (Exception e) {
       log.error("CA certificate delete is incomplete due to: ", e);
       // Rollback DB.
+      CustomCaCertificateInfo origCert = CustomCaCertificateInfo.get(customerId, certId, false);
+      // We need to ensure paths is not messed up in the custom cert table.
+      cert.setContents(origCert.getContents());
       cert.activate();
       try {
         suppressErrors = true;
@@ -340,6 +344,27 @@ public List<Map<String, String>> getPemStoreConfig() {
 
   // -------------- PKCS12 CA trust-store specific methods ------------
 
+  public List<Map<String, String>> getYBAJavaKeyStoreConfig() {
+    String storagePath = AppConfigHelper.getStoragePath();
+    String trustStoreHome = getTruststoreHome(storagePath);
+    List ybaJavaKeyStoreConfig = new ArrayList<>();
+
+    if (Files.exists(Paths.get(trustStoreHome))) {
+      String javaTrustStorePathStr = pkcs12TrustStoreManager.getYbaTrustStorePath(trustStoreHome);
+      Path javaTrustStorePath = Paths.get(javaTrustStorePathStr);
+      if (Files.exists(javaTrustStorePath)) {
+        Map<String, String> trustStoreMap = new HashMap<>();
+        trustStoreMap.put("path", javaTrustStorePathStr);
+        trustStoreMap.put("type", pkcs12TrustStoreManager.getYbaTrustStoreType());
+        trustStoreMap.put("password", new String(getTruststorePassword()));
+        ybaJavaKeyStoreConfig.add(trustStoreMap);
+      }
+    }
+
+    log.debug("YBA's custom java trust store config is {}", ybaJavaKeyStoreConfig);
+    return ybaJavaKeyStoreConfig;
+  }
+
   private KeyStore getYbaKeyStore() {
     String storagePath = AppConfigHelper.getStoragePath();
     String trustStoreHome = getTruststoreHome(storagePath);

managed/src/main/java/com/yugabyte/yw/common/certmgmt/castore/PemTrustStoreManager.java

@@ -9,6 +9,7 @@
 import com.google.inject.Singleton;
 import com.yugabyte.yw.common.PlatformServiceException;
 import com.yugabyte.yw.common.certmgmt.CertificateHelper;
+import com.yugabyte.yw.models.CustomCaCertificateInfo;
 import com.yugabyte.yw.models.FileData;
 import java.io.File;
 import java.io.FileInputStream;
@@ -24,8 +25,11 @@
 import java.security.cert.X509Certificate;
 import java.util.ArrayList;
 import java.util.Collections;
+import java.util.Iterator;
 import java.util.List;
+import java.util.stream.Collectors;
 import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.collections.CollectionUtils;
 import org.bouncycastle.openssl.jcajce.JcaPEMWriter;
 
 /** Relates to YBA's PEM trust store */
@@ -44,9 +48,9 @@ public boolean addCertificate(
       throws KeyStoreException, CertificateException, IOException, PlatformServiceException {
     log.debug("Trying to update YBA's PEM truststore ...");
 
-    Certificate newCert = null;
-    newCert = getX509Certificate(certPath);
-    if (newCert == null) {
+    List<Certificate> newCerts = null;
+    newCerts = getX509Certificate(certPath);
+    if (CollectionUtils.isEmpty(newCerts)) {
       throw new PlatformServiceException(
           BAD_REQUEST, String.format("No new CA certificate exists at %s", certPath));
     }
@@ -60,21 +64,35 @@ public boolean addCertificate(
       log.debug("Created an empty YBA PEM trust-store");
     } else {
       List<Certificate> trustCerts = getCertsInTrustStore(trustStorePath, trustStorePassword);
+      List<Certificate> addedCertChain = new ArrayList<Certificate>(newCerts);
       if (trustCerts != null) {
         // Check if such an alias already exists.
-        boolean exists = trustCerts.contains(newCert);
-        if (exists && !suppressErrors) {
-          String msg = "CA certificate with same content already exists";
-          log.error(msg);
-          throw new PlatformServiceException(BAD_REQUEST, msg);
+        for (int i = 0; i < addedCertChain.size(); i++) {
+          Certificate newCert = addedCertChain.get(i);
+          boolean exists = trustCerts.contains(newCert);
+          if (!exists) {
+            break;
+          }
+          // In case of certificate chain, we can have the same root/intermediate
+          // cert present in the chain. Throw error in case all of these exists.
+          if (exists && !suppressErrors && addedCertChain.size() - 1 == i) {
+            String msg = "CA certificate with same content already exists";
+            log.error(msg);
+            throw new PlatformServiceException(BAD_REQUEST, msg);
+          } else if (exists && addedCertChain.size() != i) {
+            newCerts.remove(i);
+          }
         }
       }
     }
 
     // Update the trust store in file system.
-    boolean append = true;
-    CertificateHelper.writeCertFileContentToCertPath(
-        (X509Certificate) newCert, trustStorePath, false, append);
+    List<X509Certificate> x509Certificates =
+        newCerts.stream()
+            .filter(certificate -> certificate instanceof X509Certificate)
+            .map(certificate -> (X509Certificate) certificate)
+            .collect(Collectors.toList());
+    CertificateHelper.writeCertBundleToCertPath(x509Certificates, trustStorePath, false, true);
     log.debug("Truststore '{}' now has the cert {}", trustStorePath, certAlias);
 
     // Backup up YBA's PEM trust store in DB.
@@ -99,18 +117,25 @@ public void replaceCertificate(
     List<Certificate> trustCerts = getCertificates(trustStorePath);
 
     // Check if such a cert already exists.
-    Certificate oldCert = getX509Certificate(oldCertPath);
-    boolean exists = trustCerts.remove(oldCert);
-    if (!exists && !suppressErrors) {
-      String msg = String.format("Certificate '%s' doesn't exist to update", certAlias);
-      log.error(msg);
-      throw new PlatformServiceException(BAD_REQUEST, msg);
+    List<Certificate> oldCerts = getX509Certificate(oldCertPath);
+    boolean exists = false;
+    for (Certificate oldCert : oldCerts) {
+      if (isCertificateUsedInOtherChain(oldCert)) {
+        log.debug("Certificate {} is part of a chain, skipping replacement.", certAlias);
+        continue;
+      }
+      exists = trustCerts.remove(oldCert);
+      if (!exists && !suppressErrors) {
+        String msg = String.format("Certificate '%s' doesn't exist to update", certAlias);
+        log.error(msg);
+        throw new PlatformServiceException(BAD_REQUEST, msg);
+      }
     }
 
     // Update the trust store.
     if (exists) {
-      Certificate newCert = getX509Certificate(newCertPath);
-      trustCerts.add(newCert);
+      List<Certificate> newCerts = getX509Certificate(newCertPath);
+      trustCerts.addAll(newCerts);
       saveTo(trustStorePath, trustCerts);
       log.info("Truststore '{}' updated with new cert at alias '{}'", trustStorePath, certAlias);
     }
@@ -144,11 +169,30 @@ public void remove(
     log.info("Removing cert {} from PEM truststore ...", certAlias);
     String trustStorePath = getTrustStorePath(trustStoreHome, TRUSTSTORE_FILE_NAME);
     List<Certificate> trustCerts = getCertificates(trustStorePath);
-
     // Check if such an alias already exists.
-    Certificate certToRemove = getX509Certificate(certPath);
-    boolean exists =
-        Iterators.removeAll(trustCerts.iterator(), Collections.singletonList(certToRemove));
+    List<Certificate> certToRemove = getX509Certificate(certPath);
+    int certToRemoveCount = certToRemove.size();
+    // Iterate through each certificate in certToRemove and check if it's used in any chain
+    boolean exists = false;
+    Iterator<Certificate> certIterator = certToRemove.iterator();
+    while (certIterator.hasNext()) {
+      Certificate cert = certIterator.next();
+      if (isCertificateUsedInOtherChain(cert)) {
+        // Certificate is part of a chain, do not remove it
+        log.debug("Certificate {} is part of a chain, skipping removal.", certAlias);
+        certToRemoveCount -= 1;
+        certIterator.remove();
+      } else {
+        // Certificate is not part of a chain
+        exists = true;
+      }
+    }
+
+    if (certToRemoveCount == 0) {
+      log.debug(
+          "Skipping removal of cert from PEM truststore, as the cert is part of other trust chain");
+      return;
+    }
 
     if (!exists && !suppressErrors) {
       String msg = String.format("Certificate '%s' does not exist to delete", certAlias);
@@ -157,14 +201,15 @@ public void remove(
     }
 
     // Delete from the trust-store.
-    if (exists) {
+    if (!certToRemove.isEmpty()) {
+      Iterators.removeAll(trustCerts.iterator(), certToRemove);
       saveTo(trustStorePath, trustCerts);
       log.debug("Certificate {} is now deleted from trust-store {}", certAlias, trustStorePath);
     }
     log.info("custom CA certs deleted from YBA's PEM truststore");
   }
 
-  private List<Certificate> getCertsInTrustStore(String trustStorePath, char[] trustStorePassword) {
+  public List<Certificate> getCertsInTrustStore(String trustStorePath, char[] trustStorePassword) {
     if (trustStorePath == null) {
       throw new PlatformServiceException(
           INTERNAL_SERVER_ERROR, "Cannot get CA certificates from empty path");
@@ -223,4 +268,21 @@ public boolean isTrustStoreEmpty(String storePathStr, char[] trustStorePassword)
       throw new PlatformServiceException(INTERNAL_SERVER_ERROR, msg);
     }
   }
+
+  private boolean isCertificateUsedInOtherChain(Certificate cert)
+      throws CertificateException, IOException {
+    // Retrieve all the certificates from `custom_ca_certificate_info` schema.
+    // In case the passed cert is substring in more than 1 cert than it is used
+    // as part of other cert chain as well.
+    // We will skip removing it from the trust-store.
+    int certChainCount = 0;
+    List<CustomCaCertificateInfo> customCACertificates = CustomCaCertificateInfo.getAll(false);
+    for (CustomCaCertificateInfo customCA : customCACertificates) {
+      List<Certificate> certChain = getX509Certificate(customCA.getContents());
+      if (certChain.contains(cert)) {
+        certChainCount++;
+      }
+    }
+    return certChainCount > 1;
+  }
 }

managed/src/main/java/com/yugabyte/yw/common/certmgmt/castore/Pkcs12TrustStoreManager.java

@@ -81,8 +81,11 @@ public boolean addCertificate(
       throw new PlatformServiceException(BAD_REQUEST, errMsg);
     }
 
-    Certificate certificate = getX509Certificate(certPath);
-    trustStore.setCertificateEntry(certAlias, certificate);
+    List<Certificate> certificates = getX509Certificate(certPath);
+    for (int i = 0; i < certificates.size(); i++) {
+      String alias = certAlias + "-" + i;
+      trustStore.setCertificateEntry(alias, certificates.get(i));
+    }
     // Update the trust store in file-system.
     saveTrustStore(trustStorePath, trustStore, trustStorePassword);
     log.debug("Truststore '{}' now has a certificate with alias '{}'", trustStorePath, certAlias);
@@ -113,17 +116,31 @@ public void replaceCertificate(
       throw new PlatformServiceException(INTERNAL_SERVER_ERROR, errMsg);
     }
 
-    // Check if such an alias already exists.
-    if (!trustStore.containsAlias(certAlias) && !suppressErrors) {
-      String errMsg = String.format("Cert by name '%s' does not exist to update", certAlias);
-      log.error(errMsg);
-      // Purge newCertPath which got created.
-      throw new PlatformServiceException(BAD_REQUEST, errMsg);
+    List<Certificate> oldCertificates = getX509Certificate(oldCertPath);
+    List<Certificate> newCertificates = getX509Certificate(newCertPath);
+    for (int i = 0; i < oldCertificates.size(); i++) {
+      // Check if such an alias already exists.
+      String alias = certAlias + "-" + i;
+      if (!trustStore.containsAlias(alias) && !suppressErrors) {
+        String errMsg = String.format("Cert by name '%s' does not exist to update", alias);
+        log.error(errMsg);
+        // Purge newCertPath which got created.
+        throw new PlatformServiceException(BAD_REQUEST, errMsg);
+      }
+    }
+
+    if (newCertificates.size() < oldCertificates.size()) {
+      for (int i = newCertificates.size(); i < oldCertificates.size(); i++) {
+        String alias = certAlias + "-" + i;
+        trustStore.deleteEntry(alias);
+      }
     }
 
     // Update the trust store.
-    Certificate newCertificate = getX509Certificate(newCertPath);
-    trustStore.setCertificateEntry(certAlias, newCertificate);
+    for (int i = 0; i < newCertificates.size(); i++) {
+      String alias = certAlias + "-" + i;
+      trustStore.setCertificateEntry(alias, newCertificates.get(i));
+    }
     saveTrustStore(trustStorePath, trustStore, trustStorePassword);
 
     // Backup up YBA's pkcs12 trust store in DB.
@@ -183,22 +200,26 @@ public void remove(
       String trustStoreHome,
       char[] trustStorePassword,
       boolean suppressErrors)
-      throws KeyStoreException {
+      throws KeyStoreException, IOException, CertificateException {
     log.info("Removing cert {} from YBA's pkcs12 truststore ...", certAlias);
 
     String trustStorePath = getTrustStorePath(trustStoreHome, TRUSTSTORE_FILE_NAME);
     KeyStore trustStore = getTrustStore(trustStorePath, trustStorePassword, false);
+    List<Certificate> certificates = getX509Certificate(certPath);
+    for (int i = 0; i < certificates.size(); i++) {
+      String alias = certAlias + "-" + i;
+
+      // Check if such an alias already exists.
+      if (!trustStore.containsAlias(alias) && !suppressErrors) {
+        String errMsg = String.format("CA certificate '%s' does not exist to delete", alias);
+        log.error(errMsg);
+        throw new PlatformServiceException(BAD_REQUEST, errMsg);
+      }
 
-    // Check if such an alias already exists.
-    if (!trustStore.containsAlias(certAlias) && !suppressErrors) {
-      String errMsg = String.format("CA certificate '%s' does not exist to delete", certAlias);
-      log.error(errMsg);
-      throw new PlatformServiceException(BAD_REQUEST, errMsg);
-    }
-
-    // Delete from the trust store.
-    if (trustStore.containsAlias(certAlias)) {
-      trustStore.deleteEntry(certAlias);
+      // Delete from the trust store.
+      if (trustStore.containsAlias(alias)) {
+        trustStore.deleteEntry(alias);
+      }
     }
     saveTrustStore(trustStorePath, trustStore, trustStorePassword);
     log.debug("Truststore '{}' now does not have a CA certificate '{}'", trustStorePath, certAlias);

managed/src/main/java/com/yugabyte/yw/common/certmgmt/castore/TrustStoreManager.java

@@ -9,17 +9,25 @@
 import java.security.cert.Certificate;
 import java.security.cert.CertificateException;
 import java.security.cert.CertificateFactory;
+import java.util.ArrayList;
+import java.util.List;
 
 public interface TrustStoreManager {
   default String getTrustStorePath(String trustStoreHome, String trustStoreFileName) {
     return String.format("%s/%s", trustStoreHome, trustStoreFileName);
   }
 
-  default Certificate getX509Certificate(String certPath) throws CertificateException, IOException {
+  default List<Certificate> getX509Certificate(String certPath)
+      throws CertificateException, IOException {
+    List<Certificate> certificates = new ArrayList<>();
     try (FileInputStream certStream = new FileInputStream(certPath)) {
       CertificateFactory certificateFactory = CertificateFactory.getInstance("X.509");
-      Certificate certificate = certificateFactory.generateCertificate(certStream);
-      return certificate;
+
+      while (certStream.available() > 0) {
+        Certificate certificate = certificateFactory.generateCertificate(certStream);
+        certificates.add(certificate);
+      }
+      return certificates;
     }
   }