add GCP credentials deployment #2118

orouz · 2024-04-10T16:43:15Z

adds a Deployment Manager template to deploy a new service account and key
adds a CI test to verify deployment of service account is useable
this is how a successful deployment looks from a user POV after they ran the service account deployment script in a GCP shell:
tested we get findings using the generated key by providing it as a JSON blob credentials option on the manual deployment option:
closes [GCP] Automated credentials creation flow for agentless #2005

mergify · 2024-04-10T16:44:05Z

This pull request does not have a backport label. Could you fix it @orouz? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit
NOTE: backport-skip has been added to this pull request.

github-actions · 2024-04-10T17:14:49Z

📊 Allure Report - 💚 No failures were reported.

Result	Count
🟥 Failed	0
🟩 Passed	359
⬜ Skipped	33

mergify · 2024-05-12T10:14:33Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b gcp_agentless_creds upstream/gcp_agentless_creds
git merge upstream/main
git push upstream gcp_agentless_creds

orouz · 2024-05-12T10:19:05Z

.github/workflows/test-gcp-dm.yml

+  # Test a GCP Deployment Manager deployment using Application Default Credentials
+  gcp_dm_adc:
+    name: CSPM GCP with ADC


this job tests the code path that uses GCP's application default credentials

cloudbeat/internal/resources/providers/gcplib/auth/credentials.go

Line 59 in e8309ef

return p.getApplicationDefaultCredentials(ctx, cfg, log)

orouz · 2024-05-12T10:19:35Z

.github/workflows/test-gcp-dm.yml

+  # Test a GCP Deployment Manager deployment using a Service Account
+  gcp_dm_sa:


this job tests the code path that uses a service account json

cloudbeat/internal/resources/providers/gcplib/auth/credentials.go

Line 63 in e8309ef

return p.getCustomCredentials(ctx, cfg, log)

orouz · 2024-05-12T10:22:25Z

.github/workflows/test-gcp-dm.yml

there is some repetitiveness in this workflow as the two jobs do mostly the same thing. we could abstract some of the steps to an action (like actions/init_env, actions/check_findings, actions/deploy_ec)

orouz · 2024-05-12T10:26:34Z

.github/workflows/test-gcp-dm.yml

+        run: |
+          # Deploys a GCP Service Account
+          cd ${{ env.DEPLOYMENT_MANAGER_DIR }}
+          export DEPLOYMENT_NAME="${{ env.GCP_SA_DEPLOYMENT_NAME }}"
+          export SERVICE_ACCOUNT_NAME="${{ env.GCP_SA_DEPLOYMENT_NAME }}-sa"
+          ./deploy_service_account.sh
+          mv KEY_FILE.json ../../${{ env.INTEGRATIONS_SETUP_DIR }}
+
+          # Installs CSPM GCP integration
+          cd ../../${{ env.INTEGRATIONS_SETUP_DIR }}
+          export SERVICE_ACCOUNT_JSON_PATH="KEY_FILE.json"
+          export DEPLOYMENT_NAME="${{ env.GCP_AGENT_DEPLOYMENT_NAME }}"
+          poetry install
+          poetry run python ./install_cspm_gcp_integration.py
+
+          # Deploys the agent using an existing service account (SERVICE_ACCOUNT_NAME)
+          cd ../../${{ env.DEPLOYMENT_MANAGER_DIR }}
+          . ./set_env.sh && ./deploy.sh


there's some trickery going on here:

we first deploy a service account using the new deployment manager template, which generates a KEY_FILE.json with the said service account

we then install a CSPM GCP integration that uses gcp.credentials.json variable to set the JSON content

then we deploy the agent, but because we've defined a service account name, it won't deploy a service account and instead use the one we deployed earlier

this is all combined into a single step because otherwise we won't be able to share the JSON file

orouz · 2024-05-12T10:27:23Z

deploy/deployment-manager/compute_engine.py

@@ -52,7 +51,7 @@ def generate_config(context):
            ),
            "serviceAccounts": [
                {
-                    "email": f"$(ref.{sa_name}.email)",
+                    "email": get_service_account_email(sa_name, project),


we can't use ref because we might be using an already-deployed service account

orouz · 2024-05-12T10:28:08Z

deploy/deployment-manager/deploy_service_account.sh

this is mostly copied from the original ./deploy.sh script

why not create a common.sh file?

orouz · 2024-05-12T10:29:21Z

deploy/deployment-manager/service_account.py

+    # pylint: disable=duplicate-code
+    service_account = {


it'd be nice to define a ./common.py file and import it to both deployment templates. not sure why that didn't work tbh

I didn't manage to do it as well.
It's because we're using deployment manager templates and other py files can not be imported to the file.

uri-weisman · 2024-05-12T11:35:25Z

deploy/deployment-manager/deploy_service_account.sh

why not create a common.sh file?

uri-weisman · 2024-05-12T11:39:06Z

deploy/deployment-manager/service_account.py

+    # pylint: disable=duplicate-code
+    service_account = {


I didn't manage to do it as well.
It's because we're using deployment manager templates and other py files can not be imported to the file.

uri-weisman · 2024-05-12T11:55:03Z

deploy/deployment-manager/deploy.sh

@@ -115,7 +116,7 @@ fi
 # Apply the deployment manager templates
 run_command "gcloud deployment-manager deployments create --automatic-rollback-on-error ${DEPLOYMENT_NAME} --project ${PROJECT_NAME} \
    --template compute_engine.py \
-    --properties elasticAgentVersion:${STACK_VERSION},fleetUrl:${FLEET_URL},enrollmentToken:${ENROLLMENT_TOKEN},allowSSH:${ALLOW_SSH},zone:${ZONE},elasticArtifactServer:${ELASTIC_ARTIFACT_SERVER},scope:${SCOPE},parentId:${PARENT_ID}"
+    --properties elasticAgentVersion:${STACK_VERSION},fleetUrl:${FLEET_URL},enrollmentToken:${ENROLLMENT_TOKEN},allowSSH:${ALLOW_SSH},zone:${ZONE},elasticArtifactServer:${ELASTIC_ARTIFACT_SERVER},scope:${SCOPE},parentId:${PARENT_ID},serviceAccountName:${SERVICE_ACCOUNT_NAME}"



Will the SERVICE_ACCOUNT_NAME param be propagated from the UI and if not we'll create one?
In case the user provides a wrong SA name, the deployment will not succeed and an error will be printed to the user.
Are we ok with it? do we want to add validation that the SA exists and if not to create one?

Will the SERVICE_ACCOUNT_NAME param be propagated from the UI and if not we'll create one?

no, SERVICE_ACCOUNT_NAME is not user-facing.

the addition of SERVICE_ACCOUNT_NAME to the deploy.sh script is done purely for testing purposes of deploying an agent with an existing service account. it simulates the new agentless flow, where a user will deploy a service account and we'd use its key in the integration credentials input.

to test the deployment of a service account is working properly, i've used the existing agent deployment script to make it so that it can operate either by deploying the full resources: VM + SA, or just a VM, while using an existing SA, which is currently passed only during tests.

In case the user provides a wrong SA name, the deployment will not succeed and an error will be printed to the user. Are we ok with it? do we want to add validation that the SA exists and if not to create one?

as mentioned, SERVICE_ACCOUNT_NAME is only used in tests, so we don't want to deploy a new service account if SERVICE_ACCOUNT_NAME doesn't exists. if for some reason SERVICE_ACCOUNT_NAME is provided but doesn't exists, failing the deployment would be the right outcome.

gurevichdmitry · 2024-05-12T12:49:48Z

tests/integrations_setup/install_cspm_gcp_integration.py

+def read_json_file(file_path):
+    """Reads a json file and returns its content"""
+    try:
+        with open(file_path, "r") as json_file:
+            return json_file.read()
+    except FileNotFoundError:
+        logger.error(f"Error: File '{file_path}' not found.")
+        sys.exit(1)
+    except IOError as e:
+        logger.error(f"Error reading file '{file_path}': {e}")
+        sys.exit(1)
+


@orouz, I think there's a bit of confusion here. The function read_json_file() and the variable json_file might suggest that they specifically handle JSON files, but they're actually generic and can read any type of file. For reading JSON files, we have a separate function defined here.

@gurevichdmitry i would prefer using read_json but it returns an empty dict when it fails (also missing decode exception) and i don't want to check for an empty dict in my use case and then exit, as it's kinda dirty. i'd prefer updating read_json to remove the try and let consumers do it and act on errors. in my case, i would sys.exit(1) if reading the file failed. can we do the same in load_data() ?

@orouz considering that read_json is only used in load_data, we can modify read_json to exit on error. This approach fulfills your requirement, and the reuse will be valid in your case.

uri-weisman

Approving deployment manager files

gurevichdmitry

@orouz Workflow and tests are approved. One more question: did you test the environment destruction when the service account is used?

orouz · 2024-05-19T08:08:39Z

@orouz Workflow and tests are approved. One more question: did you test the environment destruction when the service account is used?

yes, deletion of GCP env works

orouz force-pushed the gcp_agentless_creds branch from 04c53f0 to a02f0f8 Compare April 10, 2024 16:43

mergify bot assigned orouz Apr 10, 2024

mergify bot added the backport-skip label Apr 10, 2024

mergify bot mentioned this pull request Apr 10, 2024

add GCP credentials deployment #2097

Closed

1 task

orouz force-pushed the gcp_agentless_creds branch from a02f0f8 to 8febe94 Compare April 11, 2024 12:41

orouz changed the title ~~[test] add GCP credentials deployment~~ add GCP credentials deployment Apr 15, 2024

Omolola-Akinleye mentioned this pull request May 6, 2024

[Cloud Security] Automated creation of cloud credentials elastic/kibana#175977

Open

2 tasks

orouz force-pushed the gcp_agentless_creds branch 3 times, most recently from 773ec4b to 3a94c87 Compare May 12, 2024 09:12

orouz force-pushed the gcp_agentless_creds branch from 3a94c87 to e8309ef Compare May 12, 2024 10:16

orouz commented May 12, 2024

View reviewed changes

orouz marked this pull request as ready for review May 12, 2024 10:32

orouz requested a review from a team as a code owner May 12, 2024 10:32

orouz requested a review from uri-weisman May 12, 2024 10:32

uri-weisman reviewed May 12, 2024

View reviewed changes

gurevichdmitry reviewed May 12, 2024

View reviewed changes

uri-weisman approved these changes May 15, 2024

View reviewed changes

orouz force-pushed the gcp_agentless_creds branch from 99f9a77 to 551662d Compare May 16, 2024 12:19

orouz added 8 commits May 19, 2024 07:28

add ci test for deployment using SA

956e175

remove unused

b1e960c

cleaning

f9bdafe

review fixes

447b97a

minor fixes

668e136

use Path

167f7e7

revert json fn

d10862a

re-use read_json fn

d630861

orouz added 3 commits May 19, 2024 07:28

fix import

0354814

use json str

10aabe9

fix pylint import

cef8f43

orouz force-pushed the gcp_agentless_creds branch from a3e7ea5 to cef8f43 Compare May 19, 2024 04:28

add new paths

254546a

gurevichdmitry approved these changes May 19, 2024

View reviewed changes

orouz merged commit a2be352 into main May 19, 2024
26 checks passed

orouz deleted the gcp_agentless_creds branch May 19, 2024 09:01

orouz mentioned this pull request Jun 4, 2024

Test Credentials JSON authentication method #2127

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add GCP credentials deployment #2118

add GCP credentials deployment #2118

orouz commented Apr 10, 2024 •

edited

mergify bot commented Apr 10, 2024

github-actions bot commented Apr 10, 2024 •

edited

mergify bot commented May 12, 2024

orouz May 12, 2024

orouz May 12, 2024

orouz May 12, 2024

orouz May 12, 2024

orouz May 12, 2024

orouz May 12, 2024

uri-weisman May 12, 2024

orouz May 12, 2024

uri-weisman May 12, 2024

uri-weisman May 12, 2024

uri-weisman May 12, 2024

uri-weisman May 12, 2024

orouz May 13, 2024

gurevichdmitry May 12, 2024

orouz May 13, 2024

gurevichdmitry May 15, 2024

uri-weisman left a comment

gurevichdmitry left a comment

orouz commented May 19, 2024

		# Test a GCP Deployment Manager deployment using a Service Account
		gcp_dm_sa:

add GCP credentials deployment #2118

add GCP credentials deployment #2118

Conversation

orouz commented Apr 10, 2024 • edited

mergify bot commented Apr 10, 2024

github-actions bot commented Apr 10, 2024 • edited

📊 Allure Report - 💚 No failures were reported.

mergify bot commented May 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uri-weisman left a comment

Choose a reason for hiding this comment

gurevichdmitry left a comment

Choose a reason for hiding this comment

orouz commented May 19, 2024

orouz commented Apr 10, 2024 •

edited

github-actions bot commented Apr 10, 2024 •

edited