[Cloud Security] Backport 1.8.1

[opauloh]

Summary

This PR adds a backport version to cloud_security_posture package 1.8.*, it includes the following changes:

• Fixed Pipeline Ingest Error due to missing cluster_id used in the copy_from;
  Reasoning: Pipeline error were preventing the following pipelines from being executed + generating unnecessary data

• Sets placeholder value for azure account.id when it is not available;

• Sets placeholder value for gcp account.id when it is not available;

Reasoning: GCP and Azure integrations can push data belonging to a CSPM Organization where the cloud.account.id field can be missing, but The Cloud Security features in Kibana 8.13 require the field cloud.account.id to be always available in the Cloud Security Posture indices.

• Removes cloud.account.name when it is empty;

Reasoning: GCP integration can push data belonging to a CSPM Organization where the cloud.account.name is empty when it should be missing.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• No more gaps in Kibana when filtering cloud.provider : gcp and Grouping by Cloud account

• No records with empty cloud.account.name

• No records with missing or empty cloud.account.id or

How to test this PR locally

Follow the guide describe here to test this PR locally.

To collect the necessary data you can execute the following Reindex command in the dev tools (The pipeline is the important part!), point host and credentials to one of the long-running environments

POST _reindex?wait_for_completion=true
{
  "conflicts": "proceed", 
  "source": {
     "remote": {
      "host": "${ES_REMOTE_HOST}",
      "username": "${ES_REMOTE_USER}",
      "password": "${ES_REMOTE_PASS}"
    },
    "index": "logs-cloud_security_posture.findings_latest-default",
    "query": {
      "match_all": {}  
    }
  },
  "dest": {
    "op_type": "create",
    "index": "logs-cloud_security_posture.findings_latest-default",
    "pipeline": "logs-cloud_security_posture.findings-1.8.1"
  }
}

Related issues

• It fixes [Cloud Security] [Bug] Grouping row is hidden in the Findings page when the grouped key is an empty string kibana#178904
• It fixes https://github.com/elastic/security-team/issues/8874

Screenshots

Pipeline

[
  {
    "set": {
      "field": "ecs.version",
      "value": "8.6.0"
    }
  },
  {
    "set": {
      "field": "rule.benchmark.posture_type",
      "value": "kspm",
      "description": "Backward compatibility cloudbeat version < 8.7",
      "if": "ctx.rule?.benchmark?.posture_type == null"
    }
  },
  {
    "set": {
      "field": "orchestrator.cluster.id",
      "copy_from": "cluster_id",
      "description": "Backward compatibility cloudbeat version < 8.8",
      "if": "ctx.cluster_id != null && ctx.orchestrator?.cluster?.id == null"
    }
  },
  {
    "set": {
      "field": "cloud.account.id",
      "value": "azure",
      "description": "Sets placeholder value for azure account id when it is not available",
      "if": "(ctx.cloud?.account?.id == null || ctx.cloud?.account?.id == '') && ctx.cloud?.provider == 'azure'"
    }
  },
  {
    "set": {
      "field": "cloud.account.id",
      "value": "gcp",
      "description": "Sets placeholder value for gcp account id when it is not available",
      "if": "(ctx.cloud?.account?.id == null || ctx.cloud?.account?.id == '' ) && ctx.cloud?.provider == 'gcp'"
    }
  },
  {
    "remove": {
      "field": "cloud.account.name",
      "ignore_missing": true,
      "description": "Removes cloud.account.name when it is empty",
      "if": "ctx.cloud?.account?.name == ''"
    }
  },
  {
    "pipeline": {
      "name": "global@custom",
      "ignore_missing_pipeline": true,
      "description": "[Fleet] Global pipeline for all data streams"
    }
  },
  {
    "pipeline": {
      "name": "logs@custom",
      "ignore_missing_pipeline": true,
      "description": "[Fleet] Pipeline for all data streams of type `logs`"
    }
  },
  {
    "pipeline": {
      "name": "logs-cloud_security_posture.integration@custom",
      "ignore_missing_pipeline": true,
      "description": "[Fleet] Pipeline for all data streams of type `logs` defined by the `cloud_security_posture` integration"
    }
  },
  {
    "pipeline": {
      "name": "logs-cloud_security_posture.findings@custom",
      "ignore_missing_pipeline": true,
      "description": "[Fleet] Pipeline for the `cloud_security_posture.findings` dataset"
    }
  }
]

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 4429dde

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[kfirpeled]

LGTM

[maxcold]

You don't mention AWS here and in the description, can you remind me why the problem does not apply to AWS documents?

[opauloh]

For CSPM AWS, Benchmark Rules at the organization level that don't have an account associated with it are going to fallback to the Management Account. And as a management account is a pre-requirement to create an organization this means our native CSPM AWS will always have a cloud.account.id .

Pinging @oren-zohar to confirm that

[maxcold]

This quite a deep context one need to know to understand why we don't have this check for AWS. if we rely heavily on this field, I'd add this check for aws anyway, even if we are sure now that it can't be missing

[maxcold]

As this value goes directly to the UX, does it make sense to have something more elaborate as a value? Eg. smth like global-azure-findings or non-organisation-specific-findings. The values I came up with are just for example, maybe it worth consulting with the product

[opauloh]

I agree, however, as we are short in time as public release targets for tomorrow I am merging this patch as we would not have the time to elaborate on that properly, but we can discuss that async and patch another integration version if needed.

cc @kfirpeled

[elasticmachine]

Package cloud_security_posture - 1.8.1 containing this change is available at https://epr.elastic.co/search?package=cloud_security_posture