[Carbon Black Cloud] - Introduced data stream specific SQS queues

[ShourieG]

Type of change

• Bug
• Enhancement

Proposed commit message

Introduced data stream specific SQS queues to address the issue of data loss while using a single SQS queue for all data streams and then filtering out data using file selectors. The filtering using file selectors in a particular data stream caused the data for other data streams to get dropped by the SQS queue as described in this issue: #9289. This PR introduces an optional SQS queue per data stream while still keeping support for the original global SQS queue. The data stream specific queues will always override the global queues. The default file selector regexes have been commented out. This approach was adopted instead of just outright removing the older queues and file selectors was to maintain compatibility with the previous versions and not cause a breaking change. The updated setup instructions and update warnings have been documented to make the end user aware of the new functionality.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes [carbon_black_cloud] Data loss when using AWS SQS mode #9289

Test Results

--- Test results for package: carbon_black_cloud - START ---
╭────────────────────┬─────────────────────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE            │ DATA STREAM                 │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├────────────────────┼─────────────────────────────┼───────────┼───────────┼────────┼───────────────┤
│ carbon_black_cloud │ alert                       │ system    │ default   │ PASS   │ 48.086184125s │
│ carbon_black_cloud │ asset_vulnerability_summary │ system    │ default   │ PASS   │    30.544891s │
│ carbon_black_cloud │ audit                       │ system    │ default   │ PASS   │ 31.415609833s │
╰────────────────────┴─────────────────────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: carbon_black_cloud - END   ---

Screenshots

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[kcreddy]

Nit. Rest LGTM 👍🏼

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 1819de5

cc @ShourieG

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

Package carbon_black_cloud - 1.21.0 containing this change is available at https://epr.elastic.co/search?package=carbon_black_cloud

[elasticmachine]

Package carbon_black_cloud - 1.21.0 containing this change is available at https://epr.elastic.co/search?package=carbon_black_cloud