[AWS Billing] Add processor and leader election support

[BenB196]

Proposed commit message

• Add support for defining processors to AWS Billing
  • Useful for doing things like adding tags/data manipulation.
• Add support for enabling leaderelection.
  • Useful for if you want to run the integration on Kubernetes.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• I've added both of these settings under advanced as I think both won't see too frequent of usage.

How to test this PR locally

Integration before upgrade

inputs:
  - id: aws/metrics-billing-30d07b98-4f9b-4735-9116-aa37b164a07e
    name: aws-1
    revision: 1
    type: aws/metrics
    use_output: default
    meta:
      package:
        name: aws
        version: 2.8.5
    data_stream:
      namespace: default
    package_policy_id: 30d07b98-4f9b-4735-9116-aa37b164a07e
    streams:
      - id: aws/metrics-aws.billing-30d07b98-4f9b-4735-9116-aa37b164a07e
        data_stream:
          dataset: aws.billing
          type: metrics
        metricsets:
          - billing
        period: 24h
        include_linked_accounts: true
        cost_explorer_config.group_by_dimension_keys:
          - AZ
          - INSTANCE_TYPE
          - SERVICE
          - LINKED_ACCOUNT
        cost_explorer_config.group_by_tag_keys:
          - 'aws:createdBy'

Integration after upgrade

(Observe that neither processors nor condition are added here as it should be)

inputs:
  - id: aws/metrics-billing-30d07b98-4f9b-4735-9116-aa37b164a07e
    name: aws-1
    revision: 2
    type: aws/metrics
    use_output: default
    meta:
      package:
        name: aws
        version: 2.9.0
    data_stream:
      namespace: default
    package_policy_id: 30d07b98-4f9b-4735-9116-aa37b164a07e
    streams:
      - id: aws/metrics-aws.billing-30d07b98-4f9b-4735-9116-aa37b164a07e
        data_stream:
          dataset: aws.billing
          type: metrics
        metricsets:
          - billing
        period: 24h
        include_linked_accounts: true
        cost_explorer_config.group_by_dimension_keys:
          - AZ
          - INSTANCE_TYPE
          - SERVICE
          - LINKED_ACCOUNT
        cost_explorer_config.group_by_tag_keys:
          - 'aws:createdBy'

Leader Election enabled, Processors null

inputs:
  - id: aws/metrics-billing-30d07b98-4f9b-4735-9116-aa37b164a07e
    name: aws-1
    revision: 3
    type: aws/metrics
    use_output: default
    meta:
      package:
        name: aws
        version: 2.9.0
    data_stream:
      namespace: default
    package_policy_id: 30d07b98-4f9b-4735-9116-aa37b164a07e
    streams:
      - id: aws/metrics-aws.billing-30d07b98-4f9b-4735-9116-aa37b164a07e
        data_stream:
          dataset: aws.billing
          type: metrics
        metricsets:
          - billing
        period: 24h
        include_linked_accounts: true
        cost_explorer_config.group_by_dimension_keys:
          - AZ
          - INSTANCE_TYPE
          - SERVICE
          - LINKED_ACCOUNT
        cost_explorer_config.group_by_tag_keys:
          - 'aws:createdBy'
        condition: '${kubernetes_leaderelection.leader} == true'

Leader Election Enabled, Processors Set

inputs:
  - id: aws/metrics-billing-30d07b98-4f9b-4735-9116-aa37b164a07e
    name: aws-1
    revision: 4
    type: aws/metrics
    use_output: default
    meta:
      package:
        name: aws
        version: 2.9.0
    data_stream:
      namespace: default
    package_policy_id: 30d07b98-4f9b-4735-9116-aa37b164a07e
    streams:
      - id: aws/metrics-aws.billing-30d07b98-4f9b-4735-9116-aa37b164a07e
        data_stream:
          dataset: aws.billing
          type: metrics
        metricsets:
          - billing
        period: 24h
        include_linked_accounts: true
        cost_explorer_config.group_by_dimension_keys:
          - AZ
          - INSTANCE_TYPE
          - SERVICE
          - LINKED_ACCOUNT
        cost_explorer_config.group_by_tag_keys:
          - 'aws:createdBy'
        condition: '${kubernetes_leaderelection.leader} == true'
        processors:
          - add_tags:
              tags:
                - test

Leader Elastic disabled, Processors set

inputs:
  - id: aws/metrics-billing-30d07b98-4f9b-4735-9116-aa37b164a07e
    name: aws-1
    revision: 5
    type: aws/metrics
    use_output: default
    meta:
      package:
        name: aws
        version: 2.9.0
    data_stream:
      namespace: default
    package_policy_id: 30d07b98-4f9b-4735-9116-aa37b164a07e
    streams:
      - id: aws/metrics-aws.billing-30d07b98-4f9b-4735-9116-aa37b164a07e
        data_stream:
          dataset: aws.billing
          type: metrics
        metricsets:
          - billing
        period: 24h
        include_linked_accounts: true
        cost_explorer_config.group_by_dimension_keys:
          - AZ
          - INSTANCE_TYPE
          - SERVICE
          - LINKED_ACCOUNT
        cost_explorer_config.group_by_tag_keys:
          - 'aws:createdBy'
        processors:
          - add_tags:
              tags:
                - test

Related issues

• Relates Add support for Kubernetes Leader Election for integrations running as a K8s Daemon Set #7362
• Closes [AWS Billing] Support Processor Configuration #8409

Screenshots

[elasticmachine]

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-11-15T23:50:39.654+0000

• Duration: 3 min 9 sec

Steps errors

Expand to view the steps failures

Load a resource file from a library

• Took 0 min 0 sec . View more details here
• Description: approval-list/elastic/integrations.yml

Error signal

• Took 0 min 0 sec . View more details here
• Description: githubApiCall: The REST API call https://api.github.com/orgs/elastic/members/BenB196 return the message : java.lang.Exception: httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/BenB196 : httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/BenB196 : Code: 404Error: {"message":"User does not exist or is not a member of the organization","documentation_url":"https://docs.github.com/rest/orgs/members#check-organization-membership-for-a-user"}

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[mrodm]

Hi @BenB196, please update your branch with the latest contents from main branch. There was an important PR merged updating the CI pipelines. Thanks!

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[BenB196]

Hi, don't mean to bother, but would it be possible to get a review of this PR?

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[ishleenk17]

/test

[ishleenk17]

/test

[ishleenk17]

The changes look fine. There is a buildkite failure. I have given a retrigger. Lets wait for the CI/Cd results.
@elastic/obs-ds-hosted-services : Could you review as well ?

[ishleenk17]

/test

[ishleenk17]

Looks good! @elastic/obs-ds-hosted-services : Please review as well

[BenB196]

@ishleenk17 sorry to bother, is there anything left that needs to be addressed in this PR?

[ishleenk17]

There are other code owner reviews pending.
@elastic/security-service-integrations : Could you please take a look at it as code owners as well.

[ishleenk17]

/test

[ishleenk17]

@BenB196 : You have the approvals now. I have triggered the build.
If it succeeds, will merge.

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: eebd86d

History

• 💚 Build #9491 succeeded 67c4317
• 💔 Build #9477 failed 67c4317
• 💔 Build #9465 failed 67c4317

[elastic-sonarqube]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

Package aws - 2.14.0 containing this change is available at https://epr.elastic.co/search?package=aws