Improve system test coverage for all inputs - 2

[moxarth-rathod]

Proposed commit message

This PR adds system tests for previously untested input types in integrations. Some integrations
support multiple input types (e.g., cloud-based and API-based), but system tests were missing
for some of these inputs. 

The following integrations now include system tests for the missing input types:

- add httpjson system test:
blacklens

- add CEL system test:
checkpoint_harmony_endpoint

- add AWS system test:
cloudflare_logpush

- add azure blob storage system test:
symantec_enpoint_security

- add GCS system test:
cloudflare_logpush
symantec_enpoint_security

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

• Relates [SSI - System test] Improve system test coverage for the SSI integrations #15144

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[kcreddy]

Can you fix indentation of assert.hit_count on this file?
Also add another event and update assert.hit_count: 2 (other inputs too)

[kcreddy]

@moxarth-rathod, this still holds. Can you update it?

[moxarth-rathod]

@kcreddy done.

[kcreddy]

Can you check azure* integrations and add explicit mappings here instead of flattened?

[moxarth-rathod]

I ran into an issue during the elastic package check because adding the required fields in the Beats YAML for system tests pushed the total field count beyond the 2048 limit. To resolve this, I had to flatten some of the fields.

[kcreddy]

You could increase the total_fields.limit setting inside the manifest.yml to get around this: Example

[moxarth-rathod]

Thanks for the suggestion. I tried increasing the total_fields.limit setting in the manifest.yml, but it did not have the intended effect and the issue still persists.

[kcreddy]

May I know the full error you are getting?

[kcreddy]

@moxarth-rathod, I just tested your latest commit with total_fields.limit setting as follows:

elasticsearch:
  index_template:
    settings:
      index:
        mapping:
          total_fields:
            limit: 3000

and updated beats.yml to reflect all fields:

- name: azure
  type: group
  fields:
    - name: storage
      type: group
      fields:
        - name: container.name
          type: keyword
          multi_fields:
            - name: text
              type: text
          description: The name of the Azure Blob Storage container
        - name: blob.name
          type: keyword
          description: The name of the Azure Blob Storage blob object
          multi_fields:
            - name: text
              type: text
        - name: blob.content_type
          type: keyword
          description: The content type of the Azure Blob Storage blob object
    - name: subscription_id
      type: keyword
      description: |
        Azure subscription ID
    - name: correlation_id
      type: keyword
      description: |
        Correlation ID
    - name: tenant_id
      type: keyword
      description: |
        tenant ID
    - name: resource
      type: group
      fields:
        - name: id
          type: keyword
          description: |
            Resource ID
        - name: group
          type: keyword
          description: |
            Resource group
        - name: provider
          type: keyword
          description: |
            Resource type/namespace
        - name: namespace
          type: keyword
          description: |
            Resource type/namespace
        - name: name
          type: keyword
          description: |
            Name
        - name: authorization_rule
          type: keyword
          description: |
            Authorization rule

And I got all tests passed

--- Test results for package: symantec_endpoint_security - START ---
╭────────────────────────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE                    │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├────────────────────────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ symantec_endpoint_security │ event       │ system    │ azure     │ PASS   │ 52.463933958s │
│ symantec_endpoint_security │ event       │ system    │ default   │ PASS   │  2m5.3946485s │
│ symantec_endpoint_security │ event       │ system    │ gcs       │ PASS   │ 46.601027458s │
╰────────────────────────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: symantec_endpoint_security - END   ---
Done

Could it be that you are setting this value to low value, say 2000?

[moxarth-rathod]

The issue is not related to the system tests. It occurs during the package build process, elastic-package check command still reports the same error.

[kcreddy]

Yeah, this is coming from package-spec. I will create an issue for this. Thanks!
For now you can make them flattened.

[kcreddy]

elastic/package-spec#1046

[moxarth-rathod]

Thanks!

[kcreddy]

@ShourieG, can you please review GCS mock service used in the system tests?

[ShourieG]

LGTM for now, I will create a docker image for the mock service and upload to docker registry soon, after that we can remove the redundant mock service code from all the packages and use the image directly

[kcreddy]

Waiting on #16675 (comment)

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 7ba088f

History

• 💔 Build #36017 failed 97b2596
• 💚 Build #35825 succeeded 602e852

cc @moxarth-rathod