Add condition and processors settings to SQL Input

[BenB196]

Enhancement

What does this PR do?

This PR adds two (2) new field settings to the SQL Input integration.

1. Adds the ability to set condition.
   • The ability to set this value allows for having multiple SQL Input integrations in the same Agent policy while only having them run on specific hosts. (Most likely to be used with the host metadata; ex; host.name or host.platform.
2. Adds the ability to set processors.
   • The ability to set processors to manipulate collected data before sending to Elasticsearch.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• I added both condition and processors as "advanced" fields.

Before Upgrade

inputs:
  - id: sql/metrics-sql-12f9198a-ae28-48f4-aae6-63b23781e064
    name: sql-1
    revision: 1
    type: sql/metrics
    use_output: default
    meta:
      package:
        name: sql
        version: 0.1.0
    data_stream:
      namespace: default
    package_policy_id: 12f9198a-ae28-48f4-aae6-63b23781e064
    streams:
      - id: sql/metrics-sql.sql-12f9198a-ae28-48f4-aae6-63b23781e064
        data_stream:
          dataset: test
        metricsets:
          - query
        hosts:
          - 'root:test@tcp(127.0.0.1:3306)/'
        driver: mysql
        sql_queries:
          - query: SHOW GLOBAL STATUS LIKE 'Innodb_system%'
            response_format: variables
        raw_data.enabled: true
        period: 10s

After Upgrade (with Settings)

inputs:
  - id: sql/metrics-sql-12f9198a-ae28-48f4-aae6-63b23781e064
    name: sql-1
    revision: 3
    type: sql/metrics
    use_output: default
    meta:
      package:
        name: sql
        version: 0.2.0
    data_stream:
      namespace: default
    package_policy_id: 12f9198a-ae28-48f4-aae6-63b23781e064
    streams:
      - id: sql/metrics-sql.sql-12f9198a-ae28-48f4-aae6-63b23781e064
        data_stream:
          dataset: test
        condition: '${host.name} == "mydb-server"'
        period: 10s
        raw_data.enabled: true
        driver: mysql
        sql_queries:
          - response_format: variables
            query: SHOW GLOBAL STATUS LIKE 'Innodb_system%'
        hosts:
          - 'root:test@tcp(127.0.0.1:3306)/'
        metricsets:
          - query
        processors:
          - drop_fields:
              fields:
                - agent

How to test this PR locally

1. Install current integration version
2. Upgrade integration to this PR
3. Add/Remove condition values.

Related issues

• N/A

Screenshots

• New fields from Kibana UI
  

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-09-05T21:40:32.221+0000

• Duration: 17 min 47 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4
Skipped	0
Total	4

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[lalit-satapathy]

@ishleenk17, Can you check this PR in sql input?

[BenB196]

FYI, this PR was a bit out of date, I've rebased and fixed the merge conflicts

[aliabbas-elastic]

/test

[ishleenk17]

@BenB196 : Could you please elaborate on the need for conditions?
If you add multiple SQL Input Integrations, they will be corresponding to their own data streams. How would conditions help here?

[BenB196]

Hi @ishleenk17 regarding the need for condition, suppose I have 2 SQL databases I want to query on 2 different servers, but they differ in some way that makes them require their own unique definition of the sql_input. Without the condition field, I need to create 2 entire Elastic Agent policies, just to support these 2 databases. However, with the condition field, I can use something like ${host.name} == 'db1' on one of the integration definitions and ${host.name} == 'db2' on the other integration definition and keep them in the same Elastic Agent policy.

[ishleenk17]

@BenB196 : This is a SQL Input Integration, where the user can enter the dataset names as input as seen in below screenshot:

In this case, 2 SQL Input Integrations will point to 2 different hosts (databases) and due to 2 different datasets, the data will be mapped to the index created by a specific dataset (hence distinguishing between db1 and db2).

For generic integrations (I see you have raised this for IIS etc), the conditions would make sense as dataset names are not taken as input from user. They are fixed.

[BenB196]

Hi @ishleenk17 I think there might be some confusion here, the issue this PR tries to solve isn't related to the dataset value. It is more related to being able to use different hosts (username, password, url)/drivers/queries across multiple integration definitions within the same Elastic Agent policy, but the integration definitions are scoped to specific conditions, so that specific integration definitions only run on specific hosts/circumstances.

[ishleenk17]

Please take care of updating the processor link as mentioned in the comment.
Rest looks good!

[BenB196]

Hi @ishleenk17, wanted to check to see if there was anything else blocking the PR?

[ishleenk17]

Hi @ishleenk17, wanted to check to see if there was anything else blocking the PR?

Thanks for the contribution. The PR is merged now.

[elasticmachine]

Package sql - 0.4.0 containing this change is available at https://epr.elastic.co/search?package=sql