[GCP] Add cloudsql data streams #4126

gpop63 · 2022-09-05T09:58:07Z

What does this PR do?

Adds CloudSQL data streams:

MySQL
PostgreSQL
SQLServer

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes GCP CloudSQL integration #4086

Screenshots

elasticmachine · 2022-09-05T10:16:39Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-07-20T22:02:17.948+0000
Duration: 18 min 46 sec

Test stats 🧪

Test	Results
Failed	0
Passed	64
Skipped	0
Total	64

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2022-09-05T10:16:51Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`6/6`)	💚
Files	100.0% (`6/6`)	💚
Classes	100.0% (`6/6`)	💚
Methods	87.826% (`101/115`)	👎 -12.174
Lines	96.0% (`1464/1525`)	👍 9.169
Conditionals	100.0% (`0/0`)	💚

gpop63 · 2022-09-16T09:01:26Z

/test

elasticmachine · 2022-09-16T09:20:20Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

botelastic · 2022-11-07T20:02:26Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

endorama · 2022-11-14T14:32:58Z

@gpop63 does this PR requires elastic/beats#33066?

gpop63 · 2022-11-14T15:09:57Z

Yes, we should first merge the CloudSQL Metadata PR (elastic/beats#33066)) and then this one, because we need the metadata to identify which database type generated events.

endorama · 2022-11-23T13:15:19Z

elastic/beats#33066 has been merged, so we can move forward with this once 8.6.0 is out.

botelastic · 2023-01-14T10:05:07Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

botelastic · 2023-02-13T10:57:08Z

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!

botelastic · 2023-03-15T15:58:28Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

botelastic · 2023-06-23T08:53:25Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

endorama

Just a minor comment but looks good!

endorama · 2023-07-19T07:37:23Z

packages/gcp/data_stream/cloudsql_mysql/elasticsearch/ingest_pipeline/default.yml

+processors:
+  - drop:
+      description: Drop if database is not MySQL.
+      if: "ctx?.gcp?.labels?.cloudsql?.name != 'mysql'"


One thing I think will be useful in the future is to document how we built this check. Is not very clear where this label come from, if it can change or not and where to check in case this pipeline stops working.

Please document this in the cloudsql.md file for future reference.

endorama · 2023-07-19T07:37:35Z

packages/gcp/data_stream/cloudsql_postgresql/elasticsearch/ingest_pipeline/default.yml

+processors:
+  - drop:
+      description: Drop if database is not PostgreSQL.
+      if: "ctx?.gcp?.labels?.cloudsql?.name != 'postgres'"


Same for this check.

endorama · 2023-07-19T07:37:43Z

packages/gcp/data_stream/cloudsql_sqlserver/elasticsearch/ingest_pipeline/default.yml

+processors:
+  - drop:
+      description: Drop if database is not SQLServer.
+      if: "ctx?.gcp?.labels?.cloudsql?.name != 'sqlserver'"


Same for this check.

elasticmachine · 2023-07-20T22:51:27Z

Package gcp - 2.24.0 containing this change is available at https://epr.elastic.co/search?package=gcp

* add cloudsql mysql data stream * add cloudsql postgresql data stream * add cloudsql sqlserver data stream * add cloudsql docs * add changelog and cloudsql policy template * bump manifest version * add mysql ingest pipeline * add postgresql ingest pipeline * add sql server ingest pipeline * add mysql datastream suffixes, units and metric_types add drop processor if database is not mysql * add postgres datastream suffixes, units and metric_types add drop processor if database is not postgres * add sqlserver datastream suffixes, units and metric_types add drop processor if database is not sqlserver * update cloudsql docs * bump package version * fix field types * update docs * remove duplicated fields from agent due to elastic-package check --------- Co-authored-by: Edoardo Tenani <526307+endorama@users.noreply.github.com>

gpop63 added 5 commits September 5, 2022 12:55

add cloudsql mysql data stream

e22cc8e

add cloudsql postgresql data stream

566fcbc

add cloudsql sqlserver data stream

da26316

add cloudsql docs

1182056

add changelog and cloudsql policy template

631cc27

Merge remote-tracking branch 'upstream/main' into add_gcp-cloudsql

65adad0

gpop63 mentioned this pull request Sep 14, 2022

[Metricbeat] GCP cloudsql metadata elastic/beats#33066

Merged

6 tasks

gpop63 marked this pull request as ready for review September 14, 2022 11:03

gpop63 requested review from a team as code owners September 14, 2022 11:03

gpop63 added 2 commits September 15, 2022 12:11

Merge remote-tracking branch 'upstream/main' into add_gcp-cloudsql

b6fdb8c

bump manifest version

6c3cd9d

endorama added Integration:GCP Team:Cloud-Monitoring Label for the Cloud Monitoring team labels Sep 23, 2022

endorama mentioned this pull request Sep 23, 2022

Request integrations for the Google cloud metrics metricset for Elastic Agent #4119

Closed

8 tasks

endorama and others added 5 commits September 29, 2022 11:30

Merge branch 'main' into add_gcp-cloudsql

da89075

add mysql ingest pipeline

ee8f7ed

add postgresql ingest pipeline

c3fd95f

add sql server ingest pipeline

f15755f

Merge remote-tracking branch 'upstream/main' into add_gcp-cloudsql

0b4c9e7

botelastic bot added the Stalled label Nov 7, 2022

botelastic bot removed the Stalled label Nov 14, 2022

gpop63 added 2 commits December 15, 2022 02:36

Merge remote-tracking branch 'upstream/main' into add_gcp-cloudsql

f86653a

update cloudsql docs

d259be4

botelastic bot added the Stalled label Jan 14, 2023

botelastic bot closed this Feb 13, 2023

endorama reopened this Feb 13, 2023

botelastic bot removed the Stalled label Feb 13, 2023

Merge branch 'main' into add_gcp-cloudsql

a69baf3

botelastic bot added the Stalled label Mar 15, 2023

Merge branch 'main' into add_gcp-cloudsql

01ca9e7

botelastic bot removed the Stalled label Mar 23, 2023

endorama and others added 3 commits April 4, 2023 15:04

Merge branch 'main' into add_gcp-cloudsql

42237fa

Merge remote-tracking branch 'upstream/main' into add_gcp-cloudsql

fd6e74d

bump package version

7d38994

rameshelastic assigned gpop63 May 24, 2023

botelastic bot added the Stalled label Jun 23, 2023

gpop63 added 2 commits July 18, 2023 00:45

fix field types

e3046c8

Merge remote-tracking branch 'upstream/main' into add_gcp-cloudsql

aef8639

botelastic bot removed the Stalled label Jul 17, 2023

endorama approved these changes Jul 19, 2023

View reviewed changes

gpop63 and others added 4 commits July 19, 2023 13:33

Merge branch 'main' into add_gcp-cloudsql

a3eba72

update docs

d9c3eac

Merge remote-tracking branch 'upstream/main' into add_gcp-cloudsql

5d76a41

remove duplicated fields from agent due to elastic-package check

1ac1b89

gpop63 merged commit 27e77c6 into elastic:main Jul 20, 2023
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GCP] Add cloudsql data streams #4126

[GCP] Add cloudsql data streams #4126

gpop63 commented Sep 5, 2022 •

edited

Loading

elasticmachine commented Sep 5, 2022 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Sep 5, 2022 •

edited

Loading

gpop63 commented Sep 16, 2022

elasticmachine commented Sep 16, 2022 •

edited

Loading

botelastic bot commented Nov 7, 2022

endorama commented Nov 14, 2022

gpop63 commented Nov 14, 2022

endorama commented Nov 23, 2022

botelastic bot commented Jan 14, 2023

botelastic bot commented Feb 13, 2023

botelastic bot commented Mar 15, 2023

botelastic bot commented Jun 23, 2023

endorama left a comment

endorama Jul 19, 2023

endorama Jul 19, 2023

endorama Jul 19, 2023

elasticmachine commented Jul 20, 2023

[GCP] Add cloudsql data streams #4126

[GCP] Add cloudsql data streams #4126

Conversation

gpop63 commented Sep 5, 2022 • edited Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Sep 5, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Sep 5, 2022 • edited Loading

🌐 Coverage report

gpop63 commented Sep 16, 2022

elasticmachine commented Sep 16, 2022 • edited Loading

🚀 Benchmarks report

botelastic bot commented Nov 7, 2022

endorama commented Nov 14, 2022

gpop63 commented Nov 14, 2022

endorama commented Nov 23, 2022

botelastic bot commented Jan 14, 2023

botelastic bot commented Feb 13, 2023

botelastic bot commented Mar 15, 2023

botelastic bot commented Jun 23, 2023

endorama left a comment

Choose a reason for hiding this comment

endorama Jul 19, 2023

Choose a reason for hiding this comment

endorama Jul 19, 2023

Choose a reason for hiding this comment

endorama Jul 19, 2023

Choose a reason for hiding this comment

elasticmachine commented Jul 20, 2023

gpop63 commented Sep 5, 2022 •

edited

Loading

elasticmachine commented Sep 5, 2022 •

edited

Loading

elasticmachine commented Sep 5, 2022 •

edited

Loading

elasticmachine commented Sep 16, 2022 •

edited

Loading