[metricbeat][mysql] Add SSL support #37997

gpop63 · 2024-02-13T12:42:25Z

Overview

Adds SSL support for following metricsets:

query
performance
status
galera_status

The query metricset uses a different SQL client (helper/sql). Others use the client from the module itself.

Added a config struct as part of this change.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

genroot.sh

OPENSSL_ROOT_CA=$1

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl genrsa 2048 > certs/root-ca-key.pem

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl req -new -x509 -nodes -days 3600 \
        -subj "${OPENSSL_ROOT_CA}" \
        -key /certs/root-ca-key.pem -out /certs/root-ca.pem

genserver.sh

OPENSSL_SERVER=$1

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl req -newkey rsa:2048 -days 3600 -nodes \
        -subj "${OPENSSL_SERVER}" \
        -keyout /certs/server-key.pem -out /certs/server-req.pem

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl rsa -in /certs/server-key.pem -out /certs/server-key.pem

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl x509 -req -in /certs/server-req.pem -days 3600 \
        -CA /certs/root-ca.pem -CAkey /certs/root-ca-key.pem \
        -set_serial 01 -out /certs/server-cert.pem

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl verify -CAfile /certs/root-ca.pem /certs/server-cert.pem

genclient.sh

OPENSSL_CLIENT=$1

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl req -newkey rsa:2048 -days 3600 -nodes \
        -subj "${OPENSSL_CLIENT}" \
        -keyout /certs/client-key.pem -out /certs/client-req.pem

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl rsa -in /certs/client-key.pem -out /certs/client-key.pem

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl x509 -req -in /certs/client-req.pem -days 3600 \
        -CA /certs/root-ca.pem -CAkey /certs/root-ca-key.pem \
        -set_serial 01 -out /certs/client-cert.pem

docker run --rm -v $PWD/certs:/certs -it nginx \
    openssl verify -CAfile /certs/root-ca.pem /certs/client-cert.pem

gencerts.sh

mkdir -p certs

OPENSSL_SUBJ="/C=US/ST=California/L=Santa Clara"
OPENSSL_CA="${OPENSSL_SUBJ}/CN=fake-CA"
OPENSSL_SERVER="${OPENSSL_SUBJ}/CN=fake-server"
OPENSSL_CLIENT="${OPENSSL_SUBJ}/CN=fake-client"

sh ./genroot.sh "${OPENSSL_CA}"
sh ./genserver.sh "${OPENSSL_SERVER}"
sh ./genclient.sh "${OPENSSL_CLIENT}"

Then run sh +x gencerts.sh to generate required certificates.

Update certificates ownership and permissions so the mysql container can use them.

sudo chown :docker certs/*
sudo chmod 755 certs/*

docker-compose.up.yml

version: '3.8'
services:
  mysql:
    image: mysql/mysql-server:8.0.21
    command: [ "mysqld",
                    "--character-set-server=utf8mb4",
                    "--collation-server=utf8mb4_unicode_ci",
                    "--bind-address=0.0.0.0",
                    "--require_secure_transport=ON",
                    "--ssl-ca=/etc/certs/root-ca.pem",
                    "--ssl-cert=/etc/certs/server-cert.pem",
                    "--ssl-key=/etc/certs/server-key.pem",
                    "--default_authentication_plugin=mysql_native_password" ]
    environment:
      MYSQL_ROOT_PASSWORD: "example"
      MYSQL_ROOT_HOST: "%"
    restart: always
    volumes:
      - type: bind
        source: ./certs
        target: /etc/certs/
    ports:
      - 3306:3306

Related issues

Closes [Metricbeat] Add TLS/SSL configuration options in the MySQL module #16133

Use cases

Screenshots

Logs

mergify · 2024-02-13T12:43:03Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @gpop63? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2024-02-13T12:50:54Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2024-03-12T08:56:27.566+0000
Duration: 55 min 49 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4567
Skipped	905
Total	5472

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

shmsr

Changes look more or less good only but I feel I need to do another comprehensive review. I'll do it ASAP.

metricbeat/module/mysql/mysql.go

metricbeat/module/mysql/config.go

metricbeat/module/mysql/mysql.go

shmsr · 2024-02-14T10:43:03Z

Can you share the minimum steps to test the changes made in the "How to test this PR locally" section of the PR description?

elasticmachine · 2024-02-14T11:17:50Z

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

CHANGELOG.next.asciidoc

metricbeat/module/mysql/config.go

Co-authored-by: Aman <38116245+devamanv@users.noreply.github.com>

gpop63 · 2024-02-15T10:23:14Z

/test

rdner · 2024-02-15T12:05:50Z

metricbeat/module/mysql/mysql_integration_test.go

@@ -31,7 +31,7 @@ import (
 func TestNewDB(t *testing.T) {
 	service := compose.EnsureUp(t, "mysql")

-	db, err := NewDB(GetMySQLEnvDSN(service.Host()))
+	db, err := NewDB(GetMySQLEnvDSN(service.Host()), nil)


Should we introduce a test that validates the added SSL support?

shmsr

The changes look good. Are you planning to add integration tests (what Denis also asked)?

elasticmachine · 2024-03-12T08:58:45Z

💚 Build Succeeded

Buildkite Build
Commit: 62b2d0a

History

💚 Build #1895 succeeded 61c44e3
💚 Build #1699 succeeded 66d412c
💚 Build #1698 succeeded 162a21c
💚 Build #1697 succeeded 4fe8058
💚 Build #1695 succeeded 6c05cc2
💚 Build #1416 succeeded 2353cea

cc @gpop63

elasticmachine · 2024-03-12T08:58:54Z

💚 Build Succeeded

Buildkite Build
Commit: 62b2d0a

History

💚 Build #1051 succeeded 61c44e3
💚 Build #855 succeeded 66d412c
💚 Build #854 succeeded 162a21c
💚 Build #853 succeeded 4fe8058
💚 Build #851 succeeded 6c05cc2
💚 Build #587 succeeded 2353cea

cc @gpop63

elasticmachine · 2024-03-12T08:58:54Z

💚 Build Succeeded

Buildkite Build
Commit: 62b2d0a

History

💚 Build #1888 succeeded 61c44e3
💚 Build #1691 succeeded 66d412c
💚 Build #1690 succeeded 162a21c
💚 Build #1689 succeeded 4fe8058
💚 Build #1687 succeeded 6c05cc2
💚 Build #1423 succeeded 2353cea

cc @gpop63

elasticmachine · 2024-03-12T09:02:24Z

💔 Build Failed

Buildkite Build
Commit: 62b2d0a

Failed CI Steps

History

💔 Build #2184 failed 61c44e3
💚 Build #1988 succeeded 66d412c
💚 Build #1987 succeeded 162a21c
💚 Build #1984 succeeded 6c05cc2

cc @gpop63

elasticmachine · 2024-03-12T09:02:42Z

💔 Build Failed

Buildkite Build
Commit: 62b2d0a

Failed CI Steps

History

💔 Build #3525 failed 61c44e3
💚 Build #3327 succeeded 66d412c
💚 Build #3323 succeeded 6c05cc2
💔 Build #3070 failed 2353cea

cc @gpop63

elasticmachine · 2024-03-12T09:42:40Z

💚 Build Succeeded

Buildkite Build
Commit: 62b2d0a

History

💚 Build #3102 succeeded 61c44e3
💚 Build #2906 succeeded 66d412c
💚 Build #2902 succeeded 6c05cc2
💚 Build #2637 succeeded 2353cea
💚 Build #2626 succeeded d15293c
💔 Build #2558 failed c6b12df

cc @gpop63

rdner · 2024-03-18T12:33:16Z

metricbeat/module/mysql/_meta/config.reference.yml

+  #ssl.certificate: "/etc/pki/client/cert.crt"
+
+  # Client certificate key file
+  #ssl.key: "/etc/pki/client/cert.key"


This pasted text should've had a new line character at the end. The missing character led to the broken documentation. See the fixing PR #38367

gpop63 added 3 commits February 13, 2024 14:14

add config

c49e23f

add ssl support for metricsets

1a24e0f

update docs

4942356

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 13, 2024

mergify bot assigned gpop63 Feb 13, 2024

add changelog entry

267b60c

gpop63 added the Team:Obs-InfraObs Label for the Observability Infrastructure Monitoring team label Feb 13, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 13, 2024

gpop63 added 4 commits February 13, 2024 15:00

make update

22f8514

fix config

b19d916

add NewDB argument

310f0b5

fix imports

c6b12df

gpop63 changed the title ~~[metricbeat] [mysq] Add SSL support~~ [metricbeat] [mysql] Add SSL support Feb 14, 2024

gpop63 marked this pull request as ready for review February 14, 2024 10:29

gpop63 requested review from a team as code owners February 14, 2024 10:29

gpop63 requested review from rdner and fearful-symmetry February 14, 2024 10:29

shmsr reviewed Feb 14, 2024

View reviewed changes

metricbeat/module/mysql/mysql.go Show resolved Hide resolved

metricbeat/module/mysql/config.go Outdated Show resolved Hide resolved

metricbeat/module/mysql/mysql.go Outdated Show resolved Hide resolved

shmsr added the enhancement label Feb 14, 2024

pierrehilbert added the Team:Elastic-Agent Label for the Agent team label Feb 14, 2024

Update mysql.go

b27664a

devamanv reviewed Feb 14, 2024

View reviewed changes

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

metricbeat/module/mysql/config.go Outdated Show resolved Hide resolved

gpop63 and others added 3 commits February 14, 2024 20:48

Update config.go

6fe1c3b

Update CHANGELOG.next.asciidoc

d15293c

Co-authored-by: Aman <38116245+devamanv@users.noreply.github.com>

Merge branch 'main' into add_mysql_tls_support

2353cea

rdner reviewed Feb 15, 2024

View reviewed changes

shmsr changed the title ~~[metricbeat] [mysql] Add SSL support~~ [metricbeat][mysql] Add SSL support Feb 16, 2024

shmsr reviewed Feb 22, 2024

View reviewed changes

gpop63 and others added 2 commits February 27, 2024 20:01

Merge branch 'main' into add_mysql_tls_support

6c05cc2

add integration test with tls

162a21c

gpop63 force-pushed the add_mysql_tls_support branch from 4fe8058 to 162a21c Compare February 27, 2024 22:07

add TLS MinVersion

66d412c

gpop63 requested a review from shmsr March 4, 2024 14:08

rdner approved these changes Mar 4, 2024

View reviewed changes

pierrehilbert and others added 2 commits March 7, 2024 16:38

Merge branch 'main' into add_mysql_tls_support

61c44e3

Merge branch 'main' into add_mysql_tls_support

62b2d0a

shmsr approved these changes Mar 12, 2024

View reviewed changes

gpop63 merged commit 9172e9c into elastic:main Mar 15, 2024
40 of 47 checks passed

rdner reviewed Mar 18, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[metricbeat][mysql] Add SSL support #37997

[metricbeat][mysql] Add SSL support #37997

gpop63 commented Feb 13, 2024 •

edited

mergify bot commented Feb 13, 2024

elasticmachine commented Feb 13, 2024 •

edited

Build stats

Test stats 🧪

shmsr left a comment

shmsr commented Feb 14, 2024 •

edited

elasticmachine commented Feb 14, 2024

gpop63 commented Feb 15, 2024

rdner Feb 15, 2024

shmsr left a comment

elasticmachine commented Mar 12, 2024

elasticmachine commented Mar 12, 2024

elasticmachine commented Mar 12, 2024

elasticmachine commented Mar 12, 2024 •

edited

elasticmachine commented Mar 12, 2024 •

edited

elasticmachine commented Mar 12, 2024

rdner Mar 18, 2024

[metricbeat][mysql] Add SSL support #37997

[metricbeat][mysql] Add SSL support #37997

Conversation

gpop63 commented Feb 13, 2024 • edited

Overview

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify bot commented Feb 13, 2024

elasticmachine commented Feb 13, 2024 • edited

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

shmsr left a comment

Choose a reason for hiding this comment

shmsr commented Feb 14, 2024 • edited

elasticmachine commented Feb 14, 2024

gpop63 commented Feb 15, 2024

rdner Feb 15, 2024

Choose a reason for hiding this comment

shmsr left a comment

Choose a reason for hiding this comment

elasticmachine commented Mar 12, 2024

💚 Build Succeeded

History

elasticmachine commented Mar 12, 2024

💚 Build Succeeded

History

elasticmachine commented Mar 12, 2024

💚 Build Succeeded

History

elasticmachine commented Mar 12, 2024 • edited

💔 Build Failed

Failed CI Steps

History

elasticmachine commented Mar 12, 2024 • edited

💔 Build Failed

Failed CI Steps

History

elasticmachine commented Mar 12, 2024

💚 Build Succeeded

History

rdner Mar 18, 2024

Choose a reason for hiding this comment

gpop63 commented Feb 13, 2024 •

edited

elasticmachine commented Feb 13, 2024 •

edited

shmsr commented Feb 14, 2024 •

edited

elasticmachine commented Mar 12, 2024 •

edited

elasticmachine commented Mar 12, 2024 •

edited