elastic-agent elasticsearch CA fingerprint support #29128

michel-laterman · 2021-11-24T16:02:24Z

What does this PR do?

Add support to the elastic-agent for specifying the elasticsearch CA by fingerprint with the --fleet-server-es-ca-trusted-fingerprint flag.

Why is it important?

The elastic stack in 8.0 will turn on security by default. If no certs are provided ES will generate self-signed certs. Using the fingerprint of the CA will let components know what certs to trust.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Run an Elasticsearch 8.0 Instance with self signed certs
Start a Kibana (8.0) instance the generated enrolment credentials
Enrol an elastic-agent with the --fleet-server-es-ca-sha256 flag
Update obs doc for Elastic Agent and Fleet Server

Related issues

Relates [8.0] Support "Security on by Default" fleet-server#760

elasticmachine · 2021-11-24T17:31:19Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-12-10T00:45:23.408+0000
Duration: 95 min 24 sec
Commit: 787cdd5

Test stats 🧪

Test	Results
Failed	0
Passed	7136
Skipped	16
Total	7152

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

jlind23 · 2021-12-01T15:59:03Z

@belimawr this may be interesting for you :-)

michel-laterman · 2021-12-01T16:16:33Z

libbeat/common/transport/tlscommon/ca_pinning.go

@@ -51,7 +51,7 @@ func verifyCAPin(hashes []string, verifiedChains [][]*x509.Certificate) error {

 // Fingerprint takes a certificate and create a hash of the DER encoded public key.
 func Fingerprint(certificate *x509.Certificate) string {
-	hash := sha256.Sum256(certificate.RawSubjectPublicKeyInfo)
+	hash := sha256.Sum256(certificate.Raw)


@simitt, I've tried the change you suggested here and my tests have rejected the certificate

Could you be a bit more specific what rejected the certificate means? And did you convert the ES fingerprint from hex to base64 before configuring it?

And since it is self-sigend, you need to also configure the CA with the current libbeat code. I mentioned this in my first comment, as it might be a problem if only the fingerprint is passed down from Fleat.

both hex and basic should still be accepted right? Can we know if certs are hex or basic in order to avoid convertion when not needed?

jlind23 · 2021-12-02T13:30:11Z

@michel-laterman could you also perform some test with the standalone agent? Certificate should then be pass into the elastic agent configuration file.
ping @mostlyjason @joshdover

michel-laterman · 2021-12-02T18:06:54Z

I have tested in fleet-mode and got it to work* with a custom snapshot build of the elastic-agent and fleet-server binary.

The agent->ES communication is done through fleet-server.
The fleet-server uses libbeat to parse tls config and passes the output of

beats/libbeat/common/transport/tlscommon/tls_config.go

Line 89 in 008182a

func (c *TLSConfig) ToConfig() *tls.Config {

as the http.RoundTripper.TLSClientConfig when creating a go-elasticsearch client. This means that all certificate verification is currently done by libbeat instead of go-elasticsearch.

To build my test binaries I cherry-picked my current PR commits to 8.0 and built a snapshot for the elastic-agent.
When building the fleet-server binary I based off of 8.0 and updated the libbeat version to the latest release (v7.15.2), injected the fingerprinting code from #29229, and forced it to use the new VerificationFingerprintOnly mode.

The binaries "worked" as fleet-server was able to connect to ES and get cluster info (the agent then failed at DNS stuff).

I'll clear the fingerprint handling from this PR and let @belimawr's pr handle it.
However, in order to get fingerprinting working with elastic-agent/fleet we will need to make sure that when a sha256 fingerprint and no cert is detected the new VerificationFingerprintOnly mode is used (likely as another commit to this PR) and we will need to make another PR on fleet-server to update the version of beats/libbeat it uses to one that includes a backport from #29229.

…rint

joshdover · 2021-12-07T13:25:30Z

As the discussion has progress some things have changed. We should instead be using a CLI arg name called fleet-server-es-ca-trusted-fingerprint to match the ca_trusted_fingerprint config added in libbeat: #29229

elasticmachine · 2021-12-08T16:01:09Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

mergify · 2021-12-09T15:35:20Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b agent-es-ca-fingerprint upstream/agent-es-ca-fingerprint
git merge upstream/master
git push upstream agent-es-ca-fingerprint

blakerouse

You need to add support for this in the container as well.

https://github.com/elastic/beats/blob/master/x-pack/elastic-agent/pkg/agent/cmd/setup_config.go

…rint

blakerouse

Thanks for adding it to the container command. This overall looks great and ready to go.

* initial commit * Update Fingerprint method * Switch to using newly added CATrustedFingerprint attribute * rename flag * Fix broken flag * Add CHANGELOG * Add container support (cherry picked from commit fcb7b81)

* initial commit * Update Fingerprint method * Switch to using newly added CATrustedFingerprint attribute * rename flag * Fix broken flag * Add CHANGELOG * Add container support (cherry picked from commit fcb7b81) Co-authored-by: Michel Laterman <82832767+michel-laterman@users.noreply.github.com>

michel-laterman added enhancement backport-v8.0.0 Automated backport with mergify Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team labels Nov 24, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Nov 24, 2021

initial commit

8c86f4a

michel-laterman force-pushed the agent-es-ca-fingerprint branch from a0a5f09 to 8c86f4a Compare November 25, 2021 02:17

Update Fingerprint method

db5c2fb

michel-laterman commented Dec 1, 2021

View reviewed changes

joshdover mentioned this pull request Dec 2, 2021

[Fleet] Add ES certificate authority fingerprint argument to Fleet Server install command elastic/kibana#116620

Closed

2 tasks

michel-laterman added 3 commits December 6, 2021 11:59

Merge remote-tracking branch 'origin/master' into agent-es-ca-fingerp…

7488ebe

…rint

Switch to using newly added CATrustedFingerprint attribute

e72a330

rename flag

859cd91

nchaulet mentioned this pull request Dec 7, 2021

[Fleet] Configure ca trusted fingerprint for on prem users elastic/kibana#120549

Merged

Fix broken flag

468c210

michel-laterman mentioned this pull request Dec 7, 2021

update go mod elatic/beats reference with newer commit elastic/fleet-server#970

Merged

Add CHANGELOG

29361f5

michel-laterman marked this pull request as ready for review December 8, 2021 16:01

michel-laterman requested a review from a team December 8, 2021 16:06

blakerouse requested changes Dec 9, 2021

View reviewed changes

michel-laterman added 2 commits December 9, 2021 16:44

Add container support

8d267c2

Merge remote-tracking branch 'origin/master' into agent-es-ca-fingerp…

787cdd5

…rint

blakerouse approved these changes Dec 14, 2021

View reviewed changes

michel-laterman merged commit fcb7b81 into elastic:master Dec 14, 2021

michel-laterman deleted the agent-es-ca-fingerprint branch December 14, 2021 19:28

mergify bot mentioned this pull request Dec 14, 2021

[8.0](backport #29128) elastic-agent elasticsearch CA fingerprint support #29428

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

elastic-agent elasticsearch CA fingerprint support #29128

elastic-agent elasticsearch CA fingerprint support #29128

michel-laterman commented Nov 24, 2021 •

edited

elasticmachine commented Nov 24, 2021 •

edited by jenkins-beats-ci bot

Build stats

Test stats 🧪

jlind23 commented Dec 1, 2021

michel-laterman Dec 1, 2021

simitt Dec 2, 2021

simitt Dec 2, 2021

jlind23 Dec 2, 2021

jlind23 commented Dec 2, 2021

michel-laterman commented Dec 2, 2021 •

edited

joshdover commented Dec 7, 2021

elasticmachine commented Dec 8, 2021

mergify bot commented Dec 9, 2021

blakerouse left a comment

blakerouse left a comment

elastic-agent elasticsearch CA fingerprint support #29128

elastic-agent elasticsearch CA fingerprint support #29128

Conversation

michel-laterman commented Nov 24, 2021 • edited

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

elasticmachine commented Nov 24, 2021 • edited by jenkins-beats-ci bot

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

jlind23 commented Dec 1, 2021

michel-laterman Dec 1, 2021

Choose a reason for hiding this comment

simitt Dec 2, 2021

Choose a reason for hiding this comment

simitt Dec 2, 2021

Choose a reason for hiding this comment

jlind23 Dec 2, 2021

Choose a reason for hiding this comment

jlind23 commented Dec 2, 2021

michel-laterman commented Dec 2, 2021 • edited

joshdover commented Dec 7, 2021

elasticmachine commented Dec 8, 2021

mergify bot commented Dec 9, 2021

blakerouse left a comment

Choose a reason for hiding this comment

blakerouse left a comment

Choose a reason for hiding this comment

michel-laterman commented Nov 24, 2021 •

edited

elasticmachine commented Nov 24, 2021 •

edited by jenkins-beats-ci bot

michel-laterman commented Dec 2, 2021 •

edited