[Fleet] Support preconfigured output secrets

[hop-dev]

As part of #157458 we are adding the ability to use secrets with outputs.

Outputs can also be preconfigured in kibana.yml config. The scope of this issue is to support secrets in preconfigured outputs:

• when a new output is detected in the config, it is created, any secrets are created and secret references are written to the output.
• when an existing output secret is modified, the change is detected and the old secret is deleted and a new one created.

Implementation

See https://github.com/elastic/infosec/issues/14853 this has been raised with infosec.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[hop-dev]

Update: nevermind, see comment below this one

We need to decide how users will specify secrets in config.

Here is a preconfigured output as it stands:

xpack.fleet.outputs:
  - id: my-logstash-output
    name: preconfigured logstash output
    type:  logstash
    hosts: ["localhost:9999"]
    ssl:
      certificate: xxxxxxxxxx
      key: this_is_secret

We can't have it so that a user upgrades to 8.12 and all of a sudden their ssl.key secret is migrated to using secrets storage, mainly bevause they also need to upgrade their flee tserver and on prem users may not have done this, so we would break their policy

A couple options, in both of these we will need to update the docs to warn the user that using the old keys will store in plaintext:

1. Separate secrets block

xpack.fleet.outputs:
  - id: my-logstash-output
    name: preconfigured logstash output
    type:  logstash
    hosts: ["localhost:9999"]
    ssl:
      certificate: xxxxxxxxxx
    secrets:
        ssl:
            key:this_is_secret
  

I think this is clean but it makes the API a bit more complicated

2. prefix/postfix to keys
   something like key_secret or key_secure ?

xpack.fleet.outputs:
  - id: my-logstash-output
    name: preconfigured logstash output
    type:  logstash
    hosts: ["localhost:9999"]
    ssl:
      certificate: xxxxxxxxxx
      key_secret: this_is_secret

@juliaElastic @kpollich any opinions/ other ideas?

[hop-dev]

Actually I've just been for a walk and realised that option 2 (Edit: I meant option 1) mirrors the HTTP API so definitely makes more sense, even if it feels less ergonomic to use

[hop-dev]

@jillguyonnet I had a DM from @ismisepaul recommending we use argon2id, so I have had a go at starting an implementation, its very rough but I think it works:

#170259

[jillguyonnet]

Reopening this as #170259 was reverted.

[kpollich]

@amolnater-qasource surfacing testing instructions from Jill's second PR here since there's multiple PR's here:

Testing

1. Ensure the outputSecretsStorage experimental feature is enabled.
2. Add the following to your kibana config:

   xpack.fleet.outputs:
     - id: my-logstash-output-with-a-secret
       name: preconfigured logstash output with a secret
       type:  logstash
       hosts: ["localhost:9999"]
       ssl:
         certificate: xxxxxxxxxx
       secrets:
         ssl:
           key: thisissecret
   

3. Verify the secret has been correctly created, e.g. by issuing a GET .fleet-secrets/_search request in Dev Tools: the secret should be listed there.
4. Change the preconfigured value and wait for kibana to restart: the secret should be updated with the new value.

@jillguyonnet if there's any more detail you can provide to help the QA folks test please feel free to add anything 🙂

[jillguyonnet]

Thank you @kpollich!

It would be great to test this with the three output types that support secret values:

• Logstash (the example you pasted)
• Kafka (password and SSL key)
• Remote Elasticsearch (service token)

@amolnater-qasource please let me know if you have any questions 🙂

[harshitgupta-qasource]

Hi Team,

We have created 08 testcases under testrail for this feature under Fleet test suite at links:

• Verify that the secrets are correctly created for the preconfigured Logstash output SSL certificate key.
• Verify that the secrets get updated with the new secrets for preconfigured Logstash output SSL certificate key, after the preconfigured value is changed.
• Verify that the secrets are correctly created for the preconfigured Kafka Password
• Verify that the secrets are correctly created for the preconfigured Kafka SSL certificate key.
• Verify that the secrets get updated with the new value for preconfigured Kafka password, after the preconfigured value is changed.
• Verify that the secrets get updated with the new value for the preconfigured Kafka SSL certificate key, after the preconfigured value is changed.
• Verify that the secrets are correctly created for the preconfigured remote Elasticsearch service token.
• Verify that the secrets get updated with the new secrets for preconfigured Remote Elasticsearch service token, after the preconfigured value is changed.

Please let us know if any other scenario needs to be added from our end.

Thanks!

[harshitgupta-qasource]

Hi Team,

We have executed 08 testcases under the Feature test run for the 8.12.0 release at the link:

• Support preconfigured output secrets

Status:

PASS: 08

Build details:
VERSION: 8.12.0 BC4
BUILD: 70016
COMMIT: c2fda47
Artifact Link: https://staging.elastic.co/8.12.0-e9640208/summary-8.12.0.html

As the testing is completed on this feature, we are marking this as QA:Validated.

Please let us know if anything else is required from our end.
Thanks