Remove unnecessary PKCS#8 key conversion for Logstash

[Oddly]

Summary

The Logstash security tasks convert PEM private keys to PKCS#8 format across all three certificate modes (external, elasticsearch_ca, standalone). This conversion is unnecessary since Logstash 7.x+ and should be removed to simplify certificate management, especially for external certificate sources like certmonger/FreeIPA.

Current behavior

All three cert modes in roles/logstash/tasks/logstash-security.yml perform PKCS#8 conversion:

external mode (line 35-43)

• Copies the external key to {certs_dir}/{hostname}-pkcs8.key
• No actual PKCS#8 conversion is done (just a rename), but the -pkcs8 suffix implies it
• Creates a dependency on the copy step, preventing certmonger from managing certs directly in the target directory

elasticsearch_ca mode (line 230-251)

• Runs openssl pkcs8 -topk8 -nocrypt to convert the PEM key
• This was needed for older Logstash versions using Java's PKCS#8-only SSL implementation

standalone mode (line 334-346)

• Same openssl pkcs8 conversion as elasticsearch_ca mode

Why PKCS#8 is no longer needed

Logstash 7.x+ (and the upcoming 9.x) uses the elastic_agent/beats input plugin which accepts standard PEM keys directly. The PKCS#8 requirement was a limitation of older Java SSL implementations that has been resolved.

The Elasticsearch role already uses PEM keys directly without PKCS#8 conversion.

Impact of current behavior

For external cert mode with certmonger (FreeIPA):

1. Certmonger writes cert+key to /etc/pki/logstash/
2. The role copies them to /etc/logstash/certs/ with renamed filenames (-server.crt, -pkcs8.key)
3. On automatic certmonger renewal, post_save restarts Logstash but the copies in /etc/logstash/certs/ are stale — they still contain the old cert
4. A manual Ansible re-run is needed to re-copy the certs, defeating the purpose of automatic renewal

Proposed changes

Phase 1: Remove PKCS#8 conversion (all modes)

• external mode: use the original PEM key filename (no -pkcs8 suffix rename)
• elasticsearch_ca mode: skip openssl pkcs8 conversion, use PEM key directly
• standalone mode: skip openssl pkcs8 conversion, use PEM key directly
• Update any templates/defaults that reference -pkcs8.key filenames

Phase 2: Direct cert usage for external mode

• Instead of copying external certs to {certs_dir}/, configure Logstash to read them from their original location (e.g. /etc/pki/logstash/)
• Or: allow configuring the exact filenames in {certs_dir}/ so they match what certmonger produces
• This makes certmonger renewals work without Ansible re-runs

Phase 3: Update pipeline cert references

• Pipeline definitions that reference -pkcs8.key should reference the PEM key directly
• Document the expected cert filenames per mode

Affected files

• roles/logstash/tasks/logstash-security.yml (lines 35-43, 230-251, 334-346)
• roles/logstash/defaults/main.yml (key filename defaults)
• Any molecule scenarios that verify -pkcs8.key filenames

Testing

The logstash_custom_certs molecule scenario (if it exists) or the default logstash scenario should verify:

1. Logstash starts with a standard PEM key (no PKCS#8)
2. The elastic_agent input plugin accepts PEM keys
3. All three cert modes produce working configurations without PKCS#8

Context

Discovered while deploying the ACC environment with FreeIPA/certmonger-managed certificates. The PKCS#8 copy step prevented automatic certificate renewal from working end-to-end. A workaround using a post_save renewal script was considered but addressing the root cause in the role is cleaner.