Skip to content

Load TLS root CA directly from /config instead of /persist/certs#5553

Merged
eriknordmark merged 1 commit intolf-edge:masterfrom
shjala:remove.tls.cert.copy
Jan 29, 2026
Merged

Load TLS root CA directly from /config instead of /persist/certs#5553
eriknordmark merged 1 commit intolf-edge:masterfrom
shjala:remove.tls.cert.copy

Conversation

@shjala
Copy link
Member

@shjala shjala commented Jan 15, 2026

Description

This change fixes the security issue where TLS root certificates were copied from the integrity-protected /config partition to the mutable /persist/certs directory and referenced via SHA256 indirection.

Changes:

  • Remove certificate copy logic from InitializeCertDir() - now only creates the certs directory without copying files
  • Update GetTLSConfig() to read certificates directly from types.V2TLSBaseFile (/config/v2tlsbaseroot-certificates.pem)
  • Update UpdateTLSProxyCerts() to read from V2TLSBaseFile instead of V2TLSCertShaFilename
  • Remove V2TLSCertShaFilename constant from locationconsts.go as it's no longer needed
  • Update getSecurityInfo() in handlemetrics.go to compute SHA256 hash directly from the V2TLSBaseFile content instead of reading a pre-computed SHA from persist

PR dependencies

None

How to test and validate this PR

  1. Verify TLS connections work after the change, device should connect to the controller, receive configurations, certs, etc.
  2. Verify certificates are read from /config (not /persist):
# Should NOT contain files like: 6edbc9851d43dc2cd77bcd60d758994663b9f4bf or v2tlsbaseroot-certificates.sha256
eve exec pillar ls -la /persist/certs/
# Verify the base cert exists in /config
cat /config/v2tlsbaseroot-certificates.pem | head -5
  1. Test with proxy certificates, proxy certs are dynamically added to cert pool (unchanged by this PR):
  • Configure a network proxy with custom certificates
  • Verify device still connects through proxy correctly
  1. Validate offline tampering protection:
  • modify cert on the /config directory
  • should result in vault lock up
  1. Test upgrade from a version without this PR:
  • Device running old version (e.g., 16.0.0) with certs in /persist/certs/]
  • Upgrade to version with this fix
  • Verify device continues to work (ignores old persist copies, uses /config)
  • Old SHA files in /persist/certs/ should is harmless (as it is not read anymore)

Changelog notes

Fixed security issue where TLS root certificates could be tampered with via physical disk access, now loading certificates directly from integrity-protected /config partition.

PR Backports

Possibly all :

  • 11.0-stable
  • 12.0-stable
  • 13.4-stable
  • 14.5-stable
  • 16.0-stable

Checklist

  • I've provided a proper description
  • I've added the proper documentation
  • I've tested my PR on amd64 device
  • I've tested my PR on arm64 device
  • I've written the test verification instructions
  • I've set the proper labels to this PR

For backport PRs (remove it if it's not a backport):

  • I've added a reference link to the original PR
  • PR's title follows the template

And the last but not least:

  • I've checked the boxes above, or I've provided a good reason why I didn't
    check them.

Please, check the boxes above after submitting the PR in interactive mode.

@shjala shjala self-assigned this Jan 15, 2026
@shjala shjala added stable Should be backported to stable release(s) security Provides a security fix labels Jan 15, 2026
Copy link
Contributor

@eriknordmark eriknordmark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that soon we should figure out how to deliver a new root CA file to deployed devices, do you have ideas how to update it?

We could do the eve config mount and then to a writerename to types.V2TLSBaseFile perhaps with a .bak file to handle a power outage? Or something different?

@shjala
Copy link
Member Author

shjala commented Jan 16, 2026

Given that soon we should figure out how to deliver a new root CA file to deployed devices, do you have ideas how to update it?

We could do the eve config mount and then to a writerename to types.V2TLSBaseFile perhaps with a .bak file to handle a power outage? Or something different?

If we start storing these in rootfs, then we don't have to change the process, a system update brings the new certs too, but if we want to allow surgical update of the pinned certs, then I think storing in config is good approach. But let's discuss all the options and have a security assessment of each before commiting.

@eriknordmark
Copy link
Contributor

Given that soon we should figure out how to deliver a new root CA file to deployed devices, do you have ideas how to update it?
We could do the eve config mount and then to a writerename to types.V2TLSBaseFile perhaps with a .bak file to handle a power outage? Or something different?

If we start storing these in rootfs, then we don't have to change the process, a system update brings the new certs too, but if we want to allow surgical update of the pinned certs, then I think storing in config is good approach. But let's discuss all the options and have a security assessment of each before commiting.

Not part of this PR, but we do have /etc/ssl/certs/ca-certificates.crt in the rootfs so we could have EVE compare and update /config/v2tlsbaseroot-certificates.pem from that file (e.g., by introducing some additional onboot container). That will ensure that we update it as we update the rootfs to new (Alpine) versions.

Copy link
Contributor

@eriknordmark eriknordmark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but CI/CD has issues. Can you rebase on master to see if those issues go away?

Copy link
Contributor

@eriknordmark eriknordmark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder whether we should just drop this file from /config and instead read from the rootfs i.e., read directly from /etc/ssl/certs/ca-certificates.crt and feed that to AppendCertsFromPEM().

@shjala
Copy link
Member Author

shjala commented Jan 19, 2026

@eriknordmark OK, I marked this as draft for now, let me finish the Flexible PCR feature and then get back to this one.

@shjala
Copy link
Member Author

shjala commented Jan 26, 2026

@eriknordmark as per our discussion:

Because Eden needs to replace the root certificate and also append the Adam server certificates to the TLS certificate bundle, it is currently not possible to move these artifacts to the rootfs. Given this limitation, we should proceed with integrity protection at this stage.

Work is in progress to utilize the TPM "authorized write" functionality. If fruitful, we can leverage it to implement more robust protection for all security-sensitive configurations.

I'm removing draft status from this PR.

@shjala shjala marked this pull request as ready for review January 26, 2026 09:36
Copy link
Contributor

@eriknordmark eriknordmark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment on lines 111 to 121
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code needs to be deleted.
CI/CD tests are failing because you create a filename/pathname which includes the content of the pem file!

Comment on lines 268 to 280
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

This change fixes the security issue where TLS root certificates were
copied from the integrity-protected /config partition to the mutable
/persist/certs directory and referenced via SHA256 indirection.

Changes:
- Remove certificate copy logic from InitializeCertDir() - now only
  creates the certs directory without copying files
- Update GetTLSConfig() to read certificates directly from
  types.V2TLSBaseFile (/config/v2tlsbaseroot-certificates.pem)
- Update UpdateTLSProxyCerts() to read from V2TLSBaseFile instead of
  V2TLSCertShaFilename
- Remove V2TLSCertShaFilename constant from locationconsts.go as it's
  no longer needed
- Update getSecurityInfo() in handlemetrics.go to compute SHA256 hash
  directly from the V2TLSBaseFile content instead of reading a
  pre-computed SHA from persist

Signed-off-by: Shahriyar Jalayeri <shahriyar@posteo.de>
@shjala shjala force-pushed the remove.tls.cert.copy branch from 761132d to b04b2ec Compare January 28, 2026 11:19
@github-actions github-actions bot requested a review from eriknordmark January 28, 2026 11:20
@shjala
Copy link
Member Author

shjala commented Jan 28, 2026

@eriknordmark rebased on master and fix the issues.

@codecov
Copy link

codecov bot commented Jan 28, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 28.32%. Comparing base (2281599) to head (b04b2ec).
⚠️ Report is 253 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #5553      +/-   ##
==========================================
+ Coverage   19.52%   28.32%   +8.79%     
==========================================
  Files          19       18       -1     
  Lines        3021     2256     -765     
==========================================
+ Hits          590      639      +49     
+ Misses       2310     1475     -835     
- Partials      121      142      +21     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@eriknordmark eriknordmark merged commit aefb47f into lf-edge:master Jan 29, 2026
47 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

security Provides a security fix stable Should be backported to stable release(s)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants