Go version
All versions (tested on go1.22, go1.24, go1.26)
What operating system and processor architecture are you using?
linux/amd64 Amazon Linux 2023 (AL2023), also affects RHEL 9 and any distro that populates /etc/ssl/certs/ with individual PEM files alongside a CA bundle.
What did you do?
Called tls.Dial (which triggers x509.SystemCertPool()) from an AWS Lambda function using the provided.al2023 runtime.
What did you expect to see?
TLS handshake latency comparable to the same binary running on Amazon Linux 2 (~290ms on cold start).
What did you see instead?
TLS handshake latency of ~600ms on cold start, a 2.1x regression compared to AL2.
Root cause
loadSystemRoots() in src/crypto/x509/root_unix.go has two phases:
-
Phase 1 (bundle files): Reads the first matching file from certFiles (e.g., /etc/pki/tls/certs/ca-bundle.crt). On AL2023 this loads all 143 root CAs. Fast, single file read.
-
Phase 2 (directory scan): Scans all directories in certDirectories (/etc/ssl/certs, /etc/pki/tls/certs). On AL2023, /etc/ssl/certs/ contains:
- 143 individual
.pem files (regular files, one cert each)
- 286
.0 hash symlinks (filtered by readUniqueDirectoryEntries)
Go reads all 143 .pem files, PEM-decodes them, ASN.1-parses them, computes SHA-224 hashes, and discovers every single one is a duplicate already loaded in Phase 1. Net result: ~430+ syscalls (readdir + 143 × open/read/close + 143 × stat for the filtered symlinks) and 143 parse operations for zero new certificates.
On AWS Lambda's read-only filesystem, this I/O is especially expensive, turning a ~290ms operation into ~600ms.
Certificate layout on AL2023
/etc/pki/tls/certs/ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem (143 certs, single file)
/etc/ssl/certs/
├── ACCVRAIZ1.pem (regular file, 1 cert)
├── AC_RAIZ_FNMT-RCM.pem (regular file, 1 cert)
├── ... 141 more .pem files ...
├── 002c0b4f.0 -> GlobalSign_Root_R46.pem (same-dir symlink, filtered by Go)
├── ... 285 more .0 symlinks ...
All 143 .pem files contain the exact same certificates as the bundle. They are separate physical files (different inodes), not symlinks to the bundle.
Lambda cold start measurements
Tested with a minimal Go Lambda function performing a single tls.Dial to aws.amazon.com:443, 256 MB memory, eu-west-1, 3 cold starts each:
| Configuration |
Run 1 |
Run 2 |
Run 3 |
Avg Duration |
provided.al2 (baseline) |
286ms |
278ms |
315ms |
~293ms |
provided.al2023 (no workaround) |
663ms |
563ms |
651ms |
~626ms |
provided.al2023 + SSL_CERT_DIR=/etc/pki/tls/certs |
231ms |
212ms |
254ms |
~232ms |
Setting SSL_CERT_DIR=/etc/pki/tls/certs works because it replaces the directory list with one that only has 2 files (the bundle symlink + a trust file), skipping the 143-file /etc/ssl/certs/ scan entirely.
Prior art
The .NET runtime had the same issue on AL2023/RHEL 9 and fixed it using inode-based file deduplication: dotnet/runtime#97267
Their fix records (inode, device) tuples for files already read and skips files that resolve to the same physical file. This works for their case (two symlinks to the same bundle), but would not fully address the Go/AL2023 case where the individual .pem files are separate physical files with different inodes.
Proposed fix
Short term (inode dedup): Track (dev, ino) of files read in Phase 1. In Phase 2, stat each file before reading and skip if already seen. This handles the .NET-style case (symlinks to same bundle) and is safe and backwards-compatible.
More impactful (skip redundant directory scan): If Phase 1 successfully loaded certificates from a bundle file, skip the directory scan for default certDirectories entirely. The directory scan exists as a fallback for distros that only have individual cert files with no bundle (e.g., SLES10, now EOL). On any modern distro with a bundle, the directory scan produces only duplicates. This would fully fix the AL2023 case.
Workaround
I set the environment variable SSL_CERT_DIR=/etc/pki/tls/certs on affected applications. This directs Go to only scan /etc/pki/tls/certs (2 files) instead of also scanning /etc/ssl/certs/ (431 files).
Go version
All versions (tested on go1.22, go1.24, go1.26)
What operating system and processor architecture are you using?
linux/amd64 Amazon Linux 2023 (AL2023), also affects RHEL 9 and any distro that populates
/etc/ssl/certs/with individual PEM files alongside a CA bundle.What did you do?
Called
tls.Dial(which triggersx509.SystemCertPool()) from an AWS Lambda function using theprovided.al2023runtime.What did you expect to see?
TLS handshake latency comparable to the same binary running on Amazon Linux 2 (~290ms on cold start).
What did you see instead?
TLS handshake latency of ~600ms on cold start, a 2.1x regression compared to AL2.
Root cause
loadSystemRoots()insrc/crypto/x509/root_unix.gohas two phases:Phase 1 (bundle files): Reads the first matching file from
certFiles(e.g.,/etc/pki/tls/certs/ca-bundle.crt). On AL2023 this loads all 143 root CAs. Fast, single file read.Phase 2 (directory scan): Scans all directories in
certDirectories(/etc/ssl/certs,/etc/pki/tls/certs). On AL2023,/etc/ssl/certs/contains:.pemfiles (regular files, one cert each).0hash symlinks (filtered byreadUniqueDirectoryEntries)Go reads all 143
.pemfiles, PEM-decodes them, ASN.1-parses them, computes SHA-224 hashes, and discovers every single one is a duplicate already loaded in Phase 1. Net result: ~430+ syscalls (readdir + 143 × open/read/close + 143 × stat for the filtered symlinks) and 143 parse operations for zero new certificates.On AWS Lambda's read-only filesystem, this I/O is especially expensive, turning a ~290ms operation into ~600ms.
Certificate layout on AL2023
All 143
.pemfiles contain the exact same certificates as the bundle. They are separate physical files (different inodes), not symlinks to the bundle.Lambda cold start measurements
Tested with a minimal Go Lambda function performing a single
tls.Dialtoaws.amazon.com:443, 256 MB memory,eu-west-1, 3 cold starts each:provided.al2(baseline)provided.al2023(no workaround)provided.al2023+SSL_CERT_DIR=/etc/pki/tls/certsSetting
SSL_CERT_DIR=/etc/pki/tls/certsworks because it replaces the directory list with one that only has 2 files (the bundle symlink + a trust file), skipping the 143-file/etc/ssl/certs/scan entirely.Prior art
The .NET runtime had the same issue on AL2023/RHEL 9 and fixed it using inode-based file deduplication: dotnet/runtime#97267
Their fix records
(inode, device)tuples for files already read and skips files that resolve to the same physical file. This works for their case (two symlinks to the same bundle), but would not fully address the Go/AL2023 case where the individual.pemfiles are separate physical files with different inodes.Proposed fix
Short term (inode dedup): Track
(dev, ino)of files read in Phase 1. In Phase 2,stateach file before reading and skip if already seen. This handles the .NET-style case (symlinks to same bundle) and is safe and backwards-compatible.More impactful (skip redundant directory scan): If Phase 1 successfully loaded certificates from a bundle file, skip the directory scan for default
certDirectoriesentirely. The directory scan exists as a fallback for distros that only have individual cert files with no bundle (e.g., SLES10, now EOL). On any modern distro with a bundle, the directory scan produces only duplicates. This would fully fix the AL2023 case.Workaround
I set the environment variable
SSL_CERT_DIR=/etc/pki/tls/certson affected applications. This directs Go to only scan/etc/pki/tls/certs(2 files) instead of also scanning/etc/ssl/certs/(431 files).