[1.3] libct: reuse tmpfs for directory masks#5282
Merged
Merged
Conversation
Contributor
Contributor
|
I'm not sure we should backport this to 1.3, it's and old branch for security fixes only (if I'm not mistaken). Everyone should have switched to v1.4 already. |
Member
Author
Line 67 in 84762a5 We haven’t released ‘1.5.0’ yet, it’s still a candidate release, so we can still receive sufficient bug fixes in ‘1.3’. And I see some cloud platforms still use ‘1.3.x’ in their products. Maybe they’ll switch to ‘1.4’ after we released ‘1.5.0’. |
Previously, masked directories (e.g., /proc/acpi, /proc/scsi) were mounted as read-only tmpfs without explicit size or inode limits. Although these mounts are meant to be empty and unwritable, the lack of resource constraints means that—should an attacker bypass the read-only protection (e.g., via container escape, mount namespace manipulation, or a kernel vulnerability)—the tmpfs could consume up to 50% of system memory by default (the kernel's default tmpfs limit). To mitigate this risk in high-density container environments and adhere to the principle of least privilege, we now explicitly set: - nr_blocks=1 (sufficient for at most one block size) - nr_inodes=1 (sufficient for at most one inode) Ref: https://man7.org/linux/man-pages/man5/tmpfs.5.html These limits ensure that even if compromised, kernel memory usage remains strictly bounded and negligible. This change aligns with best practices used by other container runtimes and strengthens defense-in-depth for sensitive masked paths. Co-authored-by: Davanum Srinivas <davanum@gmail.com> Refactored-by: lifubang <lifubang@acmcoder.com> Signed-off-by: lifubang <lifubang@acmcoder.com> (cherry picked from commit e57a7a4) Signed-off-by: lifubang <lifubang@acmcoder.com>
Kubernetes may add one sysfs thermal_throttle entry per CPU to maskedPaths. On large Intel systems this can produce many directory masks for a single container. runc currently handles each directory mask with a separate read-only tmpfs mount, and therefore a separate tmpfs superblock. On Linux 4.18/RHEL 8 kernels, creating and tearing down many tmpfs superblocks can contend on the global shrinker_rwsem when containers start or stop concurrently. Use one read-only tmpfs for directory masks and bind-mount it over the remaining directory targets. The first non-procfs-fd directory mount is reopened through the container root fd before it is reused. File masks still bind /dev/null, and procfs fd targets keep the existing one-tmpfs-per-target behaviour because they are fd aliases rather than stable rootfs paths. If the bind-mount of the shared source fails (e.g. due to kernel restrictions), fall back to individual tmpfs mounts for all remaining directories. Tmpfs mounts use nr_blocks=1,nr_inodes=1 to minimise kernel resource usage. The bind mounts do not create additional tmpfs superblocks. They also retain the read-only mount flag inherited from the source vfsmount, so the masking semantics remain unchanged. xref: kubernetes/kubernetes#138512 xref: kubernetes/kubernetes#138388 xref: kubernetes/kubernetes#131018 Co-authored-by: Davanum Srinivas <davanum@gmail.com> Refactored-by: lifubang <lifubang@acmcoder.com> Signed-off-by: lifubang <lifubang@acmcoder.com> (cherry picked from commit c046c9b) Signed-off-by: lifubang <lifubang@acmcoder.com>
Co-authored-by: Davanum Srinivas <davanum@gmail.com> Signed-off-by: lifubang <lifubang@acmcoder.com> (cherry picked from commit 124772f) Signed-off-by: lifubang <lifubang@acmcoder.com>
This is a follow-up to opencontainers#5275. That change reused a single tmpfs mount to mask multiple directories, which is efficient when masking more than one path. However, it introduced unnecessary overhead when only one directory is masked. This commit restores the original behavior for the single-path case while preserving shared tmpfs logic for multiple paths. Signed-off-by: lifubang <lifubang@acmcoder.com> (cherry picked from commit e7e2f00) Signed-off-by: lifubang <lifubang@acmcoder.com>
Close the root file descriptor immediately after use in maskPaths to reduce the window during which an attacker could potentially exploit an open fd to access or manipulate the root filesystem. This follows the principle of least privilege and mitigates risks in compromised or malicious container scenarios. Co-authored-by: Kir Kolyshkin <kolyshkin@gmail.com> Signed-off-by: lifubang <lifubang@acmcoder.com> (cherry picked from commit b88635e) Signed-off-by: lifubang <lifubang@acmcoder.com>
lifubang
force-pushed
the
backport-5275-1.3
branch
from
ed0eeab to
9a2fec0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport #5275 and #5285
This is not a clean backport: the first commit from #5275 has been omitted, and the final commit from #5285 has been modified in this PR.
This PR carries forward #5262.
Wondering if a problem that showed up recently in k8s when we added more masked paths can be handled better in
runcitself? please see details below:Kubernetes may add one sysfs thermal_throttle entry per CPU to maskedPaths. On large Intel systems this can produce many directory masks for a single container. runc currently handles each directory mask with a separate read-only tmpfs mount, and therefore a separate tmpfs superblock.
On Linux 4.18/RHEL 8 kernels, creating and tearing down many tmpfs superblocks can contend on the global shrinker_rwsem when containers start or stop concurrently.
Use one read-only tmpfs for directory masks and bind-mount it over the remaining directory targets. The first non-procfs-fd directory mount is reopened through the container root fd before it is reused. File masks still bind /dev/null, and procfs fd targets keep the existing one-tmpfs-per-target behavior because they are fd aliases rather than stable rootfs paths.
The bind mounts do not create additional tmpfs superblocks. They also retain the read-only mount flag inherited from the source vfsmount, so the masking semantics remain unchanged.
xref: kubernetes/kubernetes#138512
xref: kubernetes/kubernetes#138388
xref: kubernetes/kubernetes#131018
(With some assistance from claude/codex)
This PR is a follow-up to #5275.
In that change, we started reusing a single tmpfs mount to mask multiple directories. While this optimization works well when masking more than one path, it introduces unnecessary overhead when only a single directory needs to be masked.
This change restores the original behavior for single-path masking while preserving the shared tmpfs logic for multiple paths.