[WIP] Add an OS abstraction layer for filesystem and subprocess access

[NouemanKHAL]

What does this PR do?

Adds an OS abstraction layer in datadog_checks_base that integrations use instead of calling open, os, shutil, glob, and subprocess directly, so path-based security validation can attach in one place. Migrates the integrations that perform direct I/O, adds a ddev validate os-interface guard, and includes the design RFC.

Behavior is unchanged by default. Point-of-use enforcement is off unless an operator opts in.

Core interface (datadog_checks_base/datadog_checks/base/utils/os_interface.py)

• Each method is a thin passthrough preceded by a validator hook. Under the default no-op validator, exception types and timing, permission bits, encodings, laziness, and return values match the call being replaced.
• get_subprocess_output stays its own operation rather than being folded into run, so its output decoding, empty-output handling, and logging are preserved exactly.
• No new configuration. Validation is gated by the existing integration_ignore_untrusted_file_params setting, the same switch that already governs config-field validation, and never applies to a trusted provider or an excluded check.
• Executable validation covers every program a command launches, not just argv[0]: sudo is unwrapped, and under shell=True the shell is what gets validated since that is what the OS actually runs. Bare command names are resolved through PATH.

Migration

Integrations now use self.os_interface. Module-level helpers take the interface as a required parameter, because the module-level singleton is bound to the no-op validator and passing it there would preserve parity and pass every test while enforcing nothing.

Paths handed to libraries that open them themselves are validated at the handoff, via a validate_path operation that validates and returns the value unchanged. This covers the shared TLS context builder in datadog_checks_base (tls_ca_cert, tls_cert, tls_private_key), which is how most integrations reach TLS, plus http_check, esxi, foundationdb, and duckdb. vsphere has the same exposure but VSphereAPI has no check reference and its constructor is used across many tests, so it is called out as follow-up rather than half-threaded here.

Erosion guard

ddev validate os-interface flags raw stdlib I/O and use of the non-enforcing singleton in check modules. Detection is AST-based, so a method named open or a mention inside a docstring is not misreported, and from os import scandir style imports are resolved back to the call they bind. Narrow legitimate cases use an inline # SKIP_OS_INTERFACE_VALIDATION comment. The repository is currently clean: 410 passed, 0 errors.

Adding import tracking immediately paid for itself: it surfaced directory, whose entire purpose is scanning a user-configured path and which was reaching the filesystem completely unmediated via from os import scandir / from os.path import exists. That is the clearest path-traversal case in the repository, and dotted-name matching alone had missed it. It is migrated here.

Note for reviewers: the main operational risk. Because validation reuses the existing setting rather than adding one, it cannot be staged separately from field validation. An operator who already has integration_ignore_untrusted_file_params enabled will begin enforcing at every migrated call site as soon as this ships, and there is no dry-run mode in which violations are reported without being blocked. Rollout therefore depends on the excluded-checks setting and on migrating in batches. This was a deliberate choice to avoid adding operator-facing configuration; it is called out in the RFC and the developer docs, and is the thing most worth a second opinion.

Motivation

Integrations accept inputs that are paths: a file to read, a bin_dir, a path to a binary the check executes. The trusted-provider mechanism already decides whether such an input is acceptable, but it applies that decision to config fields at load time. It does not govern the operation performed later, nor paths derived at runtime.

The exec case is the sharper one and is not hypothetical. slurm executes binaries resolved from a configurable slurm_binaries_dir and per-command *_path options, ceph runs a configurable ceph_cmd, glusterfs runs a gstatus_path, and gunicorn runs a configurable gunicorn binary. Several wrap the configured binary in sudo, so the attacker-chosen program is not even the first element of the command.

This is a mediation layer for direct standard-library I/O, not a containment boundary. A Python-level wrapper cannot intercept a path opened inside a third-party library, anything a subprocess does once launched, or what a shell string executes. Those limits are documented rather than implied. See rfc-filesystem-abstraction-layer.md.

Testing

• datadog_checks_base: 185 unit tests pass; the OS interface suite is 86 tests, 177 counting the test double. The interface module is at 100% line coverage, re-measured after the final change rather than claimed once.
• Enforcement is verified end-to-end, not just at the unit level: slurm, gunicorn, esxi, foundationdb, and directory have tests asserting that a disallowed binary, certificate path, or traversal root is never launched, handed to the library, or read, exercising the real check and its real config parsing.
• A registry test asserts every public interface method consults the validator, so a new method cannot be added without enforcement coverage.
• The new tests were mutation-tested: reverting each behavior in the source (dropping a validator hook, dropping shell=True handling, reverting to argv[0]-only validation, reverting an integration to the non-enforcing singleton, reverting directory to unmediated stdlib access, adding an unguarded public method) makes the relevant test fail. Source was restored after each.
• ddev: the validate suite passes, including 15 tests for the new validation.
• ddev test --lint clean across all 43 changed packages.
• Unit sweep across all 43 changed packages: zero failures locally. Several suites could not run in the authoring environment for reasons unrelated to this change (Docker Desktop VM disk exhaustion, cacti needing rrd.h, btrfs being Linux-only, foundationdb needing the native libfdb_c); CI has since run all of them on ubuntu-22.04 and they pass.
• ddev docs build --check passes.

Several defects in this work were caught by its own tooling or by CI rather than by review, and are worth flagging since each was an invisible failure mode:

• The test double did not intercept glob, so a fixture-based test would have silently read the real filesystem. Fixed, and there is now a drift guard asserting the double covers the full interface surface.
• .ddev/config.toml pins an explicit validation list, so the erosion guard would never have run in CI despite being registered.
• resolve_path normalized paths at library handoffs, turning a relative path absolute and breaking the parity requirement. The pre-existing test_tls.py caught it; hence validate_path.
• refresh_tls_context rebuilt the TLS context without the validator, skipping validation on refresh.
• Three new tests assumed POSIX semantics (mode bits, PATHEXT) and failed the Windows job; they now exercise both platforms rather than being skipped.
• The test double let the platform separator into a key space callers populate with forward slashes: walk re-joined children with os.path.join (/root\sub vs the registered /root/sub) and glob measured depth with os.sep. Rewriting glob also fixed a platform-independent semantic bug, since fnmatch cannot express ** matching zero or more segments. A double that under-matches makes tests pass for the wrong reason.

Those last defects were only reachable through the Windows CI job, which is a slow loop for something reproducible in-process, so there is now a fixture that points os.path at ntpath and reproduces them on any host.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

🤖 Generated with Claude Code

[datadog-prod-us1-6]

 

🛑 Gate Violations

🎯 1 Code Coverage issue detected

A Total coverage percentage gate may be blocking this PR.

• Overall coverage for service kube_apiserver_metrics: 93.28% (threshold: 75.00%)
• Overall coverage for service torchserve: 99.49% (threshold: 75.00%)
• Overall coverage for service lparstats: 75.45% (threshold: 75.00%)
• and 197 more

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 95.95%
• Overall Coverage: 88.53% (+0.14%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: a89a1f7 | Docs | Datadog PR Page | Give us feedback!}

[dd-octo-sts]

Validation Report

All 22 validations passed.

Show details

Validation	Description	Status
agent-reqs	Verify check versions match the Agent requirements file	✅
ci	Validate CI configuration and code coverage settings	✅
codeowners	Validate every integration has a CODEOWNERS entry	✅
config	Validate default configuration files against spec.yaml	✅
dep	Verify dependency pins are consistent and Agent-compatible	✅
http	Validate integrations use the HTTP wrapper correctly	✅
imports	Validate check imports do not use deprecated modules	✅
integration-style	Validate check code style conventions	✅
jmx-metrics	Validate JMX metrics definition files and config	✅
labeler	Validate PR labeler config matches integration directories	✅
legacy-signature	Validate no integration uses the legacy Agent check signature	✅
license-headers	Validate Python files have proper license headers	✅
licenses	Validate third-party license attribution list	✅
metadata	Validate metadata.csv metric definitions	✅
models	Validate configuration data models match spec.yaml	✅
openmetrics	Validate OpenMetrics integrations disable the metric limit	✅
os-interface	Validate integrations use the validated OS interface for file and subprocess access	✅
package	Validate Python package metadata and naming	✅
qa-label	Validate the pull request declares whether it needs QA for the next Agent release	✅
readmes	Validate README files have required sections	✅
saved-views	Validate saved view JSON file structure and fields	✅
version	Validate version consistency between package and changelog	✅

View full run

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a89a1f78c8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Pass the check-bound interface when reading SNMP profiles

definition_file is taken from SNMP init_config['profiles'], but this helper now opens the resolved file through the module-level os_interface singleton. When OS-interface enforcement is enabled for a provider outside the trusted list, a configured profile path or an extends entry can still be read with the no-op validator, so the new point-of-use protection is bypassed; thread the check-bound self.os_interface into the profile-loading helpers instead.

AGENTS.md reference: AGENTS.md:L84-L86

Useful? React with 👍 / 👎.

[NouemanKHAL]

Superseded by a stack of three PRs, split by package:

1. Add ddev validate os-interface #24781 — ddev validate os-interface (the erosion guard)
2. Add a validated OS interface for filesystem and subprocess access #24782 — datadog_checks_base (the interface itself)
3. Route integration filesystem and subprocess access through the OS interface #24783 — the integrations migration, which also switches the guard on

The guard is deliberately not selected in .ddev/config.toml until the last PR, so every PR in the stack is green on its own rather than the first one failing against unmigrated integrations.

Closing this one.