Fix Datadog agent for SkyPilot training jobs #3955
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes agent startup in Docker containers by checking for binary instead of systemd service, and adds automatic startup during run phase.
Fixes script exit on pgrep failure and adds better error messages
- Add metta_run_id, skypilot_task_id, and other tags to all log entries - Enables filtering logs by run ID in Datadog Logs Explorer - Fixes issue where logs weren't searchable by metta_run_id tag
- Remove wildcard paths (use explicit file paths only) - Create empty log files during setup so agent can collect immediately - Use unbuffered output (stdbuf) to ensure logs are written in real-time - Add programmatic log checking script (check_datadog_logs.py) - Fix tag formatting in log collection config
- Agent now always restarts to ensure logs_enabled and log collection config are loaded - This ensures the agent picks up config changes made during setup phase
- Add logs section to main datadog.yaml in addition to separate config file - Change custom_logs.d to custom_logs directory (standard format) - This ensures logs are collected even if separate config isn't picked up
- Change directory from custom_logs to skypilot_training.d (proper .d format) - Add logs_config section with auto_multi_line_detection and force_use_http - This follows Datadog documentation best practices for log collection
- Remove duplicate logs section from main datadog.yaml (logs come from conf.d files) - Add detailed verification in startup script to check: - Agent status and log collection status - Log collection config file existence - Training log file existence and size - This helps debug why logs aren't appearing in Datadog
- Show config file contents when found - Run agent configcheck to verify agent sees the config - List available config files if our config is missing - This helps debug why logs aren't being collected
- Set proper file permissions (644) on log config file - Log config file size for verification - Validate config contains required fields - This helps debug why agent isn't collecting logs
This ensures the Datadog agent user (dd-agent) can read logs generated by the training process user. - Set /tmp/training_logs to 777 - Set log files to 666
- Temporarily disable pipefail when starting training to avoid silent failures - Set PYTHONUNBUFFERED=1 for real-time logging - Add debugging to check if training produces output after 5s
Delays the Datadog agent startup to allow the training process (Ray/Torchrun) to fully initialize and bind ports. This prevents resource contention or conflicts that were causing the training process to hang during startup.
added 3 commits
December 6, 2025 17:08
….sh just tees to it. both attend to an env var established by launch.py
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
nishu-builder
approved these changes
Dec 7, 2025
nishu-builder
changed the title
zfogg
pushed a commit
that referenced
this pull request
Dec 20, 2025
Gets training logs from SkyPilot jobs into Datadog. The key issue was that SkyPilot's subprocess daemon recursively kills all child processes when a phase ends. Starting the DD agent in the setup phase meant it was killed before training started. the fix is to move DD agent setup from setup phase to run phase (run.sh), so it stays alive during training. run.sh also `tee`s stdout to a log file for DD to tail, as determined by an env var we pass from launch.py --------- Co-authored-by: Nishu <133812901+nishu-builder@users.noreply.github.com> Co-authored-by: Nishad <nishad@stem.ai> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Gets training logs from SkyPilot jobs into Datadog.
The key issue was that SkyPilot's subprocess daemon recursively kills all child processes when a phase ends. Starting the DD agent in the setup phase meant it was killed before training started.
the fix is to move DD agent setup from setup phase to run phase (run.sh), so it stays alive during training.
run.sh also
tees stdout to a log file for DD to tail, as determined by an env var we pass from launch.py