SlurmClusterExecutor - resubmission of crashed jobs #699
Conversation
|
""" WalkthroughA file existence check and removal step was added before dumping new HDF5 input files for tasks, ensuring old files are deleted to prevent conflicts. Corresponding tests were introduced to verify executor behavior when cache files already exist or are corrupted, including checks for correct file counts in the cache directory after executions. Changes
Sequence Diagram(s)sequenceDiagram
participant Test as Test Suite
participant Executor as FluxClusterExecutor
participant FS as File System
Test->>Executor: Submit task with input
Executor->>FS: Check if .h5 file exists
alt File exists
Executor->>FS: Remove existing .h5 file
end
Executor->>FS: Dump new .h5 input file
Executor->>FS: Execute task and manage cache
Test->>FS: Check cache directory file count
Possibly related PRs
Poem
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 Pylint (3.3.7)executorlib/task_scheduler/file/shared.pytests/test_fluxclusterexecutor.py✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #699 +/- ##
=======================================
Coverage 96.89% 96.89%
=======================================
Files 29 29
Lines 1319 1321 +2
=======================================
+ Hits 1278 1280 +2
Misses 41 41 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
tests/test_fluxclusterexecutor.py (1)
72-72: Fix whitespace formatting issue.Remove the extra space before the closing parenthesis as flagged by static analysis.
- file_path = os.path.join("executorlib_cache", file_name ) + file_path = os.path.join("executorlib_cache", file_name)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
executorlib/task_scheduler/file/shared.py(1 hunks)tests/test_fluxclusterexecutor.py(1 hunks)
🧰 Additional context used
🪛 Flake8 (7.2.0)
tests/test_fluxclusterexecutor.py
[error] 72-72: whitespace before ')'
(E202)
⏰ Context from checks skipped due to timeout of 90000ms (14)
- GitHub Check: unittest_openmpi (ubuntu-latest, 3.13)
- GitHub Check: unittest_openmpi (ubuntu-latest, 3.11)
- GitHub Check: unittest_openmpi (macos-latest, 3.13)
- GitHub Check: unittest_mpich (ubuntu-latest, 3.11)
- GitHub Check: unittest_openmpi (ubuntu-latest, 3.12)
- GitHub Check: benchmark (ubuntu-latest, 3.13, .ci_support/environment-mpich.yml)
- GitHub Check: benchmark (ubuntu-latest, 3.13, .ci_support/environment-openmpi.yml)
- GitHub Check: unittest_mpich (ubuntu-latest, 3.13)
- GitHub Check: unittest_mpich (macos-latest, 3.13)
- GitHub Check: unittest_win
- GitHub Check: notebooks
- GitHub Check: unittest_flux_mpich
- GitHub Check: unittest_flux_openmpi
- GitHub Check: notebooks_integration
🔇 Additional comments (3)
executorlib/task_scheduler/file/shared.py (1)
119-120: LGTM! Excellent defensive programming approach.The file existence check and removal before dumping new data effectively addresses the ValueError issue mentioned in the PR objectives. This ensures a clean state when recreating job files during resubmission of crashed jobs.
tests/test_fluxclusterexecutor.py (2)
44-44: Good addition of cache directory file count verification.Adding the assertion to check the number of files in the cache directory helps ensure consistent behavior across test runs.
47-88: Excellent comprehensive test coverage for existing files scenario.The test effectively validates the file removal logic by:
- Running initial execution to populate cache
- Re-running with existing cache files
- Creating corrupted .h5 files and verifying robust handling
- Ensuring consistent file count throughout
This directly supports the changes made in
shared.pyand provides confidence in the crash job resubmission functionality.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
tests/test_fluxclusterexecutor.py (1)
61-91: Excellent test coverage for existing file handling with minor formatting fix needed.This test method effectively validates the fix mentioned in the PR objectives by:
- Creating a scenario where cache files exist from a previous execution
- Simulating corrupted/existing input files by creating dummy
_i.h5files- Verifying that the executor can still function correctly when these files exist
The test logic is sound and directly addresses the issue where resubmission of crashed jobs could fail due to existing files.
Fix the whitespace issue flagged by static analysis:
- file_path = os.path.join("executorlib_cache", file_name ) + file_path = os.path.join("executorlib_cache", file_name)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
executorlib/task_scheduler/file/shared.py(1 hunks)tests/test_fluxclusterexecutor.py(2 hunks)
✅ Files skipped from review due to trivial changes (1)
- executorlib/task_scheduler/file/shared.py
🧰 Additional context used
🪛 Flake8 (7.2.0)
tests/test_fluxclusterexecutor.py
[error] 74-74: whitespace before ')'
(E202)
🔇 Additional comments (1)
tests/test_fluxclusterexecutor.py (1)
11-11: LGTM: Import addition supports new test functionality.The
dumpimport is correctly added to support the new test method that creates dummy HDF5 input files.
Before a
ValueErrorwas raised when the job file was recreated, now the previous file is removed this guarantees that the resources are correctly updated.Summary by CodeRabbit
Bug Fixes
Tests