Fix AutoMate SoftDTW import fallback#6039
Conversation
There was a problem hiding this comment.
🤖 Isaac Lab Review Bot
PR #6039 — Fix AutoMate SoftDTW import fallback
Summary
This PR adds a graceful fallback when numba.cuda fails to import, allowing the AutoMate SoftDTW module to still load and fall back to the CPU implementation. The approach is sound and addresses a real import failure scenario.
✅ Strengths
- Layered import handling — Separates
numba(jit/prange) andnumba.cudaimports, catching bothImportErrorandAttributeError. This correctly handles the reportedNPDatetimeattribute error. - Graceful degradation — When CUDA is unavailable,
SoftDTW(use_cuda=True)emits a one-timeRuntimeWarningand silently falls back to CPU instead of crashing. _identity_jitdesign — Correctly handles both bare@jitand@jit(...)decorator patterns.- Regression test — Uses
monkeypatchto simulate the failure without requiring a broken Numba install. Properly asserts the warning and fallback behavior. - Changelog fragment — Included per project conventions.
⚠️ Findings (2 minor)
-
CI:
ruff formatfailure — The pre-commit check failed because 1 file needs reformatting. Please runruff formaton the changed files and push the fix. -
Global mutable state for warning suppression —
_CUDA_FALLBACK_WARNEDusesglobalto ensure the warning fires only once. This works but consider usingwarnings.warnwith the same message text (Python\u2019s warning filter deduplicates by default viadefaultaction), which would eliminate the need for the global flag entirely. This is a style suggestion, not a blocker.
🏁 Verdict
Approve (with minor formatting fix needed)
The logic is correct and well-structured. The only blocking issue is the pre-commit formatting failure — once that\u2019s resolved, this is ready to merge.
Automated review by IsaacLab Review Bot • Commit: cdda369
Greptile SummaryThis PR makes the AutoMate
Confidence Score: 4/5Safe to merge; the fallback path is well-guarded and the CUDA code paths are unchanged when Numba is fully available. The only finding is that stacklevel=2 on the warning points to the internal forward() call rather than the user's call site — a minor usability gap that does not affect correctness or the fallback behavior itself. No files require special attention; soft_dtw_cuda.py is the only non-trivial change and its fallback logic is consistent throughout. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Module import] --> B{import numba jit/prange}
B -- success --> C[use real jit / prange]
B -- AttributeError / ImportError --> D[jit = _identity_jit\nprange = range]
C --> E{import numba.cuda}
D --> E
E -- success --> F[cuda = real numba.cuda\n_CUDA_IMPORT_ERROR = None]
E -- AttributeError / ImportError --> G[cuda = _UnavailableCuda\n_CUDA_IMPORT_ERROR = exc]
F --> H["@cuda.jit decorators applied\n(real CUDA kernels compiled)"]
G --> I["@cuda.jit decorators applied\n(identity no-op, plain Python fns)"]
H --> J[SoftDTW._get_func_dtw called]
I --> J
J --> K{use_cuda and\n_CUDA_IMPORT_ERROR?}
K -- Yes --> L[emit one-shot RuntimeWarning\nset use_cuda = False]
L --> M[return _SoftDTW.apply\nCPU path]
K -- No --> N{use_cuda and\nseq_len > 1024?}
N -- Yes --> O[print msg\nset use_cuda = False\nreturn _SoftDTW.apply]
N -- No, use_cuda=True --> P[return _SoftDTWCUDA.apply\nGPU path]
N -- No, use_cuda=False --> M
Reviews (1): Last reviewed commit: "Fix AutoMate SoftDTW import fallback" | Re-trigger Greptile |
| warnings.warn( | ||
| "SoftDTW CUDA backend is unavailable because numba.cuda failed to import " | ||
| f"({type(_CUDA_IMPORT_ERROR).__name__}: {_CUDA_IMPORT_ERROR}). " | ||
| "Falling back to CPU SoftDTW.", | ||
| RuntimeWarning, | ||
| stacklevel=2, | ||
| ) |
There was a problem hiding this comment.
stacklevel=2 makes the warning point at the internal forward() call inside this module rather than the user's code that invoked the criterion. The call stack is: user code → forward() → _get_func_dtw() → warnings.warn(). Using stacklevel=3 would surface the warning at the correct site in user code.
| warnings.warn( | |
| "SoftDTW CUDA backend is unavailable because numba.cuda failed to import " | |
| f"({type(_CUDA_IMPORT_ERROR).__name__}: {_CUDA_IMPORT_ERROR}). " | |
| "Falling back to CPU SoftDTW.", | |
| RuntimeWarning, | |
| stacklevel=2, | |
| ) | |
| warnings.warn( | |
| "SoftDTW CUDA backend is unavailable because numba.cuda failed to import " | |
| f"({type(_CUDA_IMPORT_ERROR).__name__}: {_CUDA_IMPORT_ERROR}). " | |
| "Falling back to CPU SoftDTW.", | |
| RuntimeWarning, | |
| stacklevel=3, | |
| ) |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
|
Closing this implementation. The runtime fallback was too broad; the root cause is the uv dependency override allowing an incompatible pre-release Numba/NumPy stack. Replacing with a narrower dependency-constraint fix. |
Summary
numba.cudabackend fails to import.SoftDTW(use_cuda=True)is requested but the CUDA backend is unavailable.numba.cuda.types.NPDatetimeimport failure.Reproduction / Verification
assembly_env.pyimportsSoftDTWfromsoft_dtw_cuda.py, which previously imported and decoratednumba.cudakernels at module import time.soft_dtw_cuda.pyimports successfully and a smallSoftDTW(use_cuda=True)call falls back to CPU.Tests
git diff --cached --checkast.parse/home/zhengyuz/Projects/IsaacLab.wt/develop/.venv/bin/python -m pytest source/isaaclab_tasks/test/contrib/test_automate_soft_dtw_cuda.py -q/home/zhengyuz/Projects/IsaacLab.wt/beta2-automate-collision-stack/.venv/bin/python -m pytest source/isaaclab_tasks/test/contrib/test_automate_soft_dtw_cuda.py -qbeta2-automate-collision-stack/.venv