CI error: Unexpected segmentation fault encountered in worker #459
Description
🐛 Bug
A non-deterministic CI bug has recently started coming on mac (github action), where tests/streaming/test_dataset.py::test_dataset_resume_on_future_chunks[True] fails with segfault.
Check action: https://github.com/Lightning-AI/litdata/actions/runs/13076309775/job/36489260607?pr=443
https://github.com/Lightning-AI/litdata/actions/runs/12948719378/job/36117940606
Additional context
logs:
Progress: 25%|██▌ | 2/8 [00:13<00:39, 6.57s/it]
Progress: 100%|██████████| 8/8 [00:13<00:00, 1.73s/it]
/Users/runner/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/lightning_sdk/helpers.py:43: UserWarning: A newer version of lightning-sdk is available (0.1.51). Please consider upgrading with `pip install -U lightning-sdk`. Not all functionalities of the platform can be guaranteed to work with the current version.
warnings.warn(
/Users/runner/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/lightning_sdk/helpers.py:43: UserWarning: A newer version of lightning-sdk is available (0.1.51). Please consider upgrading with `pip install -U lightning-sdk`. Not all functionalities of the platform can be guaranteed to work with the current version.
warnings.warn(
ERROR: Unexpected segmentation fault encountered in worker.
/Users/runner/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/lightning_sdk/helpers.py:43: UserWarning: A newer version of lightning-sdk is available (0.1.51). Please consider upgrading with `pip install -U lightning-sdk`. Not all functionalities of the platform can be guaranteed to work with the current version.
warnings.warn(
/Users/runner/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/lightning_sdk/helpers.py:43: UserWarning: A newer version of lightning-sdk is available (0.1.51). Please consider upgrading with `pip install -U lightning-sdk`. Not all functionalities of the platform can be guaranteed to work with the current version.
warnings.warn(
ERROR: Unexpected segmentation fault encountered in worker.
=========================== short test summary info ============================
FAILED tests/streaming/test_dataset.py::test_dataset_resume_on_future_chunks[True] - RuntimeError: DataLoader worker (pid(s) 24332) exited unexpectedly
===== 1 failed, 213 passed, 11 skipped, 20 warnings in 1317.66s (0:21:57) ======
Error: Process completed with exit code 1.