Skip to content

Conversation

@philgzl
Copy link
Contributor

@philgzl philgzl commented Jul 4, 2025

Before submitting
  • Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?

What does this PR do?

Fixes #649.

  • Automatic state loading and saving is now done at the beginning of iter() instead of at the end.
  • Automatic state loading and saving is now controlled by a new option resume.
  • Refactored ParallelStreamingDataset tests a bit.

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

@codecov
Copy link

codecov bot commented Jul 4, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83%. Comparing base (1c51664) to head (1360439).
Report is 1 commits behind head on main.

Additional details and impacted files
@@         Coverage Diff         @@
##           main   #650   +/-   ##
===================================
  Coverage    83%    83%           
===================================
  Files        49     49           
  Lines      6756   6758    +2     
===================================
+ Hits       5636   5639    +3     
+ Misses     1120   1119    -1     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bhimrazy bhimrazy self-requested a review July 8, 2025 07:20
Copy link
Collaborator

@tchaton tchaton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

real nice

@tchaton tchaton merged commit 466341c into Lightning-AI:main Jul 8, 2025
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ParallelStreamingDataset does not resume when breaking early

3 participants