fix: optimize with StreamingDataloader(parquet data) for >=5 workers

[bhimrazy]

What does this PR do?

Fixes #599
Also resolves #482

Fixes data loss when using optimize() with StreamingDataLoader on parquet data with 5+ workers.

The solution is admittedly a bit hacky but effective: we increase the total virtual items and let workers process their iterators until they're naturally exhausted, rather than depending on rigid index.

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83%. Comparing base (d11b1c9) to head (f6e69f9).
Report is 1 commits behind head on main.

Additional details and impacted files

@@         Coverage Diff         @@
##           main   #616   +/-   ##
===================================
  Coverage    83%    83%           
===================================
  Files        49     49           
  Lines      6750   6756    +6     
===================================
+ Hits       5604   5636   +32     
+ Misses     1146   1120   -26     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[bhimrazy]

Okay, so now I’ve got the minimal test for this PR. The test is working as expected, and it failed on the CI machines.

Now, let’s add the fixes back.

[bhimrazy]

Okay, it looks like the test is now working as expected.
The real issue seems to be that the item count becomes unequal — which confirms that an item is indeed getting skipped.

And also , this only appears to happen with Parquet datasets.

[Borda]

Okay, it looks like the test is now working as expected.

Nice, will you also add a fix for it?

[bhimrazy]

Nice, will you also add a fix for it?

Yes, @Borda — I’m still investigating the root cause. So far, I haven’t found a reliable fix.

[tchaton]

Nice !