Better support for streaming optimized dataset

[robTheBuildr]

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

What does this PR do?

This PR improves the reliability of the streaming. More work to be down to fully get to the bottom of occasional slowdown

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[codecov]

Codecov Report

❌ Patch coverage is 58.43373% with 69 lines in your changes missing coverage. Please review.
✅ Project coverage is 84%. Comparing base (df92bf8) to head (8009c3e).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@         Coverage Diff         @@
##           main   #727   +/-   ##
===================================
- Coverage    84%    84%   -1%     
===================================
  Files        52     52           
  Lines      7278   7328   +50     
===================================
+ Hits       6126   6129    +3     
- Misses     1152   1199   +47     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.