Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix slow file reading #1949

Merged
merged 7 commits into from
Aug 13, 2021
Merged

Fix slow file reading #1949

merged 7 commits into from
Aug 13, 2021

Conversation

kustosz
Copy link
Contributor

@kustosz kustosz commented Aug 12, 2021

Pull Request Description

There was unnecessary data copying happening when reading a file. There isn't anymore.
From the numbers reported in #1724 it brings the times down to ~1s. This is exactly the same as reading a file in pure Java, with a FileInputStream, which is pretty much the best time we can get on the JVM.

Fixes #1724

Important Notes

Checklist

Please include the following checklist in your PR:

  • The documentation has been updated if necessary.
  • All code conforms to the Scala, Java, and Rust style guides.
  • All documentation and configuration conforms to the markdown and YAML style guides.
  • All code has been tested where possible.

@kustosz kustosz added Type: Bug p-highest Should be completed ASAP labels Aug 12, 2021
@kustosz kustosz self-assigned this Aug 12, 2021
@kustosz kustosz merged commit bd04f8c into main Aug 13, 2021
@kustosz kustosz deleted the wip/mk/fix_slow_file branch August 13, 2021 14:16
iamrecursion pushed a commit that referenced this pull request Aug 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
p-highest Should be completed ASAP
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Extremely slow loading of big files
2 participants