Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-14653: [R] head() hangs on CSV datasets > 600MB #11992

Closed
wants to merge 2 commits into from

Conversation

thisisnic
Copy link
Member

This PR switches to using the asynchronous scanner by default when reading in datasets. I've tested it locally on a large dataset (2.5Gb of CSV files) and it does resolve the original issue, but due to the size of the files involved I wasn't sure this was something I could easily write tests for.

@github-actions
Copy link

@github-actions
Copy link

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

@thisisnic
Copy link
Member Author

@ursabot please benchmark lang=r

Copy link
Member

@westonpace westonpace left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will hopefully be superseded by #11991 but it can't hurt to have in the meantime.

@jonkeane
Copy link
Member

@ursabot please benchmark lang=R

@jonkeane
Copy link
Member

@ursabot please benchmark

@thisisnic thisisnic closed this in 762fad5 Jan 3, 2022
@ursabot
Copy link

ursabot commented Jan 3, 2022

Benchmark runs are scheduled for baseline = cb1897e and contender = 762fad5. 762fad5 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️1.35% ⬆️0.0%] ursa-i9-9960x
[Failed] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants