Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
CSV speed improvements and support for large CSV files #532
Read in CSVs line by line so that CSV files with more than
This is also saves more performant since we can preallocate
Fix limit on read_stan_csv (#530) without requiring a patch to base R.
Faster to read in CSV files even when they do not exceed the max size limit.
How to Verify:
Create large file via CMDStan and read in with read_stan_csv.
Copyright and Licensing
Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company):
By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:
My patch was applied to base R (wch/r-source@86f7ea9) so the root cause is fixed. However, this PR is still a speed improvement, and reduces the number of times the sample data frame must be copied in memory so you are not required to have 2X+ the available RAM to load in the sample file.
This was referenced
May 29, 2018
@aaronjg When I merged this, it was passing the tests, but a bunch of things got messed up subsequently. I tried to fix everything, but now it seems that the example and tests for
On Wed, Aug 8, 2018, 8:49 PM aaronjg ***@***.***> wrote: Sure. Where are the test cases that are failing? I looked at "./rstan/example/testreadcsv.R" but that seems to be somewhat out of date. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#532 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADOrqimrZ0eEI2MKCVLCmsH3lMxxjNodks5uO4cOgaJpZM4UNOeA> .
On Wed, Aug 8, 2018 at 10:20 PM aaronjg ***@***.***> wrote: There was an extra LF at the end of the example csv. I don't think Stan generates those CSVs with extra LF now, but I am handling it just in case. Fixed with commit 1c3236e <1c3236e> https://github.com/aaronjg/rstan/tree/for-2.18 — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#532 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADOrqsZTTCjy8_qxg33BffZiPH_dAFYEks5uO5yBgaJpZM4UNOeA> .