New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
vimc-6490: Save intermediate files out in qs format #21
Conversation
Codecov Report
@@ Coverage Diff @@
## master #21 +/- ##
===========================================
- Coverage 100.00% 99.75% -0.25%
===========================================
Files 19 19
Lines 2047 2063 +16
===========================================
+ Hits 2047 2058 +11
- Misses 0 5 +5
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. I vote for no csv
new_csv_file <- tempfile(fileext = ".csv") | ||
write.csv(x = result$cal, file = new_csv_file, | ||
row.names = FALSE) | ||
new_csv_file <- tempfile(fileext = ".qs") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
new_qs_file ?
Made a further update to support running stochastic upload with both qs files and csv files (for supporting re-uploading old outputs should we need to tear down annex at some point) |
R/stochastic_upload.R
Outdated
if (type == "csv") { | ||
read_stochastic_csv(file, is_cohort) | ||
} else if (type == "qs") { | ||
read <-read_stochastic_qs(file, is_cohort) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the read <-
intentional?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah no, good spot
So that read & writes are quicker
Also means we can remove some of the logic which checks row formatting when we read the file back in.
1 question about this - is the csv format useful? Will researchers ever get this file locally and look at it in excel? Or will they always load it into R to explore it? If the latter qs seems like a good choice, if the first maybe keeping the file as csv makes sense