Support incremental array writing from data frame object#251
Merged
eddelbuettel merged 7 commits intomasterfrom May 25, 2021
Merged
Support incremental array writing from data frame object#251eddelbuettel merged 7 commits intomasterfrom
eddelbuettel merged 7 commits intomasterfrom
Conversation
aaronwolen
requested changes
May 25, 2021
aaronwolen
approved these changes
May 25, 2021
Member
aaronwolen
left a comment
aaronwolen
left a comment
There was a problem hiding this comment.
Cool! This is a nice feature.
Contributor
Author
|
Yup. Allowing 'larger-than-memory' sized arrays to be written from 'compatible-with-memory-size' chunks is neat. And I feel like a doofus that I didn't catch the error. I will do the usual and blame Azure for being down yesterday (but I should have caught GitHub Actions) -- so thanks again for catching that. Will wait for this round to run and then merge. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR supports writing larger-than-memory arrays incrementally. To do so, it permits to set dimension domain ranges as a new optional argument to
fromDataFrame(). This allows e.g. years to be set to [1970, 2021], say, rather than to the range in submitted chunk. (String dimensions already have null so no issue there.) Additionally, we ensure that the to-be-appended data.frame is re-ordered such that dimensions precede attributes after which the existing writing code "just works".A new unit tests has been added as well.
(The tests were all fine at GH but some timed out when Azure Pipes had the flu earlier today leading to the temporary red marks. They will pass on re-run, and we can wait if need be.)