Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for writing to gzip (.gz) with fwrite #2016

Closed
MichaelChirico opened this issue Feb 5, 2017 · 5 comments · Fixed by #3288
Closed

Support for writing to gzip (.gz) with fwrite #2016

MichaelChirico opened this issue Feb 5, 2017 · 5 comments · Fixed by #3288

Comments

@MichaelChirico
Copy link
Member

@MichaelChirico MichaelChirico commented Feb 5, 2017

write.csv can handle the following:

library(data.table)
DT = data.table(a = 1:10, b = 11:20)

tmp.gz = tempfile(fileext = '.csv.gz')
gzf = gzfile(tmp.gz, 'w')
write.csv(DT, gzf)
close(gzf)

But fwrite cannot:

tmp.gz = tempfile(fileext = '.csv.gz')
gzf = gzfile(tmp.gz, 'w')
fwrite(DT, gzf)
close(gzf)

Error: is.character(file) && length(file) == 1 && !is.na(file) is not TRUE

(because the gzf object is more complicated than just being a file path).

fwrite(DT, tmp.gz), as expected, does not perform compression -- the output is the same size as fwrite(DT, tempfile()) (which is different from write.csv).

I'm using fwrite to pipe some output to Vowpal Wabbit, trying to use the --compressed argument to save on some disk space. Workaround is to fwrite then gzip, but this seems sub-optimal.

@JhossePaul
Copy link

@JhossePaul JhossePaul commented Feb 28, 2017

+1

1 similar comment
@rargelaguet
Copy link

@rargelaguet rargelaguet commented Mar 13, 2017

+1

@msgoussi
Copy link

@msgoussi msgoussi commented May 3, 2017

Dear Mr.
When I export data.table using fwrite, the size of the file is 31.3GB
then when I use gzip in order to compress the file, I get:
Error in writeBin(bfr, con = out, size = 1L) :
'Calloc' could not allocate memory (10000000 of 1 bytes)

How can I solve this problem?
Thanks

@Griefberg
Copy link

@Griefberg Griefberg commented May 31, 2018

+1

@wookietreiber
Copy link

@wookietreiber wookietreiber commented May 31, 2018

Please don't post comments with simply "+1". This adds absolutely no value. Either write something of value that adds something to the discussion or use the thumbs up emoticon / reaction on the original post to express interest in the issue and a solution to it.

I'm in favor of deleting such comments, including mine BTW, because it is of no consequence to the technical discussion of this issue as writing "+1". Please, people, please ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants