fwrite(): final items

[mattdowle]

• malloc()s and write() need a thread safe way to error() if fail
• Add progress % and ETA to appear after 2 seconds if more than 2 seconds are remaining
  Include R_CheckUserInterrupt and test closes worker team ok 1a4263f
• Date, IDate and update this quesion
• POSIXct (see fread/fwrite *base data types* directly for efficiency #1656)
• ITime
• integer64
  Add "NEW:" item to startup banner
• Confirm fwrite() writes 10GB ok on Windows (it should do) to ensure 'big' file > 4GB ok. Thanks to Hugh Parsonage for testing as we don't have Windows other than via AppVeyor for test suite.
• quote = 'auto'
• sep2 (see fwrite to support secondary separator (sep2) for vectors in list columns #806)
• match write.csv's scientific/decimal format exactly and add many tests. 6c1ed96
• add dec='.' to R level and connect to already existing option at C level
• add row.names for data.frames, default FALSE.
• rename setthreads to setDTthreads so as not to affect other packages using OpenMP, add reference to its manual in fwrite man
• refine ?fwrite
• more tests qmethod= 'double' and 'escape'

[eantonya]

Do people actually like having quote=TRUE when writing to csv? I find it to be a big nuisance and would much prefer for fwrite to have quote=FALSE by default.

[MichaelChirico]

I find quote = TRUE to be more robust -- you never know when you have a JAMES SMITH, JR in a character column and it can be a huge pain to get a .csv read when it has nuisance commas strewn about.

[mattdowle]

@eantonya Agree. I prefer quote=FALSE too. The base R thinking I believe has numbers/ids with leading 0's stored as character format ... the default ensures they get read by Excel as character and the leading 0's not lost. But fwrite could detect that situation and quote just that situation by default. Where character columns contain letters and no embedded quotes, I really don't see why quotes are needed. Plus we save a bit on file size by saving the 2 extra quotes per field.

[mattdowle]

@MichaelChirico Agree with you too. fwrite can detect that and put the quotes in those situations. fwrite already does a first-pass through all strings to calculate maximum line length before allocating buffer sizes. It could test if there are any sep or quote in the string at that point. So I guess I'm suggesting quote='auto' by default.

[MichaelChirico]

@mattdowle great, good point. Should only marginally affect speed then.

PS IIRC Excel converts "001" to 1 anyway :|

[mattdowle]

@MichaelChirico Now you mention it I do seem to remember Excel doing that. I haven't used Excel for many years now thankfully.

[jangorecki]

I would assume Excel behave in an inconsistent (os versions, office versions, os locales, office localces, 365s, etc.) way about that matter.

[rafapereirabr]

Are you planning to include the append = T ? Please?! Anyway, congrats for the great job with data.table that will become even greater with fwrite() !

[MichaelChirico]

@rafapereirabr? append = TRUE in fact works for me, are you suggesting that should be the default?

[rafapereirabr]

@MichaelChirico , I didn't know it was already implemented ! I couldn't try it as I was planning to test it tonight. Just ignore my comment then. ps. I don't think this should be the default.