Documentation request: behavior of read_*_chunked when number of rows exceeds maximum integer value

[timothy-barry]

Hello,

Thank you for the response to my previous issue.

Can I safely use read_*_chunked when the number of rows in the file exceeds R's maximum integer value of about 2 billion? I will be reading fewer than 2 billion rows per chunk. Moreover, I will not use the index or "pos" argument in the callback functions.

[jimhester]

Reading the file should 'work', though pos will wrap around.

It is going to be very slow however.

I generated a file with 4 billion lines of just '1'` and the read it with

read_lines_raw_chunked("out", function(x, pos) print(pos), chunk_size = 10000000, progress=F))

It took over an hour to get to 2 Billion rows read and start wrapping. As this is about as simple as you can get reading more complex data and doing anything useful with it seems like it would take too long to be practical.

I think you would probably be better off looking into other tools more suited to handle data of this size.

[timothy-barry]

Thanks Jim. I've noticed that read_*_chunked is slow when printing pos within the callback function. I wonder if another callback function, such as function(x, pos) return(10), would be faster?

EDIT: I'll give this a try and get back.

[timothy-barry]

Thank you. I'll give this a shot!