Obtaining Quick File Info for Large Data Files Stored On Disk

[drag05]

I wonder if it would be possible to quickly obtain some info on larger-than-memory size data stored on disk, similar to, for example:

write.table(iris, 'test.csv', sep=",",row.names=FALSE,quote=FALSE)

info = function(file) {
            tot = callr::r(
                          function(file) {
                                                  fread(file)[,.N]
                             }, package = 'data.table', args = list(file = readpath)
        )
           nmes = names(fread(file = readpath, nrows = 0L))
cat('\n found', tot, 'rows and', length(nmes), 'columns named:\n\n', sQuote(nmes), '\n')
return(invisible(list(tot, nmes)))
}

info('test.csv')

 found 150 rows and 5 columns named:

 ‘Sepal.Length’ ‘Sepal.Width’ ‘Petal.Length’ ‘Petal.Width’ ‘Species’ 

system.time(info('test.csv'))

user  system elapsed 
   0.01    0.00    0.27 

but faster than that. Thank you!

[MichaelChirico]

It seems like in the vast majority of cases you are best off just using command-line tools:

# count the number of '\n' to estimate the # of rows
wc -l "${FILE}"

# show the first line
head -n1 "${FILE}"

# count the number of fields
head -n1 "${FILE}"| awk -F',' '{print NF}'

This should also be way faster than fread could ever hope to be:

info = function(f) {
  nrow = as.integer(unlist(strsplit(system2("wc", c("-l", f), stdout=TRUE), " ", fixed=TRUE))[1L]) - 1L
  fields = unlist(strsplit(system2("head", c("-n1", f), stdout=TRUE), ",", fixed=TRUE))
  writeLines(sprintf("%s rows and %d columns named:\n\n%s", nrow, length(fields), paste(sQuote(fields), collapse = " ")))
  invisible()
}

system.time(info('test.csv'))
# 150 rows and 5 columns named:

# ‘Sepal.Length’ ‘Sepal.Width’ ‘Petal.Length’ ‘Petal.Width’ ‘Species’
#    user  system elapsed 
#   0.008   0.020   0.028 

The advantage of using fread() would be that it adds logic to understand the file if the structure is not "perfectly rectangular", e.g. there are quoted fields with embedded newlines, or the header columns are quoted with embeded commas.

Have you tried fread(verbose=TRUE) on the file? Ideally, it will run the first few steps & report something usable before crashing due to lack of memory.

[drag05]

@MichaelChirico Thanks for quick response! The apparent gain in speed in your case, on the small example file is partly due to new process creation by compiler in my case.

I have tested both info() functions on a larger file and there is no sensible difference between them on my machine:

your suggested info():

found 1048575 rows and 7 columns named:

‘scan_num’ ‘charge’ ‘RT’ ‘mz’ ‘abundance’ ‘type’ ‘% abundance’

   user  system elapsed 
   0.00    0.00    0.44 

my original example info():

found 1048575 rows and 7 columns named:

 ‘scan_num’ ‘charge’ ‘RT’ ‘mz’ ‘abundance’ ‘type’ ‘% abundance’ 

   user  system elapsed 
   0.02    0.04    0.40 

Plus, my example info() returns (invisibly) nrow and names for further processing if needed:

sapply(info(), class)
[1] "integer"   "character"

[MichaelChirico]

I don't think a difference of 0.04 seconds is worth fretting over.

I repeated the exercise on an 8GiB file (~330M rows replication of iris) and found a different result

# my info()
329200600 rows and 5 columns named:

‘Sepal.Length’ ‘Sepal.Width’ ‘Petal.Length’ ‘Petal.Width’ ‘Species’
   user  system elapsed 
  0.220   1.438   1.660 

# your info(), after correcting the error 'readpath not found'
 found 329200600 rows and 5 columns named:

 ‘Sepal.Length’ ‘Sepal.Width’ ‘Petal.Length’ ‘Petal.Width’ ‘Species’ 
   user  system elapsed 
 34.423   4.832  39.110 

It is not difficult to tweak my info() to provide the info you requested.