Think about using C API to increase read speed.

[jefferis]

TN003IgorBinaryFormat provides a simple C interface for reading/writing Igor binary files. This could be worth considering, perhaps as an optional extension. In general this probably isn't too big issue, as the some simple experiments suggest that most of the time is spent reading the file rather than processing it. However R's readBin is not that speedy, so there could be room for improvement.

As an example, two 38 Mb files read from network storage

> system.time(w10 <- read.pxp("/Volumes/JData5/JPeople/Shahar/Data/160201/nm20160201c0/nm20160201c0_010.pxp"))
   user  system elapsed 
  0.744   0.166   1.333 

> f='/Volumes/JData5/JPeople/Shahar/Data/160201/nm20160201c0/nm20160201c0_011.pxp'
> system.time(rb <- readBin(f, what='raw', n=file.info(f)$size))
   user  system elapsed 
  0.002   0.069   0.519 
> system.time(rb <- readBin(f, what='raw', n=file.info(f)$size))
   user  system elapsed 
  0.001   0.027   0.033 

> system.time(w11 <- read.pxp("/Volumes/JData5/JPeople/Shahar/Data/160201/nm20160201c0/nm20160201c0_011.pxp"))
   user  system elapsed 
  0.788   0.150   0.956 

> system.time(w10 <- read.pxp("/Volumes/JData5/JPeople/Shahar/Data/160201/nm20160201c0/nm20160201c0_010.pxp"))
   user  system elapsed 
  0.774   0.131   0.908 

So the the raw read time (which probably isn't much slower than a C implementation) is about 40% of the total file parsing time. So that (2.5x) is the kind of speedup I might imagine achieving.

For the same files coming from the built-in flash memory on a macbook

> system.time(w10 <- read.pxp("~/dev/Igor/nm20160201c0/nm20160201c0_010.pxp"))
   user  system elapsed 
  0.778   0.093   0.876 
> system.time(w10 <- read.pxp("~/dev/Igor/nm20160201c0/nm20160201c0_010.pxp"))
   user  system elapsed 
  0.754   0.083   0.840 
> f='~/dev/Igor/nm20160201c0/nm20160201c0_011.pxp'
> system.time(rb <- readBin(f, what='raw', n=file.info(f)$size))
   user  system elapsed 
  0.002   0.039   0.116 

so here I could imagine as much as an 8-fold speed improvement.

[t-b]

@jefferis
I just wanted to point out that, according to the latest version of TN003IgorBinaryFormat, that the 64bit version of Igor might create large waves in a non documented file format. I guess this would not be a regression if you switch to the C API, as I don't expect the current code to handle these type of waves.