NetCDF: Not a valid ID error when using CentOS with NetCDF library 4.6.2

[trivedi-group]

Hi,

I am using the code below on a CentOS node that is on computing cluster. It has installed and loaded R and required packages fine. It can also detect all my netCDF files in the location but as soon as it starts to create my xcmsSet, it fails with an error.

Loading required package: Biobase
Loading required package: BiocGenerics
Loading required package: parallel

Attaching package: ‘BiocGenerics’

The following objects are masked from ‘package:parallel’:

    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
    clusterExport, clusterMap, parApply, parCapply, parLapply,
    parLapplyLB, parRapply, parSapply, parSapplyLB

The following objects are masked from ‘package:stats’:

    IQR, mad, sd, var, xtabs

The following objects are masked from ‘package:base’:

    anyDuplicated, append, as.data.frame, basename, cbind, colMeans,
    colnames, colSums, dirname, do.call, duplicated, eval, evalq,
    Filter, Find, get, grep, grepl, intersect, is.unsorted, lapply,
    lengths, Map, mapply, match, mget, order, paste, pmax, pmax.int,
    pmin, pmin.int, Position, rank, rbind, Reduce, rowMeans, rownames,
    rowSums, sapply, setdiff, sort, table, tapply, union, unique,
    unsplit, which, which.max, which.min

Welcome to Bioconductor

    Vignettes contain introductory material; view with
    'browseVignettes()'. To cite Bioconductor, see
    'citation("Biobase")', and for packages 'citation("pkgname")'.

Loading required package: BiocParallel
Loading required package: MSnbase
Loading required package: mzR
Loading required package: Rcpp
Loading required package: S4Vectors
Loading required package: stats4

Attaching package: ‘S4Vectors’

The following object is masked from ‘package:base’:

    expand.grid

Loading required package: ProtGenerics

This is MSnbase version 2.8.3 
  Visit https://lgatto.github.io/MSnbase/ to get started.


Attaching package: ‘MSnbase’

The following object is masked from ‘package:stats’:

    smooth

The following object is masked from ‘package:base’:

    trimws


This is xcms version 3.4.4 


Attaching package: ‘xcms’

The following object is masked from ‘package:stats’:

    sigma

Scanning files in directory /net/lustre/mbdssdt3/test ... found 5 files
Error in R_nc4_close: NetCDF: Not a valid ID

Script:

#Set working directory
setwd("~/scratch/test")

#Load packages
library(xcms)
library(CAMERA)
library(MSnbase)
library(BiocParallel)

##Parallel Processing##
register(bpstart(MulticoreParam(32)))

#xcms
xset0 <- xcmsSet(BPPARAM = bpparam())

I have tried the same files on MacOS and they are OK. I have narrowed it down to being how XCMS works on centOS but am unable to put a finger on what is causing that error. Internet is of no use on this error code on netCDF library. Is this a familiar scenario?

Some of the raw files can be accessed here - https://www.dropbox.com/sh/m6um4ofsmvgw7ct/AABDcbjI_tr54ruHOYqGrLXHa?dl=0

Edit: I've tried (inspired from similar issue here - #299) without parallel processing and with - nothing changes. Still get same error.

[trivedi-group]

Linking related issue that might help - I think the reported error 14 days ago might hint that the problem is related to changes that were merged? - sneumann/mzR#185 (comment)

[sneumann]

Hi, quick note, does it still fail if you use plain mzR ? openMSfile(). I guess so.
If you have a docker container with exact versions that fails that could be helpful. Yours, Steffen

[trivedi-group]

Hi, quick note, does it still fail if you use plain mzR ? openMSfile(). I guess so.
If you have a docker container with exact versions that fails that could be helpful. Yours, Steffen

Hi, 'openMSfile()' can handle the file. The output I get is:

Mass Spectrometry file handle.
Filename: BATCH1_QC1301.CDF
Number of scans: 11214

I've tried a few different files and they all seem to return no error.

[jorainer]

Can you please provide the output of your sessionInfo()?

[jorainer]

With the Bioconductor 3.8 release version of xcms (3.4.4) I can confirm the error:

library(xcms)
library(msdata)
fl <- system.file("cdf/ko15.CDF", package = "msdata")

raw <- xcmsRaw(fl)
Error in R_nc4_close: NetCDF: Not a valid ID
Create profile matrix with method 'bin' and step 1 ... OK

od <- readMSData(fl, mode = "onDisk")
sps <- spectra(od)

Since this does not happen with the new xcms3/MSnbase-way to process data it has to be something within the way we read data prior to xcms3. I'll have a look.

[jorainer]

I've fixed it in the master and RELEASE_3_8 branch of xcms. We were closing the netCDF file even if it was already closed.

You can install the fixed version (assuming you have R-3.5.2 and Bioconductor 3.8 packages) with

devtools::install_github("sneumann/xcms", ref = "RELEASE_3_8")

[trivedi-group]

Can you please provide the output of your sessionInfo()?

I am guessing you don't need this info any longer?

I've fixed it in the master and RELEASE_3_8 branch of xcms. We were closing the netCDF file even if it was already closed.

You can install the fixed version (assuming you have R-3.5.2 and Bioconductor 3.8 packages) with

devtools::install_github("sneumann/xcms", ref = "RELEASE_3_8")

Thanks a lot! I've downloaded newer version, will give it a whirl!

[jorainer]

Please also close the issue when you're OK with it (i.e. we've fixed your problem).

[trivedi-group]

I have a feeling something is stopping the newer version to run. I've new set of text of wall and feel it is local machine issue rather than XCMS. Thoughts?

When I run xset = xcmsSet(), now I see this text and process stops.

rm -f massifquant/xcms_massifquant.o massifquant/TrMgr.o massifquant/Tracker.o massifquant/SegProc.o massifquant/DataKeeper.o massifquant/OpOverload.o obiwarp/mat.o obiwarp/vec.o obiwarp/xcms_dynprog.o obiwarp/xcms_lmat.o xcms_obiwarp.o fastMatch.o mzClust_hclust.o mzROI.o util.o xcms.o binners.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c massifquant/xcms_massifquant.cpp -o massifquant/xcms_massifquant.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c massifquant/TrMgr.cpp -o massifquant/TrMgr.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c massifquant/Tracker.cpp -o massifquant/Tracker.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c massifquant/SegProc.cpp -o massifquant/SegProc.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c massifquant/DataKeeper.cpp -o massifquant/DataKeeper.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c massifquant/OpOverload.cpp -o massifquant/OpOverload.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c obiwarp/mat.cpp -o obiwarp/mat.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c obiwarp/vec.cpp -o obiwarp/vec.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c obiwarp/xcms_dynprog.cpp -o obiwarp/xcms_dynprog.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c obiwarp/xcms_lmat.cpp -o obiwarp/xcms_lmat.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c xcms_obiwarp.cpp -o xcms_obiwarp.o
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c fastMatch.c -o fastMatch.o
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c mzClust_hclust.c -o mzClust_hclust.o
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c mzROI.c -o mzROI.o
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c util.c -o util.o
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c xcms.c -o xcms.o
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c binners.c -o binners.o
/usr/bin/g++ -shared -L/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/lib -L/opt/gridware/depots/8e896c5a/el7/pkg/libs/blas/3.6.0/gcc-4.8.5 -L/opt/gridware/depots/8e896c5a/el7/pkg/libs/lapack/3.5.0/gcc-4.8.5/lib -o xcms.so massifquant/xcms_massifquant.o massifquant/TrMgr.o massifquant/Tracker.o massifquant/SegProc.o massifquant/DataKeeper.o massifquant/OpOverload.o obiwarp/mat.o obiwarp/vec.o obiwarp/xcms_dynprog.o obiwarp/xcms_lmat.o xcms_obiwarp.o fastMatch.o mzClust_hclust.o mzROI.o util.o xcms.o binners.o -L/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/lib -lR
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG   -I/usr/local/include   -fpic  -g -O2  -c fastMatch.c -o fastMatch.o
/usr/bin/gcc -std=gnu99 -shared -L/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/lib -L/opt/gridware/depots/8e896c5a/el7/pkg/libs/blas/3.6.0/gcc-4.8.5 -L/opt/gridware/depots/8e896c5a/el7/pkg/libs/lapack/3.5.0/gcc-4.8.5/lib -o CAMERA.so fastMatch.o -L/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/lib -lR
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG  -I"/mnt/iusers01/pb02/mbdssdt3/gridware/share/R/3.5.2/Rcpp/include" -I/usr/local/include   -fpic  -g -O2  -c RcppExports.cpp -o RcppExports.o
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG  -I"/mnt/iusers01/pb02/mbdssdt3/gridware/share/R/3.5.2/Rcpp/include" -I/usr/local/include   -fpic  -g -O2  -c Spectrum1_class.c -o Spectrum1_class.o
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG  -I"/mnt/iusers01/pb02/mbdssdt3/gridware/share/R/3.5.2/Rcpp/include" -I/usr/local/include   -fpic  -g -O2  -c Spectrum2_class.c -o Spectrum2_class.o
/usr/bin/g++  -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG  -I"/mnt/iusers01/pb02/mbdssdt3/gridware/share/R/3.5.2/Rcpp/include" -I/usr/local/include   -fpic  -g -O2  -c imp_neighbour_avg.cpp -o imp_neighbour_avg.o
/usr/bin/gcc -std=gnu99 -I"/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/include" -DNDEBUG  -I"/mnt/iusers01/pb02/mbdssdt3/gridware/share/R/3.5.2/Rcpp/include" -I/usr/local/include   -fpic  -g -O2  -c utils.c -o utils.o
/usr/bin/g++ -shared -L/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/lib -L/opt/gridware/depots/8e896c5a/el7/pkg/libs/blas/3.6.0/gcc-4.8.5 -L/opt/gridware/depots/8e896c5a/el7/pkg/libs/lapack/3.5.0/gcc-4.8.5/lib -o MSnbase.so RcppExports.o Spectrum1_class.o Spectrum2_class.o imp_neighbour_avg.o utils.o -L/opt/gridware/depots/8e896c5a/el7/pkg/apps/R/3.5.2/gcc-4.8.5+lapack-3.5.0+blas-3.6.0/lib64/R/lib -lR

[jorainer]

This looks like text printed during the installation of xcms. Where you able to install xcms? What version do you have now (i.e. run sessionInfo() after library(xcms))?

[trivedi-group]

This looks like text printed during the installation of xcms. Where you able to install xcms? What version do you have now (i.e. run sessionInfo() after library(xcms))?

You are right. It was an issue with installation of xcms. I've now installed it after removing previous package and it has sailed through. I think it has successfully read most of the files now (got over 600 x 2gb files) but it will take longer to finish. I will close this thread as the issue seems to be resolved. If something new pops up, I'll open another thread.

Thank you for resolving this so quickly!