Errors when converting HCA h5ad file into h5seurat file

[cyxss]

Hi! I met this error when I try to convert a h5ad file in 1.6G size to h5seurat. It seems something wrong with hdf5. But when I used the same code to convert a smaller h5ad file, this error did not appear. I also checked the RAM and the storage space on the disk, they are far from exhausted.

Warning message:
“Unknown file type: h5ad”
Warning message:
“'assay' not set, setting to 'RNA'”
Creating h5Seurat file for version 3.1.5.9900

Adding X as data

Adding X as counts

Adding meta.features from var

Adding X_pca as cell embeddings for pca

Error in self$write_low_level(value, file_space = self_space_id, mem_space = mem_space_id, : HDF5-API Errors:
    error #000: H5Dio.c in H5Dwrite(): line 336: can't write data
        class: HDF5
        major: Dataset
        minor: Write failed

    error #001: H5Dio.c in H5D__write(): line 828: can't write data
        class: HDF5
        major: Dataset
        minor: Write failed

    error #002: H5Dchunk.c in H5D__chunk_write(): line 2363: unable to read raw data chunk
        class: HDF5
        major: Low-level I/O
        minor: Read failed

    error #003: H5Dchunk.c in H5D__chunk_lock(): line 3686: unable to preempt chunk(s) from cache
        class: HDF5
        major: Low-level I/O
        minor: Unable to initialize object

    error #004: H5Dchunk.c in H5D__chunk_cache_prune(): line 3352: unable to preempt one or more raw data cache entry
        class: HDF5
        major: Low-level I/O
        minor: Unable to flush data from cache

    error #005: H5Dchunk.c in H5D__chunk_cache_evict(): line 3203: cannot flush indexed storage buffer
        class: HDF5
       
Traceback:

1. Convert(paste0(path, "vascular", ".h5ad"), dest = "h5seurat", 
 .     overwrite = TRUE)
2. Convert.character(paste0(path, "vascular", ".h5ad"), dest = "h5seurat", 
 .     overwrite = TRUE)
3. Convert(source = hfile, dest = dest, assay = assay, overwrite = overwrite, 
 .     verbose = verbose, ...)
4. Convert.H5File(source = hfile, dest = dest, assay = assay, overwrite = overwrite, 
 .     verbose = verbose, ...)
5. H5ADToH5Seurat(source = source, dest = dest, assay = assay, overwrite = overwrite, 
 .     verbose = verbose)
6. Transpose(x = source[["obsm"]][[reduc]], dest = reduc.group, 
 .     dname = "cell.embeddings", verbose = FALSE)
7. Transpose.H5D(x = source[["obsm"]][[reduc]], dest = reduc.group, 
 .     dname = "cell.embeddings", verbose = FALSE)
8. tmtx$write(args = dims, value = t(x = x$read(args = rev(x = dims))))
9. self$write_low_level(value, file_space = self_space_id, mem_space = mem_space_id, 
 .     dataset_xfer_pl = dataset_xfer_pl)

Sometimes it would also show the error of failed to closed the file in hdf5 or just kill the R kernel.

Warning message:
“Unknown file type: h5ad”
Warning message:
“'assay' not set, setting to 'RNA'”
Creating h5Seurat file for version 3.1.5.9900

Adding X as data

Adding X as counts

Adding meta.features from var

Adding X_pca as cell embeddings for pca

Adding X_umap as cell embeddings for umap

Error in H5_close_any(self$id): HDF5-API Errors:
    error #000: H5F.c in H5Fclose(): line 668: closing file ID failed
        class: HDF5
        major: File accessibilty
        minor: Unable to close file

    error #001: H5Fint.c in H5F__close(): line 2047: decrementing file ID failed
        class: HDF5
        major: File accessibilty
        minor: Unable to decrement reference count

    error #002: H5I.c in H5I_dec_app_ref(): line 1309: can't decrement ID ref count
        class: HDF5
        major: Object atom
        minor: Unable to decrement reference count

    error #003: H5Fint.c in H5F__close_cb(): line 2105: can't close file
        class: HDF5
        major: File accessibilty
        minor: Unable to close file

    error #004: H5Fint.c in H5F_try_close(): line 2276: problems closing file
        class: HDF5
        major: File accessibilty
        minor: Unable to close file

    error #005: H5Fint.c in H5F__dest(): line 1299: unable to close file
        class: HDF5
        major: File accessibi
Traceback:

1. Convert(paste0(path, "ventricular", ".h5ad"), dest = "h5seurat", 
 .     overwrite = TRUE)
2. Convert.character(paste0(path, "ventricular", ".h5ad"), dest = "h5seurat", 
 .     overwrite = TRUE)
3. dfile$close_all()
4. H5_close_any(self$id)

The environment is CentOS Linux release 7.6.1810 (Core), 128G RAM, R version 3.6.1, Seurat 3.2.2, SeuratDisk 0.0.0.9013

[sbdaxia]

I am having a similar issue trying to convert '.h5ad' file to '.h5seruat'.

My environment is maxOS Catalina 10.15.4. R-4.0.3, Seurat-3.2.2, SeuratDisk-0.0.0.9013.

I tried to run the example code given in the "convert-anndata.Rmd" vignette. The code I ran in the markdown file is

url <- "https://seurat.nygenome.org/pbmc3k_final.h5ad"
curl::curl_download(url, basename(url))
Convert("pbmc3k_final.h5ad", dest = "h5seurat", overwrite = TRUE)

The error I have been getting (which is the same error I get when I tried to convert my own h5ad files) is

Unknown file type: h5ad'assay' not set, setting to 'RNA'Creating h5Seurat file for version 3.1.5.9900
Adding X as scale.data
Adding raw/X as data
Adding raw/X as counts
Adding meta.features from raw/var
Adding dispersions from scaled feature-level metadata
Adding dispersions_norm from scaled feature-level metadata
Merging gene_ids from scaled feature-level metadata
Adding highly_variable from scaled feature-level metadata
Adding means from scaled feature-level metadata
Merging n_cells from scaled feature-level metadata
Error in dfile$obj_copy_from(src_loc = source, src_name = "obs", dst_name = "meta.data") : 
  HDF5-API Errors:
    error #000: H5Ocopy.c in H5Ocopy(): line 251: unable to copy object
        class: HDF5
        major: Object header
        minor: Unable to copy object

    error #001: H5VLcallback.c in H5VL_object_copy(): line 5530: object copy failed
        class: HDF5
        major: Virtual Object Layer
        minor: Unable to copy object

    error #002: H5VLcallback.c in H5VL__object_copy(): line 5491: object copy failed
        class: HDF5
        major: Virtual Object Layer
        minor: Unable to copy object

    error #003: H5VLnative_object.c in H5VL__native_object_copy(): line 126: unable to copy object
        class: HDF5
        major: Symbol table
        minor: Unable to copy object

    error #004: H5Ocopy.c in H5O_copy(): line 313: unable to copy object
        class: HDF5
        major: Object header
        minor: Unable to copy object

    error #005: H5Ocopy.c in H5O__copy_obj(): line 1217: unable to copy object
        class: HDF5
        major
Error in private$closeFun(id) : HDF5-API Errors:
    error #000: H5F.c in H5Fclose(): line 886: decrementing file ID failed
        class: HDF5
        major: Object atom
        minor: Unable to close file

    error #001: H5I.c in H5I_dec_app_ref(): line 1422: can't decrement ID ref count
        class: HDF5
        major: Object atom
        minor: Unable to decrement reference count

    error #002: H5F.c in H5F__close_cb(): line 243: unable to close file
        class: HDF5
        major: File accessibility
        minor: Unable to close file

    error #003: H5VLcallback.c in H5VL_file_close(): line 3977: file close failed
        class: HDF5
        major: Virtual Object Layer
        minor: Unable to close file

    error #004: H5VLcallback.c in H5VL__file_close(): line 3945: file close failed
        class: HDF5
        major: Virtual Object Layer
        minor: Unable to close file

    error #005: H5VLnative_file.c in H5VL__native_file_close(): line 884: can't close file
        class: HDF5

The error seems to have something to do with the hdf5 package I installed using Homebew. I installed gcc-10.2.0 and hdf5-1.12.0_1 from Homebrew prior to installing SeuratDisk in R.

When I installed hdf5, I encounter a warning

Warning: hdf5 dependency gcc was built with a different C++ standard
library (libstdc++ from clang). This may cause problems at runtime.

However, re-installing gcc still have me the same warning when I re-installed hdf5. Not sure if this has something to do with the issue or it's something to do with the new version of R-4.0.3.

Thanks!

[schmic05]

Hi! I am facing the very same error as @sbdaxia.

My configuration is: CentOS Linux release 7.9.2009, conda 4.9.2, hdf5 1.10.4, R-4.0.3.
Here's my session info:
R version 4.0.3 (2020-10-10)
Platform: x86_64-conda-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS/LAPACK: /users/mscherer/anaconda3/envs/scran/lib/libopenblasp-r0.3.12.so

locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] parallel stats4 stats graphics grDevices utils datasets
[8] methods base

other attached packages:
[1] SeuratDisk_0.0.0.9018 SeuratData_0.2.1
[3] scran_1.18.5 SingleCellExperiment_1.12.0
[5] SummarizedExperiment_1.20.0 Biobase_2.50.0
[7] GenomicRanges_1.42.0 GenomeInfoDb_1.26.2
[9] IRanges_2.24.1 S4Vectors_0.28.1
[11] BiocGenerics_0.36.0 MatrixGenerics_1.2.1
[13] matrixStats_0.58.0 SeuratObject_4.0.0
[15] Seurat_4.0.0

loaded via a namespace (and not attached):
[1] Rtsne_0.15 colorspace_2.0-0
[3] deldir_0.2-9 ellipsis_0.3.1
[5] ggridges_0.5.3 scuttle_1.0.4
[7] bluster_1.0.0 XVector_0.30.0
[9] BiocNeighbors_1.8.2 spatstat.data_1.7-0
[11] leiden_0.3.7 listenv_0.8.0
[13] bit64_4.0.5 ggrepel_0.9.1
[15] sparseMatrixStats_1.2.1 codetools_0.2-18
[17] splines_4.0.3 polyclip_1.10-0
[19] jsonlite_1.7.2 ica_1.0-2
[21] cluster_2.1.0 png_0.1-7
[23] uwot_0.1.10 shiny_1.6.0
[25] sctransform_0.3.2 compiler_4.0.3
[27] httr_1.4.2 dqrng_0.2.1
[29] assertthat_0.2.1 Matrix_1.3-2
[31] fastmap_1.1.0 lazyeval_0.2.2
[33] cli_2.3.0 limma_3.46.0
[35] later_1.1.0.1 BiocSingular_1.6.0
[37] htmltools_0.5.1.1 tools_4.0.3
[39] rsvd_1.0.3 igraph_1.2.6
[41] gtable_0.3.0 glue_1.4.2
[43] GenomeInfoDbData_1.2.4 RANN_2.6.1
[45] reshape2_1.4.4 dplyr_1.0.4
[47] rappdirs_0.3.3 Rcpp_1.0.6
[49] spatstat_1.64-1 scattermore_0.7
[51] vctrs_0.3.6 nlme_3.1-152
[53] DelayedMatrixStats_1.12.3 lmtest_0.9-38
[55] stringr_1.4.0 globals_0.14.0
[57] beachmat_2.6.4 mime_0.9
[59] miniUI_0.1.1.1 lifecycle_0.2.0
[61] irlba_2.3.3 statmod_1.4.35
[63] goftest_1.2-2 future_1.21.0
[65] edgeR_3.32.1 MASS_7.3-53
[67] zlibbioc_1.36.0 zoo_1.8-8
[69] scales_1.1.1 promises_1.1.1
[71] spatstat.utils_2.0-0 RColorBrewer_1.1-2
[73] curl_4.3 reticulate_1.18
[75] pbapply_1.4-3 gridExtra_2.3
[77] ggplot2_3.3.3 rpart_4.1-15
[79] stringi_1.5.3 BiocParallel_1.24.1
[81] rlang_0.4.10 pkgconfig_2.0.3
[83] bitops_1.0-6 lattice_0.20-41
[85] ROCR_1.0-11 purrr_0.3.4
[87] tensor_1.5 patchwork_1.1.1
[89] htmlwidgets_1.5.3 bit_4.0.4
[91] cowplot_1.1.1 tidyselect_1.1.0
[93] parallelly_1.23.0 RcppAnnoy_0.0.18
[95] plyr_1.8.6 magrittr_2.0.1
[97] R6_2.5.0 generics_0.1.0
[99] DelayedArray_0.16.1 withr_2.4.1
[101] pillar_1.4.7 mgcv_1.8-33
[103] fitdistrplus_1.1-3 survival_3.2-7
[105] abind_1.4-5 RCurl_1.98-1.2
[107] tibble_3.0.6 future.apply_1.7.0
[109] hdf5r_1.3.3 crayon_1.4.1
[111] KernSmooth_2.23-18 plotly_4.9.3
[113] locfit_1.5-9.4 grid_4.0.3
[115] data.table_1.13.6 digest_0.6.27
[117] xtable_1.8-4 tidyr_1.1.2
[119] httpuv_1.5.5 munsell_0.5.0
[121] viridisLite_0.3.0

Any help would be much appreciated.

[HKL-97]

same error here

[joseah]

I experienced these errors too with seurat-disk@163f1aa. However, I solved it by just increasing the RAM memory resources. I imagine exporting the data into a h5seurat file creates temporary instances of memory to work.

[GouQiao]

Hi! I met this error when I try to convert a h5ad file in 1.6G size to h5seurat. It seems something wrong with hdf5. But when I used the same code to convert a smaller h5ad file, this error did not appear. I also checked the RAM and the storage space on the disk, they are far from exhausted.

Warning message:
“Unknown file type: h5ad”
Warning message:
“'assay' not set, setting to 'RNA'”
Creating h5Seurat file for version 3.1.5.9900

Adding X as data

Adding X as counts

Adding meta.features from var

Adding X_pca as cell embeddings for pca

Error in self$write_low_level(value, file_space = self_space_id, mem_space = mem_space_id, : HDF5-API Errors:
    error #000: H5Dio.c in H5Dwrite(): line 336: can't write data
        class: HDF5
        major: Dataset
        minor: Write failed

    error #001: H5Dio.c in H5D__write(): line 828: can't write data
        class: HDF5
        major: Dataset
        minor: Write failed

    error #002: H5Dchunk.c in H5D__chunk_write(): line 2363: unable to read raw data chunk
        class: HDF5
        major: Low-level I/O
        minor: Read failed

    error #003: H5Dchunk.c in H5D__chunk_lock(): line 3686: unable to preempt chunk(s) from cache
        class: HDF5
        major: Low-level I/O
        minor: Unable to initialize object

    error #004: H5Dchunk.c in H5D__chunk_cache_prune(): line 3352: unable to preempt one or more raw data cache entry
        class: HDF5
        major: Low-level I/O
        minor: Unable to flush data from cache

    error #005: H5Dchunk.c in H5D__chunk_cache_evict(): line 3203: cannot flush indexed storage buffer
        class: HDF5
       
Traceback:

1. Convert(paste0(path, "vascular", ".h5ad"), dest = "h5seurat", 
 .     overwrite = TRUE)
2. Convert.character(paste0(path, "vascular", ".h5ad"), dest = "h5seurat", 
 .     overwrite = TRUE)
3. Convert(source = hfile, dest = dest, assay = assay, overwrite = overwrite, 
 .     verbose = verbose, ...)
4. Convert.H5File(source = hfile, dest = dest, assay = assay, overwrite = overwrite, 
 .     verbose = verbose, ...)
5. H5ADToH5Seurat(source = source, dest = dest, assay = assay, overwrite = overwrite, 
 .     verbose = verbose)
6. Transpose(x = source[["obsm"]][[reduc]], dest = reduc.group, 
 .     dname = "cell.embeddings", verbose = FALSE)
7. Transpose.H5D(x = source[["obsm"]][[reduc]], dest = reduc.group, 
 .     dname = "cell.embeddings", verbose = FALSE)
8. tmtx$write(args = dims, value = t(x = x$read(args = rev(x = dims))))
9. self$write_low_level(value, file_space = self_space_id, mem_space = mem_space_id, 
 .     dataset_xfer_pl = dataset_xfer_pl)

Sometimes it would also show the error of failed to closed the file in hdf5 or just kill the R kernel.

Warning message:
“Unknown file type: h5ad”
Warning message:
“'assay' not set, setting to 'RNA'”
Creating h5Seurat file for version 3.1.5.9900

Adding X as data

Adding X as counts

Adding meta.features from var

Adding X_pca as cell embeddings for pca

Adding X_umap as cell embeddings for umap

Error in H5_close_any(self$id): HDF5-API Errors:
    error #000: H5F.c in H5Fclose(): line 668: closing file ID failed
        class: HDF5
        major: File accessibilty
        minor: Unable to close file

    error #001: H5Fint.c in H5F__close(): line 2047: decrementing file ID failed
        class: HDF5
        major: File accessibilty
        minor: Unable to decrement reference count

    error #002: H5I.c in H5I_dec_app_ref(): line 1309: can't decrement ID ref count
        class: HDF5
        major: Object atom
        minor: Unable to decrement reference count

    error #003: H5Fint.c in H5F__close_cb(): line 2105: can't close file
        class: HDF5
        major: File accessibilty
        minor: Unable to close file

    error #004: H5Fint.c in H5F_try_close(): line 2276: problems closing file
        class: HDF5
        major: File accessibilty
        minor: Unable to close file

    error #005: H5Fint.c in H5F__dest(): line 1299: unable to close file
        class: HDF5
        major: File accessibi
Traceback:

1. Convert(paste0(path, "ventricular", ".h5ad"), dest = "h5seurat", 
 .     overwrite = TRUE)
2. Convert.character(paste0(path, "ventricular", ".h5ad"), dest = "h5seurat", 
 .     overwrite = TRUE)
3. dfile$close_all()
4. H5_close_any(self$id)

The environment is CentOS Linux release 7.6.1810 (Core), 128G RAM, R version 3.6.1, Seurat 3.2.2, SeuratDisk 0.0.0.9013

Hi!
I met same error shows:
Error in self$write_low_level(value, file_space = self_space_id, mem_space = mem_space_id, :
Number of objects in robj is not the same and not a multiple of number of elements selected in file: expected are 0 but provided are 3000

How did you solve it?

Best.