Error in H5Fopen(file): HDF5. File accessibilty. Unable to open file

[brianpenghe]

I was trying to subset an ArchRProj but got these errors:

lung <- subsetArchRProject(
  ArchRProj = lung,
  cells = getCellNames(lung[lung$TSSEnrichment > 8]),
  dropCells = TRUE,
  logFile = NULL,
  threads = 1,
  force = TRUE
)
Copying ArchRProject to new outputDirectory : /mnt/210623/ArchRSubset

Copying Arrow Files...

.copyArrow : Initializing Out ArrowFile

.copyArrow : Adding Metadata to Out ArrowFile

Error in H5Fopen(file): HDF5. File accessibilty. Unable to open file.
Traceback:

Do you know why?

[brianpenghe]

Currently I brutally subsetted the object by doing this:
lung <- lung[lung$TSSEnrichment > 8]
Not sure whether it will damage the object but will keep you posted.

[rcorces]

HDF5 errors are hard to trace. usually it is because the HDF5 file got corrupted. If this repeats in the future, you can re-open this issue by commenting below.

[brianpenghe]

Currently I brutally subsetted the object by doing this:
lung <- lung[lung$TSSEnrichment > 8]
Not sure whether it will damage the object but will keep you posted.

Based on your reply #390 (comment) I guess my workaround isn't elegant.

I also tried subsetting in the R terminal outside Jupyter notebook, which produced the same error. It seems to be an issue related to writing permission because when I used sudo R to start R program the error was gone.

Does it sound right? Just want to make sure I'm not creating a bigger problem for myself while fixing this.

Thanks

[rcorces]

It could be a file permissions issue. I'm not sure I've encountered that before but I dont think you would create problems by running under sudo. Still surprised you would need to do that but it could be something specific to your environment.

[j-andrews7]

This is relatively difficult to overcome in certain environments, e.g. RStudio-server instances where setting R to run with sudo permissions is a pain.

[rcorces]

I wish I knew the root cause of why this happens on certain environments. Any ideas / suggestions are welcome. I've never been able to reproduce this.

[j-andrews7]

For what it's worth, I was able to overcome this by setting subthreading = FALSE in my createArrowFiles call. Attached are two logs - the first of which failed, the latter of which completed successfully with subthreading = FALSE. Ultimately, I think it's a permissions issue with the temporary Arrow Files, but I don't know why the multithreading causes issues.

ArchR-createArrows-7dc63059af5d-Date-2021-08-23_Time-15-21-46.log

ArchR-createArrows-7dc6336cf3df-Date-2021-08-23_Time-15-33-07.log

[rcorces]

I had forgotten about an earlier instance of this issue from a previous report. For others who end up here, see #248 for a potential workaround.

[brianpenghe]

For what it's worth, I was able to overcome this by setting subthreading = FALSE in my createArrowFiles call. Attached are two logs - the first of which failed, the latter of which completed successfully with subthreading = FALSE. Ultimately, I think it's a permissions issue with the temporary Arrow Files, but I don't know why the multithreading causes issues.

ArchR-createArrows-7dc63059af5d-Date-2021-08-23_Time-15-21-46.log

ArchR-createArrows-7dc6336cf3df-Date-2021-08-23_Time-15-33-07.log

I also realized that some errors can be avoided by not setting multithreaded parameters (aka using a single thread). Not sure whether why that is.