Re-saving gobj with overwrite breaks gobj

[rbutleriii]

Describe the Error

I was trying to update a Giotto object and save it in the same location with overwrite = TRUE, but when I try it it appears to break the object by erasing some image related files

Error in list_images(gobject) : object 'gobject' not found

When I ran it with my data it gave a different, but seemingly related error:

[1] "Saving the giotto object..."
Folder already exist and overwrite = TRUE, overwrite folder
1. Start writing feature information
For feature: rna
For feature: neg_probe
2. Start writing spatial information
For spatial information: cell
For spatial information: subregion
3. Start writing image information
For image information: fov011-composite
Error: [writeRaster] file does not exist: /oak/stanford/scg/lab_flongo/Tau-PS19_C31_cortex_snRNAseq/CosMx/6 Analysis/spatial_analysis/hippocampus/s24.PS19-V.giotto_obj/Images/fov011-composite_spatRaster
Execution halted

But that file is a bit bigger if you want me to send it to you.

To Reproduce

The easy error can be reproduced with:

library(Giotto)
library(GiottoData)

loadGiottoMini(dataset="cosmx") %>%
  saveGiotto(foldername = "my_gobj", method = 'qs', overwrite = TRUE)

loadGiotto("my_gobj") %>%
  saveGiotto(foldername = "my_gobj", method = 'qs', overwrite = TRUE)
Folder already exist and overwrite = TRUE, overwrite folder
1. Start writing feature information
1. read Giotto object
giotto object was not found
 skip loading giotto object
2. read Giotto feature information
3. read Giotto spatial information

4. read Giotto image information
Error in list_images(gobject) : object 'gobject' not found

x = loadGiotto("my_gobj")
1. read Giotto object
giotto object was not found
 skip loading giotto object
2. read Giotto feature information
3. read Giotto spatial information

4. read Giotto image information
Error in list_images(gobject) : object 'gobject' not found

System Information

> sessionInfo()
R version 4.2.2 (2022-10-31)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS/LAPACK: /home/rrbutler/.local/share/r-miniconda/envs/giotto_env/lib/libopenblasp-r0.3.25.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] GiottoData_0.2.6.2 GiottoUtils_0.1.3  Giotto_4.0.2       GiottoClass_0.1.3

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.12         pillar_1.9.0        compiler_4.2.2
 [4] qs_0.25.7           tools_4.2.2         jsonlite_1.8.8
 [7] lifecycle_1.0.4     tibble_3.2.1        gtable_0.3.4
[10] lattice_0.22-5      pkgconfig_2.0.3     png_0.1-8
[13] rlang_1.1.3         Matrix_1.6-4        cli_3.6.2
[16] parallel_4.2.2      terra_1.7-55        withr_2.5.2
[19] dplyr_1.1.4         colorRamp2_0.1.0    generics_0.1.3
[22] vctrs_0.6.5         gtools_3.9.5        GiottoVisuals_0.1.2
[25] rappdirs_0.3.3      grid_4.2.2          tidyselect_1.2.0
[28] RApiSerialize_0.1.2 reticulate_1.34.0   glue_1.7.0
[31] data.table_1.14.10  R6_2.5.1            fansi_1.0.6
[34] ggplot2_3.4.4       magrittr_2.0.3      scales_1.3.0
[37] codetools_0.2-19    colorspace_2.1-0    stringfish_0.16.0
[40] utf8_1.2.4          RcppParallel_5.1.7  munsell_0.5.0

[rbutleriii]

Hi @jiajic , so I was running this on a slurm cluster in a batch to process several slides of data, and I encountered a problem with the fix. It appears to use a common .giotto_scratch, folder for a temp directory, but it needs to be specific so that collisions do not occur, or else every gobj folder needs its own subfolder to separate them.

==> 42915043_giotto_wkflow_prep_cortex-s23-PS19-V.log <==
For image information: fov009-compartment
For image information: fov010-composite
For image information: fov010-overlay
For image information: fov010-compartment
For image information: fov013-composite
For image information: fov013-overlay
For image information: fov013-compartment
[1] TRUE
TIME
16:42

==> 42915044_giotto_wkflow_prep_cortex-s23-PS19-C31.log <==
For image information: fov035-overlay
For image information: fov035-compartment
[1] TRUE
Warning messages:
1: In dir.create(use_dir, recursive = TRUE) :
  'cortex/.giotto_scratch' already exists
2: In dir.create(feat_dir) :
  'cortex/.giotto_scratch/Features' already exists
TIME
25:10

==> 42915045_giotto_wkflow_prep_cortex-s23-WT-V.log <==
 overwrite folder
1. Start writing feature information
For feature: rna
Error: [writeVector] file exists. Use 'overwrite=TRUE' to overwrite it
In addition: Warning messages:
1: In dir.create(use_dir, recursive = TRUE) :
  'cortex/.giotto_scratch' already exists
2: In dir.create(feat_dir) :
  'cortex/.giotto_scratch/Features' already exists
Execution halted

==> 42915046_giotto_wkflow_prep_cortex-s23-WT-C31.log <==
 overwrite folder
1. Start writing feature information
For feature: rna
Error: [writeVector] file exists. Use 'overwrite=TRUE' to overwrite it
In addition: Warning messages:
1: In dir.create(use_dir, recursive = TRUE) :
  'cortex/.giotto_scratch' already exists
2: In dir.create(feat_dir) :
  'cortex/.giotto_scratch/Features' already exists
Execution halted

==> 42915047_giotto_wkflow_prep_cortex-s24-PS19-V.log <==
 overwrite folder
1. Start writing feature information
For feature: rna
Error: [writeVector] file exists. Use 'overwrite=TRUE' to overwrite it
In addition: Warning messages:
1: In dir.create(use_dir, recursive = TRUE) :
  'cortex/.giotto_scratch' already exists
2: In dir.create(feat_dir) :
  'cortex/.giotto_scratch/Features' already exists
Execution halted

==> 42915048_giotto_wkflow_prep_cortex-s24-PS19-C31.log <==
 overwrite folder
1. Start writing feature information
For feature: rna
Error: [writeVector] file exists. Use 'overwrite=TRUE' to overwrite it
In addition: Warning messages:
1: In dir.create(use_dir, recursive = TRUE) :
  'cortex/.giotto_scratch' already exists
2: In dir.create(feat_dir) :
  'cortex/.giotto_scratch/Features' already exists
Execution halted

==> 42915049_giotto_wkflow_prep_cortex-s24-WT-V.log <==
For image information: fov015-compartment
For image information: fov018-composite
For image information: fov018-overlay
For image information: fov018-compartment
For image information: fov020-composite
For image information: fov020-overlay
For image information: fov020-compartment
[1] TRUE
TIME
7:49

==> 42915050_giotto_wkflow_prep_cortex-s24-WT-C31.log <==
1. Start writing feature information
For feature: rna
Error: [writeVector] Layer creation failed
In addition: Warning messages:
1: In dir.create(use_dir, recursive = TRUE) :
  'cortex/.giotto_scratch' already exists
2: In dir.create(feat_dir) :
  'cortex/.giotto_scratch/Features' already exists
3: Failed to create file cortex/.giotto_scratch/Features/rna_feature_spatVector.shp: No such file or directory (GDAL error 1)
Execution halted

==> 42915051_giotto_wkflow_prep_cortex-s25-PS19-V.log <==
                      'Analytic Pearson residuals for normalization of single-cell RNA-seq UMI data'

 Pearson residual normalized data will be returned to the pearson Giotto slot

Folder already exist and overwrite = TRUE,
 overwrite folder
1. Start writing feature information
For feature: rna
Error: [writeVector] file exists. Use 'overwrite=TRUE' to overwrite it
Execution halted

==> 42915052_giotto_wkflow_prep_cortex-s25-PS19-C31.log <==
 overwrite folder
1. Start writing feature information
For feature: rna
Error: [writeVector] file exists. Use 'overwrite=TRUE' to overwrite it
In addition: Warning messages:
1: In dir.create(use_dir, recursive = TRUE) :
  'cortex/.giotto_scratch' already exists
2: In dir.create(feat_dir) :
  'cortex/.giotto_scratch/Features' already exists
Execution halted

==> 42915053_giotto_wkflow_prep_cortex-s25-WT-V.log <==
 overwrite folder
1. Start writing feature information
For feature: rna
Error: [writeVector] file exists. Use 'overwrite=TRUE' to overwrite it
In addition: Warning messages:
1: In dir.create(use_dir, recursive = TRUE) :
  'cortex/.giotto_scratch' already exists
2: In dir.create(feat_dir) :
  'cortex/.giotto_scratch/Features' already exists
Execution halted

==> 42915054_giotto_wkflow_prep_cortex-s25-WT-C31.log <==
For image information: fov033-overlay
For image information: fov033-compartment
For image information: fov034-composite
For image information: fov034-overlay
For image information: fov034-compartment
For image information: fov038-composite
For image information: fov038-overlay
For image information: fov038-compartment
For image information: fov039-composite
slurmstepd: error: *** JOB 42915054 ON sgiuv300-srcf-d10-01 CANCELLED AT 2024-04-24T14:22:03 ***