Creating a tar_format() to work with terra SpatVector objects [help]

[Aariq]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

This is related to #809, but the examples in there didn't quite help. I'm trying to use targets for a project where a lot of intermediate terra objects are created. If I have a target that creates a SpatVector, tar_read() gives me Error: external pointer is not valid. The solution is to use terra::wrap()—then tar_read() |> terra::unwrap() can load the object.

I'm trying to figure out how to write a custom tar_format() to make this "just work" and I'm close, but not all the way there. I thought providing wrap() and unwrap() to marshal and unmarshal, respectively, would be the way to do it, but that doesn't seem to work. Providing a custom read and write does work, but only if the pipeline is run with a single worker.

library(targets)

tar_script({
  library(terra)
  # tar_option_set(controller = crew::crew_controller_local(workers = 3))
  list(
    tar_target(shape_file, system.file("ex/lux.shp", package = "terra"), format = "file"),
    tar_target(vect, vect(shape_file)),
    tar_target(vect_wrapped, wrap(vect(shape_file))),
    tar_target(vect_marshal, vect(shape_file), format = tar_format(
      marshal = function(object) terra::wrap(object),
      unmarshal = function(object) terra::unwrap(object)
    )),
    tar_target(vect_readwrite, vect(shape_file), format = tar_format(
      read = function(path) terra::unwrap(readRDS(path)),
      write = function(object, path) saveRDS(object = terra::wrap(object), file = path)
    ))
  )

})
library(terra)
#> terra 1.7.65
tar_make()
#> terra 1.7.65
#> ▶ dispatched target shape_file
#> ● completed target shape_file [0.003 seconds]
#> ▶ dispatched target vect
#> ● completed target vect [0.025 seconds]
#> ▶ dispatched target vect_wrapped
#> ● completed target vect_wrapped [0.048 seconds]
#> ▶ dispatched target vect_marshal
#> ● completed target vect_marshal [0.006 seconds]
#> ▶ dispatched target vect_readwrite
#> ● completed target vect_readwrite [0.007 seconds]
#> ▶ completed pipeline [2.171 seconds]
tar_read(vect)
#> Error: external pointer is not valid
tar_read(vect_wrapped) |> unwrap()
#>  class       : SpatVector 
#>  geometry    : polygons 
#>  dimensions  : 12, 6  (geometries, attributes)
#>  extent      : 5.74414, 6.528252, 49.44781, 50.18162  (xmin, xmax, ymin, ymax)
#>  coord. ref. : lon/lat WGS 84 (EPSG:4326) 
#>  names       :  ID_1   NAME_1  ID_2   NAME_2  AREA   POP
#>  type        : <num>    <chr> <num>    <chr> <num> <int>
#>  values      :     1 Diekirch     1 Clervaux   312 18081
#>                    1 Diekirch     2 Diekirch   218 32543
#>                    1 Diekirch     3  Redange   259 18664
tar_read(vect_marshal)
#> Error: external pointer is not valid
tar_read(vect_readwrite)
#>  class       : SpatVector 
#>  geometry    : polygons 
#>  dimensions  : 12, 6  (geometries, attributes)
#>  extent      : 5.74414, 6.528252, 49.44781, 50.18162  (xmin, xmax, ymin, ymax)
#>  coord. ref. : lon/lat WGS 84 (EPSG:4326) 
#>  names       :  ID_1   NAME_1  ID_2   NAME_2  AREA   POP
#>  type        : <num>    <chr> <num>    <chr> <num> <int>
#>  values      :     1 Diekirch     1 Clervaux   312 18081
#>                    1 Diekirch     2 Diekirch   218 32543
#>                    1 Diekirch     3  Redange   259 18664

^{Created on 2024-01-24 with reprex v2.0.2}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.3.1 (2023-06-16)
#>  os       macOS Sonoma 14.2.1
#>  system   x86_64, darwin20
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       America/Phoenix
#>  date     2024-01-24
#>  pandoc   3.1.1 @ /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version date (UTC) lib source
#>  backports     1.4.1   2021-12-13 [1] CRAN (R 4.3.0)
#>  base64url     1.4     2018-05-14 [1] CRAN (R 4.3.0)
#>  callr         3.7.3   2022-11-02 [1] CRAN (R 4.3.0)
#>  cli           3.6.2   2023-12-11 [1] CRAN (R 4.3.0)
#>  codetools     0.2-19  2023-02-01 [2] CRAN (R 4.3.1)
#>  data.table    1.14.10 2023-12-08 [1] CRAN (R 4.3.0)
#>  digest        0.6.33  2023-07-07 [1] CRAN (R 4.3.0)
#>  evaluate      0.23    2023-11-01 [1] CRAN (R 4.3.0)
#>  fansi         1.0.6   2023-12-08 [1] CRAN (R 4.3.0)
#>  fastmap       1.1.1   2023-02-24 [1] CRAN (R 4.3.0)
#>  fs            1.6.3   2023-07-20 [1] CRAN (R 4.3.0)
#>  glue          1.7.0   2024-01-09 [1] CRAN (R 4.3.0)
#>  htmltools     0.5.7   2023-11-03 [1] CRAN (R 4.3.0)
#>  igraph        1.5.1   2023-08-10 [1] CRAN (R 4.3.0)
#>  knitr         1.45    2023-10-30 [1] CRAN (R 4.3.0)
#>  lifecycle     1.0.4   2023-11-07 [1] CRAN (R 4.3.0)
#>  magrittr      2.0.3   2022-03-30 [1] CRAN (R 4.3.0)
#>  pillar        1.9.0   2023-03-22 [1] CRAN (R 4.3.0)
#>  pkgconfig     2.0.3   2019-09-22 [1] CRAN (R 4.3.0)
#>  processx      3.8.2   2023-06-30 [1] CRAN (R 4.3.0)
#>  ps            1.7.5   2023-04-18 [1] CRAN (R 4.3.0)
#>  purrr         1.0.2   2023-08-10 [1] CRAN (R 4.3.0)
#>  R.cache       0.16.0  2022-07-21 [1] CRAN (R 4.3.0)
#>  R.methodsS3   1.8.2   2022-06-13 [1] CRAN (R 4.3.0)
#>  R.oo          1.25.0  2022-06-12 [1] CRAN (R 4.3.0)
#>  R.utils       2.12.2  2022-11-11 [1] CRAN (R 4.3.0)
#>  R6            2.5.1   2021-08-19 [1] CRAN (R 4.3.0)
#>  Rcpp          1.0.12  2024-01-09 [1] CRAN (R 4.3.0)
#>  reprex        2.0.2   2022-08-17 [1] CRAN (R 4.3.0)
#>  rlang         1.1.3   2024-01-10 [1] CRAN (R 4.3.0)
#>  rmarkdown     2.25    2023-09-18 [1] CRAN (R 4.3.0)
#>  rstudioapi    0.15.0  2023-07-07 [1] CRAN (R 4.3.0)
#>  sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.3.0)
#>  styler        1.10.2  2023-08-29 [1] CRAN (R 4.3.0)
#>  targets     * 1.4.1   2024-01-09 [1] CRAN (R 4.3.0)
#>  terra       * 1.7-65  2023-12-15 [1] CRAN (R 4.3.0)
#>  tibble        3.2.1   2023-03-20 [1] CRAN (R 4.3.0)
#>  tidyselect    1.2.0   2022-10-10 [1] CRAN (R 4.3.0)
#>  utf8          1.2.4   2023-10-22 [1] CRAN (R 4.3.0)
#>  vctrs         0.6.5   2023-12-01 [1] CRAN (R 4.3.0)
#>  withr         2.5.2   2023-10-30 [1] CRAN (R 4.3.0)
#>  xfun          0.41    2023-11-01 [1] CRAN (R 4.3.0)
#>  yaml          2.3.7   2023-01-23 [1] CRAN (R 4.3.0)
#> 
#>  [1] /Users/ericscott/Library/R/x86_64/4.3/library
#>  [2] /Library/Frameworks/R.framework/Versions/4.3-x86_64/Resources/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

With multiple workers, the vect_readwrite target errors:

library(targets)

tar_script({
  library(terra)
  tar_option_set(controller = crew::crew_controller_local(workers = 2))
  list(
    tar_target(shape_file, system.file("ex/lux.shp", package = "terra"), format = "file"),
    tar_target(vect, vect(shape_file)),
    tar_target(vect_wrapped, wrap(vect(shape_file))),
    tar_target(vect_marshal, vect(shape_file), format = tar_format(
      marshal = function(object) terra::wrap(object),
      unmarshal = function(object) terra::unwrap(object)
    )),
    tar_target(vect_readwrite, vect(shape_file), format = tar_format(
      read = function(path) terra::unwrap(readRDS(path)),
      write = function(object, path) saveRDS(object = terra::wrap(object), file = path)
    ))
  )

})
library(terra)
#> terra 1.7.65
tar_make()
#> terra 1.7.65
#> ▶ dispatched target shape_file
#> ● completed target shape_file [2.944 seconds]
#> ▶ dispatched target vect
#> ● completed target vect [0.023 seconds]
#> ▶ dispatched target vect_wrapped
#> ▶ dispatched target vect_marshal
#> ▶ dispatched (pending) target vect_readwrite
#> ● completed target vect_wrapped [0.059 seconds]
#> ✖ errored target vect_readwrite
#> ✖ errored pipeline [11.201 seconds]
#> Error:
#> ! Error running targets::tar_make()
#> Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
#> Debugging guide: https://books.ropensci.org/targets/debugging.html
#> How to ask for help: https://books.ropensci.org/targets/help.html
#> Last error message:
#>     _store_ external pointer is not valid
#> Last error traceback:
#>     No traceback available.
#> Backtrace:
#>     ▆
#>  1. └─targets::tar_make()
#>  2.   └─targets:::callr_outer(...)
#>  3.     ├─targets:::if_any(...)
#>  4.     └─targets:::callr_error(traced_condition = out, fun = fun)
#>  5.       └─targets::tar_throw_run(message, class = class(traced_condition$condition))
#>  6.         └─targets::tar_error(...)
#>  7.           └─rlang::abort(message = message, class = class, call = tar_empty_envir)
tar_read(vect)
#> Error: external pointer is not valid
tar_read(vect_wrapped) |> unwrap()
#>  class       : SpatVector 
#>  geometry    : polygons 
#>  dimensions  : 12, 6  (geometries, attributes)
#>  extent      : 5.74414, 6.528252, 49.44781, 50.18162  (xmin, xmax, ymin, ymax)
#>  coord. ref. : lon/lat WGS 84 (EPSG:4326) 
#>  names       :  ID_1   NAME_1  ID_2   NAME_2  AREA   POP
#>  type        : <num>    <chr> <num>    <chr> <num> <int>
#>  values      :     1 Diekirch     1 Clervaux   312 18081
#>                    1 Diekirch     2 Diekirch   218 32543
#>                    1 Diekirch     3  Redange   259 18664
tar_read(vect_marshal)
#> Error:
#> ! target vect_marshal not found
#> Backtrace:
#>     ▆
#>  1. └─targets::tar_read(vect_marshal)
#>  2.   └─targets::tar_read_raw(...)
#>  3.     └─targets:::tar_read_inner(name, branches, meta, path_store = store)
#>  4.       └─targets::tar_throw_validate("target ", name, " not found")
#>  5.         └─targets::tar_error(...)
#>  6.           └─rlang::abort(message = message, class = class, call = tar_empty_envir)
tar_read(vect_readwrite)
#> Warning in gzfile(file, "rb"): cannot open compressed file
#> '_targets/objects/vect_readwrite', probable reason 'No such file or directory'
#> Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'unwrap': cannot open the connection

^{Created on 2024-01-24 with reprex v2.0.2}

Is there an "elegant" way to go about this or is the best solution just work directly with the .shp file (e.g. every target that needs the SpatVector object just starts with reading in the .shp file)?

[Aariq]

I think I answered my own question. I ended up not saving any SpatVector targets and for my SpatRaster targets I'm using the following format which works with multiple workers:

format_geotiff <- tar_format(
  read = function(path) terra::rast(path),
  write = function(object, path) terra::writeRaster(x = object, filename = path, filetype = "GTiff", overwrite = TRUE),
  marshal = function(object) terra::wrap(object),
  unmarshal = function(object) terra::unwrap(object)
)

The resulting targets are quite large, but there is no avoiding that, so I'm just trying to make as few targets as possible.