[help] How to static branch over a hierarchical object?

[stulacy]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I'm trying to get static branching working over a nested data structure, represented below as JSON as this is the mental model that is most comfortable to me. The pseudo-code of what I want to do is:

• for each obj in objects:
  • data = read_data(obj$id)
  • for each species in obj$species:
    • do_something(data, obj$species$species_name, obj$species$param1)

[
    {
        "id": "a",
        "species": [
            {
                "species_name": "CO2",
                "param1": 1
            },
            {
                "species_name": "CO",
                "param1": 2
            }
        ]
    },
    {
        "id": "b",
        "species": [
            {
                "species_name": "NH3",
                "param1": 3
            },
            {
                "species_name": "NO2",
                "param1": 4
            },
            {
                "species_name": "CH3",
                "param1": 5
            },
        ]
    },
]

This feels like a trivial workflow, yet I'm struggling converting both the data structure and the logic flow to targets DSL. I've tried many different approaches: nested list, nested tibble, using dynamic branching with pattern=map(), saving the output of an initial tar_map and putting that into a second tar_map, trying the quote tip from the docs.

My 2 main problems are that targets seems to prefer a tabular data structure where you pass it a list of lists, each of which has the same length, even when using a list rather than tibble, and also that tar_map doesn't seem to accept the output of previous targets.

My current _targets.R is:

library(targets)
library(tarchetypes)

grab_data <- function(x) {
    # Read data and do something
}

config <- list(
    id=c("a", "b"),
    species=
        list(
            quote(list(
                species2=c("CO2", "CO"), params=c(1, 2)
            )),
            quote(list(
                species2=c("NH3", "NO2", "CH3"), params=c(3, 4, 5)
            ))
        )
)

ids_mapped <- tar_map(
    values=config,
    names="id",
    tar_target(
        name = data,
        command = grab_data(id)
    ),
    tar_target(
        name=get_species,
        command = species,
    )
)
list(ids_mapped)

Which has the following output to tar_manifest(), so it's definitely reading the species list in some way...

# A tibble: 4 × 2
  name          command                                                                    
  <chr>         <chr>                                                                      
1 data_b        "grab_data(\"b\")"                                                         
2 get_species_a "list(species2 = c(\"CO2\", \"CO\"), params = c(1, 2))"                    
3 get_species_b "list(species2 = c(\"NH3\", \"NO2\", \"CH3\"), params = c(3, \n     4, 5))"
4 data_a        "grab_data(\"a\")"  

But when I try to iterate over the species I get an error:

# ... as above

species_mapped <- tar_map(
    values=ids_mapped,
    names="species2",
    tar_target(
        name=foo,
        command=species_2
    )
)
list(ids_mapped, species_mapped)

Error:
! Error running targets::tar_manifest()
  Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
  Debugging guide: https://books.ropensci.org/targets/debugging.html
  How to ask for help: https://books.ropensci.org/targets/help.html
  Last error: <text>:1:1: unexpected '<'
1: <

Or a different error if I try to directly try to use the output of the get_species target:

ids_mapped <- tar_map(
    values=config,
    names="id",
    tar_target(
        name = data,
        command = grab_data(id)
    ),
    tar_target(
        name=get_species,
        command = species,
    ),
    tar_map(
        values=get_species,
        tar_target(
            name=test,
            command=species2
        )
    )
)
list(ids_mapped)

Error:
! Error running targets::tar_manifest()
  Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
  Debugging guide: https://books.ropensci.org/targets/debugging.html
  How to ask for help: https://books.ropensci.org/targets/help.html
  Last error: object 'get_species' not found

[wlandau]

The helpers in tarchetypes are designed for functional programming over a flattened list of inputs. For a hierarchical structure, it is probably clearer to write out each target manually, even if you have upwards of 10 or 20 of them. If you have hundreds, then it could get cumbersome, but you could reduce the volume of code if each inner list is a tar_map().

library(targets)
library(tarchetypes)
list(
  tar_target(data_a, get_data(id = "a")),
  tar_target(data_b, get_data(id = "b")),
  tar_map(
    values = list(species_name = c("CO2", "CO")),
    tar_target(species_a, f(data_a, species_name))
  ),
  tar_map(
    values = list(species_name = c("NH3", "NO2", "CH3")),
    tar_target(species_b, f(data_b, species_name))
  )
)

[stulacy]

Thanks @wlandau for that explanation of the idiomatic way of achieving this in tarchetypes.