S3 and file targets: changing file contents does not outdate the target

[jds485]

I am using S3 as the targets repository. I've noticed that when the files which are tracked in targets are changed, the changes are not triggering a rebuild of the targets. I'm using this target format for the files:

tar_option_set(
               resources = tar_resources(
                 aws = tar_resources_aws(bucket = "bucket_name")),
               repository = "aws",
               format = "rds"
               )
...
tar_target(
    file_csv, 
    'test.csv',
    format = 'file'
  ),
  tar_target(
    read_file, 
    read_csv(file_csv)
  ),
...

I am not sure if this is an issue or a user error.

Here is my sessionInfo()

sessionInfo()
R version 4.1.3 (2022-03-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.4 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] clustermq_0.8.95.3 targets_0.11.0    

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.8.3       pillar_1.7.0       compiler_4.1.3     class_7.3-20       tools_4.1.3       
 [6] digest_0.6.29      lifecycle_1.0.1    tibble_3.1.6       pkgconfig_2.0.3    rlang_1.0.2       
[11] igraph_1.3.0       DBI_1.1.2          cli_3.2.0          yaml_2.3.5         xfun_0.30         
[16] e1071_1.7-9        withr_2.5.0        dplyr_1.0.8        knitr_1.38         generics_0.1.2    
[21] vctrs_0.4.0        classInt_0.4-3     grid_4.1.3         tidyselect_1.1.2   glue_1.6.2        
[26] data.table_1.14.2  sf_1.0-7           R6_2.5.1           processx_3.5.3     fansi_1.0.3       
[31] base64url_1.4      purrr_0.3.4        callr_3.7.0        magrittr_2.0.3     backports_1.4.1   
[36] ps_1.6.0           codetools_0.2-18   ellipsis_0.3.2     units_0.8-0        assertthat_0.2.1  
[41] utf8_1.2.2         KernSmooth_2.23-20 proxy_0.4-26       crayon_1.5.1

[wlandau]

To send the file to your S3 bucket, you will need repository = "aws" as well as format = "file". An example is at https://books.ropensci.org/targets/data.html#usage.

[jds485]

Okay, thanks. So targets has different handling of file targets when repository = local (changes are tracked) vs. when using the cloud. I think that point should be made clear in the user manual. It seems like files that are input data (as in our case) should always be local so that changes to them will be tracked.

[wlandau]

Yesterday I did add more text to the manual and help files based on this thread. It used to be clearer because there was an "aws_file" format before I separated the "aws" part into the "repository" argument.

[januz]

@wlandau I'm also struggling with this somewhat conceptually:

• it's not intuitive for me that the _targets.R file still specifies a local path (under which the file does not exist anymore after targets uploaded it)
• if I'm renaming and pruning (or destroying) targets, the uploaded files get lost

Also, how would you imagine this working with files that do need to be changed over time. Should users change the {target_name}_file object in the _targets/objects/ directory in in the AWS bucket?!

[wlandau]

it's not intuitive for me that the _targets.R file still specifies a local path (under which the file does not exist anymore after targets uploaded it)

This lets users control how and where the file is staged before uploading. The other alternative for format = "file" is to require users to save to a specific pre-defined path, which seems more confusing.

if I'm renaming and pruning (or destroying) targets, the uploaded files get lost

If you only rename a target, you can still find the original data object in the bucket, e.g. with the AWS web console. In functions like tar_delete() and tar_destroy(), part of the intention is to remove data in the bucket. In both those functions, you can set arguments like cloud = FALSE and destroy = "local" to avoid deleting cloud data.

Also, how would you imagine this working with files that do need to be changed over time. Should users change the {target_name}_file object in the _targets/objects/ directory in in the AWS bucket?!

It depends on how the changes are meant to happen. If the file only changes through tar_make(), then downstream targets will automatically update, and there is no issue. If you expect to manually modify the file, e.g. if it is an Excel spreadsheet where you enter the data manually, then it is best to keep that file local with repository = "local".

[januz]

@wlandau Thanks so much for the further explanation! Yes, I was thinking of cases where the content of a file would be changed in an external program. I will use repository = "local" for those. If I understand correctly, I won't be able to use tarchetype::tar_file() then because I'd like for the target itself to be stored in AWS. I guess I'll have to specify a file target and an outcome target separately using base targets functionality.