Skip to content

Commit

Permalink
feat: landsat example (#50)
Browse files Browse the repository at this point in the history
@johanneskoester I'm trying to get this working with gcs, but I'm
hitting issues about flags from snakemake:

```console
__main__.py: error: unrecognized arguments: --storage-s3-retries
```
We are installing the latest snakemake from main. Thanks!

Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Co-authored-by: vsoch <vsoch@users.noreply.github.com>
  • Loading branch information
vsoch and vsoch committed May 7, 2024
1 parent b0b7614 commit 944767f
Show file tree
Hide file tree
Showing 4 changed files with 57 additions and 0 deletions.
7 changes: 7 additions & 0 deletions example/landsat/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Landsat Example

Here is an example specifying to use an s3 bucket

```bash
snakemake --jobs 1 --executor googlebatch --googlebatch-region us-central1 --googlebatch-project llnl-flux --default-storage-provider gcs --default-storage-prefix gcs://change-me --storage-gcs-project llnl-flux
```
45 changes: 45 additions & 0 deletions example/landsat/Snakefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
import os

storage:
provider="gcs",
max_requests_per_second= None

rule all:
input:
"landsat-data.txt.bz2",
"testdir"

rule copy:
input:
storage.gcs("gcs://gcp-public-data-landsat/LC08/01/001/003/LC08_L1GT_001003_20170430_20170501_01_RT/LC08_L1GT_001003_20170430_20170501_01_RT_MTL.txt"),
output:
"landsat-data.txt"
resources:
mem_mb=100
run:
# we could test volume size like this but it is currently unclear what f1-micro instances provide as boot disk size
#stats = os.statvfs('.')
#volume_gib = stats.f_bsize * stats.f_blocks / 1.074e9
#assert volume_gib > 90
shell("cp {input} {output}")

rule pack:
input:
"landsat-data.txt"
output:
"landsat-data.txt.bz2"
conda:
"env.yml"
log:
"logs/pack.log"
shell:
"bzip2 -c {input} > {output}; echo successful > {log}"


rule directory:
output:
directory("testdir")
log:
"logs/directory.log"
shell:
"mkdir -p {output}; touch {output}/test.txt"
1 change: 1 addition & 0 deletions example/landsat/config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"message": "hahaha"}
4 changes: 4 additions & 0 deletions example/landsat/env.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
channels:
- conda-forge
dependencies:
- bzip2

0 comments on commit 944767f

Please sign in to comment.