Move to .bat file and add Terraform role for bucket permissions (CU-D…

…BMI#2)

* changes for assayworks project specifics

* update shell script with bucket name

* detemplatize; gsutil instructions

* assay works docs context; simplify source dir

* bucket.get svc-acct role binding

* move gsutil work from .sh to .bat file

Co-Authored-By: Faisal Alquaddoomi <faisal.alquaddoomi@cuanschutz.edu>

* linting; remove shellcheck

* rename gsutil file, remove .sh file

* remove shellcheck

Co-authored-by: Faisal Alquaddoomi <faisal.alquaddoomi@cuanschutz.edu>

.gitignore

@@ -205,3 +205,5 @@ cue.mod/dagger.*
 
 # data ignores
 *.json
+*.zip
+*.tar.gz

.pre-commit-config.yaml

@@ -39,8 +39,3 @@ repos:
       - id: terraform_validate
       - id: terraform_tflint
       - id: terraform_tfsec
-  # checking yaml formatting
-  - repo: https://github.com/shellcheck-py/shellcheck-py
-    rev: v0.9.0.2
-    hooks:
-      - id: shellcheck

README.md

@@ -26,7 +26,8 @@ See below for steps which are required for installation.
 1. [Clone the repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) to your development environment.
 1. Install [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)
 1. Configure Terraform as necessary to your Google Cloud environment.
-1. __Optional__: make changes to script under `./utilities/example_gsutil_sync.sh` in preparation for synchronizing data to or from the bucket.
+1. __Optional__: make changes to script under `./utilities/example_gsutil_sync.bat` in preparation for synchronizing data to or from the bucket.
+
 
 ## :books:Tutorial
 
@@ -59,7 +60,8 @@ These steps cover an example of how to use the bucket with an example [gsutil](h
 1. Change directory into `./utilities`
 1. Ensure `service-account.json` key is found within `./utilities` directory (becomes available after infrastructure steps are taken with Terraform).
 1. Make changes to `gsutil rsync ...` line to specify the local data location and the target bucket.
-1. Run the `gsutil_sync.sh` script (for example: `sh ./gsutil_sync.sh`).
+1. Run the `gsutil_sync.bat` script by double clicking it or from a command line prompt (for example, by typing: `gsutil_sync.bat` and hitting the enter key).
+
 
 ## 🧑‍💻 Development
 

terraform/README.md

@@ -27,6 +27,7 @@ No modules.
 | [google_service_account.service_account](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account) | resource |
 | [google_service_account_key.key](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account_key) | resource |
 | [google_storage_bucket.target_bucket](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket) | resource |
+| [google_storage_bucket_iam_binding.bucket_get_binding](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket_iam_binding) | resource |
 | [google_storage_bucket_iam_member.member](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket_iam_member) | resource |
 | [local_file.service_account_key](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource |
 

terraform/storage.tf

@@ -9,8 +9,22 @@ resource "google_storage_bucket" "target_bucket" {
   }
 }
 
+# adds member to bucket with objectAdmin permissions
+# see the following link for more information on roles:
+# https://cloud.google.com/storage/docs/access-control/iam-roles
 resource "google_storage_bucket_iam_member" "member" {
   bucket = google_storage_bucket.target_bucket.name
   role   = "roles/storage.objectAdmin"
   member = "serviceAccount:${google_service_account.service_account.email}"
 }
+
+# granting additional role for rsync operations
+# which require storage.buckets.get access
+resource "google_storage_bucket_iam_binding" "bucket_get_binding" {
+  bucket = google_storage_bucket.target_bucket.name
+
+  role = "roles/storage.legacyBucketReader"
+  members = [
+    "serviceAccount:${google_service_account.service_account.email}",
+  ]
+}

utilities/README.md

@@ -3,8 +3,9 @@
 Thank you for your help in uploading data as part of this project! Please see the following instructions on uploading data to the Google Cloud bucket.
 
 1. Ensure `service-account.json` key is found within the same directory where script is run.
-1. Prepare data to be uploaded under `./data` directory relative to `gsutil_sync.sh` location.
-1. Run the `gsutil_sync.sh` script (for example: `sh ./gsutil_sync.sh`).
+1. Prepare data to be uploaded under `./data` directory relative to `gsutil_sync.bat` location.
+1. Run the `gsutil_sync.bat` script by double clicking it or from a command line prompt (for example, by typing: `gsutil_sync.bat` and hitting the enter key).
+
 
 Please reference the following directory tree structure for an example of what the path should contain:
 
@@ -13,12 +14,13 @@ Please reference the following directory tree structure for an example of what t
 ├── README.md
 ├── data
 │   └── <data to be synchronized>
-├── gsutil_sync.sh
+├── gsutil_sync.bat
 └── service-account.json
 ```
 
 ## Additional Notes
 
-- __Alternative data upload path__: if an alternative data upload path is preferred, please reference and update `gsutil_sync.sh` as follows:
-  - Original: `gsutil rsync ./data gs://waylab-assayworks-bucket`
+- __Alternative data upload path__: if an alternative data upload path is preferred, please reference and update `gsutil_sync.bat` as follows:
+  - Original: `gsutil rsync data gs://waylab-assayworks-bucket`
   - Updated: `gsutil rsync <new data location> gs://waylab-assayworks-bucket`
+- __Additional gsutil rsync options__: additional options for the `gsutil rsync` command may be found from the following link: <https://cloud.google.com/storage/docs/gsutil/commands/rsync>

utilities/gsutil_sync.bat

@@ -0,0 +1,24 @@
+::  This file automates how data are sync'd to a
+::  Google Cloud Cloud Storage bucket using a
+::  pre-existing service account key.
+::
+::  Notes:
+::  ----------------------------------------------------
+::  presumes gsutil has already been installed and is
+::  available in the path.
+::  see gsutil docs for more information:
+::  https://cloud.google.com/storage/docs/gsutil_install
+
+::  authenticate gcloud for the service account
+::  note: this is the preferred method for authenticating gsutil
+::  see the following for more details:
+::  https://cloud.google.com/storage/docs/gsutil/commands/config#configuring-service-account-credentials
+call gcloud auth activate-service-account --key-file=service-account.json
+
+::  synchronize data from local directory `./data`
+::  to bucket lab-initiative-bucket
+::  see the following for more details:
+::  https://cloud.google.com/storage/docs/gsutil/commands/rsync
+call gsutil rsync -r data gs://waylab-assayworks-bucket
+
+pause