diff --git a/.gitignore b/.gitignore index b025bd7..65aaa4c 100644 --- a/.gitignore +++ b/.gitignore @@ -205,3 +205,5 @@ cue.mod/dagger.* # data ignores *.json +*.zip +*.tar.gz diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 217068b..05cb755 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -39,8 +39,3 @@ repos: - id: terraform_validate - id: terraform_tflint - id: terraform_tfsec - # checking yaml formatting - - repo: https://github.com/shellcheck-py/shellcheck-py - rev: v0.9.0.2 - hooks: - - id: shellcheck diff --git a/README.md b/README.md index 38180c9..c82ed11 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,8 @@ See below for steps which are required for installation. 1. [Clone the repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) to your development environment. 1. Install [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) 1. Configure Terraform as necessary to your Google Cloud environment. -1. __Optional__: make changes to script under `./utilities/example_gsutil_sync.sh` in preparation for synchronizing data to or from the bucket. +1. __Optional__: make changes to script under `./utilities/example_gsutil_sync.bat` in preparation for synchronizing data to or from the bucket. + ## :books:Tutorial @@ -59,7 +60,8 @@ These steps cover an example of how to use the bucket with an example [gsutil](h 1. Change directory into `./utilities` 1. Ensure `service-account.json` key is found within `./utilities` directory (becomes available after infrastructure steps are taken with Terraform). 1. Make changes to `gsutil rsync ...` line to specify the local data location and the target bucket. -1. Run the `gsutil_sync.sh` script (for example: `sh ./gsutil_sync.sh`). +1. Run the `gsutil_sync.bat` script by double clicking it or from a command line prompt (for example, by typing: `gsutil_sync.bat` and hitting the enter key). + ## 🧑‍💻 Development diff --git a/terraform/README.md b/terraform/README.md index acc4cfd..b0edcd1 100644 --- a/terraform/README.md +++ b/terraform/README.md @@ -27,6 +27,7 @@ No modules. | [google_service_account.service_account](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account) | resource | | [google_service_account_key.key](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/service_account_key) | resource | | [google_storage_bucket.target_bucket](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket) | resource | +| [google_storage_bucket_iam_binding.bucket_get_binding](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket_iam_binding) | resource | | [google_storage_bucket_iam_member.member](https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/storage_bucket_iam_member) | resource | | [local_file.service_account_key](https://registry.terraform.io/providers/hashicorp/local/latest/docs/resources/file) | resource | diff --git a/terraform/storage.tf b/terraform/storage.tf index d8471c0..bc7fc71 100644 --- a/terraform/storage.tf +++ b/terraform/storage.tf @@ -9,8 +9,22 @@ resource "google_storage_bucket" "target_bucket" { } } +# adds member to bucket with objectAdmin permissions +# see the following link for more information on roles: +# https://cloud.google.com/storage/docs/access-control/iam-roles resource "google_storage_bucket_iam_member" "member" { bucket = google_storage_bucket.target_bucket.name role = "roles/storage.objectAdmin" member = "serviceAccount:${google_service_account.service_account.email}" } + +# granting additional role for rsync operations +# which require storage.buckets.get access +resource "google_storage_bucket_iam_binding" "bucket_get_binding" { + bucket = google_storage_bucket.target_bucket.name + + role = "roles/storage.legacyBucketReader" + members = [ + "serviceAccount:${google_service_account.service_account.email}", + ] +} diff --git a/utilities/README.md b/utilities/README.md index 9bd3cdf..a2556a7 100644 --- a/utilities/README.md +++ b/utilities/README.md @@ -3,8 +3,9 @@ Thank you for your help in uploading data as part of this project! Please see the following instructions on uploading data to the Google Cloud bucket. 1. Ensure `service-account.json` key is found within the same directory where script is run. -1. Prepare data to be uploaded under `./data` directory relative to `gsutil_sync.sh` location. -1. Run the `gsutil_sync.sh` script (for example: `sh ./gsutil_sync.sh`). +1. Prepare data to be uploaded under `./data` directory relative to `gsutil_sync.bat` location. +1. Run the `gsutil_sync.bat` script by double clicking it or from a command line prompt (for example, by typing: `gsutil_sync.bat` and hitting the enter key). + Please reference the following directory tree structure for an example of what the path should contain: @@ -13,12 +14,13 @@ Please reference the following directory tree structure for an example of what t ├── README.md ├── data │   └── -├── gsutil_sync.sh +├── gsutil_sync.bat └── service-account.json ``` ## Additional Notes -- __Alternative data upload path__: if an alternative data upload path is preferred, please reference and update `gsutil_sync.sh` as follows: - - Original: `gsutil rsync ./data gs://waylab-assayworks-bucket` +- __Alternative data upload path__: if an alternative data upload path is preferred, please reference and update `gsutil_sync.bat` as follows: + - Original: `gsutil rsync data gs://waylab-assayworks-bucket` - Updated: `gsutil rsync gs://waylab-assayworks-bucket` +- __Additional gsutil rsync options__: additional options for the `gsutil rsync` command may be found from the following link: diff --git a/utilities/gsutil_sync.bat b/utilities/gsutil_sync.bat new file mode 100755 index 0000000..a5fb955 --- /dev/null +++ b/utilities/gsutil_sync.bat @@ -0,0 +1,24 @@ +:: This file automates how data are sync'd to a +:: Google Cloud Cloud Storage bucket using a +:: pre-existing service account key. +:: +:: Notes: +:: ---------------------------------------------------- +:: presumes gsutil has already been installed and is +:: available in the path. +:: see gsutil docs for more information: +:: https://cloud.google.com/storage/docs/gsutil_install + +:: authenticate gcloud for the service account +:: note: this is the preferred method for authenticating gsutil +:: see the following for more details: +:: https://cloud.google.com/storage/docs/gsutil/commands/config#configuring-service-account-credentials +call gcloud auth activate-service-account --key-file=service-account.json + +:: synchronize data from local directory `./data` +:: to bucket lab-initiative-bucket +:: see the following for more details: +:: https://cloud.google.com/storage/docs/gsutil/commands/rsync +call gsutil rsync -r data gs://waylab-assayworks-bucket + +pause diff --git a/utilities/gsutil_sync.sh b/utilities/gsutil_sync.sh deleted file mode 100755 index 37620eb..0000000 --- a/utilities/gsutil_sync.sh +++ /dev/null @@ -1,24 +0,0 @@ -#!/bin/sh -# -# This file automates how data are sync'd to a -# Google Cloud Cloud Storage bucket using a -# pre-existing service account key. -# -# Notes: -# ---------------------------------------------------- -# presumes gsutil has already been installed and is -# available in the path. -# see gsutil docs for more information: -# https://cloud.google.com/storage/docs/gsutil_install - -# authenticate gcloud for the service account -# note: this is the preferred method for authenticating gsutil -# see the following for more details: -# https://cloud.google.com/storage/docs/gsutil/commands/config#configuring-service-account-credentials -gcloud auth activate-service-account --key-file=./service-account.json - -# synchronize data from local directory `./data` -# to bucket lab-initiative-bucket -# see the following for more details: -# https://cloud.google.com/storage/docs/gsutil/commands/rsync -gsutil rsync ./data gs://waylab-assayworks-bucket