Initialize Way Lab EM morphology Tokyo bucket (#1)

* initial template additions

* remove key file

* avoid tf variable error with static backend name

* update dagger ci version

* revert v4 of dagger-for-github for compatibility

* add waylab prefix to bucket name and label

* Update .gitignore

* Update LICENSE

* linting / tfdocs updates from pre-commit

* update to remove key file references

Co-Authored-By: Faisal Alquaddoomi <faisal.alquaddoomi@cuanschutz.edu>

---------

Co-authored-by: Faisal Alquaddoomi <faisal.alquaddoomi@cuanschutz.edu>

.github/workflows/test.yml

@@ -0,0 +1,24 @@
+---
+name: run tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  run_tests:
+    runs-on: ubuntu-latest
+    env:
+      DAGGER_CACHE_FROM: type=gha,scope=gc-cloud-storage-bucket
+      DAGGER_CACHE_TO: type=gha,mode=max,scope=gc-cloud-storage-bucket
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+      - name: Dagger
+        uses: dagger/dagger-for-github@v4
+        with:
+          cmds: |
+            project update
+            do test

.gitignore

@@ -1,3 +1,43 @@
+#########################################
+# Terraform gitignores
+# sourced from: https://github.com/github/gitignore/blob/main/Terraform.gitignore
+#########################################
+
+# Local .terraform directories
+**/.terraform/*
+
+# .tfstate files
+*.tfstate
+*.tfstate.*
+
+# Crash log files
+crash.log
+
+# Ignore any .tfvars files that are generated automatically for each Terraform run. Most
+# .tfvars files are managed as part of configuration and so should be included in
+# version control.
+#
+# example.tfvars
+
+# Ignore override files as they are usually used to override resources locally and so
+# are not checked in
+override.tf
+override.tf.json
+*_override.tf
+*_override.tf.json
+
+# Include override files you do wish to add to version control using negated pattern
+#
+# !example_override.tf
+
+# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
+# example: *tfplan*
+
+#########################################
+# Python gitignores
+# sourced from: https://github.com/github/gitignore/blob/main/Python.gitignore
+#########################################
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]
@@ -158,3 +198,15 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
+
+# dagger ignores
+cue.mod/pkg
+cue.mod/dagger.*
+
+# data ignores
+*.json
+*.zip
+*.tar.gz
+
+# terraform ignores
+hashicorp/

.pre-commit-config.yaml

@@ -0,0 +1,51 @@
+# See https://pre-commit.com for more information
+# See https://pre-commit.com/hooks.html for more hooks
+---
+# misc checks
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.5.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-added-large-files
+      - id: check-toml
+      - id: detect-private-key
+  # checking yaml formatting
+  - repo: https://github.com/adrienverge/yamllint
+    rev: v1.35.1
+    hooks:
+      - id: yamllint
+  # checking spelling
+  - repo: https://github.com/codespell-project/codespell
+    rev: v2.2.6
+    hooks:
+      - id: codespell
+        exclude: >
+          (?x)^(
+              .*\.lock|.*\.csv
+          )$
+  # checking markdown formatting
+  - repo: https://github.com/executablebooks/mdformat
+    rev: 0.7.17
+    hooks:
+      - id: mdformat
+        # ignore terraform autogenerated docs
+        exclude: >
+          (?x)^(
+              terraform/.* |
+              terraform/.*/.*
+          )$
+  # linting for terraform
+  - repo: https://github.com/antonbabenko/pre-commit-terraform
+    rev: v1.88.0
+    hooks:
+      - id: terraform_docs
+        args:
+          - --hook-config=--path-to-file=README.md
+          - --hook-config=--add-to-existing-file=true
+          - --hook-config=--create-file-if-not-exist=true
+      - id: terraform_fmt
+      - id: terraform_tflint
+      - id: terraform_trivy

LICENSE

@@ -1,6 +1,6 @@
 BSD 3-Clause License
 
-Copyright (c) 2024, The Way Lab
+Copyright (c) 2023, The Way Lab
 
 Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are met:

README.md

@@ -1,2 +1,95 @@
-# gc-bucket-em-morphology-tokyo
-GCP bucket for EM morphology data collaborations.
+# Google Cloud - Cloud Storage Bucket - waylab-em-morphology-tokyo-bucket
+
+```mermaid
+flowchart LR
+    subgraph upload
+      files[("file(s)")]
+      data-provider["👤 Data Provider"]
+
+    end
+    subgraph download
+      data-receiver["👤 Data Receiver"]
+      files2[("file(s)")]
+    end
+    bucket[\"Cloud Storage\n Bucket"/]
+    files --> |to| bucket
+    data-provider --> |uploads| files
+    bucket --> |provides\naccess to| files2
+    files2 --> |received by| data-receiver
+
+```
+
+Template leveraging [Cookiecutter](https://github.com/cookiecutter/cookiecutter) for creating [Cloud Storage](https://cloud.google.com/storage/) bucket on [Google Cloud](https://cloud.google.com/) with a service account and related key to enable data or file upload and use.
+
+This repository uses [Terraform](https://developer.hashicorp.com/terraform/intro) to maintain cloud resources. See [terraform/README.md](terraform/README.md) for documentation on Terraform elements.
+
+## 👥 Roles
+
+See below for an overview of roles which are important to context for various parts of this repository.
+
+- __Terraform Administrator__: this role involves administrating over cloud resources created with Terraform. Content found under the `terraform` directory and following steps under [Tutorial: Bucket Infrastructure](#%EF%B8%8F-bucket-infrastructure) apply to this role.
+- __Data Provider__: this role involves using content under `utilities/data-provider` to synchronize (add, update, or remove) data to the bucket created by a Terraform Administrator. Instructions specific to this role are provided under [`utilities/data-provider/README.md`](utilities/data-provider/README.md).
+- __Data Receiver__: this role is involved with downloading content from the bucket after it has been uploaded by the data provider. Associated content may be found under [`utilities/data-receiver/README.md`](utilities/data-receiver/README.md).
+
+## 🛠️ Install
+
+See below for steps which are required for installation.
+
+1. [Create a repository from this template](https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-repository-from-a-template).
+
+1. [Clone the repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository) to your development environment.
+
+1. Install [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli)
+
+1. Configure Terraform as necessary for your Google Cloud environment.
+
+## :books: Tutorial
+
+See below for brief tutorials on how to implement the work found in this repository for your needs.
+
+### 🎛️ State Management
+
+These steps cover how to store [Terraform state management](https://developer.hashicorp.com/terraform/language/state) files in association with the bucket infrastructure below. Terraform tracks cloud resources it creates as a statefile (`.tfstate`). If multiple people want to manage the same resources at the same time, they all need to have access to the same statefile or else they risk overwriting or corrupting state data. One option for sharing this statefile is to use a Google Cloud Bucket, which is the option used here.
+
+Note: Terraform cloud state management must be setup before it is referenced as a backend.
+
+1. Make adjustments to the content as necessary (for example, this readme file).
+1. Fill in [terraform.tfvars](terraform/state-management/terraform.tfvars) with values that make sense for your initiative (note: these are by default filled in from cookiecutter values).
+1. Terraform __init__: to ensure Terraform is initialized use command `terraform -chdir=terraform/state-management init`.
+1. Terraform __plan__: to plan the work and observe any needs use command `terraform -chdir=terraform/state-management plan` .
+1. Terraform __apply__: to apply the work and create resources use command `terraform -chdir=terraform/state-management apply`
+
+### 🏗️ Bucket Infrastructure
+
+These steps cover how to control the infrastructure found within this repository.
+
+1. Make adjustments to the content as necessary (for example, this readme file).
+1. Fill in [terraform.tfvars](terraform/operations/terraform.tfvars) with values that make sense for your initiative (note: these are by default filled in from cookiecutter values).
+1. Terraform __init__: to ensure Terraform is initialized use command `terraform -chdir=terraform/operations init`.
+1. Terraform __plan__: to plan the work and observe any needs use command `terraform -chdir=terraform/operations plan` .
+1. Terraform __apply__: to apply the work and create resources use command `terraform -chdir=terraform/operations apply`
+
+When finished with the work, optionally use the following step.
+
+- __OPTIONAL__: Terraform __destroy__: to destroy all created resources use command `terraform -chdir=terraform/operations destroy`
+
+### 📁 Using the Bucket
+
+These steps cover an example of how to use the bucket after creating the surrounding infrastructure.
+
+| <span style="text-align:left;float:left;font-weight:normal;"> ⚠️  Please note: be certain data you upload to Google Cloud abide any data governance or privacy restrictions applicable to your environment. The steps below do not inherently check or validate that data, the bucket, or the Google Cloud environment follow these policies. </span>   |
+|-----------------------------------------|
+
+- Data Upload (Data Provider): please see [`utilities/data-provider/README.md`](utilities/data-provider/README.md) for more information.
+- Data Download (Data Receiver): please see [`utilities/data-receiver/README.md`](utilities/data-receiver/README.md) for more information.
+
+## 🧑‍💻 Development
+
+Development for this repository is assisted by the following technologies:
+
+- [Poetry](https://python-poetry.org/docs/): Used to help configure pre-commit for local development work.
+- [Pre-commit](https://pre-commit.com): Used for performing checks within local development environment and via Github Actions automated testing. The following sub-items are used as checks through [pre-commit-terraform](https://github.com/antonbabenko/pre-commit-terraform) and require local installation when testing outside of Dagger:
+  - [terraform_docs](https://github.com/terraform-docs/terraform-docs/): Used to automatically generate Terraform-specific documentation.
+  - [tflint](https://github.com/terraform-linters/tflint): Used to perform static analysis (linting) on Terraform content.
+  - [tfsec](https://github.com/aquasecurity/tfsec): Used to perform security-focused static analysis (security linting) on Terraform content.
+- [Dagger](https://docs.dagger.io/): Used to help orchestrate reproducible testing within local development environment and for automated testing.

cue.mod/module.cue

@@ -0,0 +1 @@
+module: ""