Skip to content
This repository has been archived by the owner on Nov 26, 2023. It is now read-only.

Latest commit

 

History

History
44 lines (29 loc) · 1.52 KB

README.md

File metadata and controls

44 lines (29 loc) · 1.52 KB
Raw

databricks-spark-programs

Maintained Label Deprecated Label Archived Label

Code Migrated to andy-jarombek-research

Overview

Spark/Databricks programs hosted on a Databricks environment on AWS.

Commands

Set GitHub Secrets

brew install gh
gh auth login

gh secret set DATABRICKS_ACCOUNT_USERNAME
gh secret set DATABRICKS_ACCOUNT_PASSWORD
gh secret set DATABRICKS_ACCOUNT_ID

gh secret set AWS_ACCESS_KEY_ID
gh secret set AWS_SECRET_ACCESS_KEY

Directories

Directory Name Description
.github GitHub Actions for CI/CD pipelines.
infra Infrastructure for a Databricks workspace and resources within a workspace.

Version History

v1.0.0 - Initial Version

Release Date: Mar 4th, 2023

  • Terraform infrastructure successfully creates Databricks E2 workspace and objects within the workspace
  • GitHub Actions workflow for formatting Terraform
  • Databricks jobs and notebooks showing the basics of working with Spark in Databricks