Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

As a user, I want to deploy a baseline nucleus automatically. #22

Closed
2 of 3 tasks
jordanpadams opened this issue Jan 13, 2023 · 6 comments
Closed
2 of 3 tasks

As a user, I want to deploy a baseline nucleus automatically. #22

jordanpadams opened this issue Jan 13, 2023 · 6 comments
Assignees
@ramesh-maddegoda
Labels
B14.0 p.should-have requirement the current issue is a requirement sprint-backlog

Comments

@jordanpadams
Copy link
Member

jordanpadams commented Jan 13, 2023 •
edited by ramesh-maddegoda

🧑‍🔬 User Persona(s)

Sys Admin

💪 Motivation

...so that I can automatically deploy a new baseline nucleus instance with minimal manual intervention.

📖 Additional Details

  • Terraform scripts for deploying in AWS (deploy.sh?)
  • Docker container for local deployment (It was decided to use MWAA on AWS and a local deployment is not applicable with MWAA)
  • Document any manual requirements

⚖️ Acceptance Criteria

Given
When I perform
Then I expect

⚙️ Engineering Details
@jordanpadams jordanpadams added needs:triage requirement the current issue is a requirement labels Jan 13, 2023
@jordanpadams
Copy link
Member Author

Status: To be wrapped up by Thursday

@jordanpadams jordanpadams added B14.0 and removed B13.1 labels Mar 23, 2023
@tloubrieu-jpl
Copy link
Member

@ramesh-maddegoda solved a security issue when launching tasks on ECS.

@ramesh-maddegoda
Copy link
Contributor

Blocker: Cannot see any of the Managed Airflow environments (MWAA) created on NGAP anymore. Also, it seems we have lost MWAA related terraform access on this account.

Terraform Error: error reading MWAA Environment (basic-mwaa): AccessDeniedException: ..... not authorized to perform: airflow:GetEnvironment on resource

Possible Cause: The NGAP team has recently deployed some Service Control Policies for NGAP environments and that may have affected the access.

Currently waiting for NGAP team to resolve this.

@ramesh-maddegoda
Copy link
Contributor

The NGAP OPS team has fixed the permission issue. Ramesh is currently updating terraform scripts to fix a S3 bucket policy related permission issue.

@tloubrieu-jpl
Copy link
Member

@ramesh-maddegoda still has permission issues of different kinds, not blocked though

ramesh-maddegoda added a commit that referenced this issue Apr 6, 2023
@ramesh-maddegoda
ADD Initial terraform scripts to create Nucleus baseline Airflow (MWA…
ca2e1de 
…A) environment and ECS container definitions

Refer to the task #22
ramesh-maddegoda added a commit that referenced this issue Apr 6, 2023
@ramesh-maddegoda
ADD README file and a sample DAG for PDS Registry (work in progress)
4d5729e 
UPDATE MWAA Terraform with some cleanup

Refer to the task #22
@ramesh-maddegoda ramesh-maddegoda mentioned this issue Apr 6, 2023
Closed
@ramesh-maddegoda
Copy link
Contributor

ramesh-maddegoda commented Apr 6, 2023 •
edited

Created a pull request for Terraform script of Nucleus baseline system (MVP).
#42

The Terraform scripts in this pull request deploy a minimum viable product (MVP) of PDS Nucleus data pipeline system on AWS Cloud. Currently, Nucleus is based on Amazon Managed Workflows for Apache Airflow (MWAA). Therefore, as a result of the Terraform scripts in this directory following things will be created.

  • AWS Security Group for MWAA
  • AWS S3 Bucket with relevant bucket policies to keep Airflow DAG files and Python requirements file
  • Dags directory in S# bucket to keep Airflow DAG files
  • Python requirements.txt file to introduce the additional Python packages required by DAGs
  • Amazon Managed Workflows for Apache Airflow (MWAA)

Note: In addition to the above components, there are Terraform modules, container definitions and a DAG file ncluded to deploy PDS Registry related ECS tasks, a DAG and an EFS file system that can be used to demonstrate an example PDS Registry use case. However, these additional components are not part of the MVP of PDS Nucleus data pipeline. These PDS Registry related terraform modules are still under development (not part of the PDS Nucleus Baseline Deployment task) and are kept disabled in the main.tf terraform file.

ramesh-maddegoda added a commit that referenced this issue Apr 6, 2023
@ramesh-maddegoda
UPDATE README text formatting
717414e 
UPDATE main.tf file to disable project specific Terraform modules

Refer to the task #22
ramesh-maddegoda added a commit that referenced this issue Apr 27, 2023
@ramesh-maddegoda
ADD Terraform scripts to create Nucleus baseline product
1201018 
Refer to the task #22
@ramesh-maddegoda ramesh-maddegoda mentioned this issue Apr 27, 2023
Merged
@jordanpadams jordanpadams mentioned this issue May 9, 2023
Closed
tloubrieu-jpl pushed a commit that referenced this issue May 17, 2023
@ramesh-maddegoda
Terraform scripts to deploy PDS Nucleus baseline system (minimum viab…
754f2db 
…le product) and a Registry use case (#42)

* ADD Terraform scripts to create Nucleus baseline product

Refer to the task #22

* ADD A Registry use case with ECS tasks, a docker image and a DAG file

Refer to the task #24

* UPDATE code to fix a style issue

Refer to the task #24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ramesh-maddegoda ramesh-maddegoda

Labels
B14.0 p.should-have requirement the current issue is a requirement sprint-backlog
Projects
No open projects
B14.0
Status: 🏁 Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jordanpadams @tloubrieu-jpl @ramesh-maddegoda