Skip to content

Add CapePy Python library across the infrastructure#173

Merged
mehalter merged 27 commits into
mainfrom
capepy
Nov 8, 2024
Merged

Add CapePy Python library across the infrastructure#173
mehalter merged 27 commits into
mainfrom
capepy

Conversation

@mehalter
Copy link
Copy Markdown
Member

@mehalter mehalter commented Oct 29, 2024

The capepy provides development tooling and helper functions for users to more easily work with the CAPE system. This should be made available to all resources such as AWS Lambda and AWS Glue resources. This adds the library in a way that the library should be automatically available across all of the locations by adding it to the CAPE Meta component.

TODO

  • AWS Lambda
    • Make a Lambda Layer
    • Add the Lambda Layer to all Lambda Functions
    • Update all Lambda functions to use capepy
  • AWS Glue
    • Add wheel to S3
    • Always add the S3 location as an additional python module
    • Update all ETL scripts to use capepy
  • Clean up and finalize structure of things like parameters/etc. and update all scripts to use the capepy package
  • Refactor uses of CapeMeta properties to use the new self.meta accessor.
  • Force @thecaffiend to make a PyPI account and take ownership and liability for capepy

This also makes the CapeMeta resources globally available to all CapeComponentResources.

Closes #153, Closes #86, Closes #150

Comment thread capepulumi.py Outdated
Comment thread assets/etl/etl_fasta_fastq.py
Comment thread assets/etl/etl_fasta_fastq.py Outdated
Comment thread assets/lambda/new_dap_results_queue_notifier_lambda.py
Comment thread assets/lambda/new_dap_results_queue_notifier_lambda.py Outdated
Comment thread assets/lambda/new_s3obj_queue_notifier_lambda.py
Comment thread assets/lambda/sqs_dap_submit_lambda.py Outdated
Comment thread assets/lambda/sqs_etl_job_trigger_lambda.py
Comment thread capeinfra/meta/capemeta.py Outdated
Comment thread capeinfra/pipeline/batch.py Outdated
mehalter and others added 21 commits November 8, 2024 13:42
…e key 'pipeline_parameters'. added a todo to make this into a class that can also handle SerDe
@mehalter mehalter marked this pull request as ready for review November 8, 2024 19:25
@mehalter mehalter requested a review from thecaffiend November 8, 2024 19:25
Copy link
Copy Markdown
Member

@thecaffiend thecaffiend left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This killed all my remaining caffeination...

@mehalter mehalter merged commit d2c3e69 into main Nov 8, 2024
@mehalter mehalter deleted the capepy branch November 8, 2024 20:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants