Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First Pass: Containerize and schedule billing utility - Ticket #1068 #6

Merged
merged 18 commits into from
Jun 14, 2022

Conversation

unoah
Copy link
Contributor

@unoah unoah commented Jun 9, 2022

First pass with revising the billing utility to run on ECS Fargate - Ticket #1068

Overview of some changes

  • Moved some resources from running on the ECF management account to the operations account
  • Scaled back permissions around querying LZ org accounts and performing Athena queries Ticket #1067
  • Provisioning of resources in operations and management ECF accounts can now be managed by Terraform as opposed to being created from the AWS console
  • Will rely on EventBridge for initiating weekly, monthly and quarterly reports
  • Revised Athena query output bucket
    • Bucket is now created via IaC and passed as environment variable as opposed to hard coded in Python code.
    • Name revised from bcgov-aws-sea-billing-reports to bcgov-ecf-billing-reports-output-<AWS-AccountID>-<AWS-Region>. This will eliminate name collisions as we deploy across LZ0, LZ1 and LZ2 using a deployment pipeline.
  • Some cleanup on Python functions.
  • CLI arguments managed by argparse have been removed. Arguments needed for non-standard fiscal date ranges are now set as environment variables.
  • Local runs on an admin workstation for non-standard fiscal date ranges uses environment variables. Environment variables needed for local run execution can be found in Parameter Store /bcgov/billingutility/manual_run/env_vars of the ECF operations account.
  • No longer require Docker when generating reports for non-standard fiscal date ranges.
  • Some updates to readme with high level overview of revised solution.
  • Cleaned up Python dependencies, addressing conflicts and security/vulnerability issues as per dependabot alerts.

Testing

  • Ran tests for weekly, monthly, quarterly and manual (non-standard fiscal date ranges) from my local workstation.
  • Deployed resources to dev ECF operations and management accounts and validated EventBridge for weekly, monthly and quarterly executed successfully.

Other comments

  • CI/CD component not included in this PR. That is being worked on.
  • FOREX component as indicated in Ticket #1068 not revised yet
  • Additional updates needed for readme
  • Next set of PRs will be significantly smaller as those will contain incremental changes. Apologies for the large PR.

unoah added 18 commits May 19, 2022 10:15
…ial setup for manual billing report generation.
@unoah unoah requested review from wrnu and MaxWardle June 9, 2022 19:05
@unoah unoah self-assigned this Jun 9, 2022
@unoah unoah merged commit 71e32c2 into dev Jun 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants