chore(ci): source code tampering protection for release

[heitorlessa]

Issue number: #2206

Summary

Implements mechanism to seal source code, further break build job, further isolate release job with binaries only, and fixes the Git Tag creation process.

Changes

Please provide a summary of what's being changed

• Split quality check and build jobs
• Create integrity check
• Create a job to seal source code after bumping version
• Bump version and seal to prevent further boilerplate in code and workarounds
• Cache-artifact
• Restore-artifact and ensure it overwrites any checked out code
• Pin down Poetry to prevent any code tampering on build + publishing
• Update bump_version and create_tag to use sealed source code with version bumped
• Replace subsequent checkouts to use RELEASE_COMMIT and restore artifact
• Document every job with its details
• Scale down Runners to use ubuntu-latest as we no longer need those 4 cores (savings!!)
• Lower permissions further as jobs are isolated
• Add simple debug to display pyproject.toml in case sealed source code is incorrect
• Forcefully fail release if sealed source code hasn't been found or uploaded correctly
• Test new release
• Test whether tag created contains latest source code with version bumped
• Document overall process at the top
• Verify integrity hash (source and build) in key steps

Additional fixes

• bump_version needs to install poetry
• Fix create_tag as it needs to add staged changes before pushing... otherwise it pushes the current commit
• No longer need custom build machines for build/release.. but layer only

Threats to address

• Checking out newer code during the release process
  • Forcefully use RELEASE_COMMIT during checkout
  • Verifies hash throughout jobs (source and build)
• Source code tampering from dependencies or workflows
  • seal job to hash and reuse artifact throughout release
  • pipx install git+<url+SHA> for poetry to prevent rogue package manager
• Tamper build artifact during release
  • release-<build-hash> for traceability and immutability
  • pipx install git+<url+SHA> for poetry to prevent rogue package manager
  • release job doesn't install any dependencies but a pinned poetry
• Tamper git tag during release
  • seal job to hash and reuse artifact
  • No dependencies installed
  • Forcefully add pyproject.toml that changed in seal to our local git tree, then push that with the new tag only. This also solves the data race condition where PR is only merged later and our tag uses an older version.
• Push modified code to repository via 3rd-party actions / dependencies
  • Protected branch enabled in trunk
  • All source code modifications generate PR and require maintainer's review
  • Least privilege permissions are enforced per job
  • For Git tags, source code tampering check is also enforced
• Secret ex-filtration 3rd-party action / dependencies
  • GitHub Environment Secrets are used over standard Secrets
  • Encrypted secrets are only available for release environment
  • release environment requires manual approval before proceeding
• Release credentials overtake
  • Setup PyPi Trusted Publisher
  • Release credentials are temporary and only issued to what's configured in PyPi Trusted Publisher (org+repo+workflow+environment name)

User experience

Please share what the user experience looks like before and after this change

Checklist

If your change doesn't seem to apply, please leave them unchecked.

• Meet tenets criteria
• I have performed a self-review of this change
• Changes have been tested
• Changes are documented
• PR title follows conventional commit semantics

Is this a breaking change?

RFC issue number:

Checklist:

• Migration process documented
• Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

[rubenfonseca]

Reviewing now

[rubenfonseca]

Super good job, TIL that you can have your own custom actions on the same repo. Just left some questions and a tiny comment.

[rubenfonseca]

❤️

[rubenfonseca]

Amazing!

[heitorlessa]

@rubenfonseca gonna need your review one more time as I've added a hash verifier as per @leandrodamascena ask. Leandro reviewed

[heitorlessa]

Successful run publishing to Test PyPi repo (minus layers for good reasons and no change there): https://github.com/heitorlessa/aws-lambda-powertools-test/actions/runs/5053853611

[leandrodamascena]

APPROVED!! Amazing work @heitorlessa!

[heitorlessa]

Aleluia! <3