feat(ci): add scheduled oci integration-test workflows

[blackboxsw]

Blocked by #7070

Proposed Commit Message

feat(ci): add scheduled oci integration-test workflows

Create oci workflow integration test coverage for Oracle on Ubuntu
Jammy, Noble, Resolute and Stonking scheduled twice-weekly:
 - Mon & Thu, cron '2 22 * * 1,4

Convert the shared 100-dispatch-common.yml workflow to a
composite action to extract any platform-specific steps in
oci scheduled workflows to the specific oci schedule jobs.

Retain all shared platform-agnostic common operations in
a single composite action.

Refactor all scheduled workflows to pass env variables as inputs
to composite action instead of secrets for the following data:
   REQUIRED_SECRET, SSH_PUBLIC_KEY, SSH_PRIVATE_KEY,
   GITHUB_TOKEN, and PYCLOUDLIB_*OCI*.
  - clean up the oci dir in the Clean pycloudlib step

Add two mandatory secrets for oci scheduled workflows:
  PYCLOUDLIB_OCI_CONFIG_B64 and PYCLOUDLIB_OCI_KEY_B64

Additional Context

Success run on individual oci creds: https://github.com/blackboxsw/cloud-init/actions/runs/30711239669

Test Steps

1. Can adapt this PR to run on your own remote:

sed -i 's/canonical\/cloud-init/YOUR_GHUSERNAME\/cloud-init/' .github/workflows/100-common-dispatch.yml
git commit -am 'WIP: adapt to run integration tests on my github remote'
git push YOUR_REMOTE THIS_BRANCH:main # --force to ensure github actions on your remote can be run directly from /github.com/YOUR_REMOTE/cloud-init/actions 

2. provide base64 encoded pycloudlib.toml with an [oci] section documenting your config

[oci]
config_path = "$OCI_CONFIG_FILE_PATH"
region = "us-...-1"
availability_domain = "qIZq:......."
compartment_id = "ocid1.compartment......."
public_key_path = "~/.ssh/cloudinit_id_rsa.pub"
vcn_name = "default"

3. Add action SECRETS to your remote to test workflow https://github.com/YOUR_REMOTE/cloud-init/settings/secrets/actions

• Create an oci config file via new API Keys from cloud.oracle.com
  PYCLOUDLIB_OCI_CONFIG_B64 # Copy value from: cat oci.config | base64 -w 0

• Grab private key pem from cloud.oracle.com
  PYCLOUDLIB_OCI_KEY_B64 # Copy value from: cat your-team-login-date*pem | base64 -w 0

• PYCLOUDLIB_TOML_B64 # copy value from cat pycloudlib.toml | base64 -w 0

SSH_PRIVATE_KEY # Copy shared CI private key used for all integration tests
SSH_PUBLIC_KEY # ditto public key

Merge recommendations

• Squash merge using "Proposed Commit Message"
• Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)

[holmanb]

File naming scheme matches the pattern, thanks for that. This looks pretty good.

I have a couple of minor questions related to organization and reuse.

[holmanb]

Cloud-specific code in a common workflow seems messy. I worry that this shared file will become increasingly difficult to understand due to cloud-specific conditional code. Can this be done outside of the shared workflow? Some question applies for LXD as well.

[blackboxsw]

A github actions job which calls a reusable workflow via workflow_call cannot also contain steps: because the entire job is delegated to the called workflow. So we cannot prepend a step like setting up lxd or writing an OCI file, and expect that LXD is configured or that a file is persisted for the common pycloudlib integration test run because they are completely separate jobs. Called jobs appear to only obtain data files via actions/download-artifact and the preparatory job or calling job would use actions/upload-artifact to setup whatever payload we want to pass in, but that would potentially expose our jobs to transporting senstive files between jobs which may be more risky.

Our alternatives appear to be to create a separate 100-dispatch-common-oci.yml job which would would have the inline step and mandatory secret checks for OCI stuff. This would contain nearly the same duplicated steps as our 100-dispatch-common.yml job, with the exception of the mandatory secret check for PYCLOUDLIB_OCI_* secrets and the writing of the oci config and pem files.

If we decide to tease out 100-dispatch-common-oci, we'd also want to tease out lxd setup into 100-dispatch-common-lxd.yml. At that point we would have 3 nearly duplicated 100-dispatchyml actions to differentiate truly common, lxd and oci pre-requisite steps. So, it looks like we either carry around 135 lines duplicated in 3 dispatch-common.yml workflows or cope with the 40-50 lines of platform conditionals inside the single 100-dispatch-common.yml.

If we decided to stay with platform conditionals inside 100-dispatch-common.yml we can further reduce the amount of vertical space used to represent those steps by creation a composite action such as uses: .github/actions/setup-oci-keys which could encapsulate the oci secrets validation and writing keys so we don't have to grok that detail in 100-dispatch-common.yml.

I'm leaning toward keeping the 100-dispatch-common.yml, adding small composite actions to reduce the reading load on the truly shared 100-dispatch-common.yml. But, if you feel strongly that we should differentiate 100-dispatch-common-oci and common-lxd I can propose that difference for a quick look.

Here's a small spike showing what a semi-composite common workflow and a separate common-oci workflow could look compared with this current branch

[holmanb]

Our alternatives appear to be to create a separate 100-dispatch-common-oci.yml job

This adds quite a bit of both indirection and almost fully duplicates the current common file (plus it duplicates one of our number prefixes which breaks an existing pattern).

Is there any reason we couldn't have our OCI workflows do something like this:

      - name: Setup OCI credentials
        env:
          PYCLOUDLIB_OCI_CONFIG_FILE_PATH: ${{ runner.temp }}/oci/config
          PYCLOUDLIB_OCI_KEY_FILE_PATH: ${{ runner.temp }}/oci/key.pem
        run: |
          mkdir -p "$(dirname "$PYCLOUDLIB_OCI_CONFIG_FILE_PATH")"
          sh -c 'printf "%s\n" "$PYCLOUDLIB_OCI_CONFIG_B64" | base64 -d > "$PYCLOUDLIB_OCI_CONFIG_FILE_PATH"'
          sh -c 'printf "%s\n" "$PYCLOUDLIB_OCI_KEY_B64" | base64 -d  | install -m 600 /dev/stdin "$PYCLOUDLIB_OCI_KEY_FILE_PATH"'
      - name: Run integration Tests
        uses: ./.github/workflows/100-dispatch-oci.yml
        with:
...

[blackboxsw]

There are a couple of things blocking here:

1. We cannot add preliminary steps in a single job and call a reusable workflow because the reusable workflow uses a different runner and hence a difference filesystem. this results in not seeing the file we created in the separate oci workflow's steps
2. Even if we converted all the 'steps' into composite actions which can be called by multiple jobs. The oci-specific or lxd-specific job is delegated to a separate runner to call the initial steps (setup-lxd or setup-oci-cred-files) and that job filesystem/environment is not reused, so we'd have to use upload-artifacts from setup oci workflow and download-artifacts from the common workflow.

That said, if we want to reduce the size of such platform-specific steps in the common workflow, I think migrating our platform-specific steps into composite actions will make it easier to read the 100-dispatch-common.yml because we will have something like:

- name: Setup OCI creds
   uses: ./.github/actions/setup-oci-creds

instead of

      - name: Setup OCI credentials
        env:
          PYCLOUDLIB_OCI_CONFIG_FILE_PATH: ${{ runner.temp }}/oci/config
          PYCLOUDLIB_OCI_KEY_FILE_PATH: ${{ runner.temp }}/oci/key.pem
        run: |
          mkdir -p "$(dirname "$PYCLOUDLIB_OCI_CONFIG_FILE_PATH")"
          sh -c 'printf "%s\n" "$PYCLOUDLIB_OCI_CONFIG_B64" | base64 -d > "$PYCLOUDLIB_OCI_CONFIG_FILE_PATH"'
          sh -c 'printf "%s\n" "$PYCLOUDLIB_OCI_KEY_B64" | base64 -d  | install -m 600 /dev/stdin "$PYCLOUDLIB_OCI_KEY_FILE_PATH"'

[blackboxsw]

@holmanb I made further progress when I converted the entire common workflow 100-common-dispatch.yml into a composite action. The supplemental 3 commits to make that change are here:

• convert common dispatch to common composite action
• Ensure checkout is performed in scheduled workflows before trying to reference composite action
• Fix unavailable runner.temp template variables in composite action

With these three additional commits, we have a successful oci workflow run

The common structure will be the following then:

1. scheduled platform-specific jobs:

• duplicated repo assertion (only run on upstream repo)
• platform specific secret assertions
• platform specific setup steps (and initial checkout) in 15X-*oci.yml
• initial checkout
• call common composite action integration-tests

2. common composite action integration tests:

• obtain inputs for runner_temp, platform, image_type, install_source, and filter_tests(name to be changed in next PR)
• perform common required secret assertions
• setup-lxd (TODO: move out into lxd-specific scheduled runners in next PR)
• remaining integration steps

If this approach is agreeable, I'll pull in those 3 commits and push to this branch for final review. Then I can followup with two small PRs:

1. rename of filter_tests -> integration_test_params
2. move setup-lxd out to platform specific scheduled workflows

I have pushed the proposed commits mentioned above to thisa PR as I believe that best aligns with your earlier review suggestions.

[blackboxsw]

Per review comments on #7047 I had missed converting the azure jobs to use composite action in this PR. I have remedied that issue here so it's central to the review and migration away from 100-dispatch-common.yml to .github/actions/integration-test/action.yml.

Successful dispatch of Azure here

[holmanb]

filter_tests doesn't seem like the best name for this. That is one thing that can be done, but it seems like one might add other arguments to tox via this variable name.

We can fix this in a followup.

[blackboxsw]

Filed #7048 for this name change.

[blackboxsw]

@holman this PR will need review before #7047 and #7048. To ensure we want a composite action instead of a common workflow via workflow_dispatch

[blackboxsw]

Per review comments on 7047 I had forgotten to address azure dropping the use of 100-dispatch-common.yml. Top most commit handles that unaddressed aspect of the migration to composite action for azure.

[holmanb]

Can we split this in two? It both adds OCI support and refactors all of the existing integration actions.

[blackboxsw]

Can we split this in two? It both adds OCI support and refactors all of the existing integration actions.

Separated out #7070 to review just the refactor.