feat: test reusable workflow in e2e test

[sailorlqh]

fix: #128

[sailorlqh]

Added @bradegler and @verbanicm as reviewers to get some input on the terraform code as they are the experts.

[sqin2019]

Please correct me if I am wrong, you want to test both reusable workflows (upload file to gcs via reusable workflow) and code logic (manually upload file to gcs) on whatever code changes. It seems redundant. We can test reusable workflow when there are changes to them, if no changes to these workflow files then test the code logic. Therefore, we can have a separate workflow to test reusable workflow. Also smaller PRs are preferred.

Another way is to test reusable workflow -> only check if files successfully uploaded to gcs in ci, test the entire e2e in dev when doing the release, this way we don't need to change the current ci infrastructure. Only concern is what if e2e fail, we need to fix it asap and release a new version.

Just my thoughts, I will wait for other people's review.

[sailorlqh]

Please correct me if I am wrong, you want to test both reusable workflows (upload file to gcs via reusable workflow) and code logic (manually upload file to gcs) on whatever code changes. It seems redundant. We can test reusable workflow when there are changes to them, if no changes to these workflow files then test the code logic. Therefore, we can have a separate workflow to test reusable workflow.

The idea for this update is to ensure the workflow's behavior is correct. According to previous discuss with Chen, we decide to add this to the CI e2e test.

Also smaller PRs are preferred.

I don't think I can further break down this PR, as calling a reusable workflow in ci.yml will require the related terraform code change, and also the tests should be updated.

Another way is to test reusable workflow -> only check if files successfully uploaded to gcs in ci, test the entire e2e in dev when doing the release, this way we don't need to change the current ci infrastructure. Only concern is what if e2e fail, we need to fix it asap and release a new version.
To make sure the reusable workflow can upload files to GCS, we still need to call the workflow, thus the terraform code change is inevitable. And we didn't make any ci infra change, we just divided them into 3 different jobs.
Testing in dev will require us to update the cloud run images if we made modification in the code, and I don't think doing a new release every time we makes modification to workflow files is the optimal way.

The current way we can test all the things in ci project, and the current set up can ensure the env are cleaned up after test is running, these are the reason I implemented this way.