[Testing] Create a new project for each test run

[jhamon]

Problem

Since all integration tests currently run in a shared project, we have to make use of randomness to avoid collisions on user-assigned names. I have a goal to remove this randomness and make tests more deterministic in order to unblock testing tools such as VCR, but before I can do that I need to isolate each test run into its own Pinecone project.

Solution

• New actions. Implement new actions:
  • project-create. I ended up wrapping this into a reusable workflow project-setup.yaml to encapsulate some of the setup and make it easier to use in on-pr.yaml and on-merge.yaml and on-pre-dep-changes.yaml.
  • project-delete. I ended up wrapping this into a reusable workflow project-cleanup.yaml to make it easier to use in on-pr.yaml and on-merge.yaml and on-pre-dep-changes.yaml. Deleting is actually a lot more complicated than creating the project and API key because all the resources must be cleared out of a project before it can be deleted, there are several resources types (indexes, collections, backups), and a lot of different ways that a delete can fail due to precondition failed / other pending operations. I implemented a deletion queue with deque that does a lot, but not unlimited, amounts of retrying. Anything that still fails after retries are exhausted will be tried again during the nightly cleanup.
  • secret-encrypt and secret-decrypt (see below)
• New/updated workflows
  • project-setup.yaml
  • project-cleanup.yaml. Cleanup is set to run always() so it should trigger even if upstream jobs are failed or canceled. I learned the hard way that always() jobs should always be used in combination with a timeout-minutes setting because if you make a mistake (such as implementing a bug that will never succeed with infinite retries), the job will not respond to normal workflow cancel requests issued from the Github UI. If you get into this state, the only ways to recover are to call the Github REST API to force cancel or to wait for the default timeout of 6 hours (FML!!) to elapse.
  • cleanup-nightly.yaml
• Passing API keys. I needed a way to pass a secret (the API key for the newly created project) from project-create into various other jobs, but github will not set job outputs that it thinks contain secrets so I was unable to directly pass the API key. I discovered that I needed to also implement secret-encrypt and secret-decrypt so that I could transmit the api key between different parts of the workflow in an encrypted form. These were implemented with symmetric encryption using Fernet from the cryptography package.
• Repository secrets. Set new repository secrets for:
  • PINECONE_SERVICE_ACCOUNT_CLIENT_ID and PINECONE_SERVICE_ACCOUNT_CLIENT_SECRET came from the web console
  • FERNET_ENCRYPTION_KEY was generated randomly.
• Adjusting existing workflows
  • Adjust testing-integration.yaml to use API key passed through inputs
  • Adjust testing-dependency.yaml to use API key passed through inputs
  • Adjust on-pr.yaml to create and delete the project, passing the API key to the integration testing step as a workflow input.
  • Adjust on-merge.yaml to create and delete the project, passing the API key to the integration and dependency testing step as a workflow input.
  • Adjust cleanup-nightly.yaml to also clean up generated projects and the resources they contain.

While doing all this surgery on CI stuff I also made the following changes:

• Deleted several unused actions
• Moved some files from scripts/ into the directory of the github action using them to make that relationship more clear. I want scripts/ to be for stuff that is mostly manually run in development.
• Renamed several actions and workflows to use a <noun>-<verb> format so that related actions would be grouped by resource when sorted alphabetically. For example, create-index and delete-index became index-create and index-delete.
• Parameterized unit and integration tests on a python version matrix. We don't often get version-specific problems, so I think we can just run the expanded version matrix on merge.
• Temporarily disabled a few asyncio tests that seem to have some sort of performance issue that needs investigating. I decided to disable rather than fix because I wanted to keep changes in tests/ to a minimum for this diff.

These changes do not affect how tests are run locally. The PINECONE_API_KEY environment variable is still loaded from .env using dotenv.

Type of Change

• Infrastructure change (CI configs, etc)