Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: Prepare parallel builds #2737

Merged
merged 7 commits into from
Apr 22, 2024
Merged

chore: Prepare parallel builds #2737

merged 7 commits into from
Apr 22, 2024

Conversation

sfc-gh-asawicki
Copy link
Collaborator

@sfc-gh-asawicki sfc-gh-asawicki commented Apr 22, 2024 •
edited
Loading

Allow builds from multiple PRs to run in parallel. The general idea is to link all the objects created in Snowflake with the given commit and to sweep only the objects matching the suffix.

In this PR:

  • Add suffix to objects created before running acceptance tests
  • Use almost the same suffix for integration and acceptance tests
  • Unify acc and int objects naming
  • Set suffix to commit SHA on CI
  • Provide fallback if env is not set locally
  • Add env required switch (to always require on CI)
  • Adjust sweepers to use the suffix (instead of prefix)
  • Change sweepers logic to care only about objects created in a given test run

NOT in the scope:

  • Clean the sweepers up (multiple TODOs left with SNOW-867247)
  • Add the missing sweepers (should also be done as part of SNOW-867247)

In the following PRs:

  • Create account-level objects with appropriate suffix during tests (both for integration and acceptance tests)
    • change logic of random id generators
    • change all the places where `random.X()`` is used
    • replace acctest.RandStringFromCharSet with a new helper with suffix
  • Move sweepers to test code (to use test client and stuff); Use if exists/use method from helper for dropping
  • Remove special behavior in sweepers for "terraform_test_database" and "terraform_test_warehouse"
  • Test added generation helpers
  • Fix fallback generation (now two different ones are generated for acc and int tests if the env is missing)

@sfc-gh-asawicki sfc-gh-asawicki marked this pull request as ready for review April 22, 2024 12:54
@github-actions GitHub Actions
Copy link

Integration tests failure for 16893577a5762f845bfb72bb7f581335108a37ea

@github-actions GitHub Actions
Copy link

Integration tests success for d0a2163d3503cd97459fe9bba023968e8db377d6

@sfc-gh-asawicki sfc-gh-asawicki merged commit 6974e25 into main Apr 22, 2024
9 checks passed
@sfc-gh-asawicki sfc-gh-asawicki deleted the prepare-parallel-builds branch April 22, 2024 14:49
@snowflake-release-please snowflake-release-please bot mentioned this pull request Apr 22, 2024
Merged
@sfc-gh-asawicki sfc-gh-asawicki mentioned this pull request Apr 26, 2024
Merged
sfc-gh-asawicki added a commit that referenced this pull request Apr 29, 2024
@sfc-gh-asawicki
chore: Replace id generation in integration and acceptance tests part1 (
cd4a914 
#2747)

In order to move to the unified ID generation (connected with the
changes in
#2737)
to stabilize parallel builds, the following changes were introduced
inside this PR:
- IdsGenerator was introduced in TestClient. It allows us to get the ids
of the default objects and build the new identifiers with the desired
suffix (indicating the current build). Most of the helpers were moved to
use this generator to get old and new ids. There are still TODOs left
there (there are helpers that get the name of the object from the caller
- we have to ensure that the names are generated the same way, or they
have a good reason to be different). Ids generator is exposed so that
both integration and acceptance tests can use it.
- Changed the generation method of resource/datasource names to the
exposed method described below in all the acceptance tests.
- Fixed the random.go file in the SDK (so that this one is used only in
the SDK unit tests - read the content of the file below, it had the long
comment describing the problems with this file):
  - renamed to random_test.go
  - unexported all the methods
  - switch to appropriate methods in integration/acceptance tests
- Next part of the change is to check all identifier creations
throughout helpers/acceptance tests/integration tests and switch to a
proper generator. This is not finished. The following methods were
attacked (the rest will be tackled in the next PR):
  - NewExternalObjectIdentifier
  - NewExternalObjectIdentifierFromFullyQualifiedName
  - NewAccountIdentifier
  - NewAccountIdentifierFromAccountLocator (moved to ids generator)
- NewAccountIdentifierFromFullyQualifiedName (2 usages left in business
critical skipped tests)
- NewAccountObjectIdentifier (testAccCheckDatabaseExistence left with
todo, grantOwnershipToTheCurrentRole,
testAccCheckSharePrivilegesRevoked,
sdk.NewAccountObjectIdentifier("ACCOUNTADMIN"),
sdk.NewAccountObjectIdentifier("ORGADMIN"),
sdk.NewAccountObjectIdentifier("NOT_EXISTING_ACCOUNT"),
sdk.NewAccountObjectIdentifier("does_not_exist"), current role/user,
sdk.NewAccountObjectIdentifier("<DB>"))
  - NewAccountObjectIdentifierFromFullyQualifiedName
- other smaller changes:
  - removed warehouse 2 in acceptance tests for now
  - fixed `TestAcc_TaskOwnershipGrant_onFuture` test

Left to be done regarding NewAccountObjectIdentifier:
- `alterMaterializedViewQueryExternally`
- `checkDatabaseAndSchemaDataRetentionTime`
- `checkDatabaseSchemaAndTableDataRetentionTime`
- `unsafe_execute_acceptance_test.go`
- `alterViewOwnershipExternally`
- check `app_role_1`
- check hardcoded `filter_by_role`, `stproc1`, `sp_pi`, `filterByRole`,
`records`
- `CreateDatabaseWithName` (and other similar helpers)
- check `setup_test.go`
- ask @sfc-gh-jcieslak about
`sdk.NewAccountObjectIdentifier("S3_STORAGE_INTEGRATION")`,
`sdk.NewAccountObjectIdentifier("GCP_STORAGE_INTEGRATION")`,
`sdk.NewAccountObjectIdentifier("AZURE_STORAGE_INTEGRATION")`
- extract common helpers for
`sdk.NewAccountObjectIdentifier("does_not_exist"` and similar
sfc-gh-jcieslak pushed a commit that referenced this pull request May 8, 2024
@snowflake-release-please
chore(main): release 0.90.0 (#2725)
57ee66b 
🤖 I have created a release *beep* *boop*
---


##
[0.90.0](v0.89.0...v0.90.0)
(2024-05-08)


### 🎉 **What's new:**

* Adjust owner_role_type and schema_evolution_record columns
([#2740](#2740))
([424e393](424e393))


### 🔧 **Misc**

* Add a guide on creating issues
([#2758](#2758))
([2b006aa](2b006aa))
* Add missing documentation
([#2781](#2781))
([cc0a6a7](cc0a6a7))
* Add scripts to close issues from Pre Snowflake bucket
([#2762](#2762))
([44c0c37](44c0c37))
* Add small adjustments
([#2783](#2783))
([e5b0b4b](e5b0b4b))
* Adjust issue templates
([#2748](#2748))
([64ab76d](64ab76d))
* Cleanup helpers part 3
([#2730](#2730))
([eb7bee4](eb7bee4))
* Cleanup helpers part 5
([#2744](#2744))
([1f165bf](1f165bf))
* Cleanup helpers part4
([#2741](#2741))
([9475e35](9475e35))
* Cleanup helpers part6
([#2745](#2745))
([eba3029](eba3029))
* Cleanup helpers poc
([#2724](#2724))
([70b99fb](70b99fb))
* Helpers cleanup continuation
([#2726](#2726))
([a70d1af](a70d1af))
* Prepare new roadmap entry
([#2773](#2773))
([b0bf28f](b0bf28f))
* Prepare parallel builds
([#2737](#2737))
([6974e25](6974e25))
* Update the contribution guidelines (and small fixes)
([#2753](#2753))
([aafdc72](aafdc72))


### 🐛 **Bug fixes:**

* Fix issue templates
([#2760](#2760))
([d0d5048](d0d5048))
* Fix setup for two tests
([#2757](#2757))
([df025b0](df025b0))
* Fix Test (wrong order of arguments)
([#2780](#2780))
([02f467e](02f467e))
* Fix/prove issues
[#2733](#2733)
[#2735](#2735)
[#2763](#2763)
[#2767](#2767)
([#2777](#2777))
([7b1c67e](7b1c67e))
* Prove problems with procedure resource data types
([#2782](#2782))
([68d0318](68d0318))
* read after create
([#2718](#2718))
([2e9b68f](2e9b68f))
* UNSET and empty SET in network policies
([#2759](#2759))
([3eacb0b](3eacb0b))
* unset network policy should not have single quotes
([#2584](#2584))
([8f2a363](8f2a363))
* Update failover group allowed integration types
([#2776](#2776))
([efde48d](efde48d))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: snowflake-release-please[bot] <105954990+snowflake-release-please[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sfc-gh-jcieslak sfc-gh-jcieslak sfc-gh-jcieslak approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@sfc-gh-asawicki @sfc-gh-jcieslak