Chore: Reorganize engine adapter integration tests#3188
Merged
Conversation
1bdd5f7 to
5944034
Compare
eakmanrq
reviewed
Sep 26, 2024
.circleci/continue_config.yml
Outdated
| #filters: | ||
| # branches: | ||
| # only: | ||
| # - main |
Collaborator
Author
There was a problem hiding this comment.
Yes, I needed to show that all the tests were still running / passing (if there was an issue with these, it would only show up post-merge)
I'll uncomment this prior to merge
eakmanrq
approved these changes
Sep 26, 2024
Collaborator
eakmanrq
left a comment
There was a problem hiding this comment.
Thanks for laying the foundation for improving this!
df5f043 to
244ce70
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently, in order to write an integration test against one of the engines, it needs to be tacked on to
test_integration.py.This file is starting to get a bit unwieldy:
TestContextautomatically gets 3 versions:df,queryandpyspark. Often you only want one of these, typicallyquery(and almost neverpyspark) leading to a bunch of skipping logic within tests if the test_type is notqueryThis PR makes the following changes:
integrationsubfolderThe goal here is to set the foundation for breaking these up further in future. Tests affecting all adapters can stay in
test_integration.pyand then tests for a specific database can be put intest_integration_<database>.py.The immediate motivation was needing somewhere to put tests for the upcoming partition management logic in the Athena and Postgres adapters but these changes also give the option of moving eg the
pysparktests out of the main file and into a Spark-specific area