Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snowflake integration tests #336

Merged
merged 26 commits into from Apr 10, 2023
Merged

Conversation

tonykploomber
Copy link

@tonykploomber tonykploomber commented Mar 29, 2023

Describe your changes

  • Add Snowflake configuration & fixtures into integration testing
  • Make test table name with random hash string during every testing session (Reason below)
  • Modify all test cases in integration testing support dynamic table name

The reason we need the dynamic table name is now we use one single snowflake connection endpoint to perform the test.
To prevent the race condition or any data consistency issue during the testing, each test session should have standalone environment: that's why we logically add the hashing to each test table.

Example table name: taxi -> taxi-abc123 (6 chars uuid)
Example case: Two people may push the commit to github at same time, those intergration testing might conflict to each other, since they are sharing the same database, same table name on snowflake


By the way, now the developer needs to add snowflake credential in the local environment variable:
(Please ask me $$$)

conda env config vars set SF_USERNAME=$$$
conda env config vars set SF_PASSWORD=$$$

Issue number

Closes #247

Checklist before requesting a review


📚 Documentation preview 📚: https://jupysql--336.org.readthedocs.build/en/336/

@tonykploomber tonykploomber linked an issue Mar 29, 2023 that may be closed by this pull request
@idomic idomic changed the title Add: Init Snowflake integration tests Mar 29, 2023
@tonykploomber
Copy link
Author

tonykploomber commented Mar 29, 2023

@edublancas Would you check again why docs/readthedocs.org:jupysql fail again?

@edublancas
Copy link

doc building has been fixed in master, please rebase

@idomic
Copy link

idomic commented Mar 30, 2023

@tonykploomber Please don't add additional reviewers

Copy link

@idomic idomic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some comments

src/sql/_testing.py Show resolved Hide resolved
src/tests/integration/test_generic_db_opeations.py Outdated Show resolved Hide resolved
src/tests/integration/conftest.py Show resolved Hide resolved
src/tests/integration/test_generic_db_opeations.py Outdated Show resolved Hide resolved
src/tests/integration/test_generic_db_opeations.py Outdated Show resolved Hide resolved
src/tests/integration/test_generic_db_opeations.py Outdated Show resolved Hide resolved
src/tests/integration/conftest.py Show resolved Hide resolved
@idomic idomic requested a review from yafimvo April 5, 2023 17:05
@idomic
Copy link

idomic commented Apr 5, 2023

I see the tests are failing.

@tonykploomber
Copy link
Author

Yeah...I am fixing those

@tonykploomber
Copy link
Author

@idomic Ready for review

Copy link

@idomic idomic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, my main concern is that the trial expires in 6 days and it'll fail the CI.
@edublancas

setup.py Outdated Show resolved Hide resolved
src/sql/_testing.py Outdated Show resolved Hide resolved
src/tests/integration/conftest.py Show resolved Hide resolved
src/tests/integration/conftest.py Show resolved Hide resolved
src/tests/integration/conftest.py Show resolved Hide resolved
setup.py Outdated Show resolved Hide resolved
@idomic idomic merged commit 4314c1e into master Apr 10, 2023
19 checks passed
@idomic idomic deleted the 247-integration-testing-with-snowflake branch April 10, 2023 14:42
Copy link

@yafimvo yafimvo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tonykploomber
lgtm! tested on windows.

I saw it's already merged but we can patch a quick fix (added a comment)



def load_taxi_data(engine, table_name, index=True):
table_name = table_name
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove this line

@edublancas
Copy link

I don't see any docs for this in the developer guide: https://jupysql.ploomber.io/en/latest/community/developer-guide.html

@idomic
Copy link

idomic commented Apr 10, 2023

I assumed it's running as part of the integration tests: pytest src/tests/integration.
Do you suggest adding a specific entry for snowflake?

@edublancas
Copy link

what happens if I want to run them locally? Looks like I'd need to set some environment variables

@idomic
Copy link

idomic commented Apr 10, 2023

Ok, I see your point.
@tonykploomber please submit a patch for these 2 comments.

@tonykploomber
Copy link
Author

I don't see any docs for this in the developer guide: https://jupysql.ploomber.io/en/latest/community/developer-guide.html

Good point. We should mention how to test this, system environment variables has to be set up in order to perform snowflake testing on local

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

integration testing with snowflake
4 participants