Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support query_max_size for insert_rows_method chunk #679

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

aibazhang
Copy link
Contributor

Hey guys,

To fix the issue raised in
#675
I added support for query_max_size to the insert_rows_method chunk.

This addition is important to prevent errors like The query is too large. The maximum standard SQL query length is 1024.00K characters (in BigQuery) when use the insert_rows_method chunk

Test

I intended to write a unit test for this change, but I encountered errors running the current test cases even after following the procedure outlined in
https://github.com/elementary-data/dbt-data-reliability/blob/master/integration_tests/README.md

I faced conflicting dependencies errors when attempting to install the required packages.

ERROR: Cannot install elementary-data and urllib3==2.0.4 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested urllib3==2.0.4
    dbt-core 1.7.9 depends on urllib3~=1.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

After I added elementary-data and dbt-postgres to the requirements.txt , another error occurred.

requirements.txt

pytest
pytest-xdist
pytest-parametrization
pytest-html
filelock
# urllib3==2.0.4
elementary-data==0.14.1
dbt-postgres==1.7.9

error

ERROR    elementary.clients.dbt.dbt_runner:dbt_runner.py:175 Encountered an error:
Compilation Error
  dbt found 2 package(s) specified in packages.yml, but only 0 package(s) installed in dbt_packages. Run "dbt deps" to install package dependencies.

Therefore, instead of writing a unit test, I tested this PR by running dbt run in our own environment, and it worked well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant