Restore transaction semantics used by dbt-redshift prior to 1.5 #475

VersusFacit · 2023-05-31T22:27:54Z

resolves dbt-labs/dbt-core#451
resolves dbt-labs/dbt-core#454
reverses dbt-labs/dbt-core#458
resolves dbt-labs/dbt-core#425 (and others I'm sure)

Description

Catch-all for all transaction related chicanery.

Certain DDL cannot exist outside a transaction block without being a NO-OP, and when it is in one, it needs autocommit=True.

The new workflow

This has the net effect of "restoring" the pre-1.5 workflow. Users won't have to do a thing. This will solve a whole bunch of tangentially tickets. If people want true transactions, we can advise them to say so in their profiles: autocommit=False

After merge

~~I'll update the relevant docs issue to be about guidance for turning this off. I need to fix the docs issue and we will need to make sure there is a single source of truth out there for these changes.~~

Will require a 1.5 backport.

Checklist

I have read the contributing guide and understand what's expected of me
I have signed the CLA
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have opened an issue to add/update docs, or docs changes are not required/relevant for this PR
I have run changie new to create a changelog entry

github-actions · 2023-05-31T22:28:10Z

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-redshift contributing guide.

VersusFacit · 2023-06-01T00:53:06Z

Ignore the 3.7 unit test. It's a lie

tests/functional/test_autocommit.py

dbeatty10

A couple spellings to update.

tests/functional/test_autocommit.py

dbeatty10

@VersusFacit Does this solve #454 also?

Based on c613aae, it looks like the answer is "yes" 🎉

If so, could you add resolves #454 to the PR so we can auto-close it?

tests/functional/test_autocommit.py

dbt/adapters/redshift/connections.py

VersusFacit · 2023-06-01T18:17:02Z

Github is now seeing the resolve 454 properly. TIL just how restrictive the syntax is

dataders

thanks for this @VersusFacit!

some follow-on opportunities:

what do end-users need to know about this change/parameter? at the least we need an update to the docs repo's redshift setup page, right?
might test_autocommit.py actually be valuable in the generic adapter zone tests? this exercise of determining root cause has made me more suspicious than ever of how dbt handles transactions generally.

VersusFacit · 2023-06-01T19:20:08Z

what do end-users need to know about this change/parameter? At the least we need an update to the docs repo's redshift setup page, right?

I'm going to post links in issues here -- basically if you used the autocommit flag, I'm taking that away. This should affect so very few people since it wasn't really documented and has only been like for 2 weeks

I already updated the docs issue 👉 👉

might test_autocommit.py actually be valuable in the generic adapter zone tests?

No because in subtle ways transactions are handled differently by each db. We may find in time that it's worthwhile to do something like that but that would require a nontrivial investigation lift.

This is a hornet's nest we need to deliberately schedule time for if we want to learn/refine this subroutine more.

dbeatty10

Would like to resolve the conversation of autocommit vs. deactivate_autocommit prior to merge.

dataders · 2023-06-01T20:26:55Z

No because in subtle ways transactions are handled differently by each db. We may find in time that it's worthwhile to do something like that but that would require a nontrivial investigation lift.

This is a hornet's nest we need to deliberately schedule time for if we want to learn/refine this subroutine more.

agreed! which is why I've opened dbt-labs/dbt-core#9980

dbeatty10

👍

* Rename flag and get autocommit on * Add changie * Add test for issue 451 * Add test for issue #454 * Fix spelling. * Revert names * update tests * update the comment to use pep for rationale --------- Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>

#476) * Rename flag and get autocommit on * Add changie * Add test for issue 451 * Add test for issue #454 * Fix spelling. * Revert names * update tests * update the comment to use pep for rationale --------- Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>

…timeout (#3456) ## What are you changing in this pull request and why? This PR is to update the docs for the following changes in dbt-redshift: 1. **[Mapping sslmode in psycopg2 to ssl & sslmode in redshift_connector](dbt-labs/dbt-redshift#439 In dbt-redshift 1.5, Python driver switched from psycopg2 to redshift_connector. redshift_connector has both ssl and sslmode parameters, while psycopg2 only has sslmode. We've made changes in dbt-redshift to convert psycopg2' accepted values of sslmode into ssl and sslmode parameters of redshift_connector. This PR explains the conversion logic and includes explanation of the sslmode flag. #3365 2. **[Adding `autocommit` as a parameter](dbt-labs/dbt-redshift#475 Autocommit is a new flag that is defaulted to True. This PR adds explanation of this change and rationale behind this decision in dbt docs #3401 3. **[Connect_timeout default to None](dbt-labs/dbt-redshift#433 Connect_timeout had the default of 10 seconds previously. We changed this default to None. #3460  ## Checklist  - [x] Review the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) and [About versioning](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) so my content adheres to these guidelines. - [ ] Add a checklist item for anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch."

…labs#475) * Rename flag and get autocommit on * Add changie * Add test for issue 451 * Add test for issue dbt-labs#454 * Fix spelling. * Revert names * update tests * update the comment to use pep for rationale --------- Co-authored-by: Mila Page <versusfacit@users.noreply.github.com>

Rename flag and get autocommit on

5a26954

VersusFacit requested a review from a team as a code owner May 31, 2023 22:27

VersusFacit requested a review from McKnight-42 May 31, 2023 22:27

cla-bot bot added the cla:yes label May 31, 2023

Add changie

cb70137

VersusFacit changed the title ~~Rename flag and get autocommit on~~ Restore transaction semantics used by dbt-redshift prior to 1.5 May 31, 2023

VersusFacit added 2 commits May 31, 2023 16:10

Add test for issue 451

0239a91

Add test for issue #454

c613aae

VersusFacit requested review from dataders, mikealfare and dbeatty10 May 31, 2023 23:39

VersusFacit self-assigned this May 31, 2023

VersusFacit closed this May 31, 2023

VersusFacit reopened this May 31, 2023

VersusFacit added ok_to_test test all ok to test labels May 31, 2023

mikealfare reviewed Jun 1, 2023

View reviewed changes

tests/functional/test_autocommit.py Show resolved Hide resolved

dbeatty10 requested changes Jun 1, 2023

View reviewed changes

tests/functional/test_autocommit.py Outdated Show resolved Hide resolved

dbeatty10 requested changes Jun 1, 2023

View reviewed changes

tests/functional/test_autocommit.py Outdated Show resolved Hide resolved

tests/functional/test_autocommit.py Outdated Show resolved Hide resolved

tests/functional/test_autocommit.py Outdated Show resolved Hide resolved

tests/functional/test_autocommit.py Outdated Show resolved Hide resolved

mikealfare reviewed Jun 1, 2023

View reviewed changes

dbt/adapters/redshift/connections.py Outdated Show resolved Hide resolved

Fix spelling.

87a7d2b

VersusFacit requested review from dbeatty10 and mikealfare June 1, 2023 18:17

dataders approved these changes Jun 1, 2023

View reviewed changes

dbeatty10 requested changes Jun 1, 2023

View reviewed changes

VersusFacit requested a review from dbeatty10 June 1, 2023 19:57

McKnight-42 approved these changes Jun 1, 2023

View reviewed changes

dataders mentioned this pull request Apr 18, 2024

[CT-2624] [Spike] Scrutinize and clarify our approach and interface for transactions dbt-labs/dbt-core#9980

Open

3 tasks

dataders mentioned this pull request Jun 1, 2023

support simply query cycle and therefore multi-statement .execute() calls aws/amazon-redshift-python-driver#162

Open

Revert names

5eeeb98

VersusFacit force-pushed the ADAP/1.5_and_on_transaction_fixes branch from 8c40dc8 to 5eeeb98 Compare June 1, 2023 20:57

VersusFacit added 2 commits June 1, 2023 14:20

update tests

f312de4

update the comment to use pep for rationale

7ccb011

VersusFacit mentioned this pull request Jun 1, 2023

Add autocommit feature dbt-labs/docs.getdbt.com#3401

Open

1 task

dbeatty10 approved these changes Jun 1, 2023

View reviewed changes

leahwicz merged commit 876076b into main Jun 1, 2023

leahwicz deleted the ADAP/1.5_and_on_transaction_fixes branch June 1, 2023 23:34

VersusFacit mentioned this pull request Jun 1, 2023

[backport] Restore transaction semantics used by dbt-redshift prior to 1.5 (#475) #476

Merged

6 tasks

jiezhen-chen mentioned this pull request Jun 2, 2023

update dbt-redshift docs for changes to sslmode, autocommit, connect_timeout dbt-labs/docs.getdbt.com#3456

Merged

2 tasks

colin-rogers-dbt mentioned this pull request Jun 20, 2023

backport 474 to 1.5.latest #502

Merged

6 tasks

dataders mentioned this pull request Apr 18, 2024

[SPIKE] Investigate state of transactions #773

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restore transaction semantics used by dbt-redshift prior to 1.5 #475

Restore transaction semantics used by dbt-redshift prior to 1.5 #475

VersusFacit commented May 31, 2023 •

edited

Loading

github-actions bot commented May 31, 2023

VersusFacit commented Jun 1, 2023

dbeatty10 left a comment

dbeatty10 left a comment

VersusFacit commented Jun 1, 2023

dataders left a comment

VersusFacit commented Jun 1, 2023 •

edited

Loading

dbeatty10 left a comment

dataders commented Jun 1, 2023

dbeatty10 left a comment

Restore transaction semantics used by dbt-redshift prior to 1.5 #475

Restore transaction semantics used by dbt-redshift prior to 1.5 #475

Conversation

VersusFacit commented May 31, 2023 • edited Loading

Description

The new workflow

After merge

Checklist

github-actions bot commented May 31, 2023

VersusFacit commented Jun 1, 2023

dbeatty10 left a comment

Choose a reason for hiding this comment

dbeatty10 left a comment

Choose a reason for hiding this comment

VersusFacit commented Jun 1, 2023

dataders left a comment

Choose a reason for hiding this comment

VersusFacit commented Jun 1, 2023 • edited Loading

dbeatty10 left a comment

Choose a reason for hiding this comment

dataders commented Jun 1, 2023

dbeatty10 left a comment

Choose a reason for hiding this comment

VersusFacit commented May 31, 2023 •

edited

Loading

VersusFacit commented Jun 1, 2023 •

edited

Loading