Skip to content

Upsert aka On Conflict Do Update#93

Merged
bh2smith merged 9 commits intomainfrom
56/upsert
Nov 26, 2024
Merged

Upsert aka On Conflict Do Update#93
bh2smith merged 9 commits intomainfrom
56/upsert

Conversation

@bh2smith
Copy link
Copy Markdown
Owner

@bh2smith bh2smith commented Nov 25, 2024

Introducing INSERT ... ON CONFLICT ... DO UPDATE!

This works well but also requires primary key or, more loosely, a uniqueness constraint on the table in question.
We have insightful error messaging that will literally print out the table alterations require for the provided conflict_columns:

So, when table exists without index we will see this error log suggesting how to fix the issue.

The ON CONFLICT clause requires a unique or exclusion constraint on the column(s): hash. Please ensure the table 'results_4238114' has the necessary constraint.To fix this, you can run the following SQL command:
ALTER TABLE results_4238114 ADD CONSTRAINT results_4238114_hash_unique UNIQUE (hash);

Note that, additionally, the table does not need to exist first. It will be created without any validation on the uniqueness constraints at first run. We only validate the uniqueness at the first time of actual upsert.

Test Plan

For now this is

  1. Run it
docker-compose up -d
make run

If you already have the table (from previous runs) you will hit the error above.
Execute the suggested ALTER TABLE command and try again!

UPDATE:
There are now "unit" tests of all conflict column validation and upsert method works as expected.

Closes #56.

@bh2smith bh2smith requested a review from mooster531 November 25, 2024 22:13
@bh2smith bh2smith marked this pull request as ready for review November 26, 2024 13:27
Copy link
Copy Markdown
Collaborator

@mooster531 mooster531 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that if we're using an ORM, we should use the ORM as much as we can :) I simplified the upsert function using sqlalchemy's built-in "on conflict update" utility.

Comment thread .github/workflows/coverage.yaml Outdated
Comment thread Makefile
Comment thread src/destinations/postgres.py Outdated
Comment thread src/destinations/postgres.py Outdated
@bh2smith
Copy link
Copy Markdown
Owner Author

I think your comments have all been addressed.

@mooster531
Copy link
Copy Markdown
Collaborator

🚀🚀🚀

@bh2smith bh2smith merged commit f6bc26d into main Nov 26, 2024
@bh2smith bh2smith deleted the 56/upsert branch November 26, 2024 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PG Support Upsert

2 participants