pg cdc feature branch #2548

jrhizor · 2021-03-22T15:47:42Z

add postgres CDC support

jrhizor · 2021-03-22T15:48:04Z

Biggest questions are where to inject the CDC logic.

* message conversion * fmt

* update cdc catalog * A

* table selection for cdc * fix broken merge * also test double quote in name

* minor postgres cdc fixes * add test and fix check behavior * fix * improve comment

* cdc docs * Update docs/integrations/sources/postgres.md Co-authored-by: Charles <giardina.charles@gmail.com> * address gcp * learn too english * add link * add more disk space warnings * add additional cdc use case * add information on how to find postgresql.conf * add how to find the file Co-authored-by: Charles <giardina.charles@gmail.com>

* postgres cdc race condition * working? but different process * add additional logging to help debug in the future * everything done except working config * remove unintended change

* add oneof configuration for cdc postgres * fmt Co-authored-by: Charles <giardina.charles@gmail.com>

cgardens · 2021-04-09T21:14:34Z

/test connector=source-postgres

🕑 source-postgres https://github.com/airbytehq/airbyte/actions/runs/734377112
✅ source-postgres https://github.com/airbytehq/airbyte/actions/runs/734377112

jrhizor · 2021-04-09T21:29:33Z

For manual testing I ran:

docker run --rm --name source -v $(pwd)/resources/postgresql.conf:/etc/postgresql/postgresql.conf -e POSTGRES_PASSWORD=password -p 2000:5432 -d postgres -c 'config_file=/etc/postgresql/postgresql.conf'

from ~/code/airbyte/airbyte-integrations/connectors/source-postgres/src/main

Then I did:

→ docker exec -it source psql -U postgres
psql (13.1 (Debian 13.1-1.pgdg100+1))
Type "help" for help.

postgres=# CREATE TABLE cars(id INTEGER, name VARCHAR(200), PRIMARY KEY (id));
CREATE TABLE
postgres=# INSERT INTO cars VALUES (0, 'mazda');
INSERT 0 1
postgres=# INSERT INTO cars VALUES (1, 'ferrari');
INSERT 0 1
postgres=# SELECT pg_create_logical_replication_slot('slot1', 'pgoutput');
 pg_create_logical_replication_slot
------------------------------------
 (slot1,0/15E4058)
(1 row)

postgres=# CREATE PUBLICATION pub1 FOR ALL TABLES;
CREATE PUBLICATION
postgres=# INSERT INTO cars VALUES (3, 'tesla');
INSERT 0 1
postgres=# INSERT INTO cars VALUES (4, 'hotwheels');
INSERT 0 1
postgres=# DELETE FROM cars WHERE name = 'hotwheels';
DELETE 1
postgres=#

And I ran three syncs, one after making the publication, one after adding two more rows, and one after the delete.

Final output:

# jrhizor in /tmp/airbyte_local/file [14:26:46]
→ cat _airbyte_raw_public_cars.jsonl
{"_airbyte_ab_id":"099f4399-df2f-4f84-ba31-0f1978c04434","_airbyte_emitted_at":1618003545939,"_airbyte_data":{"id":0,"name":"mazda","_ab_cdc_updated_at":1618003547642,"_ab_cdc_lsn":22954984,"_ab_cdc_deleted_at":null}}
{"_airbyte_ab_id":"84e056ce-ed41-4fd5-888c-0d04762b9d7a","_airbyte_emitted_at":1618003545939,"_airbyte_data":{"id":1,"name":"ferrari","_ab_cdc_updated_at":1618003547655,"_ab_cdc_lsn":22954984,"_ab_cdc_deleted_at":null}}
{"_airbyte_ab_id":"aad4d213-5490-42e4-abf6-51933a7d2cb9","_airbyte_emitted_at":1618003589691,"_airbyte_data":{"id":3,"name":"tesla","_ab_cdc_updated_at":1618003573423,"_ab_cdc_lsn":22955080,"_ab_cdc_deleted_at":null}}
{"_airbyte_ab_id":"853e2385-8835-4fd0-b568-c54609e6b550","_airbyte_emitted_at":1618003589691,"_airbyte_data":{"id":4,"name":"hotwheels","_ab_cdc_updated_at":1618003581506,"_ab_cdc_lsn":22955576,"_ab_cdc_deleted_at":null}}
{"_airbyte_ab_id":"cf11e9b7-0277-4fa0-9162-b314ed7b8e54","_airbyte_emitted_at":1618003634540,"_airbyte_data":{"id":4,"name":null,"_ab_cdc_updated_at":1618003625921,"_ab_cdc_lsn":22955816,"_ab_cdc_deleted_at":1618003625921}}

cgardens · 2021-04-09T21:49:31Z

Output of my test:

cdc

➜  airbyte git:(jrhizor/debezium) cat /tmp/airbyte_local/json_data/cdc_test/_airbyte_raw_cdc_public_cars.jsonl
{"_airbyte_ab_id":"3c00aed2-03c8-4c7e-a3a3-bb6e3f9f013c","_airbyte_emitted_at":1618004801536,"_airbyte_data":{"id":0,"name":"mazda","_ab_cdc_updated_at":1618004802907,"_ab_cdc_lsn":267773586200,"_ab_cdc_deleted_at":null}}
{"_airbyte_ab_id":"c3574d33-f5f9-4f99-8e0e-5dea9a401a60","_airbyte_emitted_at":1618004801536,"_airbyte_data":{"id":1,"name":"ferrari","_ab_cdc_updated_at":1618004802921,"_ab_cdc_lsn":267773586200,"_ab_cdc_deleted_at":null}}
➜  airbyte git:(jrhizor/debezium) cat /tmp/airbyte_local/json_data/cdc_test/_airbyte_raw_cdc_public_plays.jsonl
{"_airbyte_ab_id":"24630352-7d90-448a-acda-3971686f5eac","_airbyte_emitted_at":1618004801536,"_airbyte_data":{"id":1,"name":"victory","_ab_cdc_updated_at":1618004802942,"_ab_cdc_lsn":267773586200,"_ab_cdc_deleted_at":null}}
{"_airbyte_ab_id":"84599002-372e-473e-abb7-badba57a4558","_airbyte_emitted_at":1618004801536,"_airbyte_data":{"id":3,"name":"the art of success","_ab_cdc_updated_at":1618004802942,"_ab_cdc_lsn":267773586200,"_ab_cdc_deleted_at":null}}
{"_airbyte_ab_id":"371dac39-f10d-4216-af2c-ad161fedc9a8","_airbyte_emitted_at":1618004801536,"_airbyte_data":{"id":2,"name":"much ado about nothing","_ab_cdc_updated_at":1618004802943,"_ab_cdc_lsn":267773586200,"_ab_cdc_deleted_at":null}}
➜  airbyte git:(jrhizor/debezium)

conventional incremental

➜  airbyte git:(jrhizor/debezium) cat /tmp/airbyte_local/json_data/cdc_test/_airbyte_raw_cdc_public_plays.jsonl
{"_airbyte_ab_id":"a3631c00-fac5-4f15-839d-8f9d76a59f97","_airbyte_emitted_at":1618004465233,"_airbyte_data":{"id":1,"name":"victory","_ab_cdc_updated_at":1618004466612,"_ab_cdc_lsn":267773585880,"_ab_cdc_deleted_at":null}}
{"_airbyte_ab_id":"1f2e656f-68d0-49c2-9386-a89b3fb6f6c9","_airbyte_emitted_at":1618004465233,"_airbyte_data":{"id":3,"name":"the art of success","_ab_cdc_updated_at":1618004466613,"_ab_cdc_lsn":267773585880,"_ab_cdc_deleted_at":null}}
{"_airbyte_ab_id":"0aeb4c8f-b88e-42e2-a7c5-9475f9422fd3","_airbyte_emitted_at":1618004465233,"_airbyte_data":{"id":2,"name":"much ado about nothing","_ab_cdc_updated_at":1618004466613,"_ab_cdc_lsn":267773585880,"_ab_cdc_deleted_at":null}}
➜  airbyte git:(jrhizor/debezium) cat /tmp/airbyte_local/json_data/cdc_test/_airbyte_raw_conventional_public_cars.jsonl
{"_airbyte_ab_id":"d5656811-ae3c-4e55-aff5-28fc3ca49495","_airbyte_emitted_at":1618004780885,"_airbyte_data":{"id":0,"name":"mazda"}}
{"_airbyte_ab_id":"de742e13-4d34-40b2-960d-bbd02d8b1358","_airbyte_emitted_at":1618004780885,"_airbyte_data":{"id":1,"name":"ferrari"}}
➜  airbyte git:(jrhizor/debezium) cat /tmp/airbyte_local/json_data/cdc_test/_airbyte_raw_conventional_public_plays.jsonl
{"_airbyte_ab_id":"24ab4a31-b9e4-468a-9e0f-c403d89c5ee0","_airbyte_emitted_at":1618004780885,"_airbyte_data":{"id":1,"name":"victory"}}
{"_airbyte_ab_id":"61f187bd-7c45-4040-9914-be60d924e1d7","_airbyte_emitted_at":1618004780885,"_airbyte_data":{"id":3,"name":"the art of success"}}
{"_airbyte_ab_id":"1d5c3bbd-5ac4-4e78-9661-45efece7316f","_airbyte_emitted_at":1618004780885,"_airbyte_data":{"id":2,"name":"much ado about nothing"}}

cgardens · 2021-04-09T21:59:13Z

/publish connector=source-postgres

❌ source-postgres https://github.com/airbytehq/airbyte/actions/runs/734471323

cgardens · 2021-04-09T22:01:58Z

/publish connector=source-postgres

❌ source-postgres https://github.com/airbytehq/airbyte/actions/runs/734477157

* add docs on creating replica identities * emphasize danger * grammar

cgardens · 2021-04-09T22:31:14Z

/test connector=source-postgres

🕑 source-postgres https://github.com/airbytehq/airbyte/actions/runs/734542603
✅ source-postgres https://github.com/airbytehq/airbyte/actions/runs/734542603

cgardens · 2021-04-09T22:31:57Z

/publish connector=source-postgres

❌ source-postgres https://github.com/airbytehq/airbyte/actions/runs/734558600

cgardens · 2021-04-09T22:48:36Z

/publish connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/734578944
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/734578944

cgardens and others added 3 commits March 16, 2021 10:19

spike

2a7d0c5

more

0c49603

debezium wip

2eb4029

jrhizor added 7 commits March 23, 2021 07:58

use oneof for configuration

320b952

iterator wrapping structure

1e28f45

push current

7101ba7

working loop

4a6170e

Merge branch 'master' into jrhizor/debezium

87f46d6

move capability into source

49ccf09

hack it into a sharable state

448280f

cgardens changed the title ~~debezium wip~~ debezium feature branch Mar 25, 2021

cgardens changed the title ~~debezium feature branch~~ pg cdc feature branch Mar 25, 2021

cgardens and others added 13 commits March 25, 2021 15:48

debezium test runner (#2617)

867b9ca

CDC Wait for Values (#2618)

9eb5190

output actual AirbyteMessages for cdc (#2631)

9be6bcf

* message conversion * fmt

add lsn extraction and comparison (#2613)

825e324

postgres cdc catalog (#2673)

79094f1

* update cdc catalog * A

table selection for cdc (#2690)

4cf62eb

* table selection for cdc * fix broken merge * also test double quote in name

Add state management to CDC (#2718)

690bb80

CDC: Fix Producer/Consumer State Machine (#2721)

ec6abf5

CDC Postgres Tests (#2777)

a2a979b

fix postgres cdc image name and run check before reading data (#2785)

13f682d

* minor postgres cdc fixes * add test and fix check behavior * fix * improve comment

remove unused props, remove todos, add some more sanity tests (#2791)

201e421

cdc: add offset store tests (#2793)

e30ab53

clean (#2798)

40834cf

cgardens marked this pull request as ready for review April 7, 2021 23:24

auto-assign bot requested review from davinchia and michel-tricot April 7, 2021 23:24

Merge branch 'master' into jrhizor/debezium

d6b1b8c

jrhizor and others added 9 commits April 7, 2021 16:38

various merge conflict fixes (#2799)

85df233

cdc standard tests (#2813)

d5ecbd1

require cdc users to create publications & update docs (#2818)

2fd3582

* postgres cdc race condition * working? but different process * add additional logging to help debug in the future * everything done except working config * remove unintended change

Use oneOf in PG CDC spec (#2827)

3d47ec4

add oneOf configuration for postgres cdc (#2831)

0893dd1

* add oneof configuration for cdc postgres * fmt Co-authored-by: Charles <giardina.charles@gmail.com>

fix test (#2834)

cdb8869

fix test

4d2364c

bump version

321c0e9

add docs on creating replica identities (#2838)

35f44bc

* add docs on creating replica identities * emphasize danger * grammar

cgardens and others added 3 commits April 9, 2021 15:50

Merge branch 'master' into jrhizor/debezium

02779a0

bump pg version in source catalog

df00a59

generate seed files

7d8b119

jrhizor merged commit 2b19da8 into master Apr 9, 2021

jrhizor deleted the jrhizor/debezium branch April 9, 2021 23:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pg cdc feature branch #2548

pg cdc feature branch #2548

jrhizor commented Mar 22, 2021 •

edited

Loading

jrhizor commented Mar 22, 2021

cgardens commented Apr 9, 2021 •

edited by github-actions bot

Loading

jrhizor commented Apr 9, 2021 •

edited

Loading

cgardens commented Apr 9, 2021 •

edited

Loading

cgardens commented Apr 9, 2021 •

edited by github-actions bot

Loading

cgardens commented Apr 9, 2021 •

edited by github-actions bot

Loading

cgardens commented Apr 9, 2021 •

edited by github-actions bot

Loading

cgardens commented Apr 9, 2021 •

edited by github-actions bot

Loading

cgardens commented Apr 9, 2021 •

edited by github-actions bot

Loading

pg cdc feature branch #2548

pg cdc feature branch #2548

Conversation

jrhizor commented Mar 22, 2021 • edited Loading

jrhizor commented Mar 22, 2021

cgardens commented Apr 9, 2021 • edited by github-actions bot Loading

jrhizor commented Apr 9, 2021 • edited Loading

cgardens commented Apr 9, 2021 • edited Loading

cgardens commented Apr 9, 2021 • edited by github-actions bot Loading

cgardens commented Apr 9, 2021 • edited by github-actions bot Loading

cgardens commented Apr 9, 2021 • edited by github-actions bot Loading

cgardens commented Apr 9, 2021 • edited by github-actions bot Loading

cgardens commented Apr 9, 2021 • edited by github-actions bot Loading

jrhizor commented Mar 22, 2021 •

edited

Loading

cgardens commented Apr 9, 2021 •

edited by github-actions bot

Loading

jrhizor commented Apr 9, 2021 •

edited

Loading

cgardens commented Apr 9, 2021 •

edited

Loading

cgardens commented Apr 9, 2021 •

edited by github-actions bot

Loading

cgardens commented Apr 9, 2021 •

edited by github-actions bot

Loading

cgardens commented Apr 9, 2021 •

edited by github-actions bot

Loading

cgardens commented Apr 9, 2021 •

edited by github-actions bot

Loading

cgardens commented Apr 9, 2021 •

edited by github-actions bot

Loading