Move postgres source to AB Protocol & various python fixes #610

sherifnada · 2020-10-19T02:53:30Z

What

The main change:

Add a new source for tap-postgres that is ABP compliant

Supporting Python/singer fixes:

Output the correct messages from ExchangerateIO & Stripe sources for check connection
Always exit with statuscode 0 when checking connection unless an "actual" exception happened
Add helper methods to SingerHelper to parse from singer types to Airbyte types

Reading Order

Main change:

postgres_singer_source/source.py
SingerPostgresSourceTest.java
remaining files in airbyte-integrations/singer/postgres_abprotocol

Supporting changes:

*/source.py
entrypoint.py
singer_helpers.py
anything else

TODO

adapt to new directory structure
Use secrets instead of config
use SingerSource instead of Source
use proper JSONSchema instead of coercing the schema

…e-postgres-to-abprotocol

'

airbyte-integrations/base-python/airbyte_protocol/entrypoint.py

michel-tricot · 2020-10-19T03:54:42Z

airbyte-integrations/singer/postgres_abprotocol/.gitignore

@@ -0,0 +1 @@
+config


See my PR this weekend: #606

airbyte-integrations/singer/postgres_abprotocol/postgres_singer_source/source.py

airbyte-integrations/singer/stripe_abprotocol/source/source_stripe_singer/source.py

cgardens · 2020-10-19T04:49:49Z

airbyte-integrations/base-singer/base_singer/singer_helpers.py

@@ -34,6 +34,36 @@ class Catalogs:


 class SingerHelper:
+    @staticmethod


having multiple types is valid jsonschema. i don't think we want to normalize it out in the integration if we are saying that you define your stream's schema using jsonschema.

oh. saw your comment here: #609 (review). maybe we're on the same page now.

Yup, we are. I was confused. JSONSchema it is.

In which case please merge your PR 😆

airbyte-integrations/base-singer/base_singer/singer_helpers.py

…e-postgres-to-abprotocol

ChristopheDuong · 2020-10-20T14:59:00Z

airbyte-integrations/base-python/airbyte_protocol/entrypoint.py

@@ -85,19 +86,17 @@ def start(self, args):
            if cmd == "check":
                check_result = self.source.check(logger, config_container)

-                if check_result.successful:
+                if check_result.status == Status.SUCCEEDED:


@sherifnada If i understand right, this change is not compatible with what is currently in the template/python-source for the ´ check' function anymore, right?

AirbyteCheckResponse has to be changed to AirbyteConnectionStatus in that file too?

@ChristopheDuong Correct - good catch

sherifnada added 18 commits October 16, 2020 13:38

template

928af6c

save

577f16b

save

c8b8cb8

Merge branch 'master' of github.com:airbytehq/airbyte into sherif/mov…

a511db7

…e-postgres-to-abprotocol

save

37e9ced

save

3b6a23a

use correct param name

7651298

save 'progress'

af37eac

'

discovery passing

f451dba

all tests passing

0f6f5f5

spotless

a7182ae

save

2d3926c

fix eraio

316b06a

remove unnecessary test

e044856

save

18e48bc

try fixing test

c1b8c3c

try more days

a85d44a

remove eraio step

41dc190

michel-tricot approved these changes Oct 19, 2020

View reviewed changes

sherifnada added 6 commits October 18, 2020 21:09

extend singer source

91771e0

cleaner if-else in stripe

e25c378

actually extend singer source

844902b

for real though

22fb146

fix the other eraio 🙃

0ca290b

add newline

4126ed8

cgardens reviewed Oct 19, 2020

View reviewed changes

fix stripe

525f4e4

jrhizor approved these changes Oct 19, 2020

View reviewed changes

airbyte-integrations/base-singer/base_singer/singer_helpers.py Outdated Show resolved Hide resolved

airbyte-integrations/base-singer/base_singer/singer_helpers.py Outdated Show resolved Hide resolved

Merge branch 'master' of github.com:airbytehq/airbyte into sherif/mov…

9f2858d

…e-postgres-to-abprotocol

jrhizor mentioned this pull request Oct 19, 2020

fix exchangerate builds on branches #622

Merged

sherifnada added 8 commits October 19, 2020 10:35

place test mount roots in /tmp

77e8617

Merge branch 'sherif/test-roots' into sherif/move-postgres-to-abprotocol

cd357cf

config -> secrets

c390e88

fix conflict

feebd91

confs

c406f21

fix python indentation issue

3cfa4c7

coerce ints into numbers'

313cadb

my code is now spotfree

e1f8f2c

sherifnada merged commit d4b6ebc into master Oct 19, 2020

sherifnada deleted the sherif/move-postgres-to-abprotocol branch October 19, 2020 21:36

ChristopheDuong reviewed Oct 20, 2020

View reviewed changes

sherifnada added a commit that referenced this pull request Oct 23, 2020

Add AB Protocol PG source & various python fixes (#610)

fb85d76

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move postgres source to AB Protocol & various python fixes #610

Move postgres source to AB Protocol & various python fixes #610

sherifnada commented Oct 19, 2020 •

edited

michel-tricot Oct 19, 2020

cgardens Oct 19, 2020

cgardens Oct 19, 2020

sherifnada Oct 19, 2020

sherifnada Oct 19, 2020

ChristopheDuong Oct 20, 2020

sherifnada Oct 20, 2020

		@@ -34,6 +34,36 @@ class Catalogs:


		class SingerHelper:
		@staticmethod

		@@ -0,0 +1 @@
		config

Move postgres source to AB Protocol & various python fixes #610

Move postgres source to AB Protocol & various python fixes #610

Conversation

sherifnada commented Oct 19, 2020 • edited

What

Reading Order

TODO

michel-tricot Oct 19, 2020

Choose a reason for hiding this comment

cgardens Oct 19, 2020

Choose a reason for hiding this comment

cgardens Oct 19, 2020

Choose a reason for hiding this comment

sherifnada Oct 19, 2020

Choose a reason for hiding this comment

sherifnada Oct 19, 2020

Choose a reason for hiding this comment

ChristopheDuong Oct 20, 2020

Choose a reason for hiding this comment

sherifnada Oct 20, 2020

Choose a reason for hiding this comment

sherifnada commented Oct 19, 2020 •

edited