[Part 1 ] : Hackday project to debug connections #33027

akashkulk · 2023-12-01T22:29:35Z

This is the Java part of the debug hack project. Specifically

Creates an DebugUtil utility debug class
Created an example debugger associated w/Postgres
Added debug hooks to ensure that every run does not modify source DB state (ack LSNs, etc)

Next steps :

Create debuggers for all certified connectors
Cleanup scripts & playbook for the other flows

vercel · 2023-12-01T22:29:41Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Dec 12, 2023 0:31am

github-actions · 2023-12-01T22:29:54Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

github-actions · 2023-12-01T22:36:13Z

Coverage report for source-postgres

File	Coverage [86.95%]	🍏
PostgresCdcCtidInitializer.java	89.1%	🍏
PostgresUtils.java	85.27%	🍏

Total Project Coverage	71.71%	🍏

postamar

You may reasonably think I'm being needlessly annoying in this review. With respect to the narrow scope of this particular change I absolutely am and I'm sorry. My interest here is to question old habits in matters of java code style, which I strongly feel need to change. We need to care more about readability. This PR happens to be greenfield development so the discussion feels warranted on my end, unlike a change involving legacy code. cc @stephane-airbyte who, I think, feels the same way about this stuff.

Wrt the review of this change in particular, having part 2 up would help me understand part 1 better.

...-sources/src/testFixtures/java/io/airbyte/cdk/integrations/debug/AbstractSourceDebugger.java

.../source-postgres/src/test/java/io/airbyte/integrations/source/postgres/PostgresDebugger.java

postamar · 2023-12-04T19:39:55Z

.../source-postgres/src/test/java/io/airbyte/integrations/source/postgres/PostgresDebugger.java

+      throw new RuntimeException("WARNING: config indicates that we are clearing the WAL log while reading data. This will mutate the WAL log"
+          + " associated with the source being debugged and is not advised.");
+    }
+    final JsonNode debugConfig = ((ObjectNode) originalConfig.deepCopy()).put("debug_mode", true);


Is there any reason why we couldn't simply decree that debug_mode be used/reserved in all connectors for this purpose? I have the feeling that this line + the isDebugMode method are going to be copy-pasted a lot otherwise.

.../source-postgres/src/test/java/io/airbyte/integrations/source/postgres/PostgresDebugger.java

...-sources/src/testFixtures/java/io/airbyte/cdk/integrations/debug/AbstractSourceDebugger.java

...source-postgres/src/test/java/io/airbyte/integrations/source/postgres/PostgresUtilsTest.java

akashkulk · 2023-12-05T17:40:51Z

You may reasonably think I'm being needlessly annoying in this review. With respect to the narrow scope of this particular change I absolutely am and I'm sorry. My interest here is to question old habits in matters of java code style, which I strongly feel need to change. We need to care more about readability. This PR happens to be greenfield development so the discussion feels warranted on my end, unlike a change involving legacy code. cc @stephane-airbyte who, I think, feels the same way about this stuff.

Wrt the review of this change in particular, having part 2 up would help me understand part 1 better.

Not at all - I appreciate the comments! I will go through these and address them. As far as Part 2 goes, that is mainly a script not related to these efforts. There is also a small follow-up which creates debuggers for MongoDb, MySql, MsSql based on the patterns established in this PR so rather part 1 will dictate how part 2 will look like.

akashkulk · 2023-12-06T06:09:44Z

I ended up rewriting a large chunk of it. I incorparated a lot of feedback from @postamar.

rodireich · 2023-12-11T19:21:37Z

airbyte-cdk/java/airbyte-cdk/README.md

Suggestion: Add a section on how to use?

I'm planning on doing a detailed write up. For now, I'm just hoping to get this in

rodireich · 2023-12-11T19:22:54Z

airbyte-integrations/connectors/source-postgres/build.gradle

@@ -15,7 +15,7 @@ java {
 airbyteJavaConnector {
    cdkVersionRequired = '0.5.3'
    features = ['db-sources']
-    useLocalCdk = false
+    useLocalCdk = true


Don't forget to change back 💁‍♂️

akashkulk · 2023-12-11T23:47:51Z

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/7174823272
✅ Successfully published Java CDK version=0.7.1!

postamar · 2023-12-12T19:04:06Z

Thanks for applying the suggestions @akashkulk and sorry for not circling back but it's all good. Much appreciated!

Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>

Initial commit of debugger

8aee954

octavia-squidington-iii added area/connectors Connector related issues CDK Connector Development Kit connectors/source/postgres labels Dec 1, 2023

akashkulk and others added 3 commits December 1, 2023 14:30

Merge branch 'master' into akash/lsn-investigate

3f263bc

Bump

ecd9705

Automated Commit - Formatting Changes

054d675

Merge branch 'master' into akash/lsn-investigate

3ea664f

akashkulk requested review from rodireich, postamar, stephane-airbyte, prateekmukhedkar, nguyenaiden and xiaohansong December 1, 2023 22:52

akashkulk marked this pull request as ready for review December 1, 2023 22:53

akashkulk requested a review from a team as a code owner December 1, 2023 22:53

postamar reviewed Dec 4, 2023

View reviewed changes

akashkulk and others added 2 commits December 5, 2023 22:06

Refactor debugger

ea2387a

Merge branch 'master' into akash/lsn-investigate

02d61a5

vercel bot deployed to Preview December 6, 2023 06:10 View deployment

akashkulk requested a review from postamar December 6, 2023 06:11

akashkulk and others added 2 commits December 6, 2023 06:11

Automated Commit - Formatting Changes

3ec2260

Merge branch 'master' into akash/lsn-investigate

778444d

vercel bot deployed to Preview December 6, 2023 16:11 View deployment

Merge branch 'master' into akash/lsn-investigate

7b0e32d

vercel bot had a problem deploying to Preview December 8, 2023 17:02 Failure

Merge branch 'master' into akash/lsn-investigate

8323a32

vercel bot deployed to Preview December 11, 2023 18:50 View deployment

rodireich approved these changes Dec 11, 2023

View reviewed changes

Merge branch 'master' into akash/lsn-investigate

eece59f

vercel bot deployed to Preview December 11, 2023 23:47 View deployment

akashkulk added 2 commits December 11, 2023 16:04

Change cdk version

6689422

bump versions

9490b5c

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Dec 12, 2023

vercel bot deployed to Preview December 12, 2023 00:31 View deployment

akashkulk merged commit f827af1 into master Dec 12, 2023
25 of 26 checks passed

akashkulk deleted the akash/lsn-investigate branch December 12, 2023 01:21

akashkulk mentioned this pull request Dec 12, 2023

Add debugger drivers for MongoDB & MySQL #33356

Merged

rishabh-cldcvr pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Dec 14, 2023

[Part 1 ] : Hackday project to debug connections (airbytehq#33027)

c6f584f

Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>

tmathew0309 pushed a commit to tmathew0309/airbyte that referenced this pull request Jan 12, 2024

[Part 1 ] : Hackday project to debug connections (airbytehq#33027)

f16f17c

Co-authored-by: akashkulk <akashkulk@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Part 1 ] : Hackday project to debug connections #33027

[Part 1 ] : Hackday project to debug connections #33027

akashkulk commented Dec 1, 2023 •

edited

Loading

vercel bot commented Dec 1, 2023 •

edited

Loading

github-actions bot commented Dec 1, 2023 •

edited by akashkulk

Loading

github-actions bot commented Dec 1, 2023 •

edited

Loading

postamar left a comment •

edited

Loading

postamar Dec 4, 2023

akashkulk commented Dec 5, 2023

akashkulk commented Dec 6, 2023

rodireich Dec 11, 2023

akashkulk Dec 11, 2023

rodireich Dec 11, 2023

akashkulk commented Dec 11, 2023 •

edited by github-actions bot

Loading

postamar commented Dec 12, 2023

[Part 1 ] : Hackday project to debug connections #33027

[Part 1 ] : Hackday project to debug connections #33027

Conversation

akashkulk commented Dec 1, 2023 • edited Loading

vercel bot commented Dec 1, 2023 • edited Loading

github-actions bot commented Dec 1, 2023 • edited by akashkulk Loading

Before Merging a Connector Pull Request

github-actions bot commented Dec 1, 2023 • edited Loading

Coverage report for source-postgres

postamar left a comment • edited Loading

Choose a reason for hiding this comment

postamar Dec 4, 2023

Choose a reason for hiding this comment

akashkulk commented Dec 5, 2023

akashkulk commented Dec 6, 2023

rodireich Dec 11, 2023

Choose a reason for hiding this comment

akashkulk Dec 11, 2023

Choose a reason for hiding this comment

rodireich Dec 11, 2023

Choose a reason for hiding this comment

akashkulk commented Dec 11, 2023 • edited by github-actions bot Loading

postamar commented Dec 12, 2023

akashkulk commented Dec 1, 2023 •

edited

Loading

vercel bot commented Dec 1, 2023 •

edited

Loading

github-actions bot commented Dec 1, 2023 •

edited by akashkulk

Loading

github-actions bot commented Dec 1, 2023 •

edited

Loading

postamar left a comment •

edited

Loading

akashkulk commented Dec 11, 2023 •

edited by github-actions bot

Loading