🐛 Destination Postgres: fix \u0000(NULL) value processing #5336

DoNotPanicUA · 2021-08-11T10:55:16Z

What

In addition, small rework of jdbc-destination and move specific implementation to the PostgresSqlOperations.
// todo (cgardens) - move this into a postgres version of this. this syntax is postgres-specific

How

Replace all \u0000 unicode values from airbyte data messages

Pre-merge Checklist

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions
Connector version bumped like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

…f SqlOperations to PostgresSqlOperations.

sherifnada · 2021-08-12T04:33:00Z

@subodh1810 would you be able to take a look at this one?

subodh1810 · 2021-08-12T22:35:28Z

Could not look into it today, will look into it tomorrow first thing

subodh1810

I have 1 comment.

subodh1810 · 2021-08-16T13:48:07Z

...stgres/src/main/java/io/airbyte/integrations/destination/postgres/PostgresSqlOperations.java

+  private List<AirbyteRecordMessage> formatRecords(List<AirbyteRecordMessage> records) {
+    // Postgres fails if json contains \u0000 unicode (NULL) in a json.
+    records.forEach(airbyteRecordMessage -> airbyteRecordMessage
+        .setData(Jsons.deserialize(Jsons.serialize(airbyteRecordMessage.getData()).replaceAll("\\\\u0000", ""))));


This looks fine but my concern is that doing the Jsons.deserialize(Jsons.serialize for each record here is going to have a performance impact. How about we move this to BufferedStreamConsumer. We already have a string conversion here so it would save us from doing the serialization twice.

Good point! Thanks ;)

…est-post-unicode

…ecessary serialization

AllanSituma · 2021-08-25T07:22:46Z

Do we sort of have a timeline when this will be ready? I am currently experiencing this issue and I can't replicate a very important table.Thanks.

…est-post-unicode

subodh1810 · 2021-08-25T18:13:58Z

/test connector=destination-postgres

🕑 destination-postgres https://github.com/airbytehq/airbyte/actions/runs/1167618494
✅ destination-postgres https://github.com/airbytehq/airbyte/actions/runs/1167618494

subodh1810 · 2021-08-25T18:14:59Z

@DoNotPanicUA can you trigger the tests for all the connectors that use BufferedStreamConsumer.java like I did here so that we are sure nothing would break
#5336 (comment)

...ors/destination-jdbc/src/main/java/io/airbyte/integrations/destination/jdbc/DataAdapter.java

subodh1810 · 2021-08-25T18:21:27Z

...ors/destination-jdbc/src/main/java/io/airbyte/integrations/destination/jdbc/DataAdapter.java

+    adaptValueNodes(null, rootNode, null);
+  }
+
+  private void adaptValueNodes(String fieldName, JsonNode node, JsonNode parentNode) {


Am not sure I follow the logic of this method. When can node.isValueNode() be true? Also I dont like the fact that fieldName can be null and we dont have a null check.

if an element contains a value - it's a value node. An element also might be an array or object.

@DoNotPanicUA can you put a comment on this method explaining how it works so that its clear for anyone who is reading this code for the first time

...stination-jdbc/src/main/java/io/airbyte/integrations/destination/jdbc/JdbcSqlOperations.java

subodh1810 · 2021-08-25T18:26:23Z

...on-jdbc/src/main/java/io/airbyte/integrations/destination/jdbc/copy/CopyConsumerFactory.java

          // TODO Truncate json data instead of throwing whole record away?
          // or should we upload it into a special rejected record folder in s3 instead?
          var emittedAt = Timestamp.from(Instant.ofEpochMilli(recordMessage.getEmittedAt()));
-          pairToCopier.get(pair).write(id, data, emittedAt);
+          pairToCopier.get(pair).write(id, Jsons.serialize(recordMessage.getData()), emittedAt);


This is a problem, we are not going to do Jsons.serialize twice for Redshift. First in the isValidData method and then here

currently, it's designed in a way that we don't have many options. Original implementation does additional serialization for all destinations. Here we have one additional serialization for one destination and only for the Copy flow. So, we already do a significant improvement here.
I propose to create a new issue for the RedShift destination improvement in order to unblock the Postgres issue.

Cool! Please create a follow up issue to resolve this

subodh1810

Please add a comment explaining the logic for method adaptValueNodes

subodh1810 · 2021-08-30T12:18:02Z

...ors/destination-jdbc/src/main/java/io/airbyte/integrations/destination/jdbc/DataAdapter.java

+    adaptValueNodes(null, rootNode, null);
+  }
+
+  private void adaptValueNodes(String fieldName, JsonNode node, JsonNode parentNode) {


@DoNotPanicUA can you put a comment on this method explaining how it works so that its clear for anyone who is reading this code for the first time

…est-post-unicode

DoNotPanicUA · 2021-08-30T16:38:36Z

/test connector=destination-postgres

🕑 destination-postgres https://github.com/airbytehq/airbyte/actions/runs/1182952718
✅ destination-postgres https://github.com/airbytehq/airbyte/actions/runs/1182952718

DoNotPanicUA · 2021-08-30T16:38:43Z

/test connector=destination-meilisearch

🕑 destination-meilisearch https://github.com/airbytehq/airbyte/actions/runs/1182952917
❌ destination-meilisearch https://github.com/airbytehq/airbyte/actions/runs/1182952917

DoNotPanicUA · 2021-08-30T16:38:52Z

/test connector=destination-mssql

🕑 destination-mssql https://github.com/airbytehq/airbyte/actions/runs/1182953304
✅ destination-mssql https://github.com/airbytehq/airbyte/actions/runs/1182953304

DoNotPanicUA · 2021-08-30T16:39:04Z

/test connector=destination-mysql

🕑 destination-mysql https://github.com/airbytehq/airbyte/actions/runs/1182954038
✅ destination-mysql https://github.com/airbytehq/airbyte/actions/runs/1182954038

DoNotPanicUA · 2021-08-30T16:39:15Z

/test connector=destination-oracle

🕑 destination-oracle https://github.com/airbytehq/airbyte/actions/runs/1182954624
❌ destination-oracle https://github.com/airbytehq/airbyte/actions/runs/1182954624

DoNotPanicUA · 2021-08-30T16:39:24Z

/test connector=destination-redshift

🕑 destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1182954911
✅ destination-redshift https://github.com/airbytehq/airbyte/actions/runs/1182954911

DoNotPanicUA · 2021-08-30T16:39:32Z

/test connector=destination-snowflake

🕑 destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1182955243
✅ destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1182955243

DoNotPanicUA · 2021-08-30T18:19:24Z

/publish connector=connectors/destination-postgres

🕑 connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/1183254176
✅ connectors/destination-postgres https://github.com/airbytehq/airbyte/actions/runs/1183254176

fix \u0000(NULL) value processing for Postgres + move postgres impl o…

379d318

…f SqlOperations to PostgresSqlOperations.

github-actions bot added the area/connectors Connector related issues label Aug 11, 2021

DoNotPanicUA requested review from cgardens, sherifnada, jrhizor, ChristopheDuong and yaroslav-hrytsaienko August 11, 2021 10:57

DoNotPanicUA self-assigned this Aug 11, 2021

changelog + format

0932f15

github-actions bot added the area/documentation Improvements or additions to documentation label Aug 11, 2021

sherifnada requested review from subodh1810 and removed request for jrhizor, ChristopheDuong and cgardens August 12, 2021 04:32

yaroslav-hrytsaienko approved these changes Aug 12, 2021

View reviewed changes

incr release version

3657e61

subodh1810 requested changes Aug 16, 2021

View reviewed changes

DoNotPanicUA added 2 commits August 17, 2021 13:30

Merge remote-tracking branch 'origin/master' into aleonets/3476-fix-d…

13a944a

…est-post-unicode

Add generic solution to adopt messages for a destination + remove unn…

b0f13a4

…ecessary serialization

subodh1810 self-requested a review August 24, 2021 18:17

DoNotPanicUA added 2 commits August 25, 2021 13:15

Merge remote-tracking branch 'origin/master' into aleonets/3476-fix-d…

c75dacd

…est-post-unicode

revert version for build

792d89b

jrhizor temporarily deployed to more-secrets August 25, 2021 18:16 Inactive

subodh1810 requested changes Aug 25, 2021

View reviewed changes

minor review fixes

9cf43bb

subodh1810 approved these changes Aug 30, 2021

View reviewed changes

DoNotPanicUA added 3 commits August 30, 2021 16:21

add comments

c2b18a2

Merge remote-tracking branch 'origin/master' into aleonets/3476-fix-d…

3edefb7

…est-post-unicode

format

2175999

jrhizor temporarily deployed to more-secrets August 30, 2021 16:41 Inactive

incr version

d374cbc

jrhizor temporarily deployed to more-secrets August 30, 2021 18:21 Inactive

DoNotPanicUA mentioned this pull request Aug 30, 2021

Redshift destination : remove extra serialization #5741

Closed

DoNotPanicUA merged commit b18bd43 into master Aug 30, 2021

DoNotPanicUA deleted the aleonets/3476-fix-dest-post-unicode branch August 30, 2021 18:41

jrhizor mentioned this pull request Sep 3, 2021

Bump Airbyte version from 0.29.13-alpha to 0.29.14-alpha #5831

Merged

karinakuz added connectors/destinations-api connectors/destination/meilisearch connectors/destinations-warehouse connectors/destination/snowflake labels Jan 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Destination Postgres: fix \u0000(NULL) value processing #5336

🐛 Destination Postgres: fix \u0000(NULL) value processing #5336

DoNotPanicUA commented Aug 11, 2021 •

edited

Loading

sherifnada commented Aug 12, 2021

subodh1810 commented Aug 12, 2021

subodh1810 left a comment

subodh1810 Aug 16, 2021

DoNotPanicUA Aug 16, 2021

AllanSituma commented Aug 25, 2021

subodh1810 commented Aug 25, 2021 •

edited by github-actions bot

Loading

subodh1810 commented Aug 25, 2021

subodh1810 Aug 25, 2021

DoNotPanicUA Aug 26, 2021

subodh1810 Aug 30, 2021

subodh1810 Aug 25, 2021

DoNotPanicUA Aug 26, 2021

subodh1810 Aug 30, 2021

subodh1810 left a comment

subodh1810 Aug 30, 2021

DoNotPanicUA commented Aug 30, 2021 •

edited by github-actions bot

Loading

DoNotPanicUA commented Aug 30, 2021 •

edited by github-actions bot

Loading

DoNotPanicUA commented Aug 30, 2021 •

edited by github-actions bot

Loading

DoNotPanicUA commented Aug 30, 2021 •

edited by github-actions bot

Loading

DoNotPanicUA commented Aug 30, 2021 •

edited by github-actions bot

Loading

DoNotPanicUA commented Aug 30, 2021 •

edited by github-actions bot

Loading

DoNotPanicUA commented Aug 30, 2021 •

edited by github-actions bot

Loading

DoNotPanicUA commented Aug 30, 2021 •

edited by github-actions bot

Loading

🐛 Destination Postgres: fix \u0000(NULL) value processing #5336

🐛 Destination Postgres: fix \u0000(NULL) value processing #5336

Conversation

DoNotPanicUA commented Aug 11, 2021 • edited Loading

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

sherifnada commented Aug 12, 2021

subodh1810 commented Aug 12, 2021

subodh1810 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AllanSituma commented Aug 25, 2021

subodh1810 commented Aug 25, 2021 • edited by github-actions bot Loading

subodh1810 commented Aug 25, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

subodh1810 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DoNotPanicUA commented Aug 30, 2021 • edited by github-actions bot Loading

DoNotPanicUA commented Aug 30, 2021 • edited by github-actions bot Loading

DoNotPanicUA commented Aug 30, 2021 • edited by github-actions bot Loading

DoNotPanicUA commented Aug 30, 2021 • edited by github-actions bot Loading

DoNotPanicUA commented Aug 30, 2021 • edited by github-actions bot Loading

DoNotPanicUA commented Aug 30, 2021 • edited by github-actions bot Loading

DoNotPanicUA commented Aug 30, 2021 • edited by github-actions bot Loading

DoNotPanicUA commented Aug 30, 2021 • edited by github-actions bot Loading

DoNotPanicUA commented Aug 11, 2021 •

edited

Loading

subodh1810 commented Aug 25, 2021 •

edited by github-actions bot

Loading

DoNotPanicUA commented Aug 30, 2021 •

edited by github-actions bot

Loading

DoNotPanicUA commented Aug 30, 2021 •

edited by github-actions bot

Loading

DoNotPanicUA commented Aug 30, 2021 •

edited by github-actions bot

Loading

DoNotPanicUA commented Aug 30, 2021 •

edited by github-actions bot

Loading

DoNotPanicUA commented Aug 30, 2021 •

edited by github-actions bot

Loading

DoNotPanicUA commented Aug 30, 2021 •

edited by github-actions bot

Loading

DoNotPanicUA commented Aug 30, 2021 •

edited by github-actions bot

Loading

DoNotPanicUA commented Aug 30, 2021 •

edited by github-actions bot

Loading