New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Postgres Source Fixed JSONB array mapping #23608
Conversation
/test connector=connectors/source-postgres
Build PassedTest summary info:
|
Affected Connector ReportNOTE
|
Connector | Version | Changelog | Publish |
---|---|---|---|
source-alloydb |
1.0.49 |
✅ | ✅ |
source-alloydb-strict-encrypt |
1.0.49 |
🔵 (ignored) |
🔵 (ignored) |
source-postgres-strict-encrypt |
1.0.51 |
🔵 (ignored) |
🔵 (ignored) |
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Destinations (0)
Connector | Version | Changelog | Publish |
---|
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Other Modules (0)
Actionable Items
(click to expand)
Category | Status | Actionable Item |
---|---|---|
Version | ❌ mismatch |
The version of the connector is different from its normal variant. Please bump the version of the connector. |
⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
|
Changelog | ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
❌ changelog missing |
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog. | |
Publish | ⚠ not in seed |
The connector is not in the seed file (e.g. source_definitions.yaml ), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug. |
❌ diff seed version |
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add tests for this
@@ -216,7 +216,7 @@ private void putJsonbArray(ObjectNode node, String columnName, ResultSet resultS | |||
final ResultSet arrayResultSet = resultSet.getArray(colIndex).getResultSet(); | |||
|
|||
while (arrayResultSet.next()) { | |||
final PGobject object = getObject(arrayResultSet, colIndex, PGobject.class); | |||
final PGobject object = getObject(arrayResultSet, 2, PGobject.class); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Am not sure I understand this. Why is the value hardcoded at 2?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you need to traverse the result set and increase the index as you go. This seems wrong
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Am not sure I understand this. Why is the value hardcoded at 2?
Please take a look how all of other arrays mapping implemented (varchar array, int array, etc)
During iteration over arrays resultSet there are only 2 indexes:
index 1 - the ordinal number of the element in the array
index 2 - value
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. Can you add a test for this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see. Can you add a test for this.
we already have test for jsonb[] datatype in AbstractPostgresSourceDatatypeTest. Updated it to contain more than 2 elements in array
/test connector=connectors/source-postgres
Build PassedTest summary info:
|
/publish connector=connectors/source-postgres |
/publish connector=connectors/source-postgres-strict-encrypt |
/publish connector=connectors/source-postgres |
/publish connector=connectors/source-postgres
if you have connectors that successfully published but failed definition generation, follow step 4 here |
/publish connector=connectors/source-postgres-strict-encrypt
if you have connectors that successfully published but failed definition generation, follow step 4 here |
@VitaliiMaltsev tests to carry out with this PR.
Please share how the data looks in destination on version 1.0.49 and with your fix |
@VitaliiMaltsev another set of tests to carry out with this PR.
Please share how the data looks in destination on version 1.0.49 and with your fix |
@subodh1810 @edgao 1.0.49 sync _airbyte_raw_table Normalized table --------------------------------------------------------------------------1.0.51 sync without schema refresh _airbyte_raw_table Normalized table --------------------------------------------------------------------------1.0.51 sync with schema refresh _airbyte_raw_table |
@VitaliiMaltsev the normalized table for 1.0.51 without schema refresh doesnt show the data, its all null. why is that? |
@edgao shouldnt the datatype of normalized column after refresh be json instead of string (i am guessing we support json data type in bigquery) |
@VitaliiMaltsev the screenshots are difficult to follow. Can you use order by in your queries so that all the screenshots have the same ordering. |
@subodh1810 because actual json schema changed to |
Screenshots have the same order
|
#23608 (comment) |
@VitaliiMaltsev for #23608 (comment) |
also I dont see the values from postgres. Can you share the table structure in postgres ( |
Original Postgres Table |
@VitaliiMaltsev Once done we need to revisit the decision making on how we want to introduce jsonb change and after thorough testing with proper backward compatibility test you can again raise a new PR.
Update : Deleted the 1.0.51 tags from docker hub |
Update : Deleted the 1.0.51 tags from docker hub |
What
Issue https://github.com/airbytehq/oncall/issues/1570
How
Describe the solution
Recommended reading order
x.java
y.python
🚨 User Impact 🚨
Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.
Pre-merge Checklist
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampledocs/integrations/README.md
airbyte-integrations/builds.md
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereUpdating a connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing/publish
command described hereConnector Generator
-scaffold
in their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates
then checking in your changes