Skip to content

Stop sync on a null value in a cursor column - #19889

Merged
Rodi Reich Zilberman (rodireich) merged 19 commits into
masterfrom
17150-database-sources-handle-when-a-cursor-value-is-null
Dec 6, 2022
Merged

Rodi Reich Zilberman (rodireich) merged 19 commits into
masterfrom
17150-database-sources-handle-when-a-cursor-value-is-null

Conversation

@rodireich

@rodireich Rodi Reich Zilberman (rodireich) commented Nov 29, 2022 •

Copy link
Copy Markdown
Contributor

What

This change will cause a sync to stop if one of the streams contains a column designated as a cursor that contains a null value.

How

This is adding another step to the verification at the beginning of a read in source postgres.
We look for each stream's cursor column and if that column is NULLABLE, we look for a null value .

Because the lookup can be expensive on a large table, we are only going to do that if the column allows null values.
Otherwise it will immediately pass this test.

Recommended reading order

  1. PostgresSource.java for query implementation
  2. AbstractDbSource.java for how the new logic is added to the read process

@rodireich

Rodi Reich Zilberman (rodireich) commented Nov 29, 2022 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3577356430
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3577356430
🐛 https://gradle.com/s/i4trimimpseg2

Build Failed

Test summary info:

Could not find result summary

@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets November 29, 2022 19:27 Inactive
@github-actions

github-actions Bot commented Nov 29, 2022 •

Copy link
Copy Markdown
Contributor

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

  • Run integration tests
  • Bump connector or module version
  • Add changelog
  • Publish the new version

⚠ Sources (25)

Connector Version Changelog Publish
source-alloydb 1.0.17 ✅ ✅
source-alloydb-strict-encrypt 1.0.17 ✅ 🔵
(ignored)
source-bigquery 0.2.3 ✅ ✅
source-clickhouse 0.1.14 ✅ ✅
source-clickhouse-strict-encrypt 0.1.14 ✅ 🔵
(ignored)
source-cockroachdb 0.1.18 ✅ ✅
source-cockroachdb-strict-encrypt 0.1.18 ✅ 🔵
(ignored)
source-db2 0.1.16 ✅ ✅
source-db2-strict-encrypt 0.1.16 ✅ 🔵
(ignored)
source-dynamodb 0.1.0 ✅ ✅
source-jdbc 0.3.5 ⚠
(doc not found)
⚠
(not in seed)
source-mongodb-strict-encrypt 0.1.19 ⚠
(doc not found)
🔵
(ignored)
source-mongodb-v2 0.1.19 ✅ ✅
source-mssql 0.4.25 ✅ ✅
source-mssql-strict-encrypt 0.4.25 ✅ 🔵
(ignored)
source-mysql 1.0.15 ✅ ✅
source-mysql-strict-encrypt 1.0.15 ✅ 🔵
(ignored)
source-oracle 0.3.21 ✅ ✅
source-oracle-strict-encrypt 0.3.21 ✅ 🔵
(ignored)
source-postgres 1.0.31 ✅ ✅
source-postgres-strict-encrypt 1.0.31 ✅ 🔵
(ignored)
source-redshift 0.3.15 ✅ ✅
source-scaffold-java-jdbc 0.1.0 ⚠
(doc not found)
⚠
(not in seed)
source-snowflake 0.1.26 ✅ ✅
source-tidb 0.2.1 ✅ ✅
  • See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector Version Changelog Publish
  • See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category Status Actionable Item
Version ❌
mismatch
The version of the connector is different from its normal variant. Please bump the version of the connector.
⚠
doc not found
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
Changelog ⚠
doc not found
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
❌
changelog missing
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish ⚠
not in seed
The connector is not in the seed file (e.g. source_definitions.yaml), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
❌
diff seed version
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version.

@rodireich

Rodi Reich Zilberman (rodireich) commented Nov 29, 2022 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3578522892
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3578522892
🐛 https://gradle.com/s/suewiodej4r4k

Build Failed

Test summary info:

Could not find result summary

@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets November 29, 2022 22:28 Inactive
@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets November 29, 2022 22:28 Inactive
@rodireich

Rodi Reich Zilberman (rodireich) commented Nov 29, 2022 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3578872975
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3578872975
🐛 https://gradle.com/s/3oobqcvwvhy44

Build Failed

Test summary info:

Could not find result summary

@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets November 29, 2022 23:28 Inactive
@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets November 29, 2022 23:28 Inactive
@rodireich

Rodi Reich Zilberman (rodireich) commented Nov 30, 2022 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3579927590
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3579927590
🐛 https://gradle.com/s/unczakb73gppk

Build Failed

Test summary info:

Could not find result summary

@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets November 30, 2022 03:02 Inactive
@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets November 30, 2022 03:02 Inactive
@rodireich

Rodi Reich Zilberman (rodireich) commented Nov 30, 2022 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3580553589

@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets November 30, 2022 05:19 Inactive
@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets November 30, 2022 05:19 Inactive
@rodireich

Rodi Reich Zilberman (rodireich) commented Nov 30, 2022 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3580900217
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3580900217
🐛

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
================= 14 passed, 5 skipped, 21 warnings in 26.05s ==================

@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets December 6, 2022 05:52 Inactive
@rodireich

Rodi Reich Zilberman (rodireich) commented Dec 6, 2022 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3631734278

@rodireich

Rodi Reich Zilberman (rodireich) commented Dec 6, 2022 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3631840630
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3631840630
🐛 https://gradle.com/s/jkabuxotu5lx2

Build Failed

Test summary info:

Could not find result summary

@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets December 6, 2022 17:20 Inactive
@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets December 6, 2022 17:21 Inactive
@rodireich

Rodi Reich Zilberman (rodireich) commented Dec 6, 2022 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3632535673

@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets December 6, 2022 18:54 Inactive
@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets December 6, 2022 18:55 Inactive
@rodireich

Rodi Reich Zilberman (rodireich) commented Dec 6, 2022 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3632695040
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3632695040
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1599    332    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
================= 14 passed, 5 skipped, 21 warnings in 37.53s ==================

@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets December 6, 2022 19:15 Inactive
@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets December 6, 2022 19:16 Inactive
@rodireich

Rodi Reich Zilberman (rodireich) commented Dec 6, 2022 •

Copy link
Copy Markdown
Contributor Author

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3633515740
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3633515740
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1599    332    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
================= 14 passed, 5 skipped, 21 warnings in 26.07s ==================

@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets December 6, 2022 21:14 Inactive
@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets December 6, 2022 21:14 Inactive
@rodireich

Rodi Reich Zilberman (rodireich) commented Dec 6, 2022 •

Copy link
Copy Markdown
Contributor Author

/publish connector=connectors/source-postgres

🕑 Publishing the following connectors:
connectors/source-postgres
https://github.com/airbytehq/airbyte/actions/runs/3633761606


Connector Did it publish? Were definitions generated?
connectors/source-postgres ✅ ✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets December 6, 2022 21:51 Inactive
@rodireich
Rodi Reich Zilberman (rodireich) temporarily deployed to more-secrets December 6, 2022 21:51 Inactive
@octavia-squidington-iii
Octavia Squidington III (octavia-squidington-iii) temporarily deployed to more-secrets December 6, 2022 22:41 Inactive
@rodireich

Rodi Reich Zilberman (rodireich) commented Dec 6, 2022 •

Copy link
Copy Markdown
Contributor Author

/publish connector=connectors/source-postgres-strict-encrypt auto-bump-version=false

🕑 Publishing the following connectors:
connectors/source-postgres-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/3634096892


Connector Did it publish? Were definitions generated?
connectors/source-postgres-strict-encrypt ✅ ❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

This branch was previously deployed

1 inactive deployment
more-secrets — 46c341e2 Deployed Dec 6, 2022 by octavia-squidington-iii
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Database Sources: Handle when a cursor value is NULL

5 participants