use replace function only when null is present #21523

subodh1810 · 2023-01-18T14:16:04Z

The method cursorCandidate.replaceAll("\u0000", ""); is costly and eats upto 2.6% of total CPU time in syncs. We should only call it if there is null present in the cursor value.

See : #20424 (comment)

github-actions · 2023-01-18T14:18:16Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (25)

Connector	Version	Changelog	Publish
`source-alloydb`	`1.0.36`	✅	✅
`source-alloydb-strict-encrypt`	`1.0.36`	🔵 (ignored)	🔵 (ignored)
`source-bigquery`	`0.2.3`	✅	✅
`source-clickhouse`	`0.1.15`	✅	✅
`source-clickhouse-strict-encrypt`	`0.1.15`	🔵 (ignored)	🔵 (ignored)
`source-cockroachdb`	`0.1.19`	✅	✅
`source-cockroachdb-strict-encrypt`	`0.1.19`	🔵 (ignored)	🔵 (ignored)
`source-db2`	`0.1.17`	✅	✅
`source-db2-strict-encrypt`	`0.1.17`	🔵 (ignored)	🔵 (ignored)
`source-dynamodb`	`0.1.0`	✅	✅
`source-jdbc`	`0.3.5`	🔵 (ignored)	🔵 (ignored)
`source-mongodb-strict-encrypt`	`0.1.19`	🔵 (ignored)	🔵 (ignored)
`source-mongodb-v2`	`0.1.19`	✅	✅
`source-mssql`	`0.4.28`	✅	✅
`source-mssql-strict-encrypt`	`0.4.28`	🔵 (ignored)	🔵 (ignored)
`source-mysql`	`1.0.21`	✅	✅
`source-mysql-strict-encrypt`	`1.0.21`	🔵 (ignored)	🔵 (ignored)
`source-oracle`	`0.3.22`	✅	✅
`source-oracle-strict-encrypt`	`0.3.22`	🔵 (ignored)	🔵 (ignored)
`source-postgres`	`1.0.42`	✅	✅
`source-postgres-strict-encrypt`	`1.0.42`	🔵 (ignored)	🔵 (ignored)
`source-redshift`	`0.3.16`	✅	✅
`source-scaffold-java-jdbc`	`0.1.0`	🔵 (ignored)	🔵 (ignored)
`source-snowflake`	`0.1.29`	✅	✅
`source-tidb`	`0.2.2`	✅	✅

See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

subodh1810 · 2023-01-18T14:18:41Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3949753893
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3949753893
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   58-65, 70-83, 246
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:22: `future_state` has a bypass reason, skipping.
================== 54 passed, 6 skipped in 718.52s (0:11:58) ===================

subodh1810 · 2023-01-27T07:54:23Z

/publish connector=connectors/source-postgres-strict-encrypt

🕑 Publishing the following connectors:
connectors/source-postgres-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/4022574089

Connector	Did it publish?	Were definitions generated?
connectors/source-postgres-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

subodh1810 · 2023-01-27T07:54:29Z

/publish connector=connectors/source-postgres

🕑 Publishing the following connectors:
connectors/source-postgres
https://github.com/airbytehq/airbyte/actions/runs/4022574459

Connector	Did it publish?	Were definitions generated?
connectors/source-postgres	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

github-actions · 2023-01-27T08:04:09Z

Airbyte Code Coverage

There is no coverage information present for the Files changed

Total Project Coverage	24%	❌

use replace function only when null is present

2d9d369

subodh1810 self-assigned this Jan 18, 2023

octavia-squidington-iv added connectors/source/relational-db area/connectors Connector related issues labels Jan 18, 2023

subodh1810 mentioned this pull request Jan 18, 2023

Profile Postgres Source Connector #20424

Closed

subodh1810 requested review from akashkulk and rodireich January 20, 2023 06:03

akashkulk approved these changes Jan 20, 2023

View reviewed changes

subodh1810 mentioned this pull request Jan 24, 2023

Date Time Parsing Slowness in Postgres Source #21801

Closed

subodh1810 added 2 commits January 27, 2023 13:19

Merge branch 'master' into fix-replace-null-call

0fb2728

upgrade version

7313f04

subodh1810 requested a review from a team as a code owner January 27, 2023 07:53

octavia-squidington-iv added area/documentation Improvements or additions to documentation connectors/source/postgres connectors/source/postgres-strict-encrypt labels Jan 27, 2023

subodh1810 temporarily deployed to more-secrets January 27, 2023 07:55 — with GitHub Actions Inactive

auto-bump connector version

4a022f1

octavia-squidington-iii temporarily deployed to more-secrets January 27, 2023 09:02 — with GitHub Actions Inactive

subodh1810 merged commit 5ceb14a into master Jan 27, 2023

subodh1810 deleted the fix-replace-null-call branch January 27, 2023 09:37

This was referenced Jan 27, 2023

Bump Airbyte version from 0.40.29 to 0.40.30 #21991

Closed

Bump Airbyte version from 0.40.29 to 0.40.30 #22045

Merged

xiaohansong mentioned this pull request Mar 15, 2024

Move cursor state decorating iterator to sourceStateIterator #35599

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use replace function only when null is present #21523

use replace function only when null is present #21523

subodh1810 commented Jan 18, 2023

github-actions bot commented Jan 18, 2023 •

edited

✅ Sources (25)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

subodh1810 commented Jan 18, 2023 •

edited by github-actions bot

subodh1810 commented Jan 27, 2023 •

edited by github-actions bot

subodh1810 commented Jan 27, 2023 •

edited by github-actions bot

github-actions bot commented Jan 27, 2023

use replace function only when null is present #21523

use replace function only when null is present #21523

Conversation

subodh1810 commented Jan 18, 2023

github-actions bot commented Jan 18, 2023 • edited

Affected Connector Report

✅ Sources (25)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

subodh1810 commented Jan 18, 2023 • edited by github-actions bot

Build Passed

subodh1810 commented Jan 27, 2023 • edited by github-actions bot

subodh1810 commented Jan 27, 2023 • edited by github-actions bot

github-actions bot commented Jan 27, 2023

Airbyte Code Coverage

github-actions bot commented Jan 18, 2023 •

edited

subodh1810 commented Jan 18, 2023 •

edited by github-actions bot

subodh1810 commented Jan 27, 2023 •

edited by github-actions bot

subodh1810 commented Jan 27, 2023 •

edited by github-actions bot