Source-postgres: Add prepareThreshold=0 to JDBC url to mitigate PGBouncer `prepared statement [X] already exists.` by nguyenaiden · Pull Request #26873 · airbytehq/airbyte

Duy Nguyen (nguyenaiden) · 2023-05-31T23:22:52Z

What

PGBouncer is intermittently throwing prepared statement [X] already exists.

How

We can mitigate this error by setting prepareThreshold=0 in the jdbcUrl. The side effect of this is that there will be no server-side caching of frequently issued statements. Per Akash Kulkarni (@akashkulk)'s findings, this would not affect Airbyte's usecase as we rarely issue the same query multiple times. And in the case that we do, this isn't the bottleneck factor for our queries. Execution of those queries is. (see link above for more info)
In addition, performance harness was used to verify that there is no performance regression.
Based on these results below, there is no significant performance deviation between having the extra param and not.

Without prepareThreshold=0:
Run 1: total secs: 43.056. total MB read: 524.6431999206543, rps: 51350.82218506132, throughput: 12.185135635466702
Run 2: total secs: 42.063. total MB read: 524.6431999206543, rps: 52563.083945510305, throughput: 12.47279556666558
Run 3: total secs: 42.059. total MB read: 524.6431999206543, rps: 52568.082931120574, throughput: 12.47398178560247

With prepareThreshold=0:
Run 1: total secs: 43.056. total MB read: 524.6431999206543, rps: 51350.82218506132, throughput: 12.185135635466702
Run 2: total secs: 42.065. total MB read: 524.6431999206543, rps: 52560.58480922382, throughput: 12.472202541796133
Run 3: total secs: 43.059. total MB read: 524.6431999206543, rps: 51347.24447850624, throughput: 12.184286674578006

🚨 User Impact 🚨

There should be no negative user impact. They should see less of the error mentioned above.

github-actions · 2023-05-31T23:23:15Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan and you've followed all steps in the Breaking Changes Checklist
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
You, or an Airbyter, have run /test successfully on this PR - or on a non-forked branch
You've updated the connector's metadata.yaml file (new!)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

Duy Nguyen (nguyenaiden) · 2023-05-31T23:25:51Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/5138634889
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/5138634889
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:598: The previous and actual discovered catalogs are identical.
=================== 70 passed, 5 skipped in 84.63s (0:01:24) ===================

Duy Nguyen (nguyenaiden) · 2023-05-31T23:25:57Z

/test connector=connectors/source-postgres-strict-encrypt

🕑 connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/5138635145
✅ connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/5138635145
No Python unittests run

Build Passed

Test summary info:

All Passed

github-actions · 2023-05-31T23:27:13Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (3)

Connector	Version	Changelog	Publish
`source-alloydb`	`2.0.28`	✅	✅
`source-alloydb-strict-encrypt`	`2.0.28`	🔵 (ignored)	🔵 (ignored)
`source-postgres-strict-encrypt`	`2.0.33`	🔵 (ignored)	🔵 (ignored)

See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the cloud or oss registry, so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that you have added a metadata.yaml file and the expected registries are enabled.

Rodi Reich Zilberman (rodireich) · 2023-05-31T23:30:30Z

/connector-performance connector=connectors/source-postgres

Note: The following `dataset=` values are supported: `1m`_(default), `10m`, `20m`,

bottleneck_stream1, bottleneck_stream_randomseed. For destinations only: you can also use stream-numbers=N to simulate N number of parallel streams. Additionally,sync-mode=incrementalis supported for destinations. For example:dataset=1m stream-numbers=2 sync-mode=incremental`

🏃 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/5138656450.

Performance test Result:

{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.s.PerformanceTest(runTest):108 total secs: 43.056. total MB read: 524.6431999206543, rps: 51350.82218506132, throughput: 12.185135635466702

Rodi Reich Zilberman (rodireich) · 2023-05-31T23:32:56Z

/connector-performance connector=connectors/source-postgres dataset=10m

Note: The following `dataset=` values are supported: `1m`_(default), `10m`, `20m`,

bottleneck_stream1, bottleneck_stream_randomseed. For destinations only: you can also use stream-numbers=N to simulate N number of parallel streams. Additionally,sync-mode=incrementalis supported for destinations. For example:dataset=1m stream-numbers=2 sync-mode=incremental`

🏃 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/5138672733.

Performance test Result:

{"type":"LOG","log":{"level":"INFO","message":"INFO i.a.i.s.PerformanceTest(runTest):108 total secs: 385.618. total MB read: 5280.909579277039, rps: 57365.86207075396, throughput: 13.69466565169945

github-actions · 2023-05-31T23:37:07Z

Coverage report for source-postgres

File	Coverage [85.87%]	🍏
PostgresSource.java	85.87%	🍏

Total Project Coverage	65.95%	🍏

Rodi Reich Zilberman (rodireich) · 2023-05-31T23:36:17Z

nit: you can give a reason or refer to issue no.

Update test

Updated docs

…on poolers Adds prepareThreshold=0 to the HikariCP data source configuration to disable server-side prepared statements. This fixes the 'prepared statement S_1 already exists' error that occurs when using connection poolers like PgBouncer or Supabase Supavisor in transaction mode. This matches the fix already applied to source-postgres in PR #26873. Resolves: airbytehq/oncall#11014 Co-Authored-By: unknown <>

Octavia Squidington III (octavia-squidington-iii) added area/connectors Connector related issues connectors/source/postgres labels May 31, 2023

Duy Nguyen (nguyenaiden) marked this pull request as ready for review May 31, 2023 23:23

Duy Nguyen (nguyenaiden) requested a review from a team as a code owner May 31, 2023 23:23

Duy Nguyen (nguyenaiden) requested review from Akash Kulkarni (akashkulk), Prateek Mukhedkar (prateekmukhedkar), Rodi Reich Zilberman (rodireich) and Subodh Chaturvedi (subodh1810) May 31, 2023 23:24

Rodi Reich Zilberman (rodireich) approved these changes Jun 1, 2023

View reviewed changes

nguyenaiden added 4 commits June 1, 2023 17:04

Append PrepareThreshold=0 to JDBC param

0c080b6

Add prepareThreshold=0

5923120

Update test

Added comment and issue number to where prepareThreshold=0 is added

caa35c3

Bumped metadata and dockerfile version

ff25060

Updated docs

Duy Nguyen (nguyenaiden) force-pushed the duy/detect-prepared-statement branch from fc63d7b to ff25060 Compare June 2, 2023 00:10

Octavia Squidington III (octavia-squidington-iii) added the area/documentation Improvements or additions to documentation label Jun 2, 2023

Duy Nguyen (nguyenaiden) added the checklist-action-run label Jun 2, 2023

Bumped versions for postgres-strict-encrypt

5df33af

Octavia Squidington III (octavia-squidington-iii) added the connectors/source/postgres-strict-encrypt label Jun 2, 2023

Duy Nguyen (nguyenaiden) merged commit 65d26bb into master Jun 2, 2023

Duy Nguyen (nguyenaiden) deleted the duy/detect-prepared-statement branch June 2, 2023 00:56

devin-ai-integration Bot mentioned this pull request Jan 22, 2026

fix(destination-postgres): Add prepareThreshold=0 to support connection poolers (AI-Triage PR) #72284

Closed

1 task

Navigation Menu

Uh oh!

Source-postgres: Add prepareThreshold=0 to JDBC url to mitigate PGBouncer `prepared statement [X] already exists.` - #26873

Source-postgres: Add prepareThreshold=0 to JDBC url to mitigate PGBouncer `prepared statement [X] already exists.`#26873
Duy Nguyen (nguyenaiden) merged 5 commits into
masterfrom
duy/detect-prepared-statement

Duy Nguyen (nguyenaiden) commented May 31, 2023 •

edited

Loading

Uh oh!

github-actions Bot commented May 31, 2023 •

edited by nguyenaiden

Loading

Uh oh!

Duy Nguyen (nguyenaiden) commented May 31, 2023 •

edited by github-actions Bot

Loading

Uh oh!

Duy Nguyen (nguyenaiden) commented May 31, 2023 •

edited by github-actions Bot

Loading

Uh oh!

github-actions Bot commented May 31, 2023 •

edited

Loading

✅ Sources (3)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

Uh oh!

Rodi Reich Zilberman (rodireich) commented May 31, 2023 •

edited by github-actions Bot

Loading

Uh oh!

Rodi Reich Zilberman (rodireich) commented May 31, 2023 •

edited by github-actions Bot

Loading

Uh oh!

github-actions Bot commented May 31, 2023 •

edited

Loading

Uh oh!

Rodi Reich Zilberman (rodireich) May 31, 2023

Uh oh!

Duy Nguyen (nguyenaiden) Jun 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Duy Nguyen (nguyenaiden) commented May 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Recommended reading order

🚨 User Impact 🚨

Uh oh!

github-actions Bot commented May 31, 2023 • edited by nguyenaiden Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Before Merging a Connector Pull Request

Uh oh!

Duy Nguyen (nguyenaiden) commented May 31, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

Duy Nguyen (nguyenaiden) commented May 31, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build Passed

Uh oh!

github-actions Bot commented May 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Affected Connector Report

✅ Sources (3)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

Uh oh!

Rodi Reich Zilberman (rodireich) commented May 31, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Note: The following dataset= values are supported: 1m(default), 10m, 20m,

Performance test Result:

Uh oh!

Rodi Reich Zilberman (rodireich) commented May 31, 2023 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Note: The following dataset= values are supported: 1m(default), 10m, 20m,

Performance test Result:

Uh oh!

github-actions Bot commented May 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage report for source-postgres

Uh oh!

Rodi Reich Zilberman (rodireich) May 31, 2023

Choose a reason for hiding this comment

Uh oh!

Duy Nguyen (nguyenaiden) Jun 1, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Duy Nguyen (nguyenaiden) commented May 31, 2023 •

edited

Loading

github-actions Bot commented May 31, 2023 •

edited by nguyenaiden

Loading

Duy Nguyen (nguyenaiden) commented May 31, 2023 •

edited by github-actions Bot

Loading

Duy Nguyen (nguyenaiden) commented May 31, 2023 •

edited by github-actions Bot

Loading

github-actions Bot commented May 31, 2023 •

edited

Loading

Rodi Reich Zilberman (rodireich) commented May 31, 2023 •

edited by github-actions Bot

Loading

Note: The following `dataset=` values are supported: `1m`_(default), `10m`, `20m`,

Rodi Reich Zilberman (rodireich) commented May 31, 2023 •

edited by github-actions Bot

Loading

Note: The following `dataset=` values are supported: `1m`_(default), `10m`, `20m`,

github-actions Bot commented May 31, 2023 •

edited

Loading