Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[postgres] close jdbc connection when close replication connection #2425

Merged
merged 2 commits into from
Sep 26, 2023

Conversation

loserwang1024
Copy link
Contributor

@loserwang1024 loserwang1024 commented Aug 23, 2023

What is the purpose of the change

In PostgresSourceEnumerator#createSlotForGlobalStreamSplit, PostgresReplicationConnection will be closed after create slot.
However, not any PostgresReplicationConnection.close will close the PostgresConnection in messageDecoder, which is generated by Hikari pool and is passed to PostgresReplicationConnection.

See AbstractMessageDecoder.close:

  • NonStreamingWal2JsonMessageDecoder.close: do nothing
  • PgProtoMessageDecoder.close: do nothing
  • StreamingWal2JsonMessageDecoder.close: do nothing
  • PgOutputMessageDecoder.close: the only one to close jdbc connection.

Thus, the Hikari pool will be full if there is too much pg source.

Verified this problem

Debug PostgresReplicationConnection.close in any test.

loserwang1024 added a commit to loserwang1024/flink-cdc-connectors that referenced this pull request Aug 23, 2023
loserwang1024 added a commit to loserwang1024/flink-cdc-connectors that referenced this pull request Aug 23, 2023
loserwang1024 added a commit to loserwang1024/flink-cdc-connectors that referenced this pull request Aug 24, 2023
@loserwang1024
Copy link
Contributor Author

@TyrantLucifer CC

Copy link
Contributor

@ruanhang1993 ruanhang1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Copy link
Contributor

@ruanhang1993 ruanhang1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@ruanhang1993 ruanhang1993 merged commit 82e3053 into apache:master Sep 26, 2023
13 of 15 checks passed
Copy link
Member

@TyrantLucifer TyrantLucifer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

e-mhui pushed a commit to e-mhui/flink-cdc-connectors that referenced this pull request Oct 18, 2023
GOODBOY008 pushed a commit to GOODBOY008/flink-cdc that referenced this pull request Oct 30, 2023
GOODBOY008 pushed a commit to GOODBOY008/flink-cdc that referenced this pull request Oct 30, 2023
GOODBOY008 pushed a commit to GOODBOY008/flink-cdc that referenced this pull request Oct 30, 2023
leonardBang pushed a commit that referenced this pull request Oct 31, 2023
zhongqishang pushed a commit to zhongqishang/flink-cdc that referenced this pull request Dec 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants