Skip to content

[fix](jdbc) fix jdbc driver bug and external datasource p2 test case issue#16033

Merged
yiguolei merged 6 commits intoapache:masterfrom
morningman:p2_bug2
Jan 18, 2023
Merged

[fix](jdbc) fix jdbc driver bug and external datasource p2 test case issue#16033
yiguolei merged 6 commits intoapache:masterfrom
morningman:p2_bug2

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Jan 17, 2023

Proposed changes

Issue Number: close #xxx

Problem summary

  1. Fix bug that when create jdbc resource with only jdbc driver file name, it will failed to do checksum
    This is because we forgot the pass the full driver url to JdbcClient.

  2. Add ResultSet.FETCH_FORWARD and set AutoCommit to false to jdbc connection, so to avoid OOM when fetching large amount of data

  3. set useCursorFetch in jdbc url for both MySQL and PostgreSQL.

  4. Fix some p2 external datasource bug

Checklist(Required)

  1. Does it affect the original behavior:
    • Yes
    • No
    • I don't know
  2. Has unit tests been added:
    • Yes
    • No
    • No Need
  3. Has document been added or modified:
    • Yes
    • No
    • No Need
  4. Does it need to update dependencies:
    • Yes
    • No
  5. Are there any changes that cannot be rolled back:
    • Yes (If Yes, please explain WHY)
    • No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@morningman morningman added area/multi-catalog dev/1.2.2 kind/fix Categorizes issue or PR as related to a bug. labels Jan 17, 2023
@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 17, 2023

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.46 seconds
load time: 496 seconds
storage size: 17122281240 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230118061933_clickbench_pr_82910.html

dataSource = new HikariDataSource(config);
conn = dataSource.getConnection();
conn.setAutoCommit(false);
if (op == TJdbcOperation.READ) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should set it to false only the op==READ

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Contributor

@Gabriel39 Gabriel39 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 18, 2023
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit 4035bd8 into apache:master Jan 18, 2023
morningman added a commit that referenced this pull request Jan 23, 2023
…issue (#16033)

Fix bug that when create jdbc resource with only jdbc driver file name, it will failed to do checksum
This is because we forgot the pass the full driver url to JdbcClient.

Add ResultSet.FETCH_FORWARD and set AutoCommit to false to jdbc connection, so to avoid OOM when fetching large amount of data

set useCursorFetch in jdbc url for both MySQL and PostgreSQL.

Fix some p2 external datasource bug
dutyu pushed a commit to dutyu/doris that referenced this pull request Feb 1, 2023
…issue (apache#16033)

Fix bug that when create jdbc resource with only jdbc driver file name, it will failed to do checksum
This is because we forgot the pass the full driver url to JdbcClient.

Add ResultSet.FETCH_FORWARD and set AutoCommit to false to jdbc connection, so to avoid OOM when fetching large amount of data

set useCursorFetch in jdbc url for both MySQL and PostgreSQL.

Fix some p2 external datasource bug
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. area/multi-catalog dev/1.2.2-merged kind/fix Categorizes issue or PR as related to a bug. kind/test reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants