Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOP-8140] Split Clickhouse and Clickhouse.Dialect to separated submodules #133

Merged
merged 1 commit into from
Aug 30, 2023

Conversation

dolfinus
Copy link
Member

@dolfinus dolfinus commented Aug 21, 2023

Change Summary

  • Split Clickhouse and Clickhouse.Dialect to submodules. Same for MSSQL, MySQL, Oracle, Postgres, Teradata.
  • Move methods of generating aliases for partitioningMode="hash" and "mod" from ReadOptions to Dialect. Because of this, connection-related classes ReadOptions and Options are not required anymore, all JDBC connection uses the same JDBCOptions and JDBCLegacyOptions.
  • Split documentation for each connection to different pages with following themes (see https://onetl--133.org.readthedocs.build/en/133/connection/db_connection/clickhouse/index.html):
    • base connection documentation (constructor, get_packages, check)
    • reading from database (DBReader and .sql with JDBCReadOptions)
    • writing to database (DBWriter with JDBCWriteOptions)
    • executing statements in database (.fetch and .execute with JDBCOptions)
  • Added tests for partitioningMode="range" with unset lowerBound/upperBound, discovered and fixed bug in Oracle dialect - expression MAX(column) AS max was returned as dataframe column with name MAX instead of max.

Related issue number

Checklist

  • Commit message and PR title is comprehensive
  • Keep the change as small as possible
  • Unit and integration tests for the changes exist
  • Tests pass on CI and coverage does not decrease
  • Documentation reflects the changes where applicable
  • docs/changelog/next_release/<pull request or issue id>.<change type>.rst file added describing change
    (see CONTRIBUTING.rst for details.)
  • My PR is ready to review.

@dolfinus dolfinus self-assigned this Aug 21, 2023
@dolfinus dolfinus added the ci:skip-changelog Add this label to skip changelog file check label Aug 21, 2023
@dolfinus dolfinus force-pushed the improvement/DOP-8140-clickhouse branch from df9d127 to 7eb02e9 Compare August 21, 2023 08:43
@dolfinus dolfinus temporarily deployed to test-pypi August 21, 2023 08:43 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 21, 2023 08:54 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi August 21, 2023 08:59 — with GitHub Actions Inactive
@dolfinus dolfinus changed the title [DOP-8140] Split Clickhouse and Clickhouse.Dialect to separated submo… [DOP-8140] Split Clickhouse and Clickhouse.Dialect to separated submodules Aug 21, 2023
@dolfinus dolfinus force-pushed the improvement/DOP-8140-clickhouse branch from 4b7a97f to 1980642 Compare August 21, 2023 09:08
@dolfinus dolfinus temporarily deployed to test-pypi August 21, 2023 09:08 — with GitHub Actions Inactive
@codecov
Copy link

codecov bot commented Aug 21, 2023

Codecov Report

Merging #133 (721d804) into develop (ae4b5c9) will increase coverage by 0.07%.
The diff coverage is 94.90%.

@@             Coverage Diff             @@
##           develop     #133      +/-   ##
===========================================
+ Coverage    94.22%   94.29%   +0.07%     
===========================================
  Files          195      207      +12     
  Lines         7684     7697      +13     
  Branches      1450     1451       +1     
===========================================
+ Hits          7240     7258      +18     
+ Misses         324      320       -4     
+ Partials       120      119       -1     
Files Changed Coverage Δ
...l/connection/db_connection/greenplum/connection.py 70.38% <ø> (ø)
onetl/connection/db_connection/hive/connection.py 90.79% <ø> (ø)
onetl/connection/db_connection/hive/dialect.py 85.71% <66.66%> (-14.29%) ⬇️
onetl/connection/db_connection/teradata/dialect.py 66.66% <66.66%> (ø)
onetl/connection/db_connection/oracle/dialect.py 88.88% <88.88%> (ø)
...netl/connection/db_connection/oracle/connection.py 86.18% <90.90%> (ø)
...tl/connection/db_connection/clickhouse/__init__.py 100.00% <100.00%> (ø)
.../connection/db_connection/clickhouse/connection.py 100.00% <100.00%> (ø)
...etl/connection/db_connection/clickhouse/dialect.py 100.00% <100.00%> (ø)
.../connection/db_connection/db_connection/dialect.py 100.00% <100.00%> (ø)
... and 15 more

@dolfinus dolfinus force-pushed the improvement/DOP-8140-clickhouse branch from 1980642 to d5d7385 Compare August 21, 2023 09:43
@dolfinus dolfinus temporarily deployed to test-pypi August 21, 2023 09:43 — with GitHub Actions Inactive
@dolfinus dolfinus force-pushed the improvement/DOP-8140-clickhouse branch from d5d7385 to d9bc06c Compare August 21, 2023 10:00
@dolfinus dolfinus temporarily deployed to test-pypi August 21, 2023 10:00 — with GitHub Actions Inactive
@dolfinus dolfinus force-pushed the improvement/DOP-8140-clickhouse branch from d9bc06c to 35339f8 Compare August 21, 2023 10:07
@dolfinus dolfinus temporarily deployed to test-pypi August 21, 2023 10:08 — with GitHub Actions Inactive
@dolfinus dolfinus force-pushed the improvement/DOP-8140-clickhouse branch from 35339f8 to 444bdd0 Compare August 21, 2023 10:45
@dolfinus dolfinus temporarily deployed to test-pypi August 21, 2023 10:45 — with GitHub Actions Inactive
@dolfinus dolfinus force-pushed the improvement/DOP-8140-clickhouse branch from 444bdd0 to 150319f Compare August 21, 2023 10:52
@dolfinus dolfinus temporarily deployed to test-pypi August 21, 2023 10:52 — with GitHub Actions Inactive
@dolfinus dolfinus force-pushed the improvement/DOP-8140-clickhouse branch from 150319f to 04832cf Compare August 21, 2023 11:25
@dolfinus dolfinus temporarily deployed to test-pypi August 21, 2023 11:25 — with GitHub Actions Inactive
@dolfinus dolfinus force-pushed the improvement/DOP-8140-clickhouse branch from 04832cf to c5479c5 Compare August 21, 2023 11:39
@dolfinus dolfinus temporarily deployed to test-pypi August 21, 2023 11:39 — with GitHub Actions Inactive
@dolfinus dolfinus removed the ci:skip-changelog Add this label to skip changelog file check label Aug 21, 2023
@dolfinus dolfinus force-pushed the improvement/DOP-8140-clickhouse branch from c5479c5 to 8f59bb8 Compare August 21, 2023 11:49
@dolfinus dolfinus temporarily deployed to test-pypi August 21, 2023 11:49 — with GitHub Actions Inactive
@dolfinus dolfinus marked this pull request as ready for review August 21, 2023 12:16
@dolfinus dolfinus force-pushed the improvement/DOP-8140-clickhouse branch from 8f59bb8 to 3201471 Compare August 22, 2023 15:40
@dolfinus dolfinus temporarily deployed to test-pypi August 22, 2023 15:40 — with GitHub Actions Inactive
@dolfinus dolfinus force-pushed the improvement/DOP-8140-clickhouse branch from 3201471 to 721d804 Compare August 30, 2023 07:48
@dolfinus dolfinus temporarily deployed to test-pypi August 30, 2023 07:48 — with GitHub Actions Inactive
@dolfinus dolfinus enabled auto-merge (rebase) August 30, 2023 07:48
@dolfinus dolfinus merged commit b7a7916 into develop Aug 30, 2023
39 checks passed
@dolfinus dolfinus deleted the improvement/DOP-8140-clickhouse branch August 30, 2023 08:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants