Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOP-6758] Fix Hive.check() behavior when Hive Metastore is not available #164

Merged
merged 1 commit into from
Oct 10, 2023

Conversation

dolfinus
Copy link
Member

@dolfinus dolfinus commented Oct 6, 2023

Change Summary

  1. Hive.check() now does not return successful result when Hive Metastore is not available.
  2. Added check to all DB and FileDF connections that Spark session is alive.

Related issue number

Checklist

  • Commit message and PR title is comprehensive
  • Keep the change as small as possible
  • Unit and integration tests for the changes exist
  • Tests pass on CI and coverage does not decrease
  • Documentation reflects the changes where applicable
  • docs/changelog/next_release/<pull request or issue id>.<change type>.rst file added describing change
    (see CONTRIBUTING.rst for details.)
  • My PR is ready to review.

@dolfinus dolfinus changed the title [DOP-6758] Fix Hive.check() behavior when Hive Metastore is not avail… [DOP-6758] Fix Hive.check() behavior when Hive Metastore is not available Oct 6, 2023
@dolfinus dolfinus temporarily deployed to test-pypi October 6, 2023 07:53 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi October 6, 2023 08:13 — with GitHub Actions Inactive
@codecov
Copy link

codecov bot commented Oct 6, 2023

Codecov Report

Merging #164 (fe52691) into develop (32c37ed) will decrease coverage by 0.04%.
The diff coverage is 81.81%.

@@             Coverage Diff             @@
##           develop     #164      +/-   ##
===========================================
- Coverage    94.33%   94.29%   -0.04%     
===========================================
  Files          209      209              
  Lines         7925     7943      +18     
  Branches      1490     1494       +4     
===========================================
+ Hits          7476     7490      +14     
- Misses         326      328       +2     
- Partials       123      125       +2     
Files Coverage Δ
onetl/connection/db_connection/hive/connection.py 91.07% <100.00%> (ø)
...etl/connection/db_connection/mongodb/connection.py 93.75% <ø> (ø)
...nnection/db_connection/db_connection/connection.py 94.11% <80.00%> (-5.89%) ⬇️
...ion/file_df_connection/spark_file_df_connection.py 88.00% <80.00%> (-1.02%) ⬇️

@dolfinus dolfinus temporarily deployed to test-pypi October 6, 2023 08:34 — with GitHub Actions Inactive
@dolfinus dolfinus self-assigned this Oct 6, 2023
@dolfinus dolfinus marked this pull request as ready for review October 6, 2023 09:00
@dolfinus dolfinus temporarily deployed to test-pypi October 6, 2023 15:43 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi October 6, 2023 15:48 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi October 6, 2023 15:57 — with GitHub Actions Inactive
@dolfinus dolfinus temporarily deployed to test-pypi October 6, 2023 16:07 — with GitHub Actions Inactive
@dolfinus
Copy link
Member Author

dolfinus commented Oct 6, 2023

Replaced Spark session aliveness probe from spark.sql("SELECT 1").collect() to checking SparkContext attribute. Previous implementation generated jobs for Spark executor, and doing that on every constructor call is a bad idea.

@dolfinus dolfinus merged commit d20b44c into develop Oct 10, 2023
38 of 39 checks passed
@dolfinus dolfinus deleted the bugfix/DOP-6758 branch October 10, 2023 08:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants