Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: integrate hive metastore for reading and write iceberg table #245

Merged
merged 2 commits into from
Aug 12, 2021
Merged

feat: integrate hive metastore for reading and write iceberg table #245

merged 2 commits into from
Aug 12, 2021

Conversation

tobegit3hub
Copy link
Collaborator

  • Please check if the PR fulfills these requirements
  • The commit message follows our guidelines
  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been added / updated (for bug fixes / features)
  • What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)

Feature. Integrate hive metastore for OpenMLDB Batch.

  • What is the current behavior? (You can also link to an open issue here)

Do not support reading table from hive metastore.

  • What is the new behavior (if this is a feature change)?

Support reading/writing iceberg tables from/to hive metastore.

  • Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

No.

  • Other information:

No.

@tobegit3hub
Copy link
Collaborator Author

Now we can read and write iceberg table with these API.

  def main(args: Array[String]): Unit = {

    val spark = SparkSession.builder()
      .master("local[*]")
      .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
      .config("spark.openmldb.hadoop.warehouse.path", "hdfs://172.27.128.215/user/tobe/iceberg_demo6/")
      .config("spark.openmldb.enable.hive.metastore", true)
      .config("hive.metastore.uris",  "thrift://127.0.0.1:9083")
      .enableHiveSupport()
      .getOrCreate()
    val sess = new OpenmldbSession(spark)

    val inputParquetPath = "file:///Users/tobe/code/4pd/OpenMLDB/cli/examples/taxi_tour_parquet_all/"
    val inputDf = spark.read.parquet(inputParquetPath).select("id", "vendor_id")
    sess.importToOfflineStorage("default", "t22", inputDf)

    sess.sql("SELECT id, vendor_id + 1 FROM t22").show()

@github-actions
Copy link
Contributor

github-actions bot commented Aug 11, 2021

Linux Test Report

     47 files  ±0     119 suites  ±0   31m 22s ⏱️ ±0s
7 508 tests ±0  7 507 ✔️ ±0  0 💤 ±0  1 ❌ ±0 

For more details on these failures, see this check.

Results for commit 9d8ecf7. ± Comparison against base commit 9d8ecf7.

♻️ This comment has been updated with latest results.

@codecov
Copy link

codecov bot commented Aug 11, 2021

Codecov Report

Merging #245 (da12197) into main (c92d7a5) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##               main     #245   +/-   ##
=========================================
  Coverage     81.63%   81.63%           
  Complexity       13       13           
=========================================
  Files           283      283           
  Lines         50128    50128           
  Branches         29       29           
=========================================
  Hits          40924    40924           
  Misses         9195     9195           
  Partials          9        9           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c92d7a5...da12197. Read the comment docs.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 11, 2021

HybridSE Linux Test Report

       71 files  ±0       219 suites  ±0   5m 26s ⏱️ ±0s
17 665 tests ±0  17 665 ✔️ ±0  0 💤 ±0  0 ❌ ±0 
17 672 runs  ±0  17 672 ✔️ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 9d8ecf7. ± Comparison against base commit 9d8ecf7.

♻️ This comment has been updated with latest results.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 11, 2021

HybridSE Mac Test Report

       71 files  ±0       219 suites  ±0   5m 32s ⏱️ ±0s
17 665 tests ±0  17 665 ✔️ ±0  0 💤 ±0  0 ❌ ±0 
17 672 runs  ±0  17 672 ✔️ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 9d8ecf7. ± Comparison against base commit 9d8ecf7.

♻️ This comment has been updated with latest results.

@imotai imotai added this to the v0.3 milestone Aug 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants