Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] failed to insert data to hive partition table if using INSERT INTO for spark-hive-connector #6078

Closed
2 of 4 tasks
FANNG1 opened this issue Feb 22, 2024 · 3 comments
Closed
2 of 4 tasks
Labels
kind:bug This is a clearly a bug priority:major

Comments

@FANNG1
Copy link

FANNG1 commented Feb 22, 2024

Code of Conduct

Search before asking

  • I have searched in the issues and found no similar issues.

Describe the bug

failed to insert data to hive partition table if using INSERT INTO for spark-hive-connector, could be reproduced by replace INSERT OVERWRITE with INSERT INTO in HiveQuerySuite.scala

  private def readPartitionedTable(format: String, hiveTable: Boolean): Unit = {
    withSparkSession() { spark =>
      val table = "hive.default.employee"
      withTempPartitionedTable(spark, table, format, hiveTable) {
        spark.sql(
          s"""
             | INSERT INTO
             | $table PARTITION(year = '2023')
             | VALUES("zhao", "09")
             |""".stripMargin)
        checkQueryResult(s"select * from $table", spark, Array(Row.apply("zhao", "2023", "09")))
      }
    }
  }

Affects Version(s)

master

Kyuubi Server Log Output

No response

Kyuubi Engine Log Output

org.apache.kyuubi.spark.connector.hive.KyuubiHiveConnectorException: Dynamic partition strict mode requires at least one static partition column. To turn this off set hive.exec.dynamic.partition.mode=nonstrict
	at org.apache.kyuubi.spark.connector.hive.write.HiveWrite.extractAndValidatePartitionCols(HiveWrite.scala:205)
	at org.apache.kyuubi.spark.connector.hive.write.HiveWrite.toBatch(HiveWrite.scala:84)
	at org.apache.spark.sql.execution.datasources.v2.V2ExistingTableWriteExec.run(WriteToDataSourceV2Exec.scala:360)
	at org.apache.spark.sql.execution.datasources.v2.V2ExistingTableWriteExec.run$(WriteToDataSourceV2Exec.scala:359)

Kyuubi Server Configurations

No response

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

  • Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
  • No. I cannot submit a PR at this time.
@FANNG1 FANNG1 added kind:bug This is a clearly a bug priority:major labels Feb 22, 2024
@pan3793
Copy link
Member

pan3793 commented Feb 22, 2024

@yikf do u have time to take a look?

@yikf
Copy link
Contributor

yikf commented Feb 23, 2024

The process of writing with Apache Spark DataSourceV2 using dynamic partitioning to handle static partitions, This exception by the issue reported should be a bug in KSHC, and will take a look.

@pan3793 pan3793 closed this as completed in 5bee05e Mar 7, 2024
pan3793 pushed a commit that referenced this issue Mar 7, 2024
… as dynamic partition at write path

# 🔍 Description
## Issue References 🔗

This pull request fixes #6078, KSHC should handle the commit of the partitioned table as dynamic partition at write path, that's beacuse the process of writing with Apache Spark DataSourceV2 using dynamic partitioning to handle static partitions.

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6082 from Yikf/KYUUBI-6078.

Closes #6078

2ae1836 [yikaifei] KSHC should handle the commit of the partitioned table as dynamic partition at write path

Authored-by: yikaifei <yikaifei@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
(cherry picked from commit 5bee05e)
Signed-off-by: Cheng Pan <chengpan@apache.org>
@FANNG1
Copy link
Author

FANNG1 commented Mar 7, 2024

thanks @yikf and @pan3793 for fixing the issue.

zhaohehuhu pushed a commit to zhaohehuhu/incubator-kyuubi that referenced this issue Mar 21, 2024
… table as dynamic partition at write path

# 🔍 Description
## Issue References 🔗

This pull request fixes apache#6078, KSHC should handle the commit of the partitioned table as dynamic partition at write path, that's beacuse the process of writing with Apache Spark DataSourceV2 using dynamic partitioning to handle static partitions.

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes apache#6082 from Yikf/KYUUBI-6078.

Closes apache#6078

2ae1836 [yikaifei] KSHC should handle the commit of the partitioned table as dynamic partition at write path

Authored-by: yikaifei <yikaifei@apache.org>
Signed-off-by: Cheng Pan <chengpan@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:bug This is a clearly a bug priority:major
Projects
None yet
3 participants