Skip to content

Comments

[SPARK-44911][SQL] Create hive table with invalid column should return error class#42609

Closed
zzzzming95 wants to merge 4 commits intoapache:masterfrom
zzzzming95:SPARK-44911
Closed

[SPARK-44911][SQL] Create hive table with invalid column should return error class#42609
zzzzming95 wants to merge 4 commits intoapache:masterfrom
zzzzming95:SPARK-44911

Conversation

@zzzzming95
Copy link
Contributor

What changes were proposed in this pull request?

create hive table with invalid column should return error class.

run sql

create table test stored as parquet as select id, date'2018-01-01' + make_dt_interval(0, id)  from range(0, 10)

before this issue , error would be :

org.apache.spark.sql.AnalysisException: Cannot create a table having a column whose name contains commas in Hive metastore. Table: `spark_catalog`.`default`.`test`; Column: DATE '2018-01-01' + make_dt_interval(0, id, 0, 0.000000)
	at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$verifyDataSchema$4(HiveExternalCatalog.scala:175)
	at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$verifyDataSchema$4$adapted(HiveExternalCatalog.scala:171)
	at scala.collection.Iterator.foreach(Iterator.scala:943)

after this issue

Exception in thread "main" org.apache.spark.sql.AnalysisException: [INVALID_HIVE_COLUMN_NAME] Cannot create the table `spark_catalog`.`default`.`parquet_ds1` having the column `DATE '2018-01-01' + make_dt_interval(0, id, 0, 0`.`000000)` whose name contains invalid characters ',' in Hive metastore.
	at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$verifyDataSchema$4(HiveExternalCatalog.scala:180)
	at org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$verifyDataSchema$4$adapted(HiveExternalCatalog.scala:171)
	at scala.collection.Iterator.foreach(Iterator.scala:943)

Why are the changes needed?

as above

Does this PR introduce any user-facing change?

no

How was this patch tested?

add UT

Was this patch authored or co-authored using generative AI tooling?

no

@wangyum
Copy link
Member

wangyum commented Aug 22, 2023

Could you change the PR title to [SPARK-44911][SQL] Create ...?

@zzzzming95 zzzzming95 changed the title [SPARK-44911] create hive table with invalid column should return error class [SPARK-44911][SQL] Create hive table with invalid column should return error class Aug 22, 2023
@zzzzming95
Copy link
Contributor Author

Could you change the PR title to [SPARK-44911][SQL] Create ...?

done , thanks for you review~

@wangyum
Copy link
Member

wangyum commented Aug 23, 2023

cc @MaxGekk

@zzzzming95 zzzzming95 requested a review from MaxGekk August 29, 2023 11:34
@zzzzming95
Copy link
Contributor Author

@MaxGekk

Can you review again on this issue ? thks!

@MaxGekk
Copy link
Member

MaxGekk commented Sep 12, 2023

+1, LGTM. Merging to master.
Thank you, @zzzzming95 and @wangyum for review.

@MaxGekk MaxGekk closed this in 1e03db3 Sep 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants