Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GLUTEN-4933][VL] Update iceberg version to 1.4.3 for Spark 3.4 and above #4967

Merged
merged 1 commit into from
Mar 21, 2024

Conversation

yma11
Copy link
Contributor

@yma11 yma11 commented Mar 15, 2024

What changes were proposed in this pull request?

Update iceberg version to 1.4.3 for Spark 3.3 and above, as v1.3.1 is not compatible with Spark 3.5.
Note that iceberg's latest version is 1.5.0 but as it's not available in mirror repo, we use 1.4.3 instead. So the version mapping is:

Spark 3.2 : v1.3.1
Spark 3.3 : v1.3.1
Spark 3.4 : v1.4.3
Spark 3.5 : v1.4.3  

How was this patch tested?

Existing UTs

Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Copy link

Run Gluten Clickhouse CI

@zhouyuan
Copy link
Contributor

related: #4933

@yma11
Copy link
Contributor Author

yma11 commented Mar 19, 2024

@liujiayi771 UT shows incompatible ParquetWriter API in Spark 3.3 + iceberg 1.4.3. I found that iceberg 1.4.3 is using parquet 1.13.1 while Spark 3.3 is using parquet 1.12.2, so we should keep using iceberg 1.3.1 for Spark 3.3, any comments?

@zhouyuan
Copy link
Contributor

@yma11 can we use different iceberg for different spark? sounds this would fix the issue

@liujiayi771
Copy link
Contributor

@yma11 I think we should keep iceberg 1.3.1 for Spark 3.3. This compatibility issue was discovered before, which is why the Iceberg version 1.3.1 was chosen at that time.

@yma11 yma11 changed the title [VL] Update iceberg version to 1.4.3 for Spark 3.3 and above [VL] Update iceberg version to 1.4.3 for Spark 3.4 and above Mar 19, 2024
Copy link

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

@zhouyuan zhouyuan changed the title [VL] Update iceberg version to 1.4.3 for Spark 3.4 and above [GLUTEN-4933][VL] Update iceberg version to 1.4.3 for Spark 3.4 and above Mar 21, 2024
Copy link

#4933

Copy link
Contributor

@zhouyuan zhouyuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@zhouyuan zhouyuan merged commit 0f5716f into apache:main Mar 21, 2024
19 checks passed
@yma11 yma11 deleted the iceberg branch April 23, 2024 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants