Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update docker-images version to 89 #19564

Merged
merged 1 commit into from
Feb 16, 2024
Merged

Update docker-images version to 89 #19564

merged 1 commit into from
Feb 16, 2024

Conversation

ebyhr
Copy link
Member

@ebyhr ebyhr commented Oct 30, 2023

Description

Relates to trinodb/docker-images#184
Fixes #17562

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.

@cla-bot cla-bot bot added the cla-signed label Oct 30, 2023
@ebyhr ebyhr changed the title Update docker-image version to 88 Update docker-images version to 88 Oct 30, 2023
@ebyhr ebyhr self-assigned this Oct 30, 2023
@ebyhr ebyhr force-pushed the ebi/iceberg-1.4.1 branch 3 times, most recently from 4442d4c to 8ef931f Compare October 31, 2023 00:08
@findinpath
Copy link
Contributor

ci / pt (default, suite-iceberg, ) (pull_request) is 🔴

@ebyhr
Copy link
Member Author

ebyhr commented Nov 17, 2023

Probably, we should burn docker-images 88. It throws NoClassDefFoundError when inserting rows into Avro table.
I tried all 3.x Hive JDBC drivers, but all of them didn't work correctly.

@findinpath
Copy link
Contributor

Any idea about this failure?

2023-11-17 10:27:59 SEVERE: Failure cause:
2023-11-17T04:42:59.6316094Z tests               | io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: org.apache.hive.service.cli.HiveSQLException: Error running query: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 217.0 failed 1 times, most recent failure: Lost task 0.0 in stage 217.0 (TID 205) (spark executor driver): java.lang.NoClassDefFoundError: org/apache/spark/sql/internal/SQLConf$LegacyBehaviorPolicy$
2023-11-17T04:42:59.6319363Z tests               | 	at org.apache.spark.sql.avro.AvroOutputWriter.<init>(AvroOutputWriter.scala:48)
2023-11-17T04:42:59.6321159Z tests               | 	at org.apache.spark.sql.avro.AvroOutputWriterFactory.newInstance(AvroOutputWriterFactory.scala:47)
2023-11-17T04:42:59.6323359Z tests               | 	at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:161)

@ebyhr ebyhr marked this pull request as draft November 21, 2023 05:07
@ebyhr ebyhr force-pushed the ebi/iceberg-1.4.1 branch 3 times, most recently from c644ce3 to 25302f4 Compare November 21, 2023 09:24
@@ -149,6 +149,7 @@ databases:
spark:
host: spark
jdbc_driver_class: org.apache.hive.jdbc.HiveDriver
jdbc_jar: /docker/jdbc/hive-jdbc.jar
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this now?

org.apache.hive.jdbc.HiveDriver in hive-jdbc.jar is same as on the classpath

<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<version>3.1.3</version>
<classifier>standalone</classifier>
<scope>runtime</scope>
<exclusions>
<exclusion>
<groupId>*</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>

<artifactItem>
<groupId>org.apache.hive</groupId>
<artifactId>hive-jdbc</artifactId>
<classifier>standalone</classifier>
<type>jar</type>
<outputDirectory>${project.build.directory}</outputDirectory>
<destFileName>hive-jdbc.jar</destFileName>
</artifactItem>

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trino-product-tests module contains another Hive JDBC driver. New Hive JDBC driver doesn't work with old Hive servers if I remember correctly. I'm losing the context of this PR. This comment might be wrong.

        <dependency>
            <groupId>io.trino.hive</groupId>
            <artifactId>hive-apache-jdbc</artifactId>
            <scope>runtime</scope>
        </dependency>

@ebyhr ebyhr force-pushed the ebi/iceberg-1.4.1 branch 2 times, most recently from 66c5a68 to 78a603b Compare December 21, 2023 10:59
@ebyhr ebyhr changed the title Update docker-images version to 88 Update docker-images version to 89 Dec 21, 2023
@ebyhr ebyhr force-pushed the ebi/iceberg-1.4.1 branch 2 times, most recently from ddc5c98 to cbc1c7a Compare December 22, 2023 03:14
@ebyhr ebyhr force-pushed the ebi/iceberg-1.4.1 branch 2 times, most recently from 20fda90 to dd7de3b Compare February 15, 2024 05:56
@ebyhr ebyhr force-pushed the ebi/iceberg-1.4.1 branch 2 times, most recently from 6135fc2 to 6a607fa Compare February 16, 2024 07:15
@ebyhr ebyhr marked this pull request as ready for review February 16, 2024 07:16
onSpark().executeQuery(format("INSERT INTO %s PARTITION(dt='-0001-01-01') VALUES (6)", sparkTableName));

assertThat(onTrino().executeQuery("SELECT \"$partition\" FROM " + trinoTableName))
assertThat(onTrino().executeQuery("SELECT value, \"$partition\" FROM " + trinoTableName))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added value column to distinguish the partition values.

@wendigo wendigo merged commit 9b0c316 into master Feb 16, 2024
99 checks passed
@wendigo wendigo deleted the ebi/iceberg-1.4.1 branch February 16, 2024 11:53
@github-actions github-actions bot added this to the 440 milestone Feb 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

Test TIMESTAMP_NTZ type in Iceberg connector
4 participants