Skip to content

[SPARK-48409][BUILD][TESTS] Upgrade MySQL & Postgres & Mariadb docker image version #46704

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
panbingkun wants to merge 3 commits into from
Closed

[SPARK-48409][BUILD][TESTS] Upgrade MySQL & Postgres & Mariadb docker image version #46704

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9bb6b02
[Only Test] MySQL & Postgre Docker Images
panbingkun May 22, 2024
1f9bc79
Merge branch 'master' into db_images_upgrade
panbingkun May 24, 2024
b8e7600
update
panbingkun May 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions ...tegration-tests/src/test/scala/org/apache/spark/sql/jdbc/MariaDBKrbIntegrationSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,9 @@ import org.apache.spark.sql.execution.datasources.jdbc.connection.SecureConnecti
import org.apache.spark.tags.DockerTest

/**
* To run this test suite for a specific version (e.g., mariadb:10.5.12):
* To run this test suite for a specific version (e.g., mariadb:10.5.25):
* {{{
* ENABLE_DOCKER_INTEGRATION_TESTS=1 MARIADB_DOCKER_IMAGE_NAME=mariadb:10.5.12
* ENABLE_DOCKER_INTEGRATION_TESTS=1 MARIADB_DOCKER_IMAGE_NAME=mariadb:10.5.25
* ./build/sbt -Pdocker-integration-tests
* "docker-integration-tests/testOnly org.apache.spark.sql.jdbc.MariaDBKrbIntegrationSuite"
* }}}
Expand All @@ -38,7 +38,7 @@ class MariaDBKrbIntegrationSuite extends DockerKrbJDBCIntegrationSuite {
override protected val keytabFileName = "mariadb.keytab"

override val db = new DatabaseOnDocker {
override val imageName = sys.env.getOrElse("MARIADB_DOCKER_IMAGE_NAME", "mariadb:10.5.12")
override val imageName = sys.env.getOrElse("MARIADB_DOCKER_IMAGE_NAME", "mariadb:10.5.25")
Copy link
Copy Markdown
Contributor Author

@panbingkun panbingkun May 24, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't see it yesterday, but it is today @yaooqinn 😄

override val env = Map(
"MYSQL_ROOT_PASSWORD" -> "rootpass"
)
Expand Down
2 changes: 1 addition & 1 deletion ...er-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/MySQLDatabaseOnDocker.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
package org.apache.spark.sql.jdbc

class MySQLDatabaseOnDocker extends DatabaseOnDocker {
override val imageName = sys.env.getOrElse("MYSQL_DOCKER_IMAGE_NAME", "mysql:8.3.0")
override val imageName = sys.env.getOrElse("MYSQL_DOCKER_IMAGE_NAME", "mysql:8.4.0")
override val env = Map(
"MYSQL_ROOT_PASSWORD" -> "rootpass"
)
Expand Down
6 changes: 3 additions & 3 deletions ...integration-tests/src/test/scala/org/apache/spark/sql/jdbc/PostgresIntegrationSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,17 +32,17 @@ import org.apache.spark.sql.types._
import org.apache.spark.tags.DockerTest

/**
* To run this test suite for a specific version (e.g., postgres:16.2):
* To run this test suite for a specific version (e.g., postgres:16.3-alpine):
* {{{
* ENABLE_DOCKER_INTEGRATION_TESTS=1 POSTGRES_DOCKER_IMAGE_NAME=postgres:16.2
* ENABLE_DOCKER_INTEGRATION_TESTS=1 POSTGRES_DOCKER_IMAGE_NAME=postgres:16.3-alpine
* ./build/sbt -Pdocker-integration-tests
* "docker-integration-tests/testOnly org.apache.spark.sql.jdbc.PostgresIntegrationSuite"
* }}}
*/
@DockerTest
class PostgresIntegrationSuite extends DockerJDBCIntegrationSuite {
override val db = new DatabaseOnDocker {
override val imageName = sys.env.getOrElse("POSTGRES_DOCKER_IMAGE_NAME", "postgres:16.2-alpine")
override val imageName = sys.env.getOrElse("POSTGRES_DOCKER_IMAGE_NAME", "postgres:16.3-alpine")
override val env = Map(
"POSTGRES_PASSWORD" -> "rootpass"
)
Expand Down
6 changes: 3 additions & 3 deletions ...egration-tests/src/test/scala/org/apache/spark/sql/jdbc/PostgresKrbIntegrationSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,9 @@ import org.apache.spark.sql.execution.datasources.jdbc.connection.SecureConnecti
import org.apache.spark.tags.DockerTest

/**
* To run this test suite for a specific version (e.g., postgres:16.2):
* To run this test suite for a specific version (e.g., postgres:16.3-alpine):
* {{{
* ENABLE_DOCKER_INTEGRATION_TESTS=1 POSTGRES_DOCKER_IMAGE_NAME=postgres:16.2
* ENABLE_DOCKER_INTEGRATION_TESTS=1 POSTGRES_DOCKER_IMAGE_NAME=postgres:16.3-alpine
* ./build/sbt -Pdocker-integration-tests
* "docker-integration-tests/testOnly *PostgresKrbIntegrationSuite"
* }}}
Expand All @@ -38,7 +38,7 @@ class PostgresKrbIntegrationSuite extends DockerKrbJDBCIntegrationSuite {
override protected val keytabFileName = "postgres.keytab"

override val db = new DatabaseOnDocker {
override val imageName = sys.env.getOrElse("POSTGRES_DOCKER_IMAGE_NAME", "postgres:16.2")
Copy link
Copy Markdown
Contributor Author

@panbingkun panbingkun May 24, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's align all postgres db docker images: 16.3-alpine

override val imageName = sys.env.getOrElse("POSTGRES_DOCKER_IMAGE_NAME", "postgres:16.3-alpine")
override val env = Map(
"POSTGRES_PASSWORD" -> "rootpass"
)
Expand Down
6 changes: 3 additions & 3 deletions ...ion-tests/src/test/scala/org/apache/spark/sql/jdbc/querytest/GeneratedSubquerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,9 @@ import org.apache.spark.tags.DockerTest

/**
* This suite is used to generate subqueries, and test Spark against Postgres.
* To run this test suite for a specific version (e.g., postgres:16.2):
* To run this test suite for a specific version (e.g., postgres:16.3-alpine):
* {{{
* ENABLE_DOCKER_INTEGRATION_TESTS=1 POSTGRES_DOCKER_IMAGE_NAME=postgres:16.2
* ENABLE_DOCKER_INTEGRATION_TESTS=1 POSTGRES_DOCKER_IMAGE_NAME=postgres:16.3-alpine
* ./build/sbt -Pdocker-integration-tests
* "docker-integration-tests/testOnly org.apache.spark.sql.jdbc.GeneratedSubquerySuite"
* }}}
Expand All @@ -39,7 +39,7 @@ import org.apache.spark.tags.DockerTest
class GeneratedSubquerySuite extends DockerJDBCIntegrationSuite with QueryGeneratorHelper {

override val db = new DatabaseOnDocker {
override val imageName = sys.env.getOrElse("POSTGRES_DOCKER_IMAGE_NAME", "postgres:16.2-alpine")
override val imageName = sys.env.getOrElse("POSTGRES_DOCKER_IMAGE_NAME", "postgres:16.3-alpine")
override val env = Map(
"POSTGRES_PASSWORD" -> "rootpass"
)
Expand Down
6 changes: 3 additions & 3 deletions ...n-tests/src/test/scala/org/apache/spark/sql/jdbc/querytest/PostgreSQLQueryTestSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,9 @@ import org.apache.spark.tags.DockerTest
* confidence, and you won't have to manually verify the golden files generated with your test.
* 2. Add this line to your .sql file: --ONLY_IF spark
*
* Note: To run this test suite for a specific version (e.g., postgres:16.2):
* Note: To run this test suite for a specific version (e.g., postgres:16.3-alpine):
* {{{
* ENABLE_DOCKER_INTEGRATION_TESTS=1 POSTGRES_DOCKER_IMAGE_NAME=postgres:16.2
* ENABLE_DOCKER_INTEGRATION_TESTS=1 POSTGRES_DOCKER_IMAGE_NAME=postgres:16.3-alpine
* ./build/sbt -Pdocker-integration-tests
* "testOnly org.apache.spark.sql.jdbc.PostgreSQLQueryTestSuite"
* }}}
Expand All @@ -45,7 +45,7 @@ class PostgreSQLQueryTestSuite extends CrossDbmsQueryTestSuite {
protected val customInputFilePath: String = new File(inputFilePath, "subquery").getAbsolutePath

override val db = new DatabaseOnDocker {
override val imageName = sys.env.getOrElse("POSTGRES_DOCKER_IMAGE_NAME", "postgres:16.2-alpine")
override val imageName = sys.env.getOrElse("POSTGRES_DOCKER_IMAGE_NAME", "postgres:16.3-alpine")
override val env = Map(
"POSTGRES_PASSWORD" -> "rootpass"
)
Expand Down
6 changes: 3 additions & 3 deletions ...egration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/PostgresIntegrationSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,17 +28,17 @@ import org.apache.spark.sql.types._
import org.apache.spark.tags.DockerTest

/**
* To run this test suite for a specific version (e.g., postgres:16.2)
* To run this test suite for a specific version (e.g., postgres:16.3-alpine)
* {{{
* ENABLE_DOCKER_INTEGRATION_TESTS=1 POSTGRES_DOCKER_IMAGE_NAME=postgres:16.2
* ENABLE_DOCKER_INTEGRATION_TESTS=1 POSTGRES_DOCKER_IMAGE_NAME=postgres:16.3-alpine
* ./build/sbt -Pdocker-integration-tests "testOnly *v2.PostgresIntegrationSuite"
* }}}
*/
@DockerTest
class PostgresIntegrationSuite extends DockerJDBCIntegrationV2Suite with V2JDBCTest {
override val catalogName: String = "postgresql"
override val db = new DatabaseOnDocker {
override val imageName = sys.env.getOrElse("POSTGRES_DOCKER_IMAGE_NAME", "postgres:16.2-alpine")
override val imageName = sys.env.getOrElse("POSTGRES_DOCKER_IMAGE_NAME", "postgres:16.3-alpine")
override val env = Map(
"POSTGRES_PASSWORD" -> "rootpass"
)
Expand Down
6 changes: 3 additions & 3 deletions ...ntegration-tests/src/test/scala/org/apache/spark/sql/jdbc/v2/PostgresNamespaceSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,16 +26,16 @@ import org.apache.spark.sql.util.CaseInsensitiveStringMap
import org.apache.spark.tags.DockerTest

/**
* To run this test suite for a specific version (e.g., postgres:16.2):
* To run this test suite for a specific version (e.g., postgres:16.3-alpine):
* {{{
* ENABLE_DOCKER_INTEGRATION_TESTS=1 POSTGRES_DOCKER_IMAGE_NAME=postgres:16.2
* ENABLE_DOCKER_INTEGRATION_TESTS=1 POSTGRES_DOCKER_IMAGE_NAME=postgres:16.3-alpine
* ./build/sbt -Pdocker-integration-tests "testOnly *v2.PostgresNamespaceSuite"
* }}}
*/
@DockerTest
class PostgresNamespaceSuite extends DockerJDBCIntegrationSuite with V2JDBCNamespaceTest {
override val db = new DatabaseOnDocker {
override val imageName = sys.env.getOrElse("POSTGRES_DOCKER_IMAGE_NAME", "postgres:16.2-alpine")
override val imageName = sys.env.getOrElse("POSTGRES_DOCKER_IMAGE_NAME", "postgres:16.3-alpine")
override val env = Map(
"POSTGRES_PASSWORD" -> "rootpass"
)
Expand Down