[Bug] [fs] Add missing s3-transfer-manager dependency for AWS SDK

[juntaozhang]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

master

Compute Engine

spark

Minimal reproduce step

  val spark = SparkSession.builder
    .appName("Spark Paimon Example")
    .master("local[*]")
    .config("spark.default.parallelism", "1")
    .config("spark.sql.shuffle.partitions", "2")
    .config(
      "spark.sql.extensions",
      "org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions")
    .config("spark.sql.codegen.wholeStage", "false")
    .config("spark.sql.catalog.paimon", "org.apache.paimon.spark.SparkCatalog")
    .config("spark.sql.catalog.paimon.warehouse", "s3://warehouse/paimon")
    .config("spark.sql.catalog.paimon.s3.path.style.access", "true")
    .config("spark.sql.catalog.paimon.s3.access-key", "test")
    .config("spark.sql.catalog.paimon.s3.secret-key", "11111111")
    .config("spark.sql.catalog.paimon.s3.endpoint", "http://localhost:9000")
    .getOrCreate
  spark.sql("USE paimon")

What doesn't meet your expectations?

currently will meet NoClassDefFoundError:

spark-sql (default)> USE paimon;
26/02/25 09:55:09 WARN HadoopUtils: Could not find Hadoop configuration via any of the supported methods
26/02/25 09:55:09 ERROR SparkSQLDriver: Failed in [USE paimon]
java.lang.NoClassDefFoundError: software/amazon/awssdk/transfer/s3/model/ObjectTransfer
	at org.apache.paimon.s3.S3FileIO.lambda$createFileSystem$0(S3FileIO.java:137)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(Unknown Source)
	at org.apache.paimon.s3.S3FileIO.createFileSystem(S3FileIO.java:121)
	at org.apache.paimon.s3.HadoopCompliantFileIO.getFileSystem(HadoopCompliantFileIO.java:147)
	at org.apache.paimon.s3.HadoopCompliantFileIO.exists(HadoopCompliantFileIO.java:104)
	at org.apache.paimon.fs.PluginFileIO.lambda$exists$4(PluginFileIO.java:71)
	at org.apache.paimon.fs.PluginFileIO.wrap(PluginFileIO.java:108)
	at org.apache.paimon.fs.PluginFileIO.exists(PluginFileIO.java:71)
	at org.apache.paimon.fs.FileIO.checkOrMkdirs(FileIO.java:303)
	at org.apache.paimon.catalog.CatalogFactory.createUnwrappedCatalog(CatalogFactory.java:98)

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!