[GLUTEN-10113][VL] Pass hadoop fs related session level configurations to native by zhouyuan · Pull Request #12367 · apache/gluten

zhouyuan · 2026-06-25T13:14:43Z

What changes are proposed in this pull request?

This patch adds the runtime hadoop related configurations to native.

How was this patch tested?

new UT

Was this patch authored or co-authored using generative AI tooling?

IBM BoB

Related issue: #10113

Signed-off-by: Yuan <yuanzhou@apache.org>

github-actions · 2026-06-25T13:15:15Z

Run Gluten Clickhouse CI on x86

Signed-off-by: Yuan <yuanzhou@apache.org>

Copilot

Pull request overview

This PR extends the Velox backend’s partition serialization so that Hadoop filesystem-related session configuration (notably fs.azure.*, fs.s3a.*, fs.gs.*) is captured on the driver and carried across the RDD partition boundary to executors, then forwarded into the native runtime via extraConf.

Changes:

Added an fsConf: Map[String, String] field to GlutenPartition to carry selected Hadoop FS config across to executors.
Updated VeloxIteratorApi.genPartitions to collect fs.* keys from the driver-side Hadoop configuration and embed them into each GlutenPartition.
Updated VeloxIteratorApi.genFirstStageIterator to merge GlutenPartition.fsConf into the native extraConf, and added a new Velox UT suite to validate propagation behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File	Description
gluten-substrait/src/main/scala/org/apache/gluten/execution/GlutenWholeStageColumnarRDD.scala	Adds `GlutenPartition.fsConf` to transport FS-related Hadoop config across partition boundaries.
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApi.scala	Captures driver-side FS Hadoop config into partitions and forwards it into native `extraConf`.
backends-velox/src/test/scala/org/apache/gluten/backendsapi/velox/VeloxIteratorApiFsConfSuite.scala	Adds unit tests verifying that selected `fs.*` keys are embedded into `GlutenPartition.fsConf`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

github-actions · 2026-06-25T13:23:44Z

Run Gluten Clickhouse CI on x86

Signed-off-by: Yuan <yuanzhou@apache.org>

github-actions · 2026-06-25T15:17:22Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

Comments suppressed due to low confidence (1)

gluten-substrait/src/main/scala/org/apache/gluten/execution/GlutenWholeStageColumnarRDD.scala:51

GlutenPartition now contains fsConf, which can include filesystem credentials (e.g., S3/ABFS secrets). Because this is a case class, the default toString will include the full fsConf map and may leak secrets into Spark logs / exception messages / UI when partitions are stringified. Override toString to avoid printing values (e.g., print only keys).

  override def preferredLocations(): Array[String] =
    splitInfos.flatMap(_.preferredLocations().asScala)
}

github-actions · 2026-06-25T17:03:26Z

Run Gluten Clickhouse CI on x86

Signed-off-by: Yuan <yuanzhou@apache.org>

github-actions · 2026-06-25T17:22:08Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

+    val partitionFsConf = inputPartition.asInstanceOf[GlutenPartition].fsConf
+    val extraConf = (partitionFsConf +
+      (GlutenConfig.COLUMNAR_CUDF_ENABLED.key -> enableCudf.toString)).asJava
    val transKernel = NativePlanEvaluator.create(BackendsApiManager.getBackendName, extraConf)


+      "fs.azure.account.auth.type.myaccount.dfs.core.windows.net" -> "OAuth",
+      "fs.azure.account.oauth.provider.type" -> "ClientCredentials"
+    ) {
+      val partitions = api.genPartitions(emptyWsCtx, Seq(Seq.empty), Seq.empty)


Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

github-actions · 2026-06-26T14:17:58Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (1)

gluten-substrait/src/main/scala/org/apache/gluten/execution/GlutenWholeStageColumnarRDD.scala:51

GlutenPartition is a case class, so its generated toString will include the full fsConf map (including credential values). This can leak secrets via exceptions/logging (e.g., $split string interpolation) or Spark UI/debug output. Consider overriding toString to avoid printing values (e.g., only print the keys).

  override def preferredLocations(): Array[String] =
    splitInfos.flatMap(_.preferredLocations().asScala)
}

+    val fsPrefixes = Seq("fs.azure.", "fs.s3a.", "fs.gs.")
+    val hadoopConf = leaves.headOption
+      .map(_ => org.apache.spark.sql.SparkSession.active.sessionState.newHadoopConf())
+      .getOrElse(org.apache.spark.SparkContext.getOrCreate().hadoopConfiguration)


zhouyuan · 2026-06-29T09:38:37Z

+    val partitionFsConf = inputPartition.asInstanceOf[GlutenPartition].fsConf
+    val extraConf = (partitionFsConf +
+      (GlutenConfig.COLUMNAR_CUDF_ENABLED.key -> enableCudf.toString)).asJava
    val transKernel = NativePlanEvaluator.create(BackendsApiManager.getBackendName, extraConf)



fixed by adding a digest on the key/password

+  test("genPartitions embeds fs.s3a.* keys from Hadoop conf into GlutenPartition.fsConf") {
+    withHadoopConf(
+      "fs.s3a.access.key" -> "AKIAIOSFODNN7EXAMPLE",
+      "fs.s3a.secret.key" -> "wJalrXUtnFEMI"
+    ) {
+      val partitions = api.genPartitions(emptyWsCtx, Seq(Seq.empty), Seq.empty)
+      assert(partitions.size == 1)
+      val fsConf = partitions.head.asInstanceOf[GlutenPartition].fsConf
+      assert(
+        fsConf.contains("fs.s3a.access.key"),
+        s"Expected fs.s3a.access.key not found; got: ${fsConf.keys.mkString(", ")}")
+      assert(fsConf("fs.s3a.access.key") == "AKIAIOSFODNN7EXAMPLE")
+      assert(fsConf.contains("fs.s3a.secret.key"))
+      assert(fsConf("fs.s3a.secret.key") == "wJalrXUtnFEMI")
+    }
+  }


+  test("genPartitions embeds fs.azure.* keys from Hadoop conf into GlutenPartition.fsConf") {
+    withHadoopConf(
+      "fs.azure.account.auth.type.myaccount.dfs.core.windows.net" -> "OAuth",
+      "fs.azure.account.oauth.provider.type" -> "ClientCredentials"
+    ) {
+      val partitions = api.genPartitions(emptyWsCtx, Seq(Seq.empty), Seq.empty)


github-actions · 2026-06-29T09:33:11Z

Run Gluten Clickhouse CI on x86

Signed-off-by: Yuan <yuanzhou@apache.org>

github-actions · 2026-06-29T09:38:19Z

Run Gluten Clickhouse CI on x86

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

+  private def stableKey(
+      backendName: String,
+      name: String,
+      extraConf: util.Map[String, String]): String = {
+    val digest = MessageDigest.getInstance("SHA-256")
+    digest.update(backendName.getBytes("UTF-8"))
+    digest.update(0.toByte) // field separator
+    digest.update(name.getBytes("UTF-8"))
+    digest.update(0.toByte)
+    // Sort keys for determinism; hash only keys, not values, to avoid leaking secrets.
+    val sortedKeys = new java.util.ArrayList(extraConf.keySet)
+    java.util.Collections.sort(sortedKeys)
+    sortedKeys.forEach {
+      k =>
+        digest.update(k.getBytes("UTF-8"))
+        digest.update(0.toByte)
+    }
+    digest.digest().map("%02x".format(_)).mkString
+  }


+    // fs.azure.* / fs.s3a.* / fs.gs.* keys captured on the driver from
+    // sessionState.newHadoopConf() and serialised here so they survive the
+    // RDD partition boundary.  Spark's withSQLConfPropagated only propagates
+    // keys that start with "spark", so these keys are otherwise invisible on
+    // the executor side (the executor's SQLConf never sees them).


+    val hadoopConf = leaves.headOption
+      .map(_ => org.apache.spark.sql.SparkSession.active.sessionState.newHadoopConf())
+      .getOrElse(org.apache.spark.SparkContext.getOrCreate().hadoopConfiguration)


+    // Hadoop configuration NOW, while we are still on the driver, and embed
+    // them in every GlutenPartition.  These keys are set by the user via
+    //   spark.conf.set("fs.azure.account.auth.type", ...)   or
+    //   sparkContext.hadoopConfiguration.set(...)


+    withHadoopConf(
+      "fs.s3a.access.key" -> "AKIAIOSFODNN7EXAMPLE",
+      "fs.s3a.secret.key" -> "wJalrXUtnFEMI"
+    ) {


jackylee-ch · 2026-06-29T10:57:11Z

+      .map(_ => org.apache.spark.sql.SparkSession.active.sessionState.newHadoopConf())
+      .getOrElse(org.apache.spark.SparkContext.getOrCreate().hadoopConfiguration)
+    val fsConf = {
+      hadoopConf.iterator().asScala


Thanks for the PR — this is a problem we've hit before. Internally we addressed it by injecting a known Hadoop key list at a specific injection point.
The hadoopConf.iterator() would return the full configuration set, which has a noticeable performance cost. Besides, some of the configu entries are also unresolved — values like .key=${...} aren't decoded, so the velox backend fails to parse them. I haven't come across a cleaner solution yet.

Good point. Hadoop Configuration has getPropsWithPrefix in the Hadoop versions Gluten supports, so I’ll replace the full iterator() scan with prefix lookups for fs.azure., fs.s3a., and fs.gs.. This keeps the same propagation behavior but avoids iterating the entire configuration set.

Signed-off-by: Yuan <yuanzhou@apache.org>

github-actions · 2026-06-29T19:12:48Z

Run Gluten Clickhouse CI on x86

jackylee-ch · 2026-06-30T02:41:42Z

+    // Spark's withSQLConfPropagated (it only forwards keys starting with
+    // "spark"), so embedding them in the serialised GlutenPartition is the
+    // only reliable transport mechanism.
+    val fsPrefixes = Seq("fs.azure.", "fs.s3a.", "fs.gs.")


nit: We can define this part as a constant and allow users to extend it (via configuration or by adding a constant prefix), which would make the user experience more friendly.

jackylee-ch

👍

LuciferYang · 2026-06-30T15:23:35Z

+    // RDD partition boundary.  Spark's withSQLConfPropagated only propagates
+    // keys that start with "spark", so these keys are otherwise invisible on
+    // the executor side (the executor's SQLConf never sees them).
+    fsConf: Map[String, String] = Map.empty


CWE‑532 / CWE‑200：Case‑class auto‑toString prints every field, so any log line or exception that stringifies a partition (or the wrapping FirstZippedPartitionsPartition) will dump fs.s3a.secret.key=… verbatim. Please override toString to print fsConf keys only — or redact values on the driver side at capture time using SECRET_REDACTION_PATTERN (spark.redaction.regex). Note SQLConf.get.stringRedactionPattern is a different config (default None) and won't help here, and SQLConf.get itself isn't safe to call off‑task. Please also add a regression test that asserts toString does not contain a seeded secret value.

I notice Copilot already flagged this security‑related risk earlier. However, the latest commit still hasn’t addressed the issue.

LuciferYang · 2026-06-30T15:24:18Z

+        digest.update(k.getBytes("UTF-8"))
+        digest.update(0.toByte)
+    }
+    digest.digest().map("%02x".format(_)).mkString


CWE‑694：stableKey hashes only the keys. Two Runtimes.contextInstance calls inside the same task that share a keyset but differ in values (e.g. IcebergWrite#write with different icebergProperties) will collide on the same TaskResources registration, and the second caller silently gets the runtime built from the first. Please hash sorted (key, value) pairs — values are still hidden from logs because only the digest escapes, and the existing key‑sort already makes the digest deterministic.

LuciferYang · 2026-06-30T15:25:14Z

+    // Capture fs.azure.* / fs.s3a.* / fs.gs.* keys from the driver-side
+    // Hadoop configuration NOW, while we are still on the driver, and embed
+    // them in every GlutenPartition.  These keys are set by the user via
+    //   spark.conf.set("fs.azure.account.auth.type", ...)   or


Two things：

The comment block at lines 128–144 still advertises spark.conf.set(...), DataFrameReader.option(...), and sessionState.newHadoopConf() as user surfaces — none of those reach the new code path, which only reads SparkContext.getOrCreate().hadoopConfiguration. The actual recipes that work are submit‑time --conf spark.hadoop.fs.*=… (which Spark copies into SparkContext.hadoopConfiguration at boot) or runtime sparkContext.hadoopConfiguration.set("fs.*", ...). spark.conf.set("spark.hadoop.fs.*", ...) set at runtime writes to SQLConf only and is invisible to the new code.

The paragraph starting "Capture fs.azure.* …" is duplicated at line 137 — looks like a merge artefact. Please collapse into one paragraph and rewrite the recipe.

LuciferYang · 2026-06-30T15:26:04Z

+
+  test("genPartitions embeds fs.s3a.* keys from Hadoop conf into GlutenPartition.fsConf") {
+    withHadoopConf(
+      "fs.s3a.access.key" -> "AKIAIOSFODNN7EXAMPLE",


AKIAIOSFODNN7EXAMPLE is AWS's documentation example and matches the AKIA/ASIA pattern used by GitHub secret scanning, gitleaks, and most fork‑level scanners. Even as a test fixture, this routinely fails CI in downstream forks and surfaces as false‑positive credential alerts. Use clearly synthetic placeholders, e.g. "dummy-access-key" / "dummy-secret-value".

LuciferYang · 2026-06-30T15:26:36Z

+    // only reliable transport mechanism.
+    val fsPrefixes = Seq("fs.azure.", "fs.s3a.", "fs.gs.")
+    // scalastyle:off hadoopconfiguration
+    val baseHadoopConf = org.apache.spark.SparkContext.getOrCreate().hadoopConfiguration


org.apache.spark.SparkContext.getOrCreate() is spelled out with full FQN; SparkContext is already imported and used by short name elsewhere in this file. Move the short name in and drop the inline FQN.

On the other hand, The simplified capture reads SparkContext.getOrCreate().hadoopConfiguration directly, which only contains keys present at SparkContext construction (from SparkConf spark.hadoop.*) plus anything mutated via sparkContext.hadoopConfiguration.set(...). It does not see per‑session overrides made via spark.conf.set("spark.hadoop.fs.*", ...) at runtime, nor session‑scoped DataFrameReader.option(...) keys — both of which live on SparkSession.sessionState.newHadoopConf(). If the original use case in issue #10113 includes runtime session‑level credential injection, the current code path won't satisfy it. Either document the supported recipes explicitly (boot‑time --conf spark.hadoop.fs.* or runtime sparkContext.hadoopConfiguration.set) or read from leaves.head.sqlContext.sessionState.newHadoopConf() when leaves are available, falling back to SparkContext.hadoopConfiguration only when they aren't.

LuciferYang · 2026-06-30T15:27:10Z

+   * purpose of this test.
+   */
+  private def emptyWsCtx: WholeStageTransformContext =
+    WholeStageTransformContext(PlanBuilder.empty())


emptyWsCtx only works because genPartitions happens to touch only wsCtx.root.toProtobuf.toByteArray, but WholeStageTransformContext has other fields — a future shape change silently breaks the helper. Either refactor genPartitions to take Array[Byte] directly (cleanest), or add a one‑line comment recording the implicit dependency. Non‑blocking.

LuciferYang · 2026-06-30T15:29:05Z

          planByteArray,
-          splitInfos.toArray
+          splitInfos.toArray,
+          fsConf = fsConf


Assigning fsConf = fsConf puts an independent serialised copy of the map on every GlutenPartition — reference sharing is driver‑side only, the wire cost scales with partition count.

LuciferYang · 2026-06-30T15:33:33Z

+    // task local properties), this is the only path these credentials can take to
+    // reach the native session config and ultimately the Velox ABFS connector.
+    val partitionFsConf = inputPartition.asInstanceOf[GlutenPartition].fsConf
+    val extraConf = (partitionFsConf +


CWE‑532: This line is where fsConf reaches extraConf and ultimately flows into the native session config printed by printConfig (cpp side). printConfig redacts on key match against spark.redaction.regex, default (?i)secret|password|token|access[.]?key. That default misses fs.azure.account.key.* (the Azure storage account key — the primary credential), fs.s3a.encryption.key and the deprecated alias fs.s3a.server-side-encryption.key, fs.gs.auth.service.account.private.key, and fs.gs.auth.service.account.json.keyfile. With spark.gluten.sql.debug=true these reach executor stdout in cleartext. The fix is on the C++ side: please hard‑redact the credential‑bearing prefixes inside printConfig in addition to applying the operator regex — minimum list: fs.azure.account.key., fs.azure.sas., fs.azure.account.oauth2., fs.s3a.access.key, fs.s3a.secret.key, fs.s3a.session.token, fs.s3a.encryption.key, fs.s3a.server-side-encryption.key, fs.gs.auth.service.account.private.key, fs.gs.auth.service.account.json.keyfile. Consider also extending Gluten's default spark.redaction.regex to cover them.

Thanks, it seems pass these configurations as a parameter will introduce the credential leakage. Let me try to use the other more secure approach

[VL] Passing hadoop related configurations to native

af6b23b

Signed-off-by: Yuan <yuanzhou@apache.org>

Copilot AI review requested due to automatic review settings June 25, 2026 13:14

github-actions Bot added CORE works for Gluten Core VELOX labels Jun 25, 2026

Copilot started reviewing on behalf of zhouyuan June 25, 2026 13:15 View session

add test

1d6f087

Signed-off-by: Yuan <yuanzhou@apache.org>

zhouyuan force-pushed the wip_pass_fs_options branch from e37d128 to 1d6f087 Compare June 25, 2026 13:19

Copilot AI reviewed Jun 25, 2026

View reviewed changes

fix

6595e1f

Signed-off-by: Yuan <yuanzhou@apache.org>

Copilot AI review requested due to automatic review settings June 25, 2026 15:16

Copilot started reviewing on behalf of zhouyuan June 25, 2026 15:17 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

fix

9b41c7f

Signed-off-by: Yuan <yuanzhou@apache.org>

zhouyuan force-pushed the wip_pass_fs_options branch from 2220efc to 9b41c7f Compare June 25, 2026 17:21

Copilot AI review requested due to automatic review settings June 25, 2026 17:21

Copilot started reviewing on behalf of zhouyuan June 25, 2026 17:29 View session

Copilot AI reviewed Jun 25, 2026

View reviewed changes

Potential fix for pull request finding

3e5abcd

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings June 26, 2026 14:17

Copilot started reviewing on behalf of zhouyuan June 26, 2026 14:17 View session

Copilot AI reviewed Jun 26, 2026

View reviewed changes

zhouyuan requested a review from marin-ma June 26, 2026 15:07

marin-ma approved these changes Jun 26, 2026

View reviewed changes

remove crednetial log

05661be

Signed-off-by: Yuan <yuanzhou@apache.org>

Copilot AI review requested due to automatic review settings June 29, 2026 09:37

zhouyuan force-pushed the wip_pass_fs_options branch from 195d39b to 05661be Compare June 29, 2026 09:37

Copilot started reviewing on behalf of zhouyuan June 29, 2026 09:38 View session

Copilot AI reviewed Jun 29, 2026

View reviewed changes

jackylee-ch reviewed Jun 29, 2026

View reviewed changes

improve on hadoop config gen

6f59d14

Signed-off-by: Yuan <yuanzhou@apache.org>

jackylee-ch reviewed Jun 30, 2026

View reviewed changes

jackylee-ch approved these changes Jun 30, 2026

View reviewed changes

LuciferYang suggested changes Jun 30, 2026

View reviewed changes

zhouyuan marked this pull request as draft July 3, 2026 12:55

Uh oh!

Conversation

zhouyuan commented Jun 25, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

github-actions Bot commented Jun 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 29, 2026

Uh oh!

github-actions Bot commented Jun 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jackylee-ch left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhouyuan commented Jun 25, 2026 •

edited by github-actions Bot

Loading