Spark memory configs read with an accessor that does not match their declared unit

[LuciferYang]

Backend

VL (Velox). The code is backend-agnostic and lives in gluten-core and gluten-substrait.

Bug description

Three Spark memory configurations are declared with a unit but read through an accessor that assumes a different one, so the off-heap budgets Gluten derives from them are wrong.

1. spark.executor.memory is declared bytesConf(ByteUnit.MiB), so a value without a suffix means MiB. SparkResourceUtil.getExecutorMemorySize read it with getSizeAsBytes, which reads a suffix-less value as bytes. spark.executor.memory=8192 gave 8192 bytes instead of 8 GiB, and the dynamic off-heap sizing path derived its budget from that.

2. spark.executor.minMemoryOverhead is a size string, MiB unless suffixed. getMemoryOverheadSize read it with conf.getLong, which is String.toLong and cannot parse 512m. On a value Spark itself accepts, that threw NumberFormatException from VeloxListenerApi#onDriverStart and aborted driver startup.

3. A ResourceProfile records executor memory amounts in MiB, while spark.gluten.memory.offHeap.size.in.bytes and spark.gluten.memory.task.offHeap.size.in.bytes are declared bytesConf(ByteUnit.BYTE). GlutenAutoAdjustStageResourceProfile#updateResourceSetting wrote the profile amount verbatim, so a 20 GiB off-heap budget became 20480 bytes.

ByteUnit.MiB.toBytes rejects a negative input but wraps past 2^43 MiB without raising. getMemoryOverheadSize ended in toBytes, so spark.executor.minMemoryOverhead=9000000t produced a negative overhead budget. VeloxListenerApi#onDriverStart writes that into spark.gluten.memory.memoryOverhead.size.in.bytes, and VeloxBackend.cc multiplies it by 0.75 to size the Velox global allocator.

updateResourceSetting computes the task slot count itself rather than calling SparkResourceUtil.getTaskSlots, and the two disagree in local mode: getTaskSlots resolves local[8] to 8 slots, while a profile reports spark.executor.cores, 1 by default. While the total was also 2^20 times too small the two errors cancelled, so the divergence had no visible effect. Correct the total and it becomes an 8x over-provision, with each of 8 concurrent tasks sized as if it owned the whole budget. The same expression has no positivity check and no floor, so a profile with CORES=1 and spark.task.cpus=2 throws ArithmeticException: / by zero on every query.

Gluten version

main (1.8.0-SNAPSHOT)

Spark version

Version-agnostic (applies to spark-3.3 / 3.4 / 3.5 / 4.0 / 4.1).

Spark configurations

The divide-by-zero needs spark.gluten.auto.adjustStageResource.enabled=true, spark.sql.adaptive.enabled=true, spark.memory.offHeap.enabled=true, spark.task.cpus=2, and master local[8].

The unit errors need spark.memory.offHeap.enabled=true plus one of spark.executor.memory=8192, spark.executor.minMemoryOverhead=512m, or spark.memory.offHeap.size=20g.

System information

Not applicable. The affected code reads configuration in gluten-core/src/main/scala/org/apache/spark/util/SparkResourceUtil.scala and gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala, and does not depend on OS or hardware.

Relevant logs

java.lang.NumberFormatException: Illegal value for config key spark.executor.minMemoryOverhead: ... Failed to parse byte string: 512m
  at org.apache.gluten.backendsapi.velox.VeloxListenerApi.onDriverStart(VeloxListenerApi.scala:116)

java.lang.ArithmeticException: / by zero
  at org.apache.spark.sql.execution.GlutenAutoAdjustStageResourceProfile$.updateResourceSetting(GlutenAutoAdjustStageResourceProfile.scala:206)

Fix direction

Read each config with the accessor that matches its declared unit. Put the MiB-to-byte conversion behind one helper in SparkResourceUtil that guards both the sign and the magnitude, and route all three boundaries through it. In updateResourceSetting, defer to SparkResourceUtil.getTaskSlots in local mode, where the profile does not reflect the thread count, and give the other branch the positivity check and one-slot floor that getTaskSlots already has. Read the exact conf bytes when the profile is the unmodified default, since reading through the profile truncates to MiB.