Skip to content

[MINOR][VL] Improve temp parquet directory in FunctionsValidateSuite#10955

Merged
wForget merged 1 commit intoapache:mainfrom
wangyum:FunctionsValidateSuite
Oct 29, 2025
Merged

[MINOR][VL] Improve temp parquet directory in FunctionsValidateSuite#10955
wForget merged 1 commit intoapache:mainfrom
wangyum:FunctionsValidateSuite

Conversation

@wangyum
Copy link
Copy Markdown
Member

@wangyum wangyum commented Oct 28, 2025

What changes are proposed in this pull request?

https://github.com/apache/incubator-gluten/blob/a8f45879d13e193733859bc9174612634df99c58/backends-velox/src/test/scala/org/apache/gluten/functions/FunctionsValidateSuite.scala#L70-L76

Spark writes output to a path. Our internal Spark distribution adds extra validation logic that rejects writes targeting a file path. This change ensures we always provide a directory path, improving compatibility with both upstream and internal Spark builds.

  Cause: org.apache.hadoop.fs.ParentNotDirectoryException: Parent path is not a directory: file:/tmp/8263767318181008243.parquet
  at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:691)
  at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:672)
  at org.apache.hadoop.fs.ChecksumFileSystem.mkdirs(ChecksumFileSystem.java:788)
  at org.apache.spark.sql.InsertOperationLockUtil$.tryLock(InsertOperationLockUtil.scala:111)
  at org.apache.spark.sql.InsertOperationLockUtil$.pickInsertOperationLock(InsertOperationLockUtil.scala:49)
  at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.detectConflict(InsertIntoHadoopFsRelationCommand.scala:389)
...

How was this patch tested?

Existing test.

@github-actions github-actions Bot added the VELOX label Oct 28, 2025
@wangyum wangyum changed the title [MINOR][VL] Improve temp parquet directory naming in FunctionsValidateSuite [MINOR][VL] Improve temp parquet directory in FunctionsValidateSuite Oct 28, 2025
Copy link
Copy Markdown
Member

@wForget wForget left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @wangyum

@wForget wForget merged commit 550ea39 into apache:main Oct 29, 2025
129 of 145 checks passed
@wangyum wangyum deleted the FunctionsValidateSuite branch October 29, 2025 02:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants