[SPARK-55097][SQL] Fix re-adding cached artifacts drops blocks silently issue #53852

pranavdev022 · 2026-01-19T10:42:57Z

What changes were proposed in this pull request?

After the introduction of the ref-counting logic for cloning sessions #52651, whenever an identical cached artifact (same session, same hash) is re-added, it incorrectly leds to deletion of the existing block.

Verified this bug locally using:

test("re-adding the same cache artifact should not remove the block") {
    val blockManager = spark.sparkContext.env.blockManager
    val remotePath = Paths.get("cache/duplicate_hash")
    val blockId = CacheId(spark.sessionUUID, "duplicate_hash")
    try {
      // First addition
      withTempPath { path =>
        Files.write(path.toPath, "test".getBytes(StandardCharsets.UTF_8))
        artifactManager.addArtifact(remotePath, path.toPath, None)
      }
      assert(blockManager.getLocalBytes(blockId).isDefined)
      blockManager.releaseLock(blockId)

      // Second addition with same hash - block should still exist
      withTempPath { path =>
        Files.write(path.toPath, "test".getBytes(StandardCharsets.UTF_8))
        artifactManager.addArtifact(remotePath, path.toPath, None)
      }
      assert(blockManager.getLocalBytes(blockId).isDefined,
        "Block should still exist after re-adding the same cache artifact")
    } finally {
      blockManager.releaseLock(blockId)
      blockManager.removeCache(spark.sessionUUID)
    }
  }

which fails assert(blockManager.getLocalBytes(blockId).isDefined check after the second addition with the same hash.

Proposed solution:

Add early exit check: if existingBlock.id == blockId, skip the addition since the block already exists.
The fix preserves the intended replacement behavior for cloned sessions (different session UUID, same hash)

Why are the changes needed?

When the same cached artifact is added twice to the same session:

BlockManager.save() detects the block exists and returns without re-adding
hashToCachedIdMap.put() returns the existing RefCountedCacheId
release() decrements ref count to 0 and deletes the block

Does this PR introduce any user-facing change?

No. This is an internal bug fix. Previously, duplicate artifact additions could silently delete cached data causing runtime errors.

How was this patch tested?

Added unit test cache artifact deduplication and replacement across sessions in ArtifactManagerSuite that validates:

Duplicate addition in same session is skipped (block survives)
Cloned session can replace inherited artifacts (different CacheId)
Reference counting works correctly across session cleanup

Was this patch authored or co-authored using generative AI tooling?

Co-authored using cursor.

github-actions · 2026-01-19T11:04:42Z

JIRA Issue Information

=== Bug SPARK-55097 ===
Summary: Re-adding cached local relations using ref-counting drops blocks silently
Assignee: None
Status: Open
Affected: ["4.1.0","4.1.1"]

This comment was automatically generated by GitHub Actions

hvanhovell · 2026-01-20T12:30:27Z

sql/core/src/main/scala/org/apache/spark/sql/artifact/ArtifactManager.scala

+            tellMaster = false)
+          updater.save()
+          val oldBlock = hashToCachedIdMap.put(blockId.hash, new RefCountedCacheId(blockId))
+          if (oldBlock != null) {


oldBlock is the same as existingBlock right? It might be better to use existingBlock...

yes, fixed.

hvanhovell · 2026-01-20T12:33:09Z

sql/core/src/main/scala/org/apache/spark/sql/artifact/ArtifactManager.scala

+            oldBlock.release(blockManager)
+          }
+        } else {
+          logDebug(s"Cache artifact with hash $hash already exists in this session, skipping.")


Please elevate this to Warning. It means that the client is sort of misbehaving...

hvanhovell · 2026-01-20T12:34:33Z

sql/core/src/test/scala/org/apache/spark/sql/artifact/ArtifactManagerSuite.scala


+  def isBlockRegistered(id: CacheId): Boolean = {
+      sparkContext.env.blockManager.getStatus(id).isDefined
+    }


NIT messed up spacing.

hvanhovell · 2026-01-20T12:35:04Z

sql/core/src/test/scala/org/apache/spark/sql/artifact/ArtifactManagerSuite.scala

    super.afterEach()
  }

+  def isBlockRegistered(id: CacheId): Boolean = {


Make this private?

hvanhovell · 2026-01-20T12:35:16Z

sql/core/src/test/scala/org/apache/spark/sql/artifact/ArtifactManagerSuite.scala

+      sparkContext.env.blockManager.getStatus(id).isDefined
+    }
+
+    def addCachedArtifact(session: SparkSession, name: String, data: String): CacheId = {


Make this private.

hvanhovell

LGTM

…ly issue ### What changes were proposed in this pull request? After the introduction of the ref-counting logic for cloning sessions #52651, whenever an identical cached artifact (same session, same hash) is re-added, it incorrectly leds to deletion of the existing block. Verified this bug locally using: ``` test("re-adding the same cache artifact should not remove the block") { val blockManager = spark.sparkContext.env.blockManager val remotePath = Paths.get("cache/duplicate_hash") val blockId = CacheId(spark.sessionUUID, "duplicate_hash") try { // First addition withTempPath { path => Files.write(path.toPath, "test".getBytes(StandardCharsets.UTF_8)) artifactManager.addArtifact(remotePath, path.toPath, None) } assert(blockManager.getLocalBytes(blockId).isDefined) blockManager.releaseLock(blockId) // Second addition with same hash - block should still exist withTempPath { path => Files.write(path.toPath, "test".getBytes(StandardCharsets.UTF_8)) artifactManager.addArtifact(remotePath, path.toPath, None) } assert(blockManager.getLocalBytes(blockId).isDefined, "Block should still exist after re-adding the same cache artifact") } finally { blockManager.releaseLock(blockId) blockManager.removeCache(spark.sessionUUID) } } ``` which fails `assert(blockManager.getLocalBytes(blockId).isDefined` check after the second addition with the same hash. Proposed solution: 1. Add early exit check: if `existingBlock.id == blockId`, skip the addition since the block already exists. 2. The fix preserves the intended replacement behavior for cloned sessions (different session UUID, same hash) ### Why are the changes needed? When the same cached artifact is added twice to the same session: - BlockManager.save() detects the block exists and returns without re-adding - hashToCachedIdMap.put() returns the existing RefCountedCacheId - release() decrements ref count to 0 and deletes the block ### Does this PR introduce _any_ user-facing change? No. This is an internal bug fix. Previously, duplicate artifact additions could silently delete cached data causing runtime errors. ### How was this patch tested? Added unit test `cache artifact deduplication and replacement across sessions` in ArtifactManagerSuite that validates: - Duplicate addition in same session is skipped (block survives) - Cloned session can replace inherited artifacts (different CacheId) - Reference counting works correctly across session cleanup ### Was this patch authored or co-authored using generative AI tooling? Co-authored using cursor. Closes #53852 from pranavdev022/fix-artifacts-duplicate-add-issue. Authored-by: pranavdev022 <pranavdev022@gmail.com> Signed-off-by: Herman van Hövell <herman@databricks.com> (cherry picked from commit dfea036) Signed-off-by: Herman van Hövell <herman@databricks.com>

hvanhovell · 2026-01-20T17:23:43Z

Merged to master/4.1

github-actions bot added the SQL label Jan 19, 2026

pranavdev022 force-pushed the fix-artifacts-duplicate-add-issue branch from 279991e to 27e1369 Compare January 19, 2026 15:06

fix re-adding cached artifacts issue

ee93b21

pranavdev022 force-pushed the fix-artifacts-duplicate-add-issue branch from 27e1369 to ee93b21 Compare January 19, 2026 19:06

pranavdev022 changed the title ~~[SQL] Fix re-adding cached artifacts drops blocks silently issue~~ [SPARK-55097][SQL] Fix re-adding cached artifacts drops blocks silently issue Jan 20, 2026

hvanhovell reviewed Jan 20, 2026

View reviewed changes

feedbacks

d599c55

hvanhovell approved these changes Jan 20, 2026

View reviewed changes

asf-gitbox-commits closed this in dfea036 Jan 20, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-55097][SQL] Fix re-adding cached artifacts drops blocks silently issue #53852

[SPARK-55097][SQL] Fix re-adding cached artifacts drops blocks silently issue #53852

pranavdev022 commented Jan 19, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Jan 19, 2026 •

edited

Loading

Uh oh!

hvanhovell Jan 20, 2026

Uh oh!

pranavdev022 Jan 20, 2026

Uh oh!

hvanhovell Jan 20, 2026

Uh oh!

hvanhovell Jan 20, 2026

Uh oh!

hvanhovell Jan 20, 2026

Uh oh!

hvanhovell Jan 20, 2026

Uh oh!

hvanhovell left a comment

Uh oh!

hvanhovell commented Jan 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[SPARK-55097][SQL] Fix re-adding cached artifacts drops blocks silently issue #53852

[SPARK-55097][SQL] Fix re-adding cached artifacts drops blocks silently issue #53852

Conversation

pranavdev022 commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions bot commented Jan 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

JIRA Issue Information

Uh oh!

hvanhovell Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

pranavdev022 Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

hvanhovell Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

hvanhovell Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

hvanhovell Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

hvanhovell Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

hvanhovell left a comment

Choose a reason for hiding this comment

Uh oh!

hvanhovell commented Jan 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pranavdev022 commented Jan 19, 2026 •

edited

Loading

github-actions bot commented Jan 19, 2026 •

edited

Loading