[CI][Java] Integration jobs with Spark fail with NoSuchMethodError:io.netty.buffer.PooledByteBufAllocator

[raulcd]

Describe the bug, including details regarding any error messages, version, and platform.

It does seem that:
#36211
Updated from PoolThreadCache to PoolArenasCache this has made our nightly integration tests with Spark previous and current development versions to fail.

• test-conda-python-3.10-spark-master
• test-conda-python-3.8-spark-v3.1.2
• test-conda-python-3.9-spark-v3.2.0

The error:

 02:07:30.759 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 0.0 in stage 24.0 (TID 30) (ab33723e6432 executor driver): java.lang.NoSuchMethodError: io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.threadCache()Lio/netty/buffer/PoolArenasCache;

Spark hasn't yet updated to 4.1.94.Final.
I am unsure on how do we fix this but does this mean we break backwards compatibility with previous Spark versions?

Component(s)

Continuous Integration, Java

[raulcd]

@BryanCutler @lidavidm @kiszk FYI
I've also opened a ticket on the JIRA for spark to let them know about the CVE.

[lidavidm]

Possibly we can do something via runtime reflection (though that would have to be carefully benchmarked)

[kiszk]

Thank you for sharing this. Here is a discussion at Spark side.

[BryanCutler]

Could we have our Spark tests use Arrow with shaded dependencies?

[lidavidm]

I think shaded Netty would be the best solution (it would also hopefully unblock downstream Spark). But we don't build such an artifact currently.

Could we force the test to use Arrow with arrow-memory-unsafe added and arrow-memory-netty excluded?

[raulcd]

@lidavidm @BryanCutler is this or should this be a blocker for 13.0.0?

[lidavidm]

I think we should evaluate if shading Netty or using arrow-memory-unsafe in place of arrow-memory-netty works, or else evaluate if something reflection-based might work.

[lidavidm]

Er, so that is to say, yes, let's consider this a blocker.

[lidavidm]

It looks like to shade Netty or use arrow-memory-unsafe, we'd have to modify the Spark pom; I'm not sure if that quite qualifies as a solution for non-HEAD Spark.

[danepitkin]

I sent a note to the Spark ML: https://lists.apache.org/thread/ndmj3ht85j2g40n8clfh92ny6qqbvd09

So far, I think we are leaning towards Spark resolving HEAD on their side and breaking backwards compatibility with older Spark versions.

[raulcd]

Thanks @danepitkin !
If this is the solution we decide to go with I suppose we can try to patch the POM to use the new Netty version (I am testing that on the related PR).
On that case this should not a blocker for the release as we will just "patch" older Spark versions (updating version on POM) to use the newer Netty version? @lidavidm

[lidavidm]

I think Spark will just have to be ignored.

[raulcd]

I never posted the JIRA ticket that was opened on SPARK. Adding it for reference: https://issues.apache.org/jira/projects/SPARK/issues/SPARK-44212

[raulcd]

@lidavidm this can be closed after this one #36928 has been merged, right?

[lidavidm]

I merged that PR - so hopefully this is fixed

[lidavidm]

I ran crossbow on that PR - looks like it does pass now

[raulcd]

Thanks @lidavidm , I am closing it then!