Upgrade spark version and cleanup dependencies

[Jackie-Jiang]

Upgrade spark 2 version from 2.4.6 to 2.4.8
Upgrade spark 3 version from 3.5.0 to 3.5.1

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 61.77%. Comparing base (59551e4) to head (307b84a).
Report is 107 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #12626      +/-   ##
============================================
+ Coverage     61.75%   61.77%   +0.02%     
+ Complexity      207      198       -9     
============================================
  Files          2436     2453      +17     
  Lines        133233   133824     +591     
  Branches      20636    20753     +117     
============================================
+ Hits          82274    82674     +400     
- Misses        44911    45038     +127     
- Partials       6048     6112      +64     

Flag	Coverage Δ
custom-integration1	<0.01% <ø> (-0.01%)	⬇️
integration	<0.01% <ø> (-0.01%)	⬇️
integration1	<0.01% <ø> (-0.01%)	⬇️
integration2	0.00% <ø> (ø)
java-11	61.73% <ø> (+0.02%)	⬆️
java-21	61.66% <ø> (+0.03%)	⬆️
skip-bytebuffers-false	61.75% <ø> (+<0.01%)	⬆️
skip-bytebuffers-true	61.64% <ø> (+33.91%)	⬆️
temurin	61.77% <ø> (+0.02%)	⬆️
unittests	61.77% <ø> (+0.02%)	⬆️
unittests1	46.93% <ø> (+0.04%)	⬆️
unittests2	27.67% <ø> (-0.07%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.