Fix tests in spark engine and kyuubi server modules with Spark 4.0

[pan3793]

🔍 Description

This PR fixes tests in spark engine and kyuubi server modules with Spark 4.0.

Types of changes 🔖

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪

Since Spark 4.0.0-preview1 is still under voting phase, this PR does not add CI, the change was tested in #6407 with Spark 4.0.0-preview1 RC1

---

Checklist 📝

• This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 58.25%. Comparing base (8244167) to head (1019864).

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #6425      +/-   ##
============================================
- Coverage     58.34%   58.25%   -0.09%     
  Complexity       24       24              
============================================
  Files           656      656              
  Lines         40268    40268              
  Branches       5498     5498              
============================================
- Hits          23494    23460      -34     
- Misses        14266    14296      +30     
- Partials       2508     2512       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pan3793]

Thanks, merged to master