[SPARK-44181][BUILD] Upgrade Apache Kafka to 3.4.1

[bjornjorgensen]

What changes were proposed in this pull request?

Upgrade Apache Kafka from 3.4.0 to 3.4.1

Why are the changes needed?

Release Notes

Kafka 3.4.1 fixes 58 issues since the 3.4.0 release.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA

[bjornjorgensen]

@dongjoon-hyun This is not the latest version. There are a 3.5.0 but that did not run as I was hoping for see my first commit.
Can we have this for spark 3.4.2 ?

[dongjoon-hyun]

Thank you for pinging me.

[bjornjorgensen]

This https://github.com/bjornjorgensen/spark/actions/runs/5365767358/jobs/9734794773 is the build log for 3.5.0 any tips on whats wrong her?

[dongjoon-hyun]

+1, LGTM (Pending CIs).

[dongjoon-hyun]

Ur, do you mean this will not pass CI?

[bjornjorgensen]

This 3.4.1 will pass CI but kafka 3.5.0 did not.

[bjornjorgensen]

oh... in 3.5.0 kafka have moved many modules.
[KAFKA-14472] - Move TransactionIndex and related to storage module
[KAFKA-14473] - Move AbstractIndex to storage module
[KAFKA-14474] - Move OffsetIndex to storage module
[KAFKA-14475] - Move TimeIndex/LazyIndex to storage module
[KAFKA-14476] - Move OffsetMap to storage module
[KAFKA-14477] - Move LogValidator and related to storage module
[KAFKA-14478] - Move LogConfig/CleanerConfig to storage module
[KAFKA-14480] - Move ProducerStateManager to storage module

https://github.com/bjornjorgensen/spark/actions/runs/5365767358/jobs/9734794665#step:7:2031

[dongjoon-hyun]

Merged to master. Could you make a backporting PR for branch-3.4, @bjornjorgensen ?

[bjornjorgensen]

@dongjoon-hyun Thank you.
Yes, I can try tomorrow.

[dongjoon-hyun]

Thank you!