KAFKA-14499: [1/N] Introduce OffsetCommit API version 9 and add new StaleMemberEpochException error

[dajac]

This patch does a few things:

1. It introduces version 9 of the OffsetCommit API. This new version has no schema changes but it can return a StaleMemberEpochException if the new consumer group protocol is used. Note the use of "latestVersionUnstable": true in the request schema. This means that this new version is not available yet unless activated.
2. It renames the generationId field in the request to GenerationIdOrMemberEpoch. This is backward compatible change.
3. It introduces the new StaleMemberEpochException error.
4. It does a minor refactoring in OffsetCommitRequest class.

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[jolshan]

Note the use of "latestVersionUnstable": true in the request schema. This means that this new version is not available yet unless activated.

This also leaves us room to add the topic ID to this version if we so choose?

[dajac]

Note the use of "latestVersionUnstable": true in the request schema. This means that this new version is not available yet unless activated.

This also leaves us room to add the topic ID to this version if we so choose?

Yeah, that's right.

[jolshan]

Something strange is going on with AuthorizerIntegrationTest in the build, but that might be unrelated. I will look into that.

[jolshan]

Generally looks good. Left one small question, but it's mostly my curiosity and nonblocking. Let's confirm the test failures are unrelated.

[dajac]

Something strange is going on with AuthorizerIntegrationTest in the build, but that might be unrelated. I will look into that.

@jolshan I found the issue related to AuthorizerIntegrationTest failures. I just pushed a fix. See last commit.

[jolshan]

Thanks for the fix @dajac let's just confirm the build and we should be good :)

[dajac]

@jolshan I was actually thinking about the AuthorizerIntegrationTest failures overnight and I found an issue with the latestVersionUnstable flag. Let me try to explain.

The latestVersionUnstable is used on the broker side to ensure that an unreleased/unstable version is not exposed by the broker. That's fine. However, it does not guarantee that a client having an unreleased/unstable version is not going to use it.

Let's take this change as an example. The version 9 will be shipped in the next release even if we don't want to use it because the schema is there. So the client knows about it and may use it if the broker eventually supports the version. The issue is that the release version my be different from the one shipped so the client would get an error.

I have updated the AbstractRequest.Builder to be defensive and only consider stable versions. If the user wants to construct a request for an unstable version, it has to specify it explicitly. This guarantee that even if an unstable version is shipped, it is never used.

[jolshan]

@jolshan I was actually thinking about the AuthorizerIntegrationTest failures overnight and I found an issue with the latestVersionUnstable flag. Let me try to explain.

I was curious if the unstable version flag was causing issues since I recall some weirdness in tests when I had an unstable version. Makes sense to require the unstable-ness to be explicit, but I will take a second look.

[jolshan]

Looking at the tests Build / JDK 20 and Scala 2.13 / kafka.server.FetchRequestTest.testCurrentEpochValidationV12() is a bit strange but it only failed on that version. Everything else seems to be familiar-ish flakes

[jolshan]

thanks for making the unstable versions explicit in the builder now. Changes lgtm

[dajac]

Failed tests are unrelated. Merging to trunk.