Pass BaseConsumer to ConsumerContext::rebalance

[loewenheim]

Description of the change

This changes the interface of ConsumerContext in the following ways:

• The &NativeClient parameter in rebalance is replaced with a &BaseConsumer<Self> parameter.
• pre_rebalance and post_rebalance also gain &BaseConsumer<Self> parameters.

This also requires

• the addition of a BaseConsumer::native_client method;
• the addition of a Sized bound to ConsumerContext.

Reason for the change

We're building an abstraction around Kafka at Sentry. It's set up in such a way that offsets are committed in batches, not individually, for performance reasons. This means, however, that when a rebalance happens, there may still be outstanding offsets that need to be committed. We want to do this by means of the pre_rebalance hook on ConsumerContext, but the problem is that this callback has no access to the BaseConsumer (or any client that could perform the commit). We previously tried holding an Arc to the BaseConsumer in the ConsumerContext, but that leads to a mire of reference cycles and deadlocks.

Passing the BaseConsumer to the callbacks would neatly enable this use case for us.

[Swatinem]

This change is a lot simpler than I imagined

[Swatinem]

This fixes #627 btw

[scanterog]

@loewenheim @Swatinem thanks a lot for working on this. We're currently investigating an issue on the latest release (as reported on #638). Once we manage to identify/solve it, we'll tackle this. Sorry for the delay.

[mindreader]

When this lands, I will be the happiest person on earth that uses this library.

[scanterog]

The PR looks good to me 👍 . I tried to find a way to pass in the configured consumer instead of the base one but I couldn't find an easy path. cc @duarten @davidblewett in case you wanna give it another view before merging. A common ask in the future could be: why are we receiving the base consumer instead of my configured one in the rebalance cb?

@loewenheim can you update the CustomContext for the simple_consumer.rs example?

[loewenheim]

@scanterog Done, thanks for the heads-up.

[davidblewett]

@loewenheim thanks for the contribution. I would like to get #644 merged with a point release cut, then I will merge this and do a version bump.

[untitaker]

@davidblewett do you know when you will be able to release a new version with this change? thank you!