docs(clients): updates the tuning content for producers and consumers

[PaulRMellor]

Documentation

Some doc updates related to producer and consumer configuration (based on the feedback for the blog posts on developing producer and consumer applications)

• adds an overview of the configuration options for securing access to Kafka
• adds the key configuration considerations for producer and consumer performance
• updates to the producer and consumer tuning content

Checklist

Please go through this checklist and make sure all applicable tasks have been done

• Write tests
• Make sure all tests pass
• Update documentation
• Check RBAC rights for Kubernetes / OpenShift roles
• Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
• Reference relevant issue(s) and close them after merging
• Update CHANGELOG.md
• Supply screenshots for visual changes, such as Grafana dashboards

[scholzj]

I left some comments. But TBH, I'm not sure this works for me. It tries to cover too much on very little space and that leads to oversimplification. So I'm not sure fro whom is this is really written. People who know Kafka will know this already. For people who do not know Kafka this does not explain enough and might just confuse them.

[fvaleri]

Hi @PaulRMellor, thanks. I think that, for every tuning recommendation, we should always add the relevant metrics that the user can check to see if the configuration change has the desired effect in their use case. We should also say that it is recommended to change one configuration at a time. You can also try them using the simple example we created for the client blog posts.

[PaulRMellor]

I left some comments. But TBH, I'm not sure this works for me. It tries to cover too much on very little space and that leads to oversimplification. So I'm not sure fro whom is this is really written. People who know Kafka will know this already. For people who do not know Kafka this does not explain enough and might just confuse them.

It's a fair point. I think the original reason for adding the tuning content was to give an idea of the common ways to tune the components to satisfy specific use cases. The idea is that a user might want to know which properties would be suitable for Increasing bandwidth for high latency connections, for example. Our content provides some tips for those kinds of use cases. I imagine that kind of content is searched for a lot. Maybe we could do better on the information we provide. Federico suggests adding metrics info, for example.

[scholzj]

I left some comments. But TBH, I'm not sure this works for me. It tries to cover too much on very little space and that leads to oversimplification. So I'm not sure fro whom is this is really written. People who know Kafka will know this already. For people who do not know Kafka this does not explain enough and might just confuse them.

It's a fair point. I think the original reason for adding the tuning content was to give an idea of the common ways to tune the components to satisfy specific use cases. The idea is that a user might want to know which properties would be suitable for Increasing bandwidth for high latency connections, for example. Our content provides some tips for those kinds of use cases. I imagine that kind of content is searched for a lot. Maybe we could do better on the information we provide. Federico suggests adding metrics info, for example.

TBH, I'm not sure we want to replace the Kafka docs. Maintaining this and testing this for every release would be a lot of load of work. Its not like a blog post where you can say that it was published a few years ago and might not apply anymore. And at least as my personal opinion - I think there is no way you can make this short and simple and yet understandable. You either need to document how Kafka works and how to write the client which is a lot of initial as well as ongoing work and I would argue world will be better of if that is done in Apache Kafka. Or you keep it short by making it arguably too simple and it would be either not helpful or spawn new questions we do not have capacity to answer.

[scholzj]

Discussed on the community call on 30.11.2023: We should think about what do we want to do with this content.

[PaulRMellor]

@mimaison , @showuon - We're currently discussing maintenance of the (broker, producer, consumer) tuning content in the Strimzi documentation and your advice would be useful in shaping any decision.

The main points of discussion are as follows:

1. Maintainability: Do we want to keep maintaining this type of content?
2. Depth: Is the content sufficiently in-depth? If not, is it better to drop the content rather than invest more time in expanding and maintaining it?
3. Inclusion in Kafka Documentation: Shouldn't this type of content be referenced from the Kafka documentation? And if there are gaps in the Kafka documentation, can that be rectified?

Do you think the tuning content is useful to have in the Strimzi documentation, and if so, how can we address the maintainability concerns? Alternatively, do you agree that it might be more appropriate to have such content in the Kafka documentation?

[mimaison]

I think this content is fine. It brings some context around the configurations and their interactions. It stays relatively high level to give users an idea of the main concepts. Diving deeper is tricky as it often depends on specific use cases. The Kafka docs on the other side are much more a reference with just the descriptions of the configurations.

[PaulRMellor]

I think this content is fine. It brings some context around the configurations and their interactions. It stays relatively high level to give users an idea of the main concepts. Diving deeper is tricky as it often depends on specific use cases. The Kafka docs on the other side are much more a reference with just the descriptions of the configurations.

Thanks @mimaison. I appreciate your perspective. Our goal is to provide practical insights into best practices and common configurations. Regarding maintenance, the content has remained relatively stable since its creation. However, if you feel there are areas we're not addressing, please let me know.

[mimaison]

LGTM, thanks!
There's just the comment about the assignment strategy which I'm not sure whether we could make a better recommendation.

[showuon]

LGTM! I also think this doc is helpful. You can check what upstream kafka's doc about important configs:

[scholzj]

Approving after discussion with @PaulRMellor. If anyone has anymore comments, please share them. Otherwise we are going to merge it.

[ppatierno]

Just a nit. LGTM.

[ppatierno]

IIRC it's not going to "reassign" in the sense that it revokes and reassigns, leaving the consumer not consuming until the reassignment happens. IIRC it doesn't revoke the partitions which should stick with that consumer. Just revokes the ones which will be moved to another consumer. @mimaison can you confirma that? If yes, @PaulRMellor could we make it clearer here?

[PaulRMellor]

That's a fair point. I don't think reassign is a good way of wording this. I've updated here: docs(review): edits to doc from review by PP

[scholzj]

You fine with this @ppatierno? Can we merge it?

[ppatierno]

Yes it's fine with me. Thanks.