New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: message.max.bytes should default to 1048576 not 1 MB #2804
Conversation
@puellanivis thanks for pointing out this discrepancy with current versions of Kafka. However, it is slightly more nuanced as this value has had different default overs the years. The Java producer since v0.8.2 has always had max.request.size of 1048576 (the base 2 version of 1 MB (MiB)) Up to Kafka v0.8.2 message.max.bytes was 1000000 server side |
Yeah, Iβm totally sure that itβs been through a lot of different values. π° Just thinking matching whatever is current is useful. I suppose in our specific use case, we should probably set it to some value greater than whatever anyone would pick as a default. That way we can avoid βchasing a default valueβ. |
I think it probably makes sense for us to match the Java producer, which uses 1048576 |
I suppose I need to rebase and add sign offs into the commit anywaysβ¦ actually, Iβd probably be faster to just open a new PR with a signed commit message. π |
Saram should match the max.request.size default from the Java producer config. https://kafka.apache.org/documentation/#producerconfigs_max.request.size Note that the server has its own limit on the record batch size which defaults to slightly higher than this, but is configurable and can also be set per topic too. https://kafka.apache.org/documentation/#brokerconfigs_message.max.bytes Co-authord-by: Cassondra Foesch <puellanivis@gmail.com> Signed-off-by: Dominic Evans <dominic.evans@uk.ibm.com>
No worries, I've done the rebase and updated the value to 1024*1024 as discussed |
Is there anything blocking now? |
No further holdups, I was just waiting for you to reply with a thumbsup before I went ahead and merged π |
π I guess in this case, I could have thumbs upped in a full comment rather than just as a reaction. |
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [github.com/IBM/sarama](https://togithub.com/IBM/sarama) | `v1.43.0` -> `v1.43.1` | [![age](https://developer.mend.io/api/mc/badges/age/go/github.com%2fIBM%2fsarama/v1.43.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/go/github.com%2fIBM%2fsarama/v1.43.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/go/github.com%2fIBM%2fsarama/v1.43.0/v1.43.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/go/github.com%2fIBM%2fsarama/v1.43.0/v1.43.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>IBM/sarama (github.com/IBM/sarama)</summary> ### [`v1.43.1`](https://togithub.com/IBM/sarama/releases/tag/v1.43.1): Version 1.43.1 (2024-03-27) [Compare Source](https://togithub.com/IBM/sarama/compare/v1.43.0...v1.43.1) <!-- Release notes generated using configuration in .github/release.yaml at main --> #### What's Changed ##### π Fixes - fix: message.max.bytes should default to [`1048576`](https://togithub.com/IBM/sarama/commit/1048576) not 1 MB by [@​puellanivis](https://togithub.com/puellanivis) in [IBM/sarama#2804 - fix: add locking around broker throttle timer to prevent race condition by [@​chengsha](https://togithub.com/chengsha) in [IBM/sarama#2826 ##### π¦ Dependency updates - chore(deps): bump go.opentelemetry.io/otel/sdk from 1.23.1 to 1.24.0 in /examples/interceptors by [@​dependabot](https://togithub.com/dependabot) in [IBM/sarama#2816 - chore(deps): bump the golang-org-x group with 1 update by [@​dependabot](https://togithub.com/dependabot) in [IBM/sarama#2825 - chore(deps): bump github.com/stretchr/testify from 1.8.4 to 1.9.0 by [@​dependabot](https://togithub.com/dependabot) in [IBM/sarama#2822 - chore(deps): bump go.opentelemetry.io/otel/exporters/stdout/stdoutmetric from 1.23.1 to 1.24.0 in /examples/interceptors by [@​dependabot](https://togithub.com/dependabot) in [IBM/sarama#2815 #### New Contributors - [@​chengsha](https://togithub.com/chengsha) made their first contribution in [IBM/sarama#2826 **Full Changelog**: IBM/sarama@v1.43.0...v1.43.1 </details> --- ### Configuration π **Schedule**: Branch creation - "on tuesday" (UTC), Automerge - At any time (no schedule defined). π¦ **Automerge**: Disabled by config. Please merge this manually once you are satisfied. β» **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. π **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/open-telemetry/opentelemetry-collector-contrib). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4yNjkuMiIsInVwZGF0ZWRJblZlciI6IjM3LjI2OS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9--> --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: opentelemetrybot <107717825+opentelemetrybot@users.noreply.github.com>
β¦etry#32091) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | [github.com/IBM/sarama](https://togithub.com/IBM/sarama) | `v1.43.0` -> `v1.43.1` | [![age](https://developer.mend.io/api/mc/badges/age/go/github.com%2fIBM%2fsarama/v1.43.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/go/github.com%2fIBM%2fsarama/v1.43.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/go/github.com%2fIBM%2fsarama/v1.43.0/v1.43.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/go/github.com%2fIBM%2fsarama/v1.43.0/v1.43.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>IBM/sarama (github.com/IBM/sarama)</summary> ### [`v1.43.1`](https://togithub.com/IBM/sarama/releases/tag/v1.43.1): Version 1.43.1 (2024-03-27) [Compare Source](https://togithub.com/IBM/sarama/compare/v1.43.0...v1.43.1) <!-- Release notes generated using configuration in .github/release.yaml at main --> #### What's Changed ##### π Fixes - fix: message.max.bytes should default to [`1048576`](https://togithub.com/IBM/sarama/commit/1048576) not 1 MB by [@&open-telemetry#8203;puellanivis](https://togithub.com/puellanivis) in [IBM/sarama#2804 - fix: add locking around broker throttle timer to prevent race condition by [@&open-telemetry#8203;chengsha](https://togithub.com/chengsha) in [IBM/sarama#2826 ##### π¦ Dependency updates - chore(deps): bump go.opentelemetry.io/otel/sdk from 1.23.1 to 1.24.0 in /examples/interceptors by [@&open-telemetry#8203;dependabot](https://togithub.com/dependabot) in [IBM/sarama#2816 - chore(deps): bump the golang-org-x group with 1 update by [@&open-telemetry#8203;dependabot](https://togithub.com/dependabot) in [IBM/sarama#2825 - chore(deps): bump github.com/stretchr/testify from 1.8.4 to 1.9.0 by [@&open-telemetry#8203;dependabot](https://togithub.com/dependabot) in [IBM/sarama#2822 - chore(deps): bump go.opentelemetry.io/otel/exporters/stdout/stdoutmetric from 1.23.1 to 1.24.0 in /examples/interceptors by [@&open-telemetry#8203;dependabot](https://togithub.com/dependabot) in [IBM/sarama#2815 #### New Contributors - [@&open-telemetry#8203;chengsha](https://togithub.com/chengsha) made their first contribution in [IBM/sarama#2826 **Full Changelog**: IBM/sarama@v1.43.0...v1.43.1 </details> --- ### Configuration π **Schedule**: Branch creation - "on tuesday" (UTC), Automerge - At any time (no schedule defined). π¦ **Automerge**: Disabled by config. Please merge this manually once you are satisfied. β» **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. π **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/open-telemetry/opentelemetry-collector-contrib). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4yNjkuMiIsInVwZGF0ZWRJblZlciI6IjM3LjI2OS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9--> --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: opentelemetrybot <107717825+opentelemetrybot@users.noreply.github.com>
https://kafka.apache.org/documentation/#brokerconfigs_message.max.bytes
Says this default value should be
1048588
, not 1 megabyte (and also weirdly not 1 Mebibyte π€·ββοΈ ) matching these expected values is important, as one language could publish an event that another language would then refuse to republish.The produced error message
ConfigurationError(fmt.Sprintf("Attempt to produce message larger than configured Producer.MaxMessageBytes: %d > %d", size, p.conf.Producer.MaxMessageBytes)
is also poorly designed to allow for event handling of that message. It would require anerrors.As
followed by astrings.HasPrefix(β¦)
which is really not what Go recommends to handle errors (that is, direct string manipulation, which is inherently fragile as the error string should be easily changeable.) This change doesnβt really address this issue, but I wanted to mention it.