-
Notifications
You must be signed in to change notification settings - Fork 595
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Using messageKeyExpression=payload.field
with StreamBridge
#2213
Comments
Hi, Thanks for this report. We will look into this soon to find out if this is a Option 1: use native encoding to true
Option 2: Use headers instead of payload as the message key expression
Then,
|
Generally it is a vey bad practice to use In your case things are even simpler since you are fully in control of Message creation. All you need to do is:
and have expression
|
When using Kafka, make sure to set the header as a |
Closing due to no activity. |
this is something we have a repro on that I can share privately. It occurs in the streambridge layers before the message provider itself. @sabbyanandan don't believe this should be closed - it's only 21 days as well. what is the age of stale issues that get closed normally? |
@sabbyanandan: I thought @sobychacko and team are looking into it. Shouldn't we just keep it open? Otherwise, the team should at least state it clearly in the official docs that
@olegz: I don't think this is a good analogy because the process of extracting and preparing the message key is within the publisher boundary. It is producer of the payload itself, so it isn't quite accurate considering it a postman. |
I'm currently using a workaround by setting Message<MyPayload> message = MessageBuilder.withPayload(payload)
.setHeader(KafkaHeaders.MESSAGE_KEY, payload.getField())
.build(); It works, but it doesn't provide (1) flexibility from configuration-based approach, (2) ability to view all keys in one glance from configuration. I believe these are the same reasons why Would really appreciate if someone can look into the issue closer and provide a fix. |
On side note, Thus I don't see any strong reason why |
what we are seeing is if we have two different bindings is that under pressure within the StreamBridge.send(binding, data/message) - that it sometimes uses the SPel expression for the OTHER binding, and vica versa -- so it seems as if there crossing of what SPel to use for that binding -- again under "pressure" . So, And since the data format is different the This was working with EventHubs -- but again we debugged and saw this issue occur |
this is dated and closed issue. Please feel free to raise a new one if you still believe there are issues that can not be addressed with explanation provided in the threads above |
well, that was the point of my initial response, why was it closed so fast and it's not closed as the error still exists. A workaround isn't a correction. |
Hi, folks. Most Spring projects typically wait for 2 weeks for a community/requester's response, and they are automatically closed when there's no activity. We don't use that Bot in Spring Cloud Stream, however, so that's why we manually closed it. Besides that, in my naive review, it appeared we had shared a few solutions, so it wasn't apparent if that was sufficient or not. If someone can share a concise description of the issue and a reproducible use case sample, we will be happy to review it. |
@sabbyanandan The move to functional (from annotations) also (originally) broke There was an earlier attempt to fix it here 74aee81
I don't see how there can be crosstalk between bindings or even concurrent sends on the same binding - the expression is evaluated in a stateless I am sure it will be much more difficult for kafka-specific expression evaluation to be implemented in the functional model, because spring-cloud-function is far away from the kafka binder and knows nothing about its internals. The problem is the message is converted/built well before it reaches the Kafka binder. For the partition key, he was able to handle it within Spring Cloud Stream, in the Perhaps some kind of pluggable interceptor mechanism could be considered, where the binder provides the interceptor and the core project applies it. However, it is not going to be simple because it would have to retain existing functionality for users that are using native encoding. In the meantime, @sobychacko 's workarounds above are available #2213 (comment) |
@garyrussell I can provide a repro privately if that works. While it's not a substantial chunk of code, its proprietary. I can even take the time with you to demonstrate the crosstalk situation on a Teams call, Google chat, Zoom, etc. Yes I agree I couldn't see how this was possible, but putting conditional breakpoints showed the crossing, and walking up the stack to the external callers the crossing wasn't there. |
I still don't understand especially in the context of a StreamBridge where you have full control of the Message, why can't you inject a specific header and have your SpEL expression to be header-based (not payload-based). Even if you have to duplicate some value from the payload. |
@olegz I agree and that's the change we made. However, there is an issue in the existing codebase that either should be fixed, documented that it doesn't work in some situation, or removed. Do you want users of the platform to go down the same path blindly thinking that spel will work and there is known issues with under pressure issues? Thereby wasting their time, and more time on GitHub issues as they post questions, bug, issues again? I think it would be beneficial to first repro the situation like I offered to @garyrussell -- and then take either a communication plan or a bug fix so as not to drain users of the platform of their time, or perhaps introduce problems in production code that goes unnoticed, essentially lowering the trust of Spring. |
We already have it documented, but it appears that it needs to be further documented in the context of SpEL. But obviously i do agree that some more documentation is missing on our end as users still making this mistake |
There is no documentation that says you shouldn't use two different bindings that have different SpeL expressions within the same process. Not sure how using the classes (StreamBridge) as documented is a mistake if is documented. Perhaps the feature as documented should be retracted if it's not working nor intended to work in all situations. But with many things, often people feel they've made a mistake but they were wrong. Again, just trying to help the Spring Community. @garyrussell let me know if there's any follow up or not. |
So, i'll label this as documentation issue. |
And we should probably deprecate all support for payload-based evaluation (routing, SpEL, etc) |
that's something I'm 100% on board with. |
Regardless of this issue regarding setting the key, if there is crosstalk possible between bindings with different An MCRE would be much appreciated. We should probably open a new issue for that, though. |
I configured
messageKeyExpression=payload.field
and sent a POJO payload usingStreamBridge
.Got an error at
KafkaExpressionEvaluatingInterceptor
because the payload has already been converted tobyte[]
.The conversion seems to happen here at
StreamBridge
beforeKafkaExpressionEvaluatingInterceptor
is invoked, despite the documentation says otherwise.Tried with both Spring Cloud
Hoxton.SR12
and2020.0.3
but no luck. Is there anything misconfigured?The text was updated successfully, but these errors were encountered: