New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Address content-type compatibility between messages generated by different versions of SCSt apps #1106
Comments
What about Kafka? Does this imply that a 2.0 Kafka consumer has to interoperate with a 1.x producer (and vice-versa)? We made the decision that 2.0 would only support >= Kafka 0.11; it sounds like this won't be possible now and we'll have to maintain support for the 0.10.x binder(s). Of course, transactions and native headers won't be available if you use such a binder. It will also be difficult to to eventually migrate to kafka 0.11 (or 1.0 when available) because it will need a big bang. To support a progressive migration, we'll need some work in the binder to not add native headers (so the 0.11 binder can talk to an 0.10 broker). When all services have migrated to 0.11 binder; they could then change out the broker; they would still then need a "big bang" to enable native headers across all the services. Documenting all this is going to be difficult. Sigh. |
Since we only support Kafka 0.11 and up in 2.0, how about we say that in
the case of Kafka based apps, 1.3 is backward/forward compatible with 2.0,
as long as it uses the latest Kafka broker (since 1.3 can work with 0.11 by
ignoring native headers and transactions). That way, we don't have to bring
support for all the 0.10.x versions of Kafka in 2.0 which is going to be
very challenging.
|
But that's not the whole picture; when using 1.3 with the 0.11 binder it does add headers. If you use 1.3 with a 0.10 binder, it doesn't use headers. 2.0 will not be able to produce messages for a 1.3/0.10 consumer, unless we add code to skip headers and reinstate support for embedded headers. |
Was a design where a 2.0 producer didn't add this legacy field considered? The downside of this approach is dragging the legacy fields along which keeps potential confusion. While somewhat less comprehensive, an approach in which the latest 1.3.x release is made to be forward compatible, by being aware of where the new 2.0 fields/headers are located. I know it is complicated, but just wanted to present another approach in case it wasn't considered. |
@markpollack that can work, just need to wrap my head around it. And it will require a new release of 1.3 pipeline (i.e., 1.3.1).
We obviously should not have problems when both producers, consumers and brokers are all of a same version and since 1.3 dropped support for pre 0.10 Kafka we don't have to worry about older brokers/clients. |
I haven't (yet) found a way for the client to figure out the broker version (only the In any case, even if we can figure out the broker version (say it's 0.11) the producer doesn't know whether the consumer is using the 0.11 client, so we can't make any assumptions based on the broker version anyway. I have issued a PR adding a new I think this covers all the bases. |
The only bases it is not covering is non-SCSt clients (producers/consumers), but as we discussed that level of the awareness may need to be delegated to such clients. Correct? |
Most non-scst peers will be ok.
|
Reopen. |
- Remove binding config property `legacyContentTypeHeaderEnabled` that was introduced to enable legacy content type handling - Enable LegacyContentTypeInterceptor in 2.0 always, but bypass any legacy content type handling if the message received is from a 2.0 producer by checking on the version header - Introdce a new BinderHeader property for version - Fix the LegacyContentType related tests - Remove the check for originalContentType in ReceivingHandler when the payload received is a byte[] - Remove unnecessary deserializePayloadIfNecessary calls in ReceivingHandler - Remove deprecated deserializePayload methods in AbstractBinder and MessageSerializationUtils Partly fixes spring-cloud#1106 Fixes spring-cloud#1110
…ity issues - This PR builds on the previous PR from Soby 6c259be (pr/1112) - Added support in AbstractBinderTests to create bindable channel based on determining channel input type based on it's name (i.e., *input*) - Added LegacyContentTypeHeaderInterceptor to TestSupportBinder to restor the previous behavior of MessageCollector for cases where Message's payload content type is a variant of 'text'. - Restored tests that use MessageCollector to depend on proper payload type - Added 'deserialize' routine back to MessageSerializationUtils - Polished MessageConverterConfigurer.LegacyContentTypeHeaderInterceptor to remove conditional original-content-type header logic - Removed default contentType from BindingProperties
…ity issues Fixes spring-cloud#1106 - This PR builds on the previous PR with commit hash 6c259be... (pr/1112) - Added support in AbstractBinderTests to create bindable channel based on determining channel input type based on it's name (i.e., *input*) - Added LegacyContentTypeHeaderInterceptor to TestSupportBinder to restor the previous behavior of MessageCollector for cases where Message's payload content type is a variant of 'text'. - Restored tests that use MessageCollector to depend on proper payload type - Added 'deserialize' routine back to MessageSerializationUtils - Polished MessageConverterConfigurer.LegacyContentTypeHeaderInterceptor to remove conditional original-content-type header logic - Removed default contentType from BindingProperties - Restored tests that use MessageCollector to depend on proper payload type - polishing - fixed Kryo/Java serialization - Fixed ser/de for "application/json" and contentType equals - Fixed of ser/de of JSON strings to ensure that Strings are not re-quoted - Fixed how we comparing contentTypes - more polishing - Fixed NPE in MessageSerializationUtils - Make bindings and consumer groups in new tests added to AbstractBinderTests mutually exclusive
olegz commentedOct 19, 2017
•
edited
Due to the recent content-type negotiation improvements we unintentionally introduced compatibility issue where pre-2.0 consumers may not be able to properly negotiate the content type of the message generated by 2.0+ producers.
This is due to the fact that pre-2.0 producers and consumers were relying on the
original-content-type
header which was removed in 2.0.Another change that was introduced as part of this effort deals with registering additional flag (
spring.cloud.stream.bindings.input.legacyContentTypeHeaderEnabled
) and interceptor (LegacyContentTypeHeaderInterceptor) to allow 2.0+ consumers to successfully process pre-2.0 messages. This issue is that it makes consumer statically aware of the producer and its version and could result in more issues if such consumer receives messages from the producers of different versions.Proposal:
original-content-type
header. While this header will be ignored by 2.0 consumers it will be used by pre-2.0 consumers as before. It should also generate one additional header such as version (e.g.,version=2.x
). This additional header will be used by 2.0+ consumers to ignore the re-introducedoriginal-content-type
header (more details below).version
header before checking for the existence of theoriginal-content-type
header. If 'version' header is pre-2.0 or is not present (which is what's going to be in the case of pre-2.0 producers) then we know it's a legacy message and content-type negotiation should be done using theoriginal-content-type
header, otherwise such header will be ignored. Basically the version header will play the same role as newly introducedspring.cloud.stream.bindings.input.legacyContentTypeHeaderEnabled
flag, but it will come as part of the message instead of static configuration of the consumer. Further more, making it aversion
instead of a boolean will give us more flexibility in the future when it comes to similar issues especially if we decide to drop support for certain version of messages.The text was updated successfully, but these errors were encountered: