INT-1807 Add Mechanism For Headers with TCP #763

Merged
merged 1 commit into from Aug 16, 2013

Projects

None yet

2 participants

@garyrussell
Member

TCP streams have no standard message structure. Therefore, the
TCP implementation previously only transferred the message
payload.

If someone wanted to convey header information, they would have
to write their own wrapper and/or use Java serialization for
the entire message.

This change provides a strategy to allow users to determine
which headers are transferred, and how.

A MessageConvertingMessageMapper is now provided that invokes
any MessageConverter. A MapMessageConverter is provided that
converts the payload, and selected heades to a Map with two
entries ("payload") and ("headers").

A MapJsonSerializer is provided that converts a Map to/from
JSON. Jackson can't delimit multiple objects in a stream
so another serializer is required to encode/decode structure.
A ByteArrayLfSerializer is used by default, inserting a
linefeed between JSON objects.

The combination of these elements now allows header
information to be transferred over TCP. Of course, users
can implment their own (de)serializer to format the
bits on the wire exactly as needed by their application.

INT-1807 Polishing

Add a test that uses a Map MessageConverter with a
Java (de)serializer.

@artembilan
Member

Hi, Gary!
Let me review a bit.
I like TcpMessageMapper with MessageConverter strategy, but as I see MapMessageConverter and MapJsonSerializer are some limited implementations, when we expect on the other side Spring Integration aplication too, with the same configuration, of course, or some JSON-aware reader. Am I right?
In my case for ISO-8583 I convert SI-Message to the ISOMsg as payload and then delegate serialization to the jpos packager for send over TCP and something similar versa.
I mean, that I'm not sure in MapMessageConverter and MapJsonSerializer implementations yet. Maybe the same result we can achieve with ObjectToMapTransformer or ObjectToJsonTransformer, but if you had provided them, you had had some reason... My point - DRY. However, I can live with them.
Further - on the lines.

@artembilan artembilan commented on the diff Apr 3, 2013
...ntegration/support/converter/MapMessageConverter.java
+ this.headerNames = newHeaderNames;
+ }
+
+ /**
+ * By default all headers on Map passed to {@link #toMessage(Object)}
+ * will be mapped. Set this property
+ * to 'true' if you wish to limit the inbound headers to those in
+ * the #headerNames.
+ * @param filterHeadersInToMessage
+ */
+ public void setFilterHeadersInToMessage(boolean filterHeadersInToMessage) {
+ this.filterHeadersInToMessage = filterHeadersInToMessage;
+ }
+
+ public <P> Message<P> toMessage(Object object) {
+ Assert.isInstanceOf(Map.class, object, "This converter expects a Map");
@artembilan
artembilan Apr 3, 2013 Spring member

Maybe add generic type to the MessageConverter? Or it can take one type on toMessage, but produce another one from fromMessage?

@garyrussell
garyrussell Apr 3, 2013 Spring member

Right - we can take any payload type in fromMessage; the resulting payload type here depends on what the deserializer puts in the Map's 'payload' element. So we have to leave it unspecified.

@artembilan artembilan and 1 other commented on an outdated diff Apr 3, 2013
...ntegration/support/converter/MapMessageConverter.java
+ * @since 3.0
+ *
+ */
+public class MapMessageConverter implements MessageConverter {
+
+ private volatile Set<String> headerNames = new HashSet<String>();
+
+ private volatile boolean filterHeadersInToMessage;
+
+ /**
+ * Headers to be converted in {@link #fromMessage(Message)}.
+ * {@link #toMessage(Object)} will populate all headers found in
+ * the map, unless {@link #filterHeadersInToMessage} is true.
+ * @param headerNames
+ */
+ public void setHeaderNames(Collection<String> headerNames) {
@artembilan
artembilan Apr 3, 2013 Spring member

How about to make just it like this:

public void setHeaderNames(String... headerNames) {
        this.headerNames = headerNames;
    }

In most cases it will be configured as <bean> and comma-delimited value for this property will be converted perfectly.
And further in the toMessage use retainAll:

if (headers != null) {
    if (this.filterHeadersInToMessage) {
        headers.keySet().retainAll(Arrays.asList(this.headerNames));
    }
    messageBuilder.copyHeaders(headers);
}
@garyrussell
garyrussell Apr 3, 2013 Spring member

Nice! However, I will still create a Set in setHeaderNames() because retainAll() uses contains() which is inefficient for a List.

Also this.headerNames = headerNames would allow the caller to mutate the list afterwards.

@artembilan artembilan commented on the diff Apr 3, 2013
...tcp/connection/MessageConvertingTcpMessageMapper.java
+
+ private final MessageConverter messageConverter;
+
+ public MessageConvertingTcpMessageMapper(MessageConverter messageConverter) {
+ Assert.notNull(messageConverter, "'messasgeConverter' must not be null");
+ this.messageConverter = messageConverter;
+ }
+
+ @Override
+ public Message<Object> toMessage(TcpConnection connection) throws Exception {
+ Object data = connection.getPayload();
+ if (data != null) {
+ Message<Object> message = this.messageConverter.toMessage(data);
+ MessageBuilder<Object> messageBuilder = MessageBuilder.fromMessage(message);
+ this.addStandardHeaders(connection, messageBuilder);
+ this.addCustomHeaders(connection, messageBuilder);
@artembilan
artembilan Apr 3, 2013 Spring member

In some recent PR we've already discussed about opening messageBuilder instance to the end-developer API...
WDYT now? Is there some other strategy to not show messageBuilder as parameter in these methods?

@garyrussell
garyrussell Apr 3, 2013 Spring member

Note that I made super.addStandardHeaders() and super.addCustomHeaders() final so subclasses can't override them and get a handle to the framework message builder (I remembered our earlier conversation about not allowing subclasses to be able to make arbitrary changes to the message).

@artembilan
artembilan Apr 3, 2013 Spring member

Got it! Thanks, So, nevermind.

@garyrussell
Member

Regarding your general comment - yes there is some overlap between converters and transformers, but this wouldn't be the first place for that. The MapJsonSerializer is really only provided as an example (of course, someone could use it) of how to take the map and put it in some format on the wire.

Also, a MessageConverter converts between an SI Message<?> and some external representation whereas a <transformer/> transforms one SI message to another.

@garyrussell
Member

Rebased, pushed.

@artembilan artembilan and 1 other commented on an outdated diff Apr 5, 2013
.../integration/ip/tcp/serializer/MapJsonSerializer.java
+ * Note that Feature.AUTO_CLOSE_SOURCE and
+ * JsonGenerator.Feature.AUTO_CLOSE_TARGET
+ * and will be disabled to avoid closing the connection's
+ * InputStream and OutputStream.
+ * <p/>
+ * The jackson deserializer can't delimit multiple JSON
+ * objects. Therefore another (de)serializer is used to
+ * apply structure to the stream. By default, this is a
+ * simple {@link ByteArrayLfSerializer}, which inserts/expects
+ * LF (0x0a) between messages.
+ *
+ * @author Gary Russell
+ * @since 3.0
+ *
+ */
+public class MapJsonSerializer implements Serializer<Map<?, ?>>, Deserializer<Map<?, ?>>,
@artembilan
artembilan Apr 5, 2013 Spring member

Gary, regarding this implementation.
It fully depends from Jackson 1.
However we are planning to support both #774
So, maybe we push MapJsonSerializer feature to future after merging my PR?
My point - to make dependency as less as posible, otherwise we will should provide MapJsonSerializer for Jackson 2.
WDYT?

@garyrussell
garyrussell Apr 5, 2013 Spring member

Yes; I agree - I will try to get your PR reviewed/merged today or Monday.

And, I think I can just make this serializer J2 (no need for both, I think - if someone really, really wants a 1.x version, they can make their own).

@artembilan artembilan and 1 other commented on an outdated diff May 8, 2013
.../integration/ip/tcp/serializer/MapJsonSerializer.java
+ }
+
+ /**
+ * A {@link Serializer} that will delimit the full JSON content in
+ * the stream. Default is
+ * {@link ByteArrayLfSerializer}.
+ * @param packetSerializer the packetSerializer
+ */
+ public void setPacketSerializer(Serializer<byte[]> packetSerializer) {
+ Assert.notNull(packetSerializer, "'packetSerializer' cannot be null");
+ this.packetSerializer = packetSerializer;
+ }
+
+ public void afterPropertiesSet() throws Exception {
+ this.objectMapper.configure(Feature.AUTO_CLOSE_SOURCE, false);
+ this.objectMapper.configure(JsonGenerator.Feature.AUTO_CLOSE_TARGET, false);
@artembilan
artembilan May 8, 2013 Spring member

Gery, let's get back to this class after merging JsonObjectMapper abstraction.
I dwell on these two lines because I didn't introduce any configuration hook and I didn't know how to abstract it in case auto-detecation of Jackson lib at CLASSPATH.
Do you have any idea?

@garyrussell
garyrussell May 8, 2013 Spring member

Ugh; you are right; let me think a while.

@artembilan
artembilan Jun 10, 2013 Spring member

this.objectMapper.configure(JsonGenerator.Feature.AUTO_CLOSE_TARGET, false);
Does not make sense: it applies for local ByteArrayOutputStream in the serialize below.
From other side, Gary, comes to mind a solution with some generic JsonObjectMapperConfiguration, which will be mapped to concrate configuration based on JsonObjectMapper implementation. E.g. GSON doesn't support similar featuers, but it has GsonBuilder. In the end it may look like thankless work to try provide hooks for all existing configuration features in different JSON mapping engines...
WDYT?

@garyrussell
garyrussell Aug 11, 2013 Spring member

When we get the package tangle code merged, I'll re-visit this PR.

@garyrussell
garyrussell Aug 15, 2013 Spring member

Artem; since this is brand new, I am inclined to support only Jackson2 out of the box (wire up a JsonObjectMapper with a properly configured Jackson2 mapper), but allow the user to inject his own JsonObjectMapper if he wishes to use Jackson1 or some other implementation.

WDYT?

@artembilan
artembilan Aug 16, 2013 Spring member

What you say is right. But my general concern here that you are using feature
this.objectMapper.configure(Feature.AUTO_CLOSE_SOURCE, false);
I think I'll play today with JsonObjectMapper here, before you wake up. 😄

@artembilan artembilan commented on an outdated diff Jun 6, 2013
...ation/support/converter/MapMessageConverterTests.java
+ Message<String> message = MessageBuilder.withPayload("foo")
+ .setHeader("bar", "baz")
+ .setHeader("baz", "qux")
+ .build();
+ MapMessageConverter converter = new MapMessageConverter();
+ converter.setHeaderNames("bar");
+ @SuppressWarnings("unchecked")
+ Map<String, Object> map = (Map<String, Object>) converter.fromMessage(message);
+
+ map.remove("payload");
+
+ try {
+ converter.toMessage(map);
+ fail("Expected exception");
+ }
+ catch (IllegalArgumentException e) {}
@artembilan
artembilan Jun 6, 2013 Spring member

Hi!
This test looks not full.

@artembilan
Member

Gary, take a look, please, into my polishing:
artembilan@06259ef
And if it is OK, there is need to change a bit IP doc.

@garyrussell garyrussell INT-1807 Add Mechanism For Headers with TCP
TCP streams have no standard message structure. Therefore, the
TCP implementation previously only transferred the message
payload.

If someone wanted to convey header information, they would have
to write their own wrapper and/or use Java serialization for
the entire message.

This change provides a strategy to allow users to determine
which headers are transferred, and how.

A MessageConvertingMessageMapper is now provided that invokes
any MessageConverter. A MapMessageConverter is provided that
converts the payload, and selected heades to a Map with two
entries ("payload") and ("headers").

A MapJsonSerializer is provided that converts a Map to/from
JSON. Jackson can't delimit multiple objects in a stream
so another serializer is required to encode/decode structure.
A ByteArrayLfSerializer is used by default, inserting a
linefeed between JSON objects.

The combination of these elements now allows header
information to be transferred over TCP. Of course, users
can implment their own (de)serializer to format the
bits on the wire exactly as needed by their application.

INT-1807 Polishing

Add a test that uses a Map MessageConverter with a
Java (de)serializer.

INT-1807: Polishing

INT-1807: Rebased and polished

Change `MapJsonSerializer` to use `JsonObjectMapper` abstraction

Doc Polishing
0699fdc
@garyrussell
Member

Thanks Artem; I applied your commit on top and polished the docs; I realize now that those Jackson settings were no longer required after I added the second serializer (LF) to delineate between json objects. Now that the objectmapper is no longer acting on the socket's inputstream it doesn't matter that it closes the stream (it is simply closing a byte array stream that is already at the end.

Merging...

@garyrussell garyrussell merged commit 0699fdc into spring-projects:master Aug 16, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment