Support outbound message extras #581

QuintinWillison · 2020-06-10T16:04:35Z

Fixes #580. Conforms, IMO, with TM2i (with a little bit of idiomatic artistic license).

Tasks outstanding:

More MessageExtras unit tests (both constructors and message pack)
Integration test conforming to RSL6a2 (as, clearly, that didn't exist!)

…a JsonObject. Some refactoring here to improve our use of gson also.

…ras.

…reated from JSON does not return a delta extras instance.

… MessageExtras.

… bidirectional coding of 'raw' arbitrary JSON.

paddybyers

I realise that this is still draft. Do you think we should also now support PushExtras ?

paddybyers · 2020-06-11T07:37:38Z

lib/src/main/java/io/ably/lib/types/MessageExtras.java

+	private final DeltaExtras delta;
+
+	/**
+	 * Only intended for use with MessageExtras instances created to be sent to Ably's servers (outbound).


I'm not sure I understand the comment. Supporting raw is for content that isn't understood by this library, and I think that content can go in both directions.

Fair enough. I think it's evolved since our discussion later yesterday. I shall refine this commentary.

QuintinWillison · 2020-06-11T10:21:30Z

@paddybyers In terms of supporting PushExtras in a strongly typed and compiler verifiable manner, then perhaps in future, yes. For now, No.

I know that the customer who is waiting on us fixing this omission is already generating the necessary gson object in their own code - according to this mould/example that they gave us:

JsonUtils.`object`().add(
    "push", JsonUtils.`object`().add(
        "fcm", JsonUtils.`object`()
        // more config here
    )
).toJson()

(via internal Slack message)

As such, they can keep that same code and just utilise the new constructor I've added to pass 'raw' gson. I'm happy to change that terminology if it's considered incorrect or somehow otherwise misleading. 😁

…raw is available, they only use that value. Also adds a toString, which assists with test debugging apart from anything else.

It fails for the text protocol at the moment due to an issue with inbound JSON decoding.

…only inspect raw if it's available.

…e old version we had in place (November 2015).

…n message extras), fixing the implementation to support this.

QuintinWillison · 2020-06-11T17:13:55Z

Back to the drawing board again, I'm afraid.

Some of this suite failing:

-> Test crypto_publish[text_protocol](io.ably.lib.test.rest.RestCryptoTest)

io.ably.lib.test.rest.RestSuite > io.ably.lib.test.rest.RestCryptoTest.crypto_publish[text_protocol] FAILED
    java.lang.AssertionError at RestCryptoTest.java:75
-> Test crypto_publish_key_mismatch[text_protocol](io.ably.lib.test.rest.RestCryptoTest)

io.ably.lib.test.rest.RestSuite > io.ably.lib.test.rest.RestCryptoTest.crypto_publish_key_mismatch[text_protocol] FAILED
    java.lang.AssertionError at RestCryptoTest.java:207
-> Test crypto_send_encrypted_unhandled[text_protocol](io.ably.lib.test.rest.RestCryptoTest)

io.ably.lib.test.rest.RestSuite > io.ably.lib.test.rest.RestCryptoTest.crypto_send_encrypted_unhandled[text_protocol] FAILED
    java.lang.AssertionError at RestCryptoTest.java:288
-> Test crypto_publish_256[text_protocol](io.ably.lib.test.rest.RestCryptoTest)

io.ably.lib.test.rest.RestSuite > io.ably.lib.test.rest.RestCryptoTest.crypto_publish_256[text_protocol] FAILED
    java.lang.AssertionError at RestCryptoTest.java:123
-> Test crypto_publish_alt[text_protocol](io.ably.lib.test.rest.RestCryptoTest)

io.ably.lib.test.rest.RestSuite > io.ably.lib.test.rest.RestCryptoTest.crypto_publish_alt[text_protocol] FAILED
    java.lang.AssertionError at RestCryptoTest.java:169

…was causing REST crypto tests for text protocol to fail.

paddybyers · 2020-06-11T20:22:45Z

lib/src/main/java/io/ably/lib/types/MessageExtras.java

+
+	/**
+	 * @since 1.2.0
+	 */
 	public MessageExtras(final DeltaExtras delta) {


I'm still not clear on how this API is going to grow to accommodate more known extras members. We're going to have push, metadata, etc. How would you construct a MessageExtras if you have multiple of those values to supply?

The kind of thing you're seeing in other languages now is where the type for each of the known members implements a specific interface, eg

interface MessageExtra {} class DeltaExtras implements MessageExtra { ... } class PushExtras implements MessageExtra { ... } public MessageExtras(JSONObject raw, MessageExtra ...members) { .... }

Interesting. Depending on pressure we're under to release 1.2.1 it would definitely be worth having a call to discuss this as you suggested elsewhere.

The main thing I don't want to do is make any breaking changes between 1.2.0 and 1.2.1, but what you propose merely widens the type acceptance of this method so I think that's ok.

@paddybyers I'm currently disinclined to do anything about this in order to get this fix out... we can think how to architect this properly outside of the scope of this PR, if that's ok with you?

In terms of what I mentioned above... If we added a super interface at some point in the future to DeltaExtras then this method could be safely refactored to accept any object that implements that interface without that being a breaking API change.

public MessageExtras(final DeltaExtras delta)

who will use this API?

For some reason or another it looks like it's only used by RealtimeDeltaDecoderTest at the moment. I'll have a look as to whether that's needed. Perhaps I can replace it with a mock.

Meanwhile, of course, do we simply ignore the fact that we exposed this for 1.2.0? Maybe the best option is to simply get rid of it from the public API. What do you think, @paddybyers?

(equally I guess you could follow that argument through and just revert this portion of the change to the type of message extras wholesale!)

I've removed that API: d6ff228

paddybyers · 2020-06-11T20:24:49Z

lib/src/main/java/io/ably/lib/types/MessageExtras.java

-	/* package private */ JsonElement toJsonElement() {
-		return Serialisation.gson.toJsonTree(this);
+	/* package private */ JsonObject getRaw() {
+		return raw;


Doesn't this need to be public ? If someone validly sends me a message containing a member in extras that this library doesn't know about, then I'm going to have to process the raw JSON.

If we are going to make it public, then perhaps it needs to be asJSON() or something like that.

Happy to make that public, yes. I was just trying to minimise exposure but I can see that's a clearly valid use case. I agree that, in that case, the name would need to change too.

I've now made that method public: e41c806

…lic. Also: - some renaming for clarity - add code to the DeltaExtras constructor to ensure that a JsonObject is always available, making the asJsonObject method easier to reason about

…ound.

…blic API.

…ase where a key is populated but with JSON null.

paddybyers

LGTM.

marto83 · 2020-06-15T15:19:49Z

@QuintinWillison I like the idea of keeping an internal JsonObject inside MessageExtras and reading Deltas out of it.

I think we can improve the way we hydrate the DeltaExtras object.
DeltasExtras is read-only so we can simplify the logic of that class. You don't need to have /* package private */ void write(MessagePacker packer) throws IOException. You always populate it out of the json object.

MessageExtras can become simpler as well. In the constructor you will just take the json and parse it to hydrate the deltas object. No need to have different parsing for json and message pack. You can use the same logic you have when parsing the json. Something like this:

public MessageExtras(final JsonObject jsonObject) {
	if (null == jsonObject) {
		this.jsonObject = new JsonObject();
	} else {
		this.jsonObject = jsonObject;
	}
	parseJson();
}

private void parseJson() {
	final JsonElement deltaElement = jsonObject.get(DELTA);
	if (deltaElement instanceof JsonObject) {
		delta = DeltaExtras.read((JsonObject)deltaElement);
	} else {
		if (null != deltaElement) {
			// Not sure if we should throw an exception here 
			throw MessageDecodeException.fromDescription("The value under the delta key is of the wrong type \"" + deltaElement.getClass() + "\" when expected a map.");
		}
	}
}

This way we can easily add more logic there to extract other read-only or read-wrtie objects. Objects that will support writing to the json will have to have an instance of the internal json object passed to them.

The next thing to note is that we should always return a copy of the jsonObject because it is mutable and we don't want people to be changing it under our feet. If somebody wants to update the MessageExtras of a Message they should use the constructor.

QuintinWillison · 2020-06-15T15:27:34Z

Thanks @marto83 ... if this is all implementation detail then I'm inclined to leave things as they are so we can get this release out. I'm happy to work on internal refactoring later and am also happy for you to take an alternative approach when you make the same change to the .NET client library.

marto83 · 2020-06-15T15:30:51Z

I agree, they are all internal details. I'm happy with the external interface.

Quintin Willison added 9 commits June 10, 2020 15:58

Add constructor to MessageExtras allowing it to be instantiated from …

a050229

…a JsonObject. Some refactoring here to improve our use of gson also.

Add instructions for running pure unit tests to the read me.

0f8ee1d

Add unit test as basic smoke test for the new constructor.

e8d9335

Implement equals and hashCode on MessageExtras and DeltaExtras.

13dd407

Add unit test for the MessageExtras constructor that takes a DeltaExt…

00ad728

…ras.

Refine existing unit test to validate that a MessageExtras instance c…

f2cc321

…reated from JSON does not return a delta extras instance.

Add unit test for MessagePack encode and decode of DeltaExtras within…

ef99b5b

… MessageExtras.

Add commentary.

f3dfaf9

Refactor the message pack support within the MessageExtras to support…

0752b58

… bidirectional coding of 'raw' arbitrary JSON.

paddybyers reviewed Jun 11, 2020

View reviewed changes

QuintinWillison self-assigned this Jun 11, 2020

Quintin Willison added 8 commits June 11, 2020 11:33

Remove misleading commentary.

c56c859

Improve message extras hash and equality implementations so that, if …

8a65f7e

…raw is available, they only use that value. Also adds a toString, which assists with test debugging apart from anything else.

Add integration test for RSL6a2.

cb6bc7c

It fails for the text protocol at the moment due to an issue with inbound JSON decoding.

Fix test now that MessageExtras quality check has been refactored to …

41d8857

…only inspect raw if it's available.

Update gson dependency to latest release (October 2019), replacing th…

65b2b67

…e old version we had in place (November 2015).

Simplify method names.

d41635b

Add SLF4J binding when running tests to help debugging.

3cbb7d9

Add integration test for RSL6a2, covering the opaque case (raw JSON i…

deb0f59

…n message extras), fixing the implementation to support this.

QuintinWillison marked this pull request as ready for review June 11, 2020 15:45

QuintinWillison requested a review from paddybyers June 11, 2020 15:46

QuintinWillison marked this pull request as draft June 11, 2020 17:12

Revert some of the manual JSON deserialisation code I injected as it …

db2a17f

…was causing REST crypto tests for text protocol to fail.

QuintinWillison marked this pull request as ready for review June 11, 2020 18:51

paddybyers reviewed Jun 11, 2020

View reviewed changes

Make the API to obtain a JsonObject from a MessageExtras instance pub…

e41c806

…lic. Also: - some renaming for clarity - add code to the DeltaExtras constructor to ensure that a JsonObject is always available, making the asJsonObject method easier to reason about

QuintinWillison requested a review from paddybyers June 12, 2020 11:21

Add unit tests for constructor null arguments on MessageExtras.

99e3407

Quintin Willison added 4 commits June 12, 2020 15:35

Remove the unnecessary DeltaExtras constructor on MessageExtras.

d6ff228

Remove DeltaExtras JSON serialisation support as it's not needed outb…

00f1167

…ound.

Make the DeltaExtras constructor private as it's not needed in the pu…

7fbc7b1

…blic API.

Fix oversight in my base message deserialisation code, covering the c…

5101238

…ase where a key is populated but with JSON null.

paddybyers approved these changes Jun 12, 2020

View reviewed changes

QuintinWillison requested a review from marto83 June 12, 2020 16:12

marto83 approved these changes Jun 15, 2020

View reviewed changes

This was referenced Jun 15, 2020

MessageExtras asJsonObject() should return a copy #583

Open

DeltaExtras does not need MessagePack write support #584

Open

MessageExtras to comply with TM2i ably/ably-dotnet#422

Closed

QuintinWillison merged commit 77a261c into master Jun 15, 2020

QuintinWillison deleted the feature/580-message-extras branch June 15, 2020 15:47

sacOO7 mentioned this pull request Aug 23, 2020

Refactor MessageExtras #595

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support outbound message extras #581

Support outbound message extras #581

QuintinWillison commented Jun 10, 2020 •

edited by sacOO7

paddybyers left a comment •

edited

paddybyers Jun 11, 2020

QuintinWillison Jun 11, 2020

QuintinWillison Jun 11, 2020

QuintinWillison commented Jun 11, 2020

QuintinWillison commented Jun 11, 2020

paddybyers Jun 11, 2020

QuintinWillison Jun 11, 2020

QuintinWillison Jun 12, 2020

paddybyers Jun 12, 2020

QuintinWillison Jun 12, 2020

QuintinWillison Jun 12, 2020

paddybyers Jun 11, 2020

QuintinWillison Jun 11, 2020

QuintinWillison Jun 12, 2020

paddybyers left a comment

marto83 commented Jun 15, 2020

QuintinWillison commented Jun 15, 2020

marto83 commented Jun 15, 2020

Support outbound message extras #581

Support outbound message extras #581

Conversation

QuintinWillison commented Jun 10, 2020 • edited by sacOO7

paddybyers left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

QuintinWillison commented Jun 11, 2020

QuintinWillison commented Jun 11, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paddybyers left a comment

Choose a reason for hiding this comment

marto83 commented Jun 15, 2020

QuintinWillison commented Jun 15, 2020

marto83 commented Jun 15, 2020

QuintinWillison commented Jun 10, 2020 •

edited by sacOO7

paddybyers left a comment •

edited