String.getBytes("UTF-16") returns different value on Mac

[yshrsmz]

TL; DR

Implementation of Entity#getBytes is currently mName.getBytes("UTF-16"), but it should be either mName.getBytes("UTF-16BE") or mName.getBytes("UTF-16LE")

detail

I am trying to implement java cli app to decrypt files encrypted by conceal, and getting tag mismatch error all the time.
And after some investigation I realized that the result of String.getBytes("UTF-16") is different between Android and Mac.

when I execute Arrays.toString("teststring".getBytes("UTF-16")) on Android, it returns

[-1, -2, 116, 0, 101, 0, 115, 0, 116, 0, 115, 0, 116, 0, 114, 0, 105, 0, 110, 0, 103, 0]

but when I execute same code on mac as gradle task or standard java project, it returns

[-2, -1, 0, 116, 0, 101, 0, 115, 0, 116, 0, 115, 0, 116, 0, 114, 0, 105, 0, 110, 0, 103]

Apparently this is why I got tag mismatch error all the time :(

To fix this issue, I create custom Entity class whose getBytes method encodes mName with UTF-16LE.

But I think this should be done in library side. any thought?

[helios175]

You're totally right. We didn't bump into this problem as we used the encryption mainly for local storage, not for sharing stored data with other agents. I was thinking that UTF-8 could have even better byte distribution (as it doesn't pad with 0's).

But whatever we choose it should be the same among platforms.
Also: we should provide a way to read data persisted with current version to avoid backward incompatibility. I'll look into that soon.

Thanks for the report!

[helios175]

Solved in https://github.com/facebook/conceal/releases/tag/v1.1.2
Existing code calling new Entity will still have the UTF-16 platform dependent implementation (to avoid breaking old code).

Entity constructor is still public so you can still use the Entity child class that fixes the getBytes, athough maybe you want either:

• just use the new library
• be able to pass the bytes to the private constructor. In that case we could maybe make that constructor protected so it can be extended.

[yshrsmz]

Thanks for the fix!

be able to pass the bytes to the private constructor. In that case we could maybe make that constructor protected so it can be extended.

yes, this would be nice :) but I think it's better to just update the library and migrate to the new Entity implementation.

will definitely try this!