Fix ItemResourceOSGiTest on Windows

[wborn]

Fixes #1002

---

With this PR and #1148, a full build with tests on Windows builds successfully for me @lolodomo.

[kaikreuzer]

Hm, I would have thought that there was a good reason to pin UTF-8 encoding here. After all, we are talking about the REST API, which should use a strictly defined encoding.
Are you sure that it isn't rather the test that might need to be modified?

[lolodomo]

I have no opinion if this fix is what we need, but yes with this fix, the test is then ok.

[wborn]

I will do some testing with a real openHAB instance with/without this change running on
both Linux/Windows and a UTF-8 encoded items file and see if these characters properly show on both Linux/Windows.

By coincidence I just read in OH 2.4 there were issues with this so it would be nice to know if this is currently resolved and if this change would thus be either a fix or causing a regression. 😉

[wborn]

After some testing it still seems to work properly with this change when running openHAB on both Linux/Windows and consuming the various /rest/items resources via the REST API using both Linux/Windows. So it seems the UTF-8 encoding is still taken care of at another level.

I also tested 2.4.0 and saw that a GET on /rest/items/Item_Name was returning ????? when running openHAB on Windows and using Greek/Russian symbols with the REST API on both Linux/Windows. It also showed these question marks with items list :

openHAB 2.4.0 (Windows)

openhab> items list
German_Item (Type=StringItem, State=German state äßöüÄÖÜ, Label=German label äßöüÄÖÜ, Category=null)
Dutch_Item (Type=StringItem, State=Dutch state abcDEF, Label=Dutch label abcDEF, Category=null)
Greek_Item (Type=StringItem, State=Greek state ?????, Label=Greek label ?????, Category=null)
French_Item (Type=StringItem, State=French state àêîÎç€, Label=French label àêîÎç€, Category=null)
Russian_Item (Type=StringItem, State=Russian state ??????, Label=Russian label ??????, Category=null)

openHAB 2.5.0.M4 (Linux/Windows)

openhab> items list                                                                                                                                                                                 
German_Item (Type=StringItem, State=German state äßöüÄÖÜ, Label=German label äßöüÄÖÜ, Category=null)
Dutch_Item (Type=StringItem, State=Dutch state abcDEF, Label=Dutch label abcDEF, Category=null)
Greek_Item (Type=StringItem, State=Greek state αβγΔΘ, Label=Greek label αβγΔΘ, Category=null)
French_Item (Type=StringItem, State=French state àêîÎç€, Label=French label àêîÎç€, Category=null)
Russian_Item (Type=StringItem, State=Russian state АВГДЄЧ, Label=Russian label АВГДЄЧ, Category=null)

[kaikreuzer]

So it seems the UTF-8 encoding is still taken care of at another level.

Hm, I would really prefer to understand why it might still work.
The method without the encoding parameter states in the JavaDoc "using the default character encoding of the platform". So clearly this should make a difference, if the platform is not using UTF-8. What is set on your Windows machine as encoding, @wborn? And which encoding uses the command line window?

@htreu Do you as the original author happen to remember any details on this matter?

[wborn]

This is what PowerShell reports on my Windows 10 VM:

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\Users\Wouter> [System.Text.Encoding]::Default


IsSingleByte      : True
BodyName          : iso-8859-1
EncodingName      : Western European (Windows)
HeaderName        : Windows-1252
WebName           : Windows-1252
WindowsCodePage   : 1252
IsBrowserDisplay  : True
IsBrowserSave     : True
IsMailNewsDisplay : True
IsMailNewsSave    : True
EncoderFallback   : System.Text.InternalEncoderBestFitFallback
DecoderFallback   : System.Text.InternalDecoderBestFitFallback
IsReadOnly        : True
CodePage          : 1252

[wborn]

After following lots of method calls I may have found the other level in the org.eclipse.smarthome.io.rest.core.internal.GsonProvider which seems to take care of all JSON serialization/deserialization and always uses UTF-8 for this:

openhab-core/bundles/org.openhab.core.io.rest.core/src/main/java/org/eclipse/smarthome/io/rest/core/internal/GsonProvider.java

Lines 66 to 74 in be69d22

	@Override
	public void writeTo(T object, Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType,
	MultivaluedMap<String, Object> httpHeaders, OutputStream entityStream)
	throws IOException, WebApplicationException {
	try (OutputStream stream = entityStream) {
	entityStream.write(gson.toJson(object).getBytes(StandardCharsets.UTF_8));
	entityStream.flush();
	}
	}

openhab-core/bundles/org.openhab.core.io.rest.core/src/main/java/org/eclipse/smarthome/io/rest/core/internal/GsonProvider.java

Lines 81 to 88 in be69d22

	@Override
	public T readFrom(Class<T> type, Type genericType, Annotation[] annotations, MediaType mediaType,
	MultivaluedMap<String, String> httpHeaders, InputStream entityStream)
	throws IOException, WebApplicationException {
	try (InputStreamReader reader = new InputStreamReader(entityStream, StandardCharsets.UTF_8)) {
	return gson.fromJson(reader, type);
	}
	}

[wborn]

It might also be better if the Stream2JSONInputStream doesn't convert between encodings. There is a charset directive in the Content-Type header (see MDN) which we may want to support one day. The conversion in the Stream2JSONInputStream to UTF-8 would then result in unnecessary processing on some platforms.

The name of the Stream2JSONInputStream also suggests the entity wil always be JSON which may also be problematic if we add an additional provider for supporting text/xml.

[kaikreuzer]

Thanks for your elaborate analysis, just for convincing me. But I can clearly say that you've succeeded 😉.

The name of the Stream2JSONInputStream also suggests the entity wil always be JSON

Right, I think this also cause part of my concern: I somehow was under the impression, that it directly produces JSON from the input stream, but you are right that the deserialisation is done elsewhere.

[wborn]

Great! I also like knowing the true reason why it all keeps working and it was also nice to see it all working with some real openHAB instances cross OS-es. :-)