added Reconciliation Services disappear on restart

[chsteiner]

Describe the bug
All added Reconciliation Services other than the default "Wikidata (en)" disappear on restart.
The only hint I get in the console could be this line: [FileProjectManager] com.fasterxml.jackson.databind.JsonMappingException: Invalid UTF-8 middle byte 0x72
at [Source: (File); line: 1, column: 2280] (through reference chain: com.google.refine.io.FileProjectManager["preferences"]->com.google.refine.preference.PreferenceStore["entries"]) (368ms)

This only happens with OpenRefine 3.3 and 3.4 (Beta). I also tested 3.1 and 3.2 and encountered no problems there!

I tested it on two different Windows machines with the same bug persisting on both.

To Reproduce
Steps to reproduce the behavior:
Just add any reconciliation service, e.g. https://lobid.org/gnd/reconcile (GND service), shut down OpenRefine and restart

Expected behavior
all added Reconciliation Services should be preserved on restart

Desktop (please complete the following information):

• OS: Windows 10
• Browser Version: Chrome Version 81.0.4044.138 (64-Bit)
• JRE or JDK Version: java version "1.8.0_251"

OpenRefine (please complete the following information):

• Version 3.3

Datasets
all of my datasets are affected

[tfmorris]

Sorry to hear about your trouble. Could we get a copy of your workspace.json file (and workspace.old.json if it exists)? To find it, click on "Open Project" then "Browse Workspace Directory" at the bottom of the page.

I suspect that there is something corrupted in it (probably not properly escaped by us).

[chsteiner]

Hi, thanks for offering help! Both files are incuded in this zip file:

workspace.zip

[tfmorris]

Thanks. I don't see anything obviously bad about those files and when I add the GND service, it remains after restart.
Do you get the error every time or was it just a one time thing?
How are you shutting down / restarting OpenRefine?

[chsteiner]

I get it every time and I get it on two different Laptops, so I thought it should be reproducable.
I am shutting down using ctrl+c in the console.

[chsteiner]

any ideas what I can try? this is really slowing down my work. I also teach some untis about OpenRefine and literally everyone using a Windows machine has this problem. So it must be reproducible for you. This is a problem that persists in 3.4 Beta btw!

i can narrow it down to the problem happening with the POST /command/core/set-preference command. Every time this command is run, after restart we get this:

FileProjectManager] com.fasterxml.jackson.databind.JsonMappingException: Invalid UTF-8 middle byte 0x72
 at [Source: (File); line: 1, column: 2277] (through reference chain: com.google.refine.io.FileProjectManager["preferences"]->com.google.refine.preference.PreferenceStore["entries"]) (353ms)

[tfmorris]

Googling the error makes me think that it's due to the wrong file encoding being used during JSON serialization for platforms which don't have UTF-8 as their default encoding (e.g. Windows). It's also only likely to be trigger when the JSON has non ASCII characters in it (e.g. "Körperschaft").

As a workaround, you could try adding -Dfile.encoding=UTF-8 to the Java command line in your refine.bat file. I'm not sure offhand if this will have any negative side effects on other file I/O, but I suspect not (everyone should be using UTF-8 these days anyway).

literally everyone using a Windows machine has this problem. So it must be reproducible for you.

That assumes, of course, that I'm running Windows, which is not the case.

I'll investigate a permanent fix to the JSON serializer.

[tfmorris]

Perhaps related to #2543 #2544

[wetneb]

@tfmorris I can try to reproduce this on a Windows machine in mid-June and check your PR too.

[chsteiner]

I have acutally always set this parameter in the JRE. But it didn't seem to help with this problem.
However, adding -Dfile.encoding=UTF-8 to openrefine.l4j.ini did the trick!
Many thanks for taking care of it!