Backslashes can cause serialization exceptions

[AaronSchulz]

Try to do:

update V content {"id":"15646406","claims":{"P357":[{"id":"Q15646406$75752992-210C-44AC-AEF4-FEE9929F876A","mainsnak":{"snaktype":"value","property":"P357","datatype":"string","datavalue":{"value":"Sub\urban","type":"string"}},"type":"statement","rank":"normal","references":[{"hash":"004ec6fbee857649acdbdbad4f97b2c8571df97b","snaks":{"P143":[{"snaktype":"value","property":"P143","datatype":"wikibase-item","datavalue":{"value":{"entity-type":"item","numeric-id":48183},"type":"wikibase-entityid"}}]},"snaks-order":["P143"]}]}],"P392":[{"id":"Q15646406$1357D555-7E9A-4C0B-B4E6-E766FD477FA5","mainsnak":{"snaktype":"value","property":"P392","datatype":"string","datavalue":{"value":"Zeitschrift f\u00fcr kritische Stadtforschung","type":"string"}},"type":"statement","rank":"normal","references":[{"hash":"004ec6fbee857649acdbdbad4f97b2c8571df97b","snaks":{"P143":[{"snaktype":"value","property":"P143","datatype":"wikibase-item","datavalue":{"value":{"entity-type":"item","numeric-id":48183},"type":"wikibase-entityid"}}]},"snaks-order":["P143"]}]}],"P31":[{"id":"Q15646406$1F7FFC3A-9084-4776-B6B1-47C251E1D13C","mainsnak":{"snaktype":"value","property":"P31","datatype":"wikibase-item","datavalue":{"value":{"entity-type":"item","numeric-id":1092563},"type":"wikibase-entityid"}},"type":"statement","rank":"normal","references":[{"hash":"004ec6fbee857649acdbdbad4f97b2c8571df97b","snaks":{"P143":[{"snaktype":"value","property":"P143","datatype":"wikibase-item","datavalue":{"value":{"entity-type":"item","numeric-id":48183},"type":"wikibase-entityid"}}]},"snaks-order":["P143"]}]}],"P236":[{"id":"Q15646406$7F9C4F82-DE2A-4393-B30D-9248C698DC97","mainsnak":{"snaktype":"value","property":"P236","datatype":"string","datavalue":{"value":"2197-2567","type":"string"}},"type":"statement","rank":"normal","references":[{"hash":"004ec6fbee857649acdbdbad4f97b2c8571df97b","snaks":{"P143":[{"snaktype":"value","property":"P143","datatype":"wikibase-item","datavalue":{"value":{"entity-type":"item","numeric-id":48183},"type":"wikibase-entityid"}}]},"snaks-order":["P143"]}]}],"P1042":[{"id":"Q15646406$6D7F267B-BE1F-498E-9758-7C252071D5BD","mainsnak":{"snaktype":"value","property":"P1042","datatype":"string","datavalue":{"value":"2750090-1","type":"string"}},"type":"statement","rank":"normal","references":[{"hash":"004ec6fbee857649acdbdbad4f97b2c8571df97b","snaks":{"P143":[{"snaktype":"value","property":"P143","datatype":"wikibase-item","datavalue":{"value":{"entity-type":"item","numeric-id":48183},"type":"wikibase-entityid"}}]},"snaks-order":["P143"]}]}]},"sitelinks":["dewiki#Suburban (Zeitschrift)"],"pids":["357","392","31","236","1042"],"iids":[1092563]} where id='15646406'

Then try:

update V content {"id":"15646406","claims":{"P357":[{"id":"Q15646406$75752992-210C-44AC-AEF4-FEE9929F876A","mainsnak":{"snaktype":"value","property":"P357","datatype":"string","datavalue":{"value":"Suburban","type":"string"}},"type":"statement","rank":"normal","references":[{"hash":"004ec6fbee857649acdbdbad4f97b2c8571df97b","snaks":{"P143":[{"snaktype":"value","property":"P143","datatype":"wikibase-item","datavalue":{"value":{"entity-type":"item","numeric-id":48183},"type":"wikibase-entityid"}}]},"snaks-order":["P143"]}]}],"P392":[{"id":"Q15646406$1357D555-7E9A-4C0B-B4E6-E766FD477FA5","mainsnak":{"snaktype":"value","property":"P392","datatype":"string","datavalue":{"value":"Zeitschrift f\u00fcr kritische Stadtforschung","type":"string"}},"type":"statement","rank":"normal","references":[{"hash":"004ec6fbee857649acdbdbad4f97b2c8571df97b","snaks":{"P143":[{"snaktype":"value","property":"P143","datatype":"wikibase-item","datavalue":{"value":{"entity-type":"item","numeric-id":48183},"type":"wikibase-entityid"}}]},"snaks-order":["P143"]}]}],"P31":[{"id":"Q15646406$1F7FFC3A-9084-4776-B6B1-47C251E1D13C","mainsnak":{"snaktype":"value","property":"P31","datatype":"wikibase-item","datavalue":{"value":{"entity-type":"item","numeric-id":1092563},"type":"wikibase-entityid"}},"type":"statement","rank":"normal","references":[{"hash":"004ec6fbee857649acdbdbad4f97b2c8571df97b","snaks":{"P143":[{"snaktype":"value","property":"P143","datatype":"wikibase-item","datavalue":{"value":{"entity-type":"item","numeric-id":48183},"type":"wikibase-entityid"}}]},"snaks-order":["P143"]}]}],"P236":[{"id":"Q15646406$7F9C4F82-DE2A-4393-B30D-9248C698DC97","mainsnak":{"snaktype":"value","property":"P236","datatype":"string","datavalue":{"value":"2197-2567","type":"string"}},"type":"statement","rank":"normal","references":[{"hash":"004ec6fbee857649acdbdbad4f97b2c8571df97b","snaks":{"P143":[{"snaktype":"value","property":"P143","datatype":"wikibase-item","datavalue":{"value":{"entity-type":"item","numeric-id":48183},"type":"wikibase-entityid"}}]},"snaks-order":["P143"]}]}],"P1042":[{"id":"Q15646406$6D7F267B-BE1F-498E-9758-7C252071D5BD","mainsnak":{"snaktype":"value","property":"P1042","datatype":"string","datavalue":{"value":"2750090-1","type":"string"}},"type":"statement","rank":"normal","references":[{"hash":"004ec6fbee857649acdbdbad4f97b2c8571df97b","snaks":{"P143":[{"snaktype":"value","property":"P143","datatype":"wikibase-item","datavalue":{"value":{"entity-type":"item","numeric-id":48183},"type":"wikibase-entityid"}}]},"snaks-order":["P143"]}]}]},"sitelinks":["dewiki#Suburban (Zeitschrift)"],"pids":["357","392","31","236","1042"],"iids":[1092563]} where id='15646406'

The first will fail but not the second. The error is:
a) Error: com.orientechnologies.orient.core.exception.OSerializationException: Error on unmarshalling JSON content for record ...
b) Error: java.lang.NumberFormatException: For input string: "rban"

The only difference using "Suburban" instead of "Sub\urban" for one of the values early in the JSON string. This may be related to issue #2424 .

[AaronSchulz]

Note that github misformats my comment a bit. The JSON in the first statement should have two backslash in "Suburban", not just one.

[AaronSchulz]

Simplified repro (note that there should be two backslashes in "Suburban", not one):

update V content {"mainsnak":{"datavalue":{"value":"Sub\urban"}}} where id=10

update V content {"datavalue":{"value":"Sub\urban"}} where id=10

The first fails, the later doesn't.

[lvca]

@AaronSchulz Are you using 2.0-SNAPSHOT?

[AaronSchulz]

I'm on 2.0M3 now and still need to hack around this :/

[lvca]

@luigidellaquila could you take a look at this for 2.0RC?

[luigidellaquila]

working on this

[luigidellaquila]

Hi @AaronSchulz

I fixed this and pushed on develop branch. Could you try it and let me know?

Thanks

Luigi

[AaronSchulz]

I build SNAPSHOT, but I'm running into 'Error: Could not find or load main class 8192' errors trying to start it. If I can get past that, I'll try this out again.

[AaronSchulz]

Gah, sill error on my part. Looking at this now.

[AaronSchulz]

Test case seems to work now.

[luigidellaquila]

this fix caused other problems, we had to revert it.
It's in high priority and is being addressed right now

[luigidellaquila]

Hi @AaronSchulz
I fixed this on develop branch, could you try it again from source?
If you cannot build from source we will publish a snapshot shortly

thanks

[AaronSchulz]

I'll take a look when I get a chance. Hopefully #2690 is next.

[AaronSchulz]

Trailing backslashes still fail (note the "sq":"vibw\"):

update Item set id=332707, labels={"zh-hans":"Viber","zh-hant":"Viber","zh-hk":"Viber","zh-cn":"Viber","zh-sg":"Viber","zh-tw":"Viber","fr":"Viber","ko":"Viber","ru":"\u0412\u0438\u0431\u0435\u0440","es":"Viber","en":"Viber","tr":"Viber","th":"\u0e44\u0e27\u0e40\u0e1a\u0e2d\u0e23\u0e4c","it":"Viber","de":"Viber","ja":"Viber","cs":"Viber","ar":"\u0641\u0627\u064a\u0628\u0631","nl":"Viber","fa":"\u0648\u0627\u06cc\u0628\u0631","zh":"Viber","id":"Viber","hr":"Viber","hu":"Viber","vi":"Viber","is":"Viber","uk":"Viber","sr":"Viber","mzn":"\u0648\u0627\u06cc\u0628\u0631","he":"Viber","pt":"Viber","ms":"Viber","ckb":"\u06a4\u0627\u06cc\u0628\u06d5\u0631","be-tarask":"Viber","bg":"Viber","hy":"viber","ca":"Viber","sq":"vibw"}, sitelinks={"zhwiki":"zhwiki#Viber","frwiki":"frwiki#Viber","kowiki":"kowiki#Viber","ruwiki":"ruwiki#Viber","eswiki":"eswiki#Viber","enwiki":"enwiki#Viber","trwiki":"trwiki#Viber","thwiki":"thwiki#\u0e44\u0e27\u0e40\u0e1a\u0e2d\u0e23\u0e4c","dewiki":"dewiki#Viber","jawiki":"jawiki#Viber","cswiki":"cswiki#Viber","simplewiki":"simplewiki#Viber","arwiki":"arwiki#\u0641\u0627\u064a\u0628\u0631","nlwiki":"nlwiki#Viber","fawiki":"fawiki#\u0648\u0627\u06cc\u0628\u0631","idwiki":"idwiki#Viber","itwiki":"itwiki#Viber","srwiki":"srwiki#Viber","hewiki":"hewiki#Viber","hrwiki":"hrwiki#Viber","viwiki":"viwiki#Viber","ukwiki":"ukwiki#Viber","huwiki":"huwiki#Viber","be_x_oldwiki":"be_x_oldwiki#Viber","mznwiki":"mznwiki#\u0648\u0627\u06cc\u0628\u0631","iswiki":"iswiki#Viber","mswiki":"mswiki#Viber","ptwiki":"ptwiki#Viber","ckbwiki":"ckbwiki#\u06a4\u0627\u06cc\u0628\u06d5\u0631","bgwiki":"bgwiki#Viber","cawiki":"cawiki#Viber"}, deleted=NULL, claims={"306":[{"snaktype":"value","valuetype":"wikibase-entityid","datavalue":880016,"rank":0,"best":1},{"snaktype":"value","valuetype":"wikibase-entityid","datavalue":94,"rank":0,"best":1},{"snaktype":"value","valuetype":"wikibase-entityid","datavalue":48493,"rank":0,"best":1}],"856":[{"snaktype":"value","valuetype":"string","datavalue":"https://www.viber.com/","rank":0,"best":1}],"646":[{"snaktype":"value","valuetype":"string","datavalue":"/m/0hncjr3","rank":0,"best":1}],"154":[{"snaktype":"value","valuetype":"string","datavalue":"Viber-logo.png","rank":0,"best":1}],"31":[{"snaktype":"value","valuetype":"wikibase-entityid","datavalue":783794,"rank":0,"best":1}]}, pids=[306,856,646,154,31], iids=[48493,94,880016,783794] where id=332707

Fatal error: Uncaught exception 'Exception' with message 'Command failed (500). Got:
java.lang.IllegalStateException: Missing closed string character: '"', position: 2415' in N:\Dev\WikiDataQueryOrient\lib
\WdqUpdater.php:644

[AaronSchulz]

Simplified repro:

update Test content {"mainsnak":{"datavalue":{"value":"Sub\urban\"}}} where id=10

Also, my update script got about 1/2 through and died with:

Error on command:
update Item set id=15646406, labels={"de":"Suburban (Zeitschrift)"}, sitelinks={"dewiki":"dewiki#Suburban (Zeitschrift)"
}, deleted=NULL, claims={"357":[{"snaktype":"value","valuetype":"string","datavalue":"Sub\urban","rank":0,"best":1}],"3
92":[{"snaktype":"value","valuetype":"string","datavalue":"Zeitschrift f\u00fcr kritische Stadtforschung","rank":0,"best
":1}],"31":[{"snaktype":"value","valuetype":"wikibase-entityid","datavalue":1092563,"rank":0,"best":1}],"236":[{"snaktyp
e":"value","valuetype":"string","datavalue":"2197-2567","rank":0,"best":1}],"1042":[{"snaktype":"value","valuetype":"str
ing","datavalue":"2750090-1","rank":0,"best":1}]}, pids=[357,392,31,236,1042], iids=[1092563] where id=15646406

PHP Fatal error: Uncaught exception 'Exception' with message 'Command failed (500). Got:
java.lang.NumberFormatException: For input string: "rban"' in N:\Dev\WikiDataQueryOrient\lib\WdqUpdater.php:644

[luigidellaquila]

Hi @AaronSchulz

I fixed it (again ;-) ). I hope it's the last one

Thanks

[AaronSchulz]

OK, now my scripts makes it through all 16million vertexes (and the many more edges too).