Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fatal error when SMWDIGeoCoord unserializes localized Lat/Long (format : 0,xxx, 0,xxxx) #3153

Closed
otisyves opened this issue May 3, 2018 · 9 comments
Closed

Fatal error when SMWDIGeoCoord unserializes localized Lat/Long (format : 0,xxx, 0,xxxx) #3153

otisyves opened this issue May 3, 2018 · 9 comments
Labels
bug Occurrence of an unintended or unanticipated behaviour that causes a vulnerability or fatal error

Comments

@otisyves
Copy link

otisyves commented May 3, 2018 •
edited by kghbln

Setup and configuration

  • SMW version: 2.5.6
  • MW version: 1.30.0-rc.0 (0a1acb0)
  • Maps version: 5.3.0 and 5.0.2 respectively
  • PHP version: 7.0.30-1~dotdeb+8.1 (fpm-fcgi)
  • DB system: MySQL 5.6.36

2 examples (public wikis):

Issue

We upgraded all our wikis to 1.30 from 1.28.x/1.29.x and made the updates of all our extensions at the same time (SMW 2.3.x and 2.4.x -> 2.5.x). All of our wikis are setup in French as the main language.

Following the upgrade, pages with maps and tables that print Geographical coordinates stop working [Fatal error type « InvalidArgumentException »].

Ex. :

It seems that the SMWDIGeoCoord (line 119) does not accept/is not able to deserialize the geo coordinates stored in the DB - which use the comma as a decimal delimiter for Lat/Long -> [0,xxx, 0,xxx].

The edition of pages with geo coordinates (via Page Forms) is not affected.
Ex. :

Stack trace

[1e7153e0bcaf538d6a010beb] /index.php?title=D%C3%A9finition_des_communs SMW\Exception\DataItemException from line 119 of /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/includes/dataitems/SMW_DI_GeoCoord.php: Unserialization of coordinates failed

Backtrace:

#0 /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/src/SQLStore/EntityStore/DIHandlers/DIGeoCoordinateHandler.php(120): SMWDIGeoCoord::doUnserialize(string)
#1 /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/includes/storage/SQLStore/SMW_Sql3StubSemanticData.php(166): SMW\SQLStore\EntityStore\DIHandlers\DIGeoCoordinateHandler->dataItemFromDBKeys(string)
#2 /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/includes/storage/SQLStore/SMW_SQLStore3_Readers.php(233): SMWSql3StubSemanticData->getPropertyValues(SMW\DIProperty)
#3 /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/src/SQLStore/EntityStore/DirectEntityLookup.php(64): SMWSQLStore3Readers->getPropertyValues(SMW\DIWikiPage, SMW\DIProperty, NULL)
#4 /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/includes/storage/SQLStore/SMW_SQLStore3.php(213): SMW\SQLStore\EntityStore\DirectEntityLookup->getPropertyValues(SMW\DIWikiPage, SMW\DIProperty, NULL)
#5 /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/src/Query/Result/ResultFieldMatchFinder.php(296): SMWSQLStore3->getPropertyValues(SMW\DIWikiPage, SMW\DIProperty, NULL)
#6 /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/src/Query/Result/ResultFieldMatchFinder.php(280): SMW\Query\Result\ResultFieldMatchFinder->doFetchPropertyValues(array, SMWPropertyValue)
#7 /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/src/Query/Result/ResultFieldMatchFinder.php(188): SMW\Query\Result\ResultFieldMatchFinder->getResultContent(SMW\DIWikiPage)
#8 /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/src/Query/Result/ResultFieldMatchFinder.php(144): SMW\Query\Result\ResultFieldMatchFinder->getResultsForProperty(SMW\DIWikiPage)
#9 /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/includes/storage/SMW_ResultArray.php(272): SMW\Query\Result\ResultFieldMatchFinder->findAndMatch(SMW\DIWikiPage)
#10 /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/includes/storage/SMW_ResultArray.php(155): SMWResultArray->loadContent()
#11 /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/includes/storage/SMW_ResultArray.php(189): SMWResultArray->getNextDataItem()
#12 /var/www/wiki.remixthecommons.org/public_html/core/extensions/Maps/SemanticMaps/src/queryprinters/SM_QueryHandler.php(259): SMWResultArray->getNextDataValue()
#13 /var/www/wiki.remixthecommons.org/public_html/core/extensions/Maps/SemanticMaps/src/queryprinters/SM_QueryHandler.php(238): SMQueryHandler->handleResultRow(array)
#14 /var/www/wiki.remixthecommons.org/public_html/core/extensions/Maps/SemanticMaps/src/queryprinters/SM_QueryHandler.php(229): SMQueryHandler->findShapes()
#15 /var/www/wiki.remixthecommons.org/public_html/core/extensions/Maps/SemanticMaps/src/queryprinters/SM_MapPrinter.php(165): SMQueryHandler->getShapes()
#16 /var/www/wiki.remixthecommons.org/public_html/core/extensions/Maps/SemanticMaps/src/queryprinters/SM_MapPrinter.php(89): SMMapPrinter->handleMarkerData(array, SMQueryHandler)
#17 /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/includes/queryprinters/ResultPrinter.php(261): SMMapPrinter->getResultText(SMWQueryResult, integer)
#18 /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/includes/queryprinters/ResultPrinter.php(226): SMW\ResultPrinter->buildResult(SMWQueryResult)
#19 /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/includes/query/SMW_QueryProcessor.php(423): SMW\ResultPrinter->getResult(SMWQueryResult, array, integer)
#20 /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/src/ParserFunctions/AskParserFunction.php(210): SMWQueryProcessor::getResultFromQuery(SMWQuery, array, integer, integer)
#21 /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/src/ParserFunctions/AskParserFunction.php(135): SMW\ParserFunctions\AskParserFunction->doFetchResultsFromFunctionParameters(array)
#22 /var/www/wiki.remixthecommons.org/public_html/core/extensions/SemanticMediaWiki/src/ParserFunctionFactory.php(298): SMW\ParserFunctions\AskParserFunction->parse(array)
#23 /var/www/wiki.remixthecommons.org/public_html/core/includes/parser/Parser.php(3383): SMW\ParserFunctionFactory->SMW\{closure}(Parser, string, string, string, string, string, string, string, string, string, string, string)
#24 /var/www/wiki.remixthecommons.org/public_html/core/includes/parser/Parser.php(3106): Parser->callParserFunction(PPFrame_DOM, string, array)
#25 /var/www/wiki.remixthecommons.org/public_html/core/includes/parser/Preprocessor_DOM.php(1229): Parser->braceSubstitution(array, PPFrame_DOM)
#26 /var/www/wiki.remixthecommons.org/public_html/core/includes/parser/Parser.php(2921): PPFrame_DOM->expand(DOMElement, integer)
#27 /var/www/wiki.remixthecommons.org/public_html/core/includes/parser/Parser.php(1277): Parser->replaceVariables(string)
#28 /var/www/wiki.remixthecommons.org/public_html/core/includes/parser/Parser.php(451): Parser->internalParse(string)
#29 /var/www/wiki.remixthecommons.org/public_html/core/includes/content/WikitextContent.php(329): Parser->parse(string, Title, ParserOptions, boolean, boolean, integer)
#30 /var/www/wiki.remixthecommons.org/public_html/core/includes/content/AbstractContent.php(516): WikitextContent->fillParserOutput(Title, integer, ParserOptions, boolean, ParserOutput)
#31 /var/www/wiki.remixthecommons.org/public_html/core/includes/poolcounter/PoolWorkArticleView.php(144): AbstractContent->getParserOutput(Title, integer, ParserOptions)
#32 /var/www/wiki.remixthecommons.org/public_html/core/includes/poolcounter/PoolCounterWork.php(123): PoolWorkArticleView->doWork()
#33 /var/www/wiki.remixthecommons.org/public_html/core/includes/page/Article.php(585): PoolCounterWork->execute()
#34 /var/www/wiki.remixthecommons.org/public_html/core/includes/actions/ViewAction.php(68): Article->view()
#35 /var/www/wiki.remixthecommons.org/public_html/core/includes/MediaWiki.php(499): ViewAction->show()
#36 /var/www/wiki.remixthecommons.org/public_html/core/includes/MediaWiki.php(293): MediaWiki->performAction(Article, Title)
#37 /var/www/wiki.remixthecommons.org/public_html/core/includes/MediaWiki.php(851): MediaWiki->performRequest()
#38 /var/www/wiki.remixthecommons.org/public_html/core/includes/MediaWiki.php(523): MediaWiki->main()
#39 /var/www/wiki.remixthecommons.org/public_html/core/index.php(43): MediaWiki->run()
#40 {main}
@kghbln kghbln added the bug Occurrence of an unintended or unanticipated behaviour that causes a vulnerability or fatal error label May 3, 2018
@kghbln
Copy link
Member

kghbln commented May 3, 2018

Thanks for reporting this issue. @JeroenDeDauw This may also be of interest to you.

@mwjames
Copy link
Contributor

mwjames commented May 3, 2018 via email

@otisyves
Copy link
Author

otisyves commented May 3, 2018

Well, no luck! I just tried to rebuildData.php on both wikis and the process stopped when it encounters a page displaying geocord datas.

  • Remix - same error as above (unSerialize)
  • Arbre-evolution - error generated with Maps
[5a2597b20ecc500041508a88] /index.php?title=Parcelle_00008 InvalidArgumentException from line 40 of /var/www/wiki.arbre-evolution.org/docs/core/extensions/Maps/src/Elements/Circle.php: $circleRadius must be greater than zero

Backtrace:

#0 /var/www/wiki.arbre-evolution.org/docs/core/extensions/Maps/includes/parsers/CircleParser.php(40): Maps\Elements\Circle->__construct(DataValues\Geo\Values\LatLongValue, double)
#1 /var/www/wiki.arbre-evolution.org/docs/core/vendor/param-processor/param-processor/src/Param.php(300): Maps\CircleParser->parse(string)
#2 /var/www/wiki.arbre-evolution.org/docs/core/vendor/param-processor/param-processor/src/Param.php(263): ParamProcessor\Param->parseAndValidateValue(Maps\CircleParser, string)
#3 /var/www/wiki.arbre-evolution.org/docs/core/vendor/param-processor/param-processor/src/Param.php(219): ParamProcessor\Param->parseAndValidate(ParamProcessor\Options)
#4 /var/www/wiki.arbre-evolution.org/docs/core/vendor/param-processor/param-processor/src/Processor.php(367): ParamProcessor\Param->process(array, array, ParamProcessor\Options)
#5 /var/www/wiki.arbre-evolution.org/docs/core/vendor/param-processor/param-processor/src/Processor.php(341): ParamProcessor\Processor->processOneParam()
#6 /var/www/wiki.arbre-evolution.org/docs/core/vendor/param-processor/param-processor/src/Processor.php(270): ParamProcessor\Processor->doParamProcessing()
#7 /var/www/wiki.arbre-evolution.org/docs/core/extensions/Validator/src/legacy/ParserHook.php(372): ParamProcessor\Processor->validateParameters()
#8 /var/www/wiki.arbre-evolution.org/docs/core/extensions/Validator/src/legacy/ParserHook.php(294): ParserHook->validateAndRender(array, integer)

By the way, we're using the PageForms input types googlemaps, leaflet, openlayer to enter the coordinates or geocode the address. I notice in the wikicode of the page (say https://wiki.remixthecommons.org/index.php?title=D%C3%A9finir_le_Bien_Commun:_Alberto_Acosta_%C3%A0_Berl%C3%ADn) that the geocodes are in a correct form :

  • -0.18065, -78.46784
    But in the database it is stored as
  • -0,18065,-78,46784

Sure I can do a conversion on the tables for the existing data...

Reviewing our installs to check for clues. Thanks for your help.

@mwjames
Copy link
Contributor

mwjames commented May 3, 2018 via email

@otisyves
Copy link
Author

otisyves commented May 4, 2018

Not just yet. First it should be established as to why non ISO norm data are stored! If you create a new page with geo related annotations, does the same behaviour occur?
I found that the $wgShellLocale was responsible to the « non-ISO geocords » in the DB.

In LocalSettings.php, it was set like this:

#$wgShellLocale = "en_US.utf8";
$wgShellLocale = "fr_CA.utf8";

I switched to English and the data comes out fine in the DB. I rebuilt the data and everything is back. Strange that it worked for so long. Now looking at the Maps circle thing.

Thanks for your time!

@kghbln
Copy link
Member

kghbln commented May 4, 2018

In LocalSettings.php, it was set like this:

Nice find. On sandbox I traditionally use $wgShellLocale = 'en_GB.utf8'; however it is strongly recommended for MW 1.30 to switch to $wgShellLocale = 'C.UTF-8'; so I will switch to this locale tomorrow. MediaWiki has changed things around this configuration parameter.

@mwjames
Copy link
Contributor

mwjames commented May 12, 2018 •
edited by kghbln

I found that the $wgShellLocale was responsible to the « non-ISO geocords » in the DB.

Nice.

$wgShellLocale = "fr_CA.utf8";
I switched to English and the data comes out fine in the DB. I rebuilt the data and everything is back.

$wgShellLocale [0] uses setlocale( LC_ALL, $wgShellLocale ); [1] and hereby influences the formatting of date/time and numeric values (e.g. '.' versus ',' as the decimal separator).

MW version: 1.30.0-rc.0 (0a1acb0)
Strange that it worked for so long.

I'm guessing due to " ... is applied for all requests. In MediaWiki 1.29 and older it only affected LC_CTYPE. ..." as noted in [2].

LC_CTYPE only applies to character classification and conversion [1].

Tasks

  • @JeroenDeDauw could Maps make sure when building a SMWDIGeoCoord object [3] that it uses ISO norm data (i.e. . formatted values) instead of a setlocale dependant string parsing?
  • @kghbln we should probably have a note somewhere (installation instructions) about $wgShellLocale and this issue

[0] https://github.com/wikimedia/mediawiki/blob/e028df0ed505273feeb86ac812493d15426222e1/includes/Setup.php#L134
[1] http://php.net/manual/en/function.setlocale.php
[2] https://www.mediawiki.org/wiki/Manual:$wgShellLocale
[3] https://github.com/JeroenDeDauw/Maps/blob/master/src/Semantic/DataValues/CoordinateValue.php#L99-L113

@mwjames
Copy link
Contributor

mwjames commented May 12, 2018

I'm closing this issue as not actionable tasks (expect for the documentation part) can be derived for SMW core. Improving the Maps interface should be handled separately by the extension.

@mwjames mwjames closed this as completed May 12, 2018
kghbln added a commit to ProfessionalWiki/Maps that referenced this issue May 12, 2018
@kghbln
Update INSTALL.md
7f2cb6a 
Add information about the locale as mentioned in SemanticMediaWiki/SemanticMediaWiki#3153
@kghbln kghbln mentioned this issue May 12, 2018
Merged
@kghbln
Copy link
Member

kghbln commented May 12, 2018 •
edited

Documented:

Reported:

@kghbln kghbln mentioned this issue May 12, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Occurrence of an unintended or unanticipated behaviour that causes a vulnerability or fatal error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kghbln @mwjames @otisyves