rebuildData.php prematurely stops without error message

[jamesmontalvo3]

I just upgraded to SMW 2.1. After update I ran rebuildData.php, but the script stopped without giving any error after printing the line "(49) Processing ID 59 ...". I have more than 10000 pages in the wiki, so this is very prematurely ending.

Next I ran "php rebuildData.php -v -s 51" since I wasn't sure if the -s option meant the number in the parenthetical (e.g. 49) or the ID (e.g. 59). Again it stopped prematurely and without error after printing the line "(9) Processing ID 59 ...".

So next I ran "php rebuildData.php -v -s 60" and it appears to be running well. When it ends I'm not sure how I'll know if it is really done or if it simply failed silently again, though. Is there a way to turn on a debug mode to give me more insight into whatever error occurred?

Environtment details:

• SMW 2.0 and 2.1
• MW 1.23.2
• PHP 5.4.16
• MySQL 5.6.20
• Windows Server 2008 R2 with IIS 7.5

[mwjames]

Is there a way to turn on a debug mode to give me more insight into whatever error occurred?

There is no real debug mode, the only debug that you can have is provided by MW's maintenance base class which makes use of the stack trace options (see smw.org@stack trace).

[kghbln]

How do you know that it stopped? As coincidence has it I also just upgrade several wikis to SMW 2.1 and for one wiki I thought there may be a problem too since nothing seemed to happen for quite a while. However after about 4 Minutes the script continued with the next ID. The second time it happened for this wiki I had a look a the process and it was really sucking power but was working.

[mwjames]

However after about 4 Minutes the script continued with the next ID. The second time it happened for this wiki I had a look a the process and it was really sucking power but was working.

Do we know the cause (or for that matter the page that causes such behaviour)?

[kghbln]

Do we know the cause (or for that matter the page that causes such behaviour)?

Unfortunately not since it is a private wiki and I just was happy that things still continued to roll. I expect this to be a very complex template with transclusions, queries and properties. I already told that person to rethink the data model in the past because of similar observations.

[mwjames]

I expect this to be a very complex template with transclusions, queries and properties. I already told that person to rethink the data model in the past because of similar observations.

Just to clarify, rebuildData has not seen any changes, and the ContentParser and InTextAnnotationParser also have not been changed (those parts that are responsible for filtering a [[::]] and replacing it with an annotation).

[kghbln]

I am indeed not blaming SMW in general or rebuildData in special. I am talking about a familypedia style wiki. There is nothing one can do about this kind of "misuse".

[mwjames]

Unfortunately not since it is a private wiki and I just was happy that things still continued to roll.

Can you not isolate the ID in order to run rebuildData again but this time with wgDebugDumpSql enabled which should output all sql activities (MW + SMW) in the logs.

[JeroenDeDauw]

SMW 2.0

Update to 2.1 now and save a server kitteh for free. You want to save the server kittehs!

[jamesmontalvo3]

Apologies, I was originally on SWM 2.1 and had this issue, so I reverted to 2.0 and tried it again and still had the issue. It was present on both 2.0 and 2.1.

@kghbln RE: how did I know that it stopped? I had barely hit "enter" to start the process and it ended. We have way more than pages with semantic properties on them than the ~50 IDs processed, so I knew something was wrong.

@mwjames RE: Do we know the page that caused it? At this point I do not, but I will try to get more information later using the stack trace recommendation you made. Are you saying that if I turn on debug info as stated in [1] that rebuildData.php may provide me with more information?

[1] http://semantic-mediawiki.org/wiki/Help:Identifying_bugs

[mwjames]

Are you saying that if I turn on debug info as stated in [1] that rebuildData.php may provide me with more information?

If you add all those options then yes but you should be careful with wgDebugDumpSql because it will spit out every sql query (delete, insert, update etc.) during the execution and may provide some clues as to what is happening and where (you should limit the ID's while wgDebugDumpSql is enabled) .

[jamesmontalvo3]

I ran the following:

Z:\wiki\extensions\SemanticMediaWiki\maintenance>php rebuildData.php -v -s 58

Which resulted in the following:

Selected storage SMWSQLStore3 for update!

Refreshing all semantic data in the database!
---
 Some versions of PHP suffer from memory leaks in long-running scripts.
 If your machine gets very slow after many pages (typically more than
 1000) were refreshed, please abort with CTRL-C and resume this script
 at the last processed page id using the parameter -s (use -v to display
 page ids during refresh). Continue this until all pages were refreshed.
---
Processing all IDs from 58 to last ID ...
(1) Processing ID 58 ...
(2) Processing ID 59 ...

Fatal error: Call to a member function getText() on a non-object in Z:\wiki\extensions\ExternalData\ED_ParserFunctions.php on line 208

Call Stack:
    0.0027     157280   1. {main}() Z:\wiki\extensions\SemanticMediaWiki\maintenance\rebuildData.php:0
    0.0242     370816   2. require_once('Z:\wiki\maintenance\doMaintenance.php') Z:\wiki\extensions\SemanticMediaWiki\maintenance\rebuildData.php:135
    2.4196    9357832   3. SMW\Maintenance\RebuildData->execute() Z:\wiki\maintenance\doMaintenance.php:104
    2.4602    9669232   4. SMW\Store\Maintenance\DataRebuilder->rebuild() Z:\wiki\extensions\SemanticMediaWiki\maintenance\rebuildData.php:115
    2.4603    9669488   5. SMW\Store\Maintenance\DataRebuilder->rebuildAll() Z:\wiki\extensions\SemanticMediaWiki\includes\src\Store\Maintenance\DataRebuilder.php:144
    8.6621   24349528   6. SMWSQLStore3->refreshData() Z:\wiki\extensions\SemanticMediaWiki\includes\src\Store\Maintenance\DataRebuilder.php:212
    8.6621   24349528   7. SMWSQLStore3SetupHandlers->refreshData() Z:\wiki\extensions\SemanticMediaWiki\includes\storage\SQLStore\SMW_SQLStore3.php:415
    8.6653   24356088   8. SMW\MediaWiki\Jobs\UpdateJob->run() Z:\wiki\extensions\SemanticMediaWiki\includes\storage\SQLStore\SMW_SQLStore3_SetupHandlers.php:402
    8.6654   24299952   9. SMW\MediaWiki\Jobs\UpdateJob->doUpdate() Z:\wiki\extensions\SemanticMediaWiki\includes\src\MediaWiki\Jobs\UpdateJob.php:57
    8.6655   24299952  10. SMW\MediaWiki\Jobs\UpdateJob->doParseContentForData() Z:\wiki\extensions\SemanticMediaWiki\includes\src\MediaWiki\Jobs\UpdateJob.php:81
    8.6655   24300216  11. SMW\ContentParser->parse() Z:\wiki\extensions\SemanticMediaWiki\includes\src\MediaWiki\Jobs\UpdateJob.php:93
    8.6655   24300248  12. SMW\ContentParser->fetchFromParser() Z:\wiki\extensions\SemanticMediaWiki\includes\ContentParser.php:129
    8.6721   24323736  13. Parser->parse() Z:\wiki\extensions\SemanticMediaWiki\includes\ContentParser.php:189
    8.6737   24239256  14. Parser->internalParse() Z:\wiki\includes\parser\Parser.php:395
    8.6738   24243832  15. Parser->replaceVariables() Z:\wiki\includes\parser\Parser.php:1216
    8.6875   24244824  16. PPFrame_DOM->expand() Z:\wiki\includes\parser\Parser.php:3153
    8.6876   24248096  17. Parser->braceSubstitution() Z:\wiki\includes\parser\Preprocessor_DOM.php:1113
    8.6877   24249744  18. Parser->callParserFunction() Z:\wiki\includes\parser\Parser.php:3336
    8.6955   24304088  19. call_user_func_array:{Z:\wiki\includes\parser\Parser.php:3620}() Z:\wiki\includes\parser\Parser.php:3620
    8.6955   24304528  20. EDParserFunctions::doGetDBData() Z:\wiki\includes\parser\Parser.php:3620

Evidently I was on an old version of ExternalData (1.3.4), since upgrading to the latest (1.8.1) removed the issue. Apologies for wasting peoples' time with a non-SMW issue. Thanks for teaching me how to debug a maintenance script.

[jamesmontalvo3]

Oh, also, when I reverted from SMW 2.1 to 2.0 to see if that caused the issue, I initially got the following error:

PHP Fatal error:  Interface 'SMW\ContextAware' not found in D:\inetpub\wwwroot\Wiki\EVA\extensions\SemanticMediaWiki\includes\Setup.php on line 39

To fix this I had to run:

composer dump-autoload --optimize

Perhaps that should be documented somewhere.

[mwjames]

Apologies for wasting peoples' time with a non-SMW issue. Thanks for teaching me how to debug a maintenance script.

You found a solution to a problem (and I hope the documentation provided on smw.org was helpful) that's all that matters.

Perhaps that should be documented somewhere.

Feel free to amend the smw.org@Installation documentation.

[mwjames]

Added --debug (SMW 2.2) to the rebuildData.php which should allow for a more elaborated debug output without the need to adjust the LocalSettings.php.

[jamesmontalvo3]

Is there a way to determine how many IDs there are to process before processing? Perhaps the script should print something like "Beginning processing of 13035 IDs" so you know if it ends at 3000 that something went wrong. Even better would be if it could catch that something caused it to end prematurely and it said "Aborting script at ID 3000 for unknown reason. Run with --debug for more info and file a bug at https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues if required"

[mwjames]

Is there a way to determine how many IDs there are to process before processing? Perhaps the script should print something like "Beginning processing of 13035 IDs" so you know if it ends at 3000 that

Moving discussion to #778