Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rebuildData.php prematurely stops without error message #766

Closed
jamesmontalvo3 opened this Issue Jan 31, 2015 · 16 comments

Comments

Projects
None yet
4 participants
@jamesmontalvo3
Copy link
Contributor

jamesmontalvo3 commented Jan 31, 2015

I just upgraded to SMW 2.1. After update I ran rebuildData.php, but the script stopped without giving any error after printing the line "(49) Processing ID 59 ...". I have more than 10000 pages in the wiki, so this is very prematurely ending.

Next I ran "php rebuildData.php -v -s 51" since I wasn't sure if the -s option meant the number in the parenthetical (e.g. 49) or the ID (e.g. 59). Again it stopped prematurely and without error after printing the line "(9) Processing ID 59 ...".

So next I ran "php rebuildData.php -v -s 60" and it appears to be running well. When it ends I'm not sure how I'll know if it is really done or if it simply failed silently again, though. Is there a way to turn on a debug mode to give me more insight into whatever error occurred?

Environtment details:

  • SMW 2.0 and 2.1
  • MW 1.23.2
  • PHP 5.4.16
  • MySQL 5.6.20
  • Windows Server 2008 R2 with IIS 7.5
@mwjames

This comment has been minimized.

Copy link
Contributor

mwjames commented Jan 31, 2015

Is there a way to turn on a debug mode to give me more insight into whatever error occurred?

There is no real debug mode, the only debug that you can have is provided by MW's maintenance base class which makes use of the stack trace options (see smw.org@stack trace).

@kghbln

This comment has been minimized.

Copy link
Member

kghbln commented Jan 31, 2015

How do you know that it stopped? As coincidence has it I also just upgrade several wikis to SMW 2.1 and for one wiki I thought there may be a problem too since nothing seemed to happen for quite a while. However after about 4 Minutes the script continued with the next ID. The second time it happened for this wiki I had a look a the process and it was really sucking power but was working.

@mwjames

This comment has been minimized.

Copy link
Contributor

mwjames commented Jan 31, 2015

However after about 4 Minutes the script continued with the next ID. The second time it happened for this wiki I had a look a the process and it was really sucking power but was working.

Do we know the cause (or for that matter the page that causes such behaviour)?

@kghbln

This comment has been minimized.

Copy link
Member

kghbln commented Jan 31, 2015

Do we know the cause (or for that matter the page that causes such behaviour)?

Unfortunately not since it is a private wiki and I just was happy that things still continued to roll. I expect this to be a very complex template with transclusions, queries and properties. I already told that person to rethink the data model in the past because of similar observations.

@mwjames

This comment has been minimized.

Copy link
Contributor

mwjames commented Jan 31, 2015

I expect this to be a very complex template with transclusions, queries and properties. I already told that person to rethink the data model in the past because of similar observations.

Just to clarify, rebuildData has not seen any changes, and the ContentParser and InTextAnnotationParser also have not been changed (those parts that are responsible for filtering a [[::]] and replacing it with an annotation).

@kghbln

This comment has been minimized.

Copy link
Member

kghbln commented Jan 31, 2015

I am indeed not blaming SMW in general or rebuildData in special. I am talking about a familypedia style wiki. There is nothing one can do about this kind of "misuse".

@mwjames

This comment has been minimized.

Copy link
Contributor

mwjames commented Jan 31, 2015

Unfortunately not since it is a private wiki and I just was happy that things still continued to roll.

Can you not isolate the ID in order to run rebuildData again but this time with wgDebugDumpSql enabled which should output all sql activities (MW + SMW) in the logs.

@JeroenDeDauw JeroenDeDauw added the bug label Jan 31, 2015

@JeroenDeDauw

This comment has been minimized.

Copy link
Member

JeroenDeDauw commented Jan 31, 2015

SMW 2.0

Update to 2.1 now and save a server kitteh for free. You want to save the server kittehs!

kitteh

@jamesmontalvo3

This comment has been minimized.

Copy link
Contributor Author

jamesmontalvo3 commented Jan 31, 2015

Apologies, I was originally on SWM 2.1 and had this issue, so I reverted to 2.0 and tried it again and still had the issue. It was present on both 2.0 and 2.1.

@kghbln RE: how did I know that it stopped? I had barely hit "enter" to start the process and it ended. We have way more than pages with semantic properties on them than the ~50 IDs processed, so I knew something was wrong.

@mwjames RE: Do we know the page that caused it? At this point I do not, but I will try to get more information later using the stack trace recommendation you made. Are you saying that if I turn on debug info as stated in [1] that rebuildData.php may provide me with more information?

[1] http://semantic-mediawiki.org/wiki/Help:Identifying_bugs

@mwjames

This comment has been minimized.

Copy link
Contributor

mwjames commented Jan 31, 2015

Are you saying that if I turn on debug info as stated in [1] that rebuildData.php may provide me with more information?

If you add all those options then yes but you should be careful with wgDebugDumpSql because it will spit out every sql query (delete, insert, update etc.) during the execution and may provide some clues as to what is happening and where (you should limit the ID's while wgDebugDumpSql is enabled) .

@jamesmontalvo3

This comment has been minimized.

Copy link
Contributor Author

jamesmontalvo3 commented Feb 1, 2015

I ran the following:

Z:\wiki\extensions\SemanticMediaWiki\maintenance>php rebuildData.php -v -s 58

Which resulted in the following:

Selected storage SMWSQLStore3 for update!

Refreshing all semantic data in the database!
---
 Some versions of PHP suffer from memory leaks in long-running scripts.
 If your machine gets very slow after many pages (typically more than
 1000) were refreshed, please abort with CTRL-C and resume this script
 at the last processed page id using the parameter -s (use -v to display
 page ids during refresh). Continue this until all pages were refreshed.
---
Processing all IDs from 58 to last ID ...
(1) Processing ID 58 ...
(2) Processing ID 59 ...

Fatal error: Call to a member function getText() on a non-object in Z:\wiki\extensions\ExternalData\ED_ParserFunctions.php on line 208

Call Stack:
    0.0027     157280   1. {main}() Z:\wiki\extensions\SemanticMediaWiki\maintenance\rebuildData.php:0
    0.0242     370816   2. require_once('Z:\wiki\maintenance\doMaintenance.php') Z:\wiki\extensions\SemanticMediaWiki\maintenance\rebuildData.php:135
    2.4196    9357832   3. SMW\Maintenance\RebuildData->execute() Z:\wiki\maintenance\doMaintenance.php:104
    2.4602    9669232   4. SMW\Store\Maintenance\DataRebuilder->rebuild() Z:\wiki\extensions\SemanticMediaWiki\maintenance\rebuildData.php:115
    2.4603    9669488   5. SMW\Store\Maintenance\DataRebuilder->rebuildAll() Z:\wiki\extensions\SemanticMediaWiki\includes\src\Store\Maintenance\DataRebuilder.php:144
    8.6621   24349528   6. SMWSQLStore3->refreshData() Z:\wiki\extensions\SemanticMediaWiki\includes\src\Store\Maintenance\DataRebuilder.php:212
    8.6621   24349528   7. SMWSQLStore3SetupHandlers->refreshData() Z:\wiki\extensions\SemanticMediaWiki\includes\storage\SQLStore\SMW_SQLStore3.php:415
    8.6653   24356088   8. SMW\MediaWiki\Jobs\UpdateJob->run() Z:\wiki\extensions\SemanticMediaWiki\includes\storage\SQLStore\SMW_SQLStore3_SetupHandlers.php:402
    8.6654   24299952   9. SMW\MediaWiki\Jobs\UpdateJob->doUpdate() Z:\wiki\extensions\SemanticMediaWiki\includes\src\MediaWiki\Jobs\UpdateJob.php:57
    8.6655   24299952  10. SMW\MediaWiki\Jobs\UpdateJob->doParseContentForData() Z:\wiki\extensions\SemanticMediaWiki\includes\src\MediaWiki\Jobs\UpdateJob.php:81
    8.6655   24300216  11. SMW\ContentParser->parse() Z:\wiki\extensions\SemanticMediaWiki\includes\src\MediaWiki\Jobs\UpdateJob.php:93
    8.6655   24300248  12. SMW\ContentParser->fetchFromParser() Z:\wiki\extensions\SemanticMediaWiki\includes\ContentParser.php:129
    8.6721   24323736  13. Parser->parse() Z:\wiki\extensions\SemanticMediaWiki\includes\ContentParser.php:189
    8.6737   24239256  14. Parser->internalParse() Z:\wiki\includes\parser\Parser.php:395
    8.6738   24243832  15. Parser->replaceVariables() Z:\wiki\includes\parser\Parser.php:1216
    8.6875   24244824  16. PPFrame_DOM->expand() Z:\wiki\includes\parser\Parser.php:3153
    8.6876   24248096  17. Parser->braceSubstitution() Z:\wiki\includes\parser\Preprocessor_DOM.php:1113
    8.6877   24249744  18. Parser->callParserFunction() Z:\wiki\includes\parser\Parser.php:3336
    8.6955   24304088  19. call_user_func_array:{Z:\wiki\includes\parser\Parser.php:3620}() Z:\wiki\includes\parser\Parser.php:3620
    8.6955   24304528  20. EDParserFunctions::doGetDBData() Z:\wiki\includes\parser\Parser.php:3620

Evidently I was on an old version of ExternalData (1.3.4), since upgrading to the latest (1.8.1) removed the issue. Apologies for wasting peoples' time with a non-SMW issue. Thanks for teaching me how to debug a maintenance script.

@jamesmontalvo3

This comment has been minimized.

Copy link
Contributor Author

jamesmontalvo3 commented Feb 1, 2015

Oh, also, when I reverted from SMW 2.1 to 2.0 to see if that caused the issue, I initially got the following error:

PHP Fatal error:  Interface 'SMW\ContextAware' not found in D:\inetpub\wwwroot\Wiki\EVA\extensions\SemanticMediaWiki\includes\Setup.php on line 39

To fix this I had to run:

composer dump-autoload --optimize

Perhaps that should be documented somewhere.

@mwjames mwjames added analysis discussion and removed bug labels Feb 1, 2015

@mwjames

This comment has been minimized.

Copy link
Contributor

mwjames commented Feb 1, 2015

Apologies for wasting peoples' time with a non-SMW issue. Thanks for teaching me how to debug a maintenance script.

You found a solution to a problem (and I hope the documentation provided on smw.org was helpful) that's all that matters.

Perhaps that should be documented somewhere.

Feel free to amend the smw.org@Installation documentation.

@mwjames mwjames closed this Feb 1, 2015

mwjames added a commit that referenced this issue Feb 1, 2015

Merge pull request #770 from SemanticMediaWiki/rebuilddata-script
rebuildData add --no-cache/--debug option, refs #749, #766
@mwjames

This comment has been minimized.

Copy link
Contributor

mwjames commented Feb 1, 2015

Added --debug (SMW 2.2) to the rebuildData.php which should allow for a more elaborated debug output without the need to adjust the LocalSettings.php.

@jamesmontalvo3

This comment has been minimized.

Copy link
Contributor Author

jamesmontalvo3 commented Feb 1, 2015

Is there a way to determine how many IDs there are to process before processing? Perhaps the script should print something like "Beginning processing of 13035 IDs" so you know if it ends at 3000 that something went wrong. Even better would be if it could catch that something caused it to end prematurely and it said "Aborting script at ID 3000 for unknown reason. Run with --debug for more info and file a bug at https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues if required"

@mwjames

This comment has been minimized.

Copy link
Contributor

mwjames commented Feb 1, 2015

Is there a way to determine how many IDs there are to process before processing? Perhaps the script should print something like "Beginning processing of 13035 IDs" so you know if it ends at 3000 that

Moving discussion to #778

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.