SQLStore / rebuildData + refreshData memory leaks #480

mwjames opened this Issue Aug 4, 2014 · 3 comments


None yet

2 participants


Email from Stephan where he wrote:

Well, apart from that pesky memory leak issue [1] causing rebuildData.php to bail out every few thousand pages.


When running a test refresh [0] I have observed that only when the Parser actively is re-parsing content of a page the memory consumption peaks without the garbage collector freeing the memory after the process and therefore remains at the level of the last invocation. Over the time when running a stack of re-parses the memory usage is accumulated without a release which will eventually lead to an overflow.

[0] php rebuildData -v --report-runtime; 5674 IDs refreshed; Memory used: 33810352 (b: 7238112, a: 41048464) with a runtime of 2347.31144 sec

EDIT: As for the method of analyzing this issue, one has to run several rebuild processes with different combinations of s/e ( -s 1 -e 1000 / -s 1 -e 3000 / -s 1 -e 5000 etc.) in order to calculate the standard deviation/variance and see when the memory is exceptional high compared to other runs indicating a specific problem for the combination tested.

Semantic MediaWiki member

My No. 1 from January 2015: 440163 IDs (1141680 values), Memory: 667858080, 33455.35 sec (MW 1.23.8, SMW 2.1.0, PHP 5.4, MySQL 5.5). No issues on the server, did not leave the room though (well, sort of).


Nothing new here, you can try to use --shallow-update but aside from that not much actionable on this as long as the Parser can not household its memory usage.

@mwjames mwjames closed this Mar 12, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment