SQLStore / rebuildData + refreshData memory leaks #480

Closed
mwjames opened this Issue Aug 4, 2014 · 3 comments

Projects

None yet

2 participants

@mwjames

Email from Stephan where he wrote:

Well, apart from that pesky memory leak issue [1] causing rebuildData.php to bail out every few thousand pages.

@mwjames

When running a test refresh [0] I have observed that only when the Parser actively is re-parsing content of a page the memory consumption peaks without the garbage collector freeing the memory after the process and therefore remains at the level of the last invocation. Over the time when running a stack of re-parses the memory usage is accumulated without a release which will eventually lead to an overflow.

[0] php rebuildData -v --report-runtime; 5674 IDs refreshed; Memory used: 33810352 (b: 7238112, a: 41048464) with a runtime of 2347.31144 sec

EDIT: As for the method of analyzing this issue, one has to run several rebuild processes with different combinations of s/e ( -s 1 -e 1000 / -s 1 -e 3000 / -s 1 -e 5000 etc.) in order to calculate the standard deviation/variance and see when the memory is exceptional high compared to other runs indicating a specific problem for the combination tested.

@kghbln
Semantic MediaWiki member

My No. 1 from January 2015: 440163 IDs (1141680 values), Memory: 667858080, 33455.35 sec (MW 1.23.8, SMW 2.1.0, PHP 5.4, MySQL 5.5). No issues on the server, did not leave the room though (well, sort of).

@mwjames

Nothing new here, you can try to use --shallow-update but aside from that not much actionable on this as long as the Parser can not household its memory usage.

@mwjames mwjames closed this Mar 12, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment