Better callForAllUsers memory usage #24327
Conversation
|
By analyzing the blame information on this pull request, we identified @icewind1991, @DeepDiver1975 and @blizzz to be potential reviewers |
nickvergessen
changed the title
| continue; | ||
| } | ||
| $user = $this->getUserObject($uid, $backend, false); | ||
| $return = $callback($user); |
There was a problem hiding this comment.
I'd say there is a unset($user) missing ...
... assuming we iterate once over all users (500k+ e.g.) - we only need each user once - but we should free the memory after calling the callback
There was a problem hiding this comment.
php should gc the old $user instance when the variable is re-asigned
There was a problem hiding this comment.
is there a way to force gc on php?
I know from other systems that gc cycles need some idle time to be invoked - no idea if this applies to php as well
There was a problem hiding this comment.
Not related to this pr but useful knowledge for any php memory manipulations:
The most reliable way I have found to have php cleanup it's memory is to have a variable go out of scope (i.e. move the memory consuming task to a new function so all newly allocated objects go out of scope the moment the task is done)
Re-assigning a variable or unsetting it all seem to only work if the gc feels like it
There was a problem hiding this comment.
@DeepDiver1975 an unset($user) here wont help. I think @nickvergessen is on the spot - the problem is $this->cachedUsers[$uid] = new User($uid, $backend, $this, $this->config); I like @icewind1991 s proposal to use a CappedMemoryCache.
For what it is worth, gc_collect_cycles() should run the garbage collector. It won't free the $this->cachedUsers[$uid] however.
There was a problem hiding this comment.
For what it is worth, gc_collect_cycles() should run the garbage collector. It won't free the $this->cachedUsers[$uid] however.
sure - those are all cached and as a result will no gc'd 🙈
|
A better approach might be to replace the that way it also fixes the memory usage if any other situation where create query a lot of user objects |
@PVince81 suggested the same at lunch, but I think in general it's good to know where things like that happen and callForAllUsers should only happen on install/update/background job anyway, so this is my prefered way. |
that's the current use case - but for sure not limited to this ... |
|
So do we want to switch to cappedcache now, or use this for backport and use the cappedcache only in 9.1+ ? |
|
Bump @DeepDiver1975 @PVince81 |
|
Yes - let's move this in 👍 |
|
backport request added @karlitschek |
|
nice. makes total sense. please backport 👍 |
@nickvergessen can I ask you to submit the pr? THX |
|
Backport in #24384 Not sure if we have to (should) force this into 9.0.2 so shortly |
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Should help to reduce the memory usage when callForAllUsers() is called https://github.com/owncloud/enterprise/issues/1298
@DeepDiver1975 @butonic