Optimizing menu item parent calculation #8863

Hackwar · 2016-01-08T20:30:29Z

mod_menu checks for every menu item if the item is a parent of another menu item. The method to do that via $menu->getItems() is pretty expensive and can create great delays on bigger sites. The proposed code will reduce this immensely by only checking those menu items that can have a child due to lft and rgt differing by more than one. Unfortunately that will not automatically mean that a menu item actually has children, since children could be present, but unpublished or in a different language. In those cases lft and rgt differ, but there is no child item anyway.

This will not change existing sites, but will mean a performance improvement. So the best way to test this would be to compare execution times on sites with a lot of menu items (>500). Another way would be a code review by a maintainer. 😄

This came up because of the insistence of @volandku in the Joomla IRC channel on Freenode who made me look into this. Thank you for being persistent. 😄

andrepereiradasilva · 2016-01-09T12:30:50Z

I have tested this item 🔴 unsuccessfully on 51de1eb

I get this php notices:

Notice: Undefined property: stdClass::$rgt in /path/to/joomla-cms/modules/mod_menu/helper.php on line 77
Notice: Undefined property: stdClass::$lft in /path/to/joomla-cms/modules/mod_menu/helper.php on line 77

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/8863.}

andrepereiradasilva · 2016-01-09T13:05:26Z

@Hackwar, instead of doing this:

foreach ($items as $i => $item)
{
    /* Code removed for simplifying */
    $item->deeper     = false;
    $item->shallower  = false;
    $item->level_diff = 0;

    if (isset($items[$lastitem]))
    {
        $items[$lastitem]->deeper     = ($item->level > $items[$lastitem]->level);
        $items[$lastitem]->shallower  = ($item->level < $items[$lastitem]->level);
        $items[$lastitem]->level_diff = ($items[$lastitem]->level - $item->level);
    }

    $item->parent = (boolean) ($item->rgt > $item->lft + 1) && $menu->getItems('parent_id', (int) $item->id, true);
    /* Code removed for simplifying */
}

if (isset($items[$lastitem]))
{
    $items[$lastitem]->deeper     = (($start?$start:1) > $items[$lastitem]->level);
    $items[$lastitem]->shallower  = (($start?$start:1) < $items[$lastitem]->level);
    $items[$lastitem]->level_diff = ($items[$lastitem]->level - ($start?$start:1));
}

Can't we do something like this?:

foreach ($items as $i => $item)
{
    /* Code removed for simplifying */
    $item->deeper     = false;
    $item->shallower  = false;
    $item->level_diff = 0;
    $item->parent     = 0;

    if (isset($items[$lastitem]))
    {
        $items[$lastitem]->deeper     = ($item->level > $items[$lastitem]->level);
        $items[$lastitem]->shallower  = ($item->level < $items[$lastitem]->level);
        $items[$lastitem]->level_diff = ($items[$lastitem]->level - $item->level);
        $items[$lastitem]->parent     = ($items[$lastitem]->level_diff == -1);
    }
    /* Code removed for simplifying */
}

if (isset($items[$lastitem]))
{
    $items[$lastitem]->deeper     = (($start?$start:1) > $items[$lastitem]->level);
    $items[$lastitem]->shallower  = (($start?$start:1) < $items[$lastitem]->level);
    $items[$lastitem]->level_diff = ($items[$lastitem]->level - ($start?$start:1));
    $items[$lastitem]->parent     = ($items[$lastitem]->level_diff == -1);
}

In other words, can't we just don't use the getItems to calculate if current menu item is a parent or not and use the already calculated level_diff to do that.

If we can do that we get even further performance improvements.

I run a test with a menu with several levels (thank for batch processing :)) and the for cycle got from:

25 published items: ~6.5 ms to ~4.7 ms.
50 published items: ~15 ms to ~9 ms.
100 published items: ~30 ms to ~19 ms.
200 published items: ~78 ms to ~38 ms.
400 published items: ~232 ms to ~76 ms.
800 published items: ~730 ms to ~155 ms.
and so on ...

joomla-cms-bot · 2016-01-09T21:39:19Z

This PR has received new commits.

CC: @andrepereiradasilva

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/8863.}

Hackwar · 2016-01-09T21:40:31Z

First of all, sorry for the broken code. Yes, you are right, your proposal is even better. I added your code. I'm setting the parent to false by default to have it set at all times.

andrepereiradasilva · 2016-01-09T21:57:07Z

I have tested this item ✅ successfully on df6acd2

Thank you for find that needed performance optimization.
This can have some performance impact on sites with a lot of menu items.

Correct me if i'm wrong but the last element can't be the parent of anything, since it's the last.
That is why you didn't put the $items[$lastitem]->parent = ($items[$lastitem]->level_diff == -1); in the setting of last item after the for cycle as i suggested.
Makes sense.

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/8863.}

HLeithner · 2016-01-21T09:13:29Z

I have tested this PR 🔴 unsuccessfully on df6acd2.

Simple test, I take default installation added 5 Menu Items:

Item 1
- Item 1a
Item 2
- Item 2a
  - Item 2b

Module with "Show Sub-menu Items" off shows on active "Item 1" a difference at the last item, in this case "Item 2".

Without patch: parent = true
With patch: parent = false

If "Show Sub-menu Items" is on the "Item 2" parent value is correct (true).

Also if "Show Sub-menu Items" is on the the complete testcase is correct.

andrepereiradasilva · 2016-01-26T10:54:50Z

@EFERIS we know the last item parent value can be not ok. See my comment when i tested.

Correct me if i'm wrong but the last element can't be the parent of anything, since it's the last.
That is why you didn't put the $items[$lastitem]->parent = ($items[$lastitem]->level_diff == -1); in the setting of last item after the for cycle as i suggested.
Makes sense.

But in what that affected your test? Did the menu not appeared correctly?

HLeithner · 2016-01-26T11:24:54Z

@andrepereiradasilva sure because the li-tag for the last element doesn't get the "parent" class, that could lead to a visual problem if you use this class (the only way) to mark the link as parent item.

andrepereiradasilva · 2016-01-26T11:27:02Z

Ok, let's wait for @Hackwar to solve that.

joomla-cms-bot · 2016-01-27T11:14:13Z

This PR has received new commits.

CC: @andrepereiradasilva

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/8863.}

joomla-cms-bot · 2016-01-27T11:15:33Z

This PR has received new commits.

CC: @andrepereiradasilva

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/8863.}

Hackwar · 2016-01-27T11:17:52Z

Please test this again. This should solve the previous issues.

HLeithner · 2016-01-28T09:07:19Z

Hi,

while testing this I think I came to a better solution.

Add this to the beginning of the foreach loop before filtering the unseen items.

$item->parent = false;
if (isset($items[$lastitem])
  && $items[$lastitem]->id == $item->parent_id)
{
  $items[$lastitem]->parent = true;
}

btw. is it normal that the joomla patch tester overrides the complete file with a incompatible version?

andrepereiradasilva · 2016-01-28T09:20:35Z

I think the prior solution (before revert) works, if you add this line inside the if after the for cycle (https://github.com/Hackwar/joomla-cms/blob/df6acd259cb3062ad0ec1b1499f72aa6cea3e10e/modules/mod_menu/helper.php#-L126-L131) to set the last parent:

$items[$lastitem]->parent     = ($items[$lastitem]->level_diff == -1);

HLeithner · 2016-01-28T09:44:32Z

@andrepereiradasilva I tested this first, but no that will not work because we don't have the child at this point. if you can please try my solution.

andrepereiradasilva · 2016-01-28T13:45:40Z

ok i will check. thanks

andrepereiradasilva · 2016-01-28T14:18:08Z

@EFERIS i tried you solution and it worked. And without the getItems. Good work.

How did i test:

Use the current php file (before this PR) and removed this line: https://github.com/joomla/joomla-cms/blob/staging/modules/mod_menu/helper.php#L85.
As you said, put your code in this line https://github.com/joomla/joomla-cms/blob/staging/modules/mod_menu/helper.php#L56, so it becomes

foreach ($items as $i => $item)
{
    $item->parent = false;
    if (isset($items[$lastitem]) && $items[$lastitem]->id == $item->parent_id)
    {
        $items[$lastitem]->parent = true;
    }
    // code removed for brevity
}

@Hackwar can you check? With @EFERIS solution i think we can get rid of getItems and with that improving performance even further.

brianteeman · 2016-04-12T12:30:52Z

@Hackwar any comment on the proposed improvement from @EFERIS would be good to move this PR forward if we can

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/8863.}

roland-d · 2016-06-24T21:35:36Z

@Hackwar Another nudge so we can move this PR forward.

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/8863.}

Hackwar · 2016-06-25T16:03:40Z

I don't know which one is @EFERIS. So far this PR is complete the way it is.

HLeithner · 2016-06-27T09:48:51Z

@Hackwar sorry I renamed the github Username

Instead of trying to get the ->parent value based on the level_diff its better to set it at the beginning before we delete the information.

Add the following code to the beginning of the foreach loop at Line 57, see comment from @andrepereiradasilva

    $item->parent = false;
    if (isset($items[$lastitem]) && $items[$lastitem]->id == $item->parent_id)
    {
        $items[$lastitem]->parent = true;
    }`

joomla-cms-bot · 2016-06-28T10:07:49Z

This PR has received new commits.

CC: @andrepereiradasilva

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/8863.}

Hackwar · 2016-06-28T10:11:09Z

I've added the change like you proposed @HLeithner.

joomla-cms-bot · 2016-06-28T10:11:16Z

This PR has received new commits.

CC: @andrepereiradasilva

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/8863.}

HLeithner · 2016-06-28T12:36:58Z

One thing, adding ", m.lft, m.rgt" to the query is no longer necessary.

roland-d · 2016-06-29T06:01:26Z

modules/mod_menu/helper.php

@@ -54,6 +54,12 @@ public static function getList(&$params)
 			{
 				foreach ($items as $i => $item)
 				{
+					$item->parent = false;
+					if (isset($items[$lastitem]) && $items[$lastitem]->id == $item->parent_id)


CS: We need an empty line before the if statement.

joomla-cms-bot · 2016-06-29T06:52:34Z

This PR has received new commits.

CC: @andrepereiradasilva

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/8863.}

andrepereiradasilva · 2016-06-29T18:43:56Z

One thing, adding ", m.lft, m.rgt" to the query is no longer necessary.

andrepereiradasilva · 2016-08-23T22:53:09Z

Menu with 500+ items

Before patch

After patch

So aprox. half the time

andrepereiradasilva · 2016-08-23T22:53:13Z

I have tested this item ✅ successfully on f1aa5a6

as comment above

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/8863.}

gwsdesk · 2016-09-06T17:03:00Z

I have tested this item ✅ successfully on c4d5c8c

I have tested this on a stock Joomla distro (3.6.3) will "Learning Sample data" in 4 languages and it has a visible faster loading time compared to pre-patch

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/8863.}

laoneo · 2016-09-07T07:38:35Z

This issue has two successful tests.

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/8863.}

Remove some lines

wilsonge · 2016-10-05T20:20:47Z

RTC

_{This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/8863.}

brianteeman · 2016-10-17T09:22:33Z

Milestone changed to 3.7 as it is now not planned to have a 3.6.4 release

Optimizing parent calculation

51de1eb

joomla-cms-bot added the PR-staging label Jan 8, 2016

Hackwar mentioned this pull request Jan 8, 2016

Optimizing JMenu::getItems() #8864

Merged

Removing the usage of JMenu::getItems() from ModMenuHelper

df6acd2

Reverting to previous solution

2e7f355

Adding lft, rgt to query

f21e205

Optimizing parent calculation as proposed by @HLeitner

9450d16

Merge branch 'staging' into patch-1

8ee65b3

roland-d reviewed Jun 29, 2016
View reviewed changes

Codestyle

f1aa5a6

roland-d mentioned this pull request Jul 3, 2016

joomla backend get slow with a lot of menu items #10997

Closed

Merge branch 'staging' into patch-1

c4d5c8c

wilsonge and others added 2 commits October 5, 2016 21:19

Remove some lines

6f304fe

Merge pull request #21 from wilsonge/patch-5

5d0a513

Remove some lines

joomla-cms-bot added the RTC This Pull Request is Ready To Commit label Oct 5, 2016

wilsonge added this to the Joomla! 3.6.4 milestone Oct 5, 2016

wilsonge added RTC This Pull Request is Ready To Commit and removed RTC This Pull Request is Ready To Commit labels Oct 5, 2016

brianteeman modified the milestones: Joomla 3.7.0, Joomla! 3.6.4 Oct 17, 2016

rdeutz merged commit 5b0d2cf into joomla:staging Oct 18, 2016

brianteeman removed the RTC This Pull Request is Ready To Commit label Oct 18, 2016

Hackwar deleted the patch-1 branch April 27, 2019 07:53

Optimizing menu item parent calculation #8863

Optimizing menu item parent calculation #8863

Hackwar commented Jan 8, 2016 •

edited by joomla-cms-bot

andrepereiradasilva commented Jan 9, 2016

andrepereiradasilva commented Jan 9, 2016

joomla-cms-bot commented Jan 9, 2016

Hackwar commented Jan 9, 2016

andrepereiradasilva commented Jan 9, 2016

HLeithner commented Jan 21, 2016

andrepereiradasilva commented Jan 26, 2016

HLeithner commented Jan 26, 2016

andrepereiradasilva commented Jan 26, 2016

joomla-cms-bot commented Jan 27, 2016

joomla-cms-bot commented Jan 27, 2016

Hackwar commented Jan 27, 2016

HLeithner commented Jan 28, 2016

andrepereiradasilva commented Jan 28, 2016

HLeithner commented Jan 28, 2016

andrepereiradasilva commented Jan 28, 2016

andrepereiradasilva commented Jan 28, 2016

brianteeman commented Apr 12, 2016

roland-d commented Jun 24, 2016

Hackwar commented Jun 25, 2016

HLeithner commented Jun 27, 2016

joomla-cms-bot commented Jun 28, 2016

Hackwar commented Jun 28, 2016

joomla-cms-bot commented Jun 28, 2016

HLeithner commented Jun 28, 2016

roland-d Jun 29, 2016

joomla-cms-bot commented Jun 29, 2016

andrepereiradasilva commented Jun 29, 2016

andrepereiradasilva commented Aug 23, 2016 •

edited

andrepereiradasilva commented Aug 23, 2016

gwsdesk commented Sep 6, 2016

laoneo commented Sep 7, 2016

wilsonge commented Oct 5, 2016

brianteeman commented Oct 17, 2016

Optimizing menu item parent calculation #8863

Optimizing menu item parent calculation #8863

Conversation

Hackwar commented Jan 8, 2016 • edited by joomla-cms-bot

andrepereiradasilva commented Jan 9, 2016

andrepereiradasilva commented Jan 9, 2016

joomla-cms-bot commented Jan 9, 2016

Hackwar commented Jan 9, 2016

andrepereiradasilva commented Jan 9, 2016

HLeithner commented Jan 21, 2016

andrepereiradasilva commented Jan 26, 2016

HLeithner commented Jan 26, 2016

andrepereiradasilva commented Jan 26, 2016

joomla-cms-bot commented Jan 27, 2016

joomla-cms-bot commented Jan 27, 2016

Hackwar commented Jan 27, 2016

HLeithner commented Jan 28, 2016

andrepereiradasilva commented Jan 28, 2016

HLeithner commented Jan 28, 2016

andrepereiradasilva commented Jan 28, 2016

andrepereiradasilva commented Jan 28, 2016

brianteeman commented Apr 12, 2016

roland-d commented Jun 24, 2016

Hackwar commented Jun 25, 2016

HLeithner commented Jun 27, 2016

joomla-cms-bot commented Jun 28, 2016

Hackwar commented Jun 28, 2016

joomla-cms-bot commented Jun 28, 2016

HLeithner commented Jun 28, 2016

roland-d Jun 29, 2016

Choose a reason for hiding this comment

joomla-cms-bot commented Jun 29, 2016

andrepereiradasilva commented Jun 29, 2016

andrepereiradasilva commented Aug 23, 2016 • edited

andrepereiradasilva commented Aug 23, 2016

gwsdesk commented Sep 6, 2016

laoneo commented Sep 7, 2016

wilsonge commented Oct 5, 2016

brianteeman commented Oct 17, 2016

Hackwar commented Jan 8, 2016 •

edited by joomla-cms-bot

andrepereiradasilva commented Aug 23, 2016 •

edited