Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#20950] JAccess Preload Patches #6728

Merged
merged 7 commits into from May 2, 2016

Conversation

Projects
None yet
10 participants
@orware
Copy link
Contributor

commented Apr 10, 2015

It's a little tricky to describe this one, but it's actually an old issue I reported way back in 2010:
http://joomlacode.org/gf/project/joomla/tracker/?action=TrackerItemEdit&tracker_item_id=20950&start=0

Working with the SQL Optimisation Working Group this week reminded me about it and I decided to take a fresh look at it again and see if I could get it updated for Joomla 3.

So I spent the better part of the last two days going through it and optimizing it quite a bit over what I had previously and I hope the results can speak for themselves (I don't have any larger sites than the example one I've been working with this week which has about 4500 articles).

Issue it Tries to Solve (Mainly: Speed/Efficiency)

Back in 2010, Joomla 1.6 was still in progress I believe but it was close to being complete and so I had started using it to develop the website for the college I work for (I had been waiting for the new ACL system to be able to do that effectively).

One thing that seemed to be missed during the development process of JAccess that I noticed at the time, was that the number of queries / speed ended up dropping dramatically when the user logged into the backend was not a Super Admin...if the user was an Administrator or any other User Group which has its permissions thoroughly checked by JAccess the number of queries and slowdown in speed was fairly apparent. This issue also affects the frontend too.

In that early investigation I found out that the queries used by JAccess were pretty slow and there was quite a bit of repetition in data that was being retrieved from the database so extra time was required when JAccess had to do any sort of significant work.

I ended up crafting a set of additional methods that could be used to preload two sets of data:

  1. Component Rules (for any components a user may have installed). This list is usually fairly small and doesn't grow too much so preloading it is a great idea and saves a lot of queries by itself.
  2. Asset Rules for "Types" for example, if I'm going to be using JAccess to check 'com_content.article.1', 'com_content.article.2', etc. it makes sense to go ahead and load these up in bulk all at once from the database rather than querying for each set of data individually.

I believe the first one is fairly straightforward.

The second one though can be a bit more complicated and is what I ended up spending the bulk of my time on.

In my initial version from 2010, I ended up retrieving all of the assets in bulk and at the same time I went ahead and created the JAccessRule instances for each of them.

Looking at things more closely here in 2015 I could see that approach would definitely cause issues quickly because during my profiling I could see that creating all of the JAccessRule instances up front ate up a lot more memory than I was expecting. So I ended up changing things around a bit and made the JAccessRule loading more just-in-time than the previous approach.

Preloading still takes a bit more memory (particularly with smaller Joomla installations) but I believe overall the zippier response times more than make up for the extra memory that would be used.

What it's not for (Yet) - Super Large Sites (with > 10,000 Content Items for a single extension)

I don't know exactly what the distribution in usage is for Joomla sites (in terms of size).

Preloading should work fairly well for a large majority of sites I feel (particularly any that have a decent amount of content). This would be in the several thousand content items range.

At the moment I'm not sure how this would perform on a super large site with a few million articles.

One of the drawbacks with my current approach is that I'm essentially doing a flat inclusion of all rules for an extension once an individual asset is attempted to be checked in JAccess (so for example, if we try and check "com_content.article.1" this will cause JAccess to load up all asset information for "com_content.article." (all articles) and "com_content.category." (all content categories). For a super large site with a few million articles this is probably not the best approach to take...essentially what we'd need to do is make each extension (or JAccess somehow) aware of exactly all of the items that are going to be displayed on the current page and allow the preload mechanism to retrieve only the information for just those items from the database in bulk (rather than everything for that particular extension).

However, if I'm correct in assuming that probably the majority of sites have less than a few thousand content items, the current solution will save a whole bunch of processing time for a lot of people.

How to Test

Testing this should be relatively easy.

First I would turn on debugging in the backend (that way you can see some of the profling information and can see the before/after difference for the number of queries executed).

If you don't already have an Administrator user then you'll need to create one (a Super Admin will bypass JAccess completely and you won't notice any difference in performance).

Go ahead and logout and then login as the Administrator user.

First take a look at a few pages in the administration area (below I used the Default Dashboard Page after logging on, the Add New Article screen and the Article List View). With the profiling information you can note the timing, memory and the number of SQL queries that were run and save that somewhere in a text file on your end.

Now with the Patch Tester component installed you would just apply the patch for this Pull Request and then go ahead and run through testing the same pages you had just tested prior to applying the patch.

Note the differences...and hopefully everything is positively affected by the patch :-).

Example Results:

The timing info from the next few sections are from PhpEd's Profiler Tool.

Add a New Article with Current JAccess (with about 1100 categories):

When I had Joomla debugging enabled, initially with all of the example categories I had added in I could no longer create a new article (or category). With debugging turned off the amount of time it took to load the page was still fairly high, but at least it loaded.

Timing Summary: 10.66 sec

Memory Summary: 13.90 MB

SQL Queries: 9.634 sec

Add a New Article with JAccess Preloading (with about 1100 categories):

With preloading you can see it's much faster.

Timing Summary: 772.0 ms

Memory Summary: 15.87 MB

SQL Queries: 79.48 ms

Normal JAccess (Loading 100 Articles in the Backend):

Timing Summary: 1.907 sec

Memory Summary: 10.42 MB

SQL Queries: 1.324 sec

New JAccess with Preloading (Loading Same 100 Articles in the Backend):

This one here illustrates how in the example site I had on my end (with 4500 articles) can use up a little more memory even when only 100 articles are being displayed.

Timing Summary: 1.032 sec

Memory Summary: 16.57 MB

SQL Queries: 281.3 ms

Joomla Profiling Examples:

Default JAccess (Admin Dashboard Page):
Time: 472.6 ms
Memory: 9.50 MB

Database queries total: 171.7 ms
55 Queries Logged 171.7 ms

JAccess with Preloading (Admin Dashboard Page):

Time: 475.7 ms
Memory: 9.27 MB

Database queries total: 58.9 ms
36 Queries Logged 58.9 ms

Default JAccess (Articles List View w/1 Article):

Time: 722.0 ms

Memory: 9.06 MB

Database queries total: 188.5 ms
52 Queries Logged 188.5 ms

JAccess with Preloading (Articles List View w/1 Article):

Time: 378.1 ms
Memory: 8.91 MB

Database queries total: 27.7 ms
34 Queries Logged 27.7 ms

@joomla-cms-bot joomla-cms-bot changed the title JAccess Preload Patches [#20950] JAccess Preload Patches Apr 10, 2015

@orware

This comment has been minimized.

Copy link
Contributor Author

commented Apr 10, 2015

I'll try and fix the issues Travis CI mentioned (mainly spaces being used instead of tabs in my editor it looks like and some other code style type issues). Though I believe anyone wanting to test it out should be able to in the meantime.

orware added some commits Apr 10, 2015

Added additional base case to automatically break out of the while lo…
…op in iterative version of getAssetAncestors(). An infinite loop would have been possible if a non-existent asset ID happened to be passed in (wouldn't be a likely scenario to occur, but still good to prevent it).
@orware

This comment has been minimized.

Copy link
Contributor Author

commented Apr 11, 2015

All the Travis CI Issues have been fixed at this point...it would be nice to start having a few testers try it out and see what the feedback is like ;-).

Thanks!

@zero-24

This comment has been minimized.

Copy link
Contributor

commented Apr 11, 2015

@orware i have just send you a CS PR at https://github.com/orware/joomla-cms/pull/1 I will try to test if i get some time

@JoshCalico

This comment has been minimized.

Copy link

commented Nov 6, 2015

I'm willing to test this. Is there still an active interest in this PR?


This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/6728.

@orware

This comment has been minimized.

Copy link
Contributor Author

commented Nov 6, 2015

Hi Joshua,

If you're interested in testing it that would be great! I didn't really hear a lot of feedback after my initial work back in April, but it would be nice to get some more testers in and see if they notice the same sorts of performance improvements I've noticed with this sort of approach (it's not perfect of course, but I think for the sites that most people have, it will result in a noticeable performance improvement).

I would definitely be interested to hear how it goes for you!

-Omar

@orware

This comment has been minimized.

Copy link
Contributor Author

commented Dec 1, 2015

@JoshuaLewis Were you able to try testing this at all on your end?

-Omar


This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/6728.

@JoshCalico

This comment has been minimized.

Copy link

commented Dec 2, 2015

Will be super busy until mid Friday (big project). I'll bookmark this PR so that I can return to it later this week. :-)


This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/6728.

@peteruoi

This comment has been minimized.

Copy link

commented Dec 5, 2015

I have tested this item successfully on 1c64c35

I did not use a profiling tool. I just tested them on a xamp local environment. I find the results really good . Especially the show 100 articles is 75% faster while there is not bigger consumption in memory.
Admin Dashboard Page (current/patched)

Time: 1144/876 ms
Memory: 11.29/11.24 MB
queries: 77 Queries Logged 189.0 ms/41 Queries Logged 40.0 ms

Add a New Article (60 categories in site) (current/patched)
Time: 1983/1456 ms
Memory: 13,23/12.52 MB
queries: 133 Queries Logged 365.0 ms/44 Queries Logged 40.0 ms

Loading 100 Articles in the Backend
Time: 1639/966 ms
Memory: 14,68/12.97 MB
queries: 182 Queries Logged 512.0 ms/42 Queries Logged 36.0 ms


This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/6728.
@orware

This comment has been minimized.

Copy link
Contributor Author

commented Dec 12, 2015

Thank you @peteruoi for testing and providing your results!

Hopefully @JoshuaLewis will have some time soon to do the same and confirm the improvements a second time :-).

-Omar


This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/6728.

@Gerlof

This comment has been minimized.

Copy link

commented Dec 12, 2015

I have tested this item successfully on 1c64c35

Tested in a live environment, using statistics from Joomla! Debug Console.

  • Logged in as Administrator
  • Control Panel > Content > Articles (ID descending)

Results:

without patch (memory usage/queries/time):

show 25 articles: 15,94MB/96/91.2ms
show 50 articles: 17,16MB/123/88.8ms
show 100 articles: 19,58MB/173/160.6ms

with patch (memory usage/queries/time):

show 25 articles: 15,0MB/39/71.7ms
show 50 articles: 15,66MB/40/46.9ms
show 100 articles: 16,72MB/40/50.3ms

The performance gain is impressive!


This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/6728.
@orware

This comment has been minimized.

Copy link
Contributor Author

commented Dec 12, 2015

Thank you @Gerlof for testing and sharing your results!


This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/6728.

@brianteeman

This comment has been minimized.

Copy link
Contributor

commented Jan 27, 2016

RTC - thanks


This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/6728.

@joomla-cms-bot joomla-cms-bot added the RTC label Jan 27, 2016

@ssnobben

This comment has been minimized.

Copy link

commented Feb 26, 2016

Will this be merged?

@wilsonge wilsonge added this to the Joomla! 3.6.0 milestone Apr 12, 2016

@rdeutz rdeutz merged commit 4ba8d17 into joomla:staging May 2, 2016

2 checks passed

JTracker/HumanTestResults Human Test Results: 2 Successful 0 Failed.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@joomla-cms-bot joomla-cms-bot removed the RTC label May 2, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.