Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Performance] Category manager / Tag Manager, get all counters via single query, reduce 20, 50, 100 queries (page limit) to 1 #22117

Merged
merged 8 commits into from Nov 1, 2018

Conversation

Projects
None yet
9 participants
@ggppdk
Copy link
Contributor

commented Sep 10, 2018

Pull Request for Issue # .

Summary of Changes

Category manager creates 1 query per category to get the assigned item counts by state
These are used to displays the columns
total published, total unpublished, total archived, total trashed

Testing Instructions

  1. Open 2 Tabs of Category manager and 2 Tabs of Tag manager
  2. Filter the 2 Tabs of Tag manager by "Tag Type" article (click search tools button)
  3. Apply patch
  4. Refresh 2nd TAB of Category manager and 2nd Tab of Tag manager

Both 1st and 2nd tabs should show the same counted items per state


Category managers to test

  • articles, banners, contact, newsfeeds, and for fieldgroups management

Tags manager:

  • repeat test for every listed Tag Type, in the Tag Type Filter (click search tools)

Expected result

1 query to get the total assigned items for the categories shown by current page

Actual result

20, 50, 100 queries to get the assigned items for the categories

Documentation Changes Required

None

@alikon

This comment has been minimized.

Copy link
Contributor

commented Sep 10, 2018

very well done 👍
.....testing right now....

// Get relation counts for all category objects with single query
$query = $db->getQuery(true)
->select('catid, ' . $state_column . ' AS state, count(*) AS count')
->from($db->qn('#__' . $related_tbl))

This comment has been minimized.

Copy link
@csthomas

csthomas Sep 10, 2018

Contributor

Replace $db->qn by $db->quoteName

This comment has been minimized.

Copy link
@alikon

alikon Sep 10, 2018

Contributor
  • use $db->quoteName() where needed
@alikon

This comment has been minimized.

Copy link
Contributor

commented Sep 10, 2018

the same "optimization" should be done on every "category" banner, contact etc....

// Get relation counts for all category objects with single query
$query = $db->getQuery(true)
->select('catid, ' . $state_column . ' AS state, count(*) AS count')

This comment has been minimized.

Copy link
@Quy

Quy Sep 10, 2018

Contributor

Capitalize count(*)

@alikon

This comment has been minimized.

Copy link
Contributor

commented Sep 10, 2018

even better can be done on some high level "category" CLASS/HELPER

@ggppdk ggppdk force-pushed the ggppdk:single_counter_query branch from dae4586 to f10fb8b Sep 11, 2018

@ggppdk

This comment has been minimized.

Copy link
Contributor Author

commented Sep 11, 2018

even better can be done on some high level "category" CLASS/HELPER

yes that was my thought too
that is why those local variables in the first commit for the columns names, that seemed redundant ...

I intended to replace countItems() method everywhere, but i see that countTagItems() has the same problem of 20, 50, 100 queries instead of 1

so i have added a countRelations() method to parent helper class that replace both methods,
actually the methods are not replaced instead they call the parent class method like this
parent::countRelations($items, ...,$config);

@ggppdk ggppdk changed the title [Performance] Category manager, get all counters via single query, reduce 20, 50, 100 queries (page limit) to 1 [Performance] Category manager / Tag Manager, get all counters via single query, reduce 20, 50, 100 queries (page limit) to 1 Sep 11, 2018

@alikon

This comment has been minimized.

Copy link
Contributor

commented Sep 11, 2018

countTagItems() you can accuse me for that 😨

// The relation query does not return a value for cases without relations of a particular state / condition, set zero as default
foreach ($items as $item)
{
foreach($counter_names as $n)

This comment has been minimized.

Copy link
@Quy

Quy Sep 11, 2018

Contributor

Add space before (

This comment has been minimized.

Copy link
@ggppdk

ggppdk Sep 11, 2018

Author Contributor

Thanks, done this and other CS errors

@HLeithner

This comment has been minimized.

Copy link
Member

commented Sep 11, 2018

We have a PR for such a function for j4 #21667 I will rework it to do the same optimization as this PR. It's really need I ask my self why I didn't do this.

Atm I wait for #20693 to get merged to get the direction how to solve the component naming issue.

@ggppdk

This comment has been minimized.

Copy link
Contributor Author

commented Sep 11, 2018

That PR has a completely different purpose than PR #21667,
PR #21667 tries to make a reusable method in a trait

Furthermore this PR not only replaces multiple queries for category assignments with 1 query (this is the purpose of this PR)
but this PR also fixes same issue with Tag counting which PR #21667 does not even touch

A note
the code here is agnostic in regards to which state / condition exist
thus it can be rather easily ported to J4

@ggppdk

This comment has been minimized.

Copy link
Contributor Author

commented Sep 11, 2018

Of course if you would want to do same optimization inside PR
#21667 for J4 then please do, as i lack time to do more work now
thanks

*
* @return stdClass[]
*
* @since 3.6

This comment has been minimized.

Copy link
@Quy

Quy Sep 11, 2018

Contributor

Change to __DEPLOY_VERSION__

This comment has been minimized.

Copy link
@ggppdk

ggppdk Sep 11, 2018

Author Contributor

thanks, lol, i remember adding DEPLOY_VERSION for 2 methods,
but then the 2 methods became 1 because 80% of their code was the same

@alikon

This comment has been minimized.

Copy link
Contributor

commented Sep 11, 2018

please, can I ask you if can we concentrate our effort on 3.x series for a while !!
if we can improve performance in the current staging....it will'be a real gain
4.x ..... it will come later......

@ggppdk

This comment has been minimized.

Copy link
Contributor Author

commented Sep 11, 2018

please, can I ask you if can we concentrate our effort on 3.x series for a while !!
if we can improve performance in the current staging....it will'be a real gain
4.x ..... it will come later......

sure, just i mention that the configuration object of the method countRelations() is structured having in mind to make easy

  • to be used by J3 components with custom states / custom states subsets
  • to be used (after ported) with a custom subset of "conditions" for J4 too
@alikon

This comment has been minimized.

Copy link
Contributor

commented Sep 11, 2018

sorry for my idiosyncrasy... i just want to plant my foots on earth and not on mars

@laoneo

This comment has been minimized.

Copy link
Member

commented Sep 12, 2018

20, 50, 100 queries to get the assigned items for the categories

This function is used only in the back end category list view. As far as I can see it, it is executed only once. But please prove that I'm wrong.

@ggppdk

This comment has been minimized.

Copy link
Contributor Author

commented Sep 12, 2018

This function is used only in the back end category list view. As far as I can see it, it is executed only once. But please prove that I'm wrong.

Sure here is the proof
please notice the existing code inside
helper classes
e.g. for categories of articles ($items contains category objects)

		foreach ($items as $item)
		{
			$item->count_trashed = 0;
			$item->count_archived = 0;
			$item->count_unpublished = 0;
			$item->count_published = 0;
			$query = $db->getQuery(true);
			$query->select('state, count(*) AS count')
				->from($db->qn('#__content'))
				->where('catid = ' . (int) $item->id)
				->group('state');
			$db->setQuery($query);
			$articles = $db->loadObjectList();
		...
		}

and then also i tested to confirm that they are executed, and then spent considerable time for this PR also revising code for tags assignments too

You can confirm the multiple queries by looking at the Debug queries slider,

and then also inside the items loop after:
$articles = $db->loadObjectList();

you can add

echo 'Executed : ' . $query . '<br><br>';
@ggppdk

This comment has been minimized.

Copy link
Contributor Author

commented Sep 12, 2018

I speak of the multiple sql queries (20, 50, 100) that a single call to the methods will do

i never said that the method is called multiple times

@laoneo

This comment has been minimized.

Copy link
Member

commented Sep 12, 2018

Now I got you.

@laoneo

This comment has been minimized.

Copy link
Member

commented Sep 12, 2018

What's the idea behind the config variable?

@HLeithner

This comment has been minimized.

Copy link
Member

commented Sep 12, 2018

ggppdk

Furthermore this PR not only replaces multiple queries for category assignments with 1 query (this is the purpose of this PR)
but this PR also fixes same issue with Tag counting which PR #21667 does not even touch

I didn't noticed that you changed the complete PR to more then changing the DB query sorry I missed this I talked about this part.

@ggppdk

This comment has been minimized.

Copy link
Contributor Author

commented Sep 12, 2018

What's the idea behind the config variable?

basically it makes possible the method to be reusable
we now have 1 method in the parent class and 5 countItems() and 3 counTagItems() methods in descendants helpers classes call it

also copying this from my previous answer

sure, just i mention that the configuration object of the method countRelations() is structured having in mind to make easy

  • to be used by J3 components with custom states / custom states subsets
  • to be used (after ported) with a custom subset of "conditions" for J4 too

e.g. banner has different table name and different column name than content

also the counting for fields in fieldgroups does not use 'catid' column, the column name is 'group_id'

@alikon

This comment has been minimized.

Copy link
Contributor

commented Sep 16, 2018

I have tested this item successfully on 0028cc8


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

@Quy

This comment has been minimized.

Copy link
Contributor

commented Sep 27, 2018

I have tested this item successfully on 3d7db8c


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

@Quy

This comment has been minimized.

Copy link
Contributor

commented Sep 27, 2018

RTC


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

@joomla-cms-bot joomla-cms-bot added the RTC label Sep 27, 2018

@csthomas

This comment has been minimized.

Copy link
Contributor

commented Sep 29, 2018

@ggppdk Can you explain why you have:

	public static function countItems(&$items, $config = null)
	{
		$config = $config ?: (object) array(
			'related_tbl'   => 'banners',
			'state_col'     => 'state',
			'group_col'     => 'catid',
			'relation_type' => 'category_or_group',
		);
		return parent::countRelations($items, $config);
	}

IMO this would be better:

	public static function countItems(&$items)
	{
		$config = (object) array(
			'related_tbl'   => 'banners',
			'state_col'     => 'state',
			'group_col'     => 'catid',
			'relation_type' => 'category_or_group',
		);
		return parent::countRelations($items, $config);
	}

If someone would like to change some parameter then should use BannersHelper::countRelations($items, $config) directly.

@ggppdk

This comment has been minimized.

Copy link
Contributor Author

commented Sep 29, 2018

@csthomas

indeed if someone would want to count relations with a custom table / custom case,
then someone could call somenameHelper::countRelations($items, $config)

which is the method inherited by the parent class

So indeed it seems to me that it can be as you say,
we do not need $config = null variable in methods

public static function countItems(&$items, $config = null)
public static function countTagItems(&$items, $extension, $config = null)

they can be (have same signature like before this)

public static function countItems(&$items)
public static function countTagItems(&$items)
@ggppdk

This comment has been minimized.

Copy link
Contributor Author

commented Sep 30, 2018

Please remove RTC

in last CS commit the table aliases are no longer added correctly (i think they are not, i will test)
plus i will remove the unused / redundant parameter as suggested by @csthomas

@franz-wohlkoenig

This comment has been minimized.

Copy link
Member

commented Sep 30, 2018

Status back on Pending as stated above.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

@joomla-cms-bot joomla-cms-bot removed the RTC label Sep 30, 2018

@ggppdk ggppdk force-pushed the ggppdk:single_counter_query branch from 3d7db8c to f6591df Oct 12, 2018

ggppdk added some commits Oct 12, 2018

CS
@ggppdk

This comment has been minimized.

Copy link
Contributor Author

commented Oct 12, 2018

Ok
the table aliases were correct
so just rebased, removed unused $config variable for methods (as suggested by @csthomas),
and make a CS fix required by appveyor

i retested, everything seems corrects,
but this needs again testing

@csthomas

This comment has been minimized.

Copy link
Contributor

commented Oct 22, 2018

I have tested this item successfully on 3a580e4ba0f8ba273cf85fa63cfeda8ab8c30a8.

There is a problem with mark this test as success on github.
I got error "Bad credentials" at
https://issues.joomla.org/tracker/joomla-cms/22117

@csthomas

This comment has been minimized.

Copy link
Contributor

commented Oct 22, 2018

I have tested this item successfully but I got error "Bad credentials" at https://issues.joomla.org/tracker/joomla-cms/22117

@Quy

This comment has been minimized.

Copy link
Contributor

commented Oct 24, 2018

I have tested this item successfully on 23a580e


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

@Quy

This comment has been minimized.

Copy link
Contributor

commented Oct 24, 2018

RTC


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

@joomla-cms-bot joomla-cms-bot added the RTC label Oct 24, 2018

@mbabker mbabker added this to the Joomla 3.9.1 milestone Nov 1, 2018

@mbabker mbabker merged commit 8b48522 into joomla:staging Nov 1, 2018

5 checks passed

Hound No violations found. Woof!
JTracker/HumanTestResults Human Test Results: 2 Successful 0 Failed.
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/drone/pr the build was successful
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@joomla-cms-bot joomla-cms-bot removed the RTC label Nov 1, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.