Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Performance] Category manager / Tag Manager, get all counters via single query, reduce 20, 50, 100 queries (page limit) to 1 #22117

Merged
merged 8 commits into from Nov 1, 2018

Conversation

@ggppdk
Copy link
Contributor

@ggppdk ggppdk commented Sep 10, 2018

Pull Request for Issue # .

Summary of Changes

Category manager creates 1 query per category to get the assigned item counts by state
These are used to displays the columns
total published, total unpublished, total archived, total trashed

Testing Instructions

  1. Open 2 Tabs of Category manager and 2 Tabs of Tag manager
  2. Filter the 2 Tabs of Tag manager by "Tag Type" article (click search tools button)
  3. Apply patch
  4. Refresh 2nd TAB of Category manager and 2nd Tab of Tag manager

Both 1st and 2nd tabs should show the same counted items per state


Category managers to test

  • articles, banners, contact, newsfeeds, and for fieldgroups management

Tags manager:

  • repeat test for every listed Tag Type, in the Tag Type Filter (click search tools)

Expected result

1 query to get the total assigned items for the categories shown by current page

Actual result

20, 50, 100 queries to get the assigned items for the categories

Documentation Changes Required

None

@alikon
Copy link
Contributor

@alikon alikon commented Sep 10, 2018

very well done 👍
.....testing right now....

Loading

// Get relation counts for all category objects with single query
$query = $db->getQuery(true)
->select('catid, ' . $state_column . ' AS state, count(*) AS count')
->from($db->qn('#__' . $related_tbl))
Copy link
Contributor

@csthomas csthomas Sep 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace $db->qn by $db->quoteName

Loading

Copy link
Contributor

@alikon alikon Sep 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • use $db->quoteName() where needed

Loading

@alikon
Copy link
Contributor

@alikon alikon commented Sep 10, 2018

the same "optimization" should be done on every "category" banner, contact etc....

Loading


// Get relation counts for all category objects with single query
$query = $db->getQuery(true)
->select('catid, ' . $state_column . ' AS state, count(*) AS count')
Copy link
Contributor

@Quy Quy Sep 10, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capitalize count(*)

Loading

@alikon
Copy link
Contributor

@alikon alikon commented Sep 10, 2018

even better can be done on some high level "category" CLASS/HELPER

Loading

@ggppdk ggppdk force-pushed the single_counter_query branch from dae4586 to f10fb8b Sep 11, 2018
@ggppdk
Copy link
Contributor Author

@ggppdk ggppdk commented Sep 11, 2018

even better can be done on some high level "category" CLASS/HELPER

yes that was my thought too
that is why those local variables in the first commit for the columns names, that seemed redundant ...

I intended to replace countItems() method everywhere, but i see that countTagItems() has the same problem of 20, 50, 100 queries instead of 1

so i have added a countRelations() method to parent helper class that replace both methods,
actually the methods are not replaced instead they call the parent class method like this
parent::countRelations($items, ...,$config);

Loading

@ggppdk ggppdk changed the title [Performance] Category manager, get all counters via single query, reduce 20, 50, 100 queries (page limit) to 1 [Performance] Category manager / Tag Manager, get all counters via single query, reduce 20, 50, 100 queries (page limit) to 1 Sep 11, 2018
@alikon
Copy link
Contributor

@alikon alikon commented Sep 11, 2018

countTagItems() you can accuse me for that 😨

Loading

// The relation query does not return a value for cases without relations of a particular state / condition, set zero as default
foreach ($items as $item)
{
foreach($counter_names as $n)
Copy link
Contributor

@Quy Quy Sep 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add space before (

Loading

Copy link
Contributor Author

@ggppdk ggppdk Sep 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, done this and other CS errors

Loading

@HLeithner
Copy link
Member

@HLeithner HLeithner commented Sep 11, 2018

We have a PR for such a function for j4 #21667 I will rework it to do the same optimization as this PR. It's really need I ask my self why I didn't do this.

Atm I wait for #20693 to get merged to get the direction how to solve the component naming issue.

Loading

@ggppdk
Copy link
Contributor Author

@ggppdk ggppdk commented Sep 11, 2018

That PR has a completely different purpose than PR #21667,
PR #21667 tries to make a reusable method in a trait

Furthermore this PR not only replaces multiple queries for category assignments with 1 query (this is the purpose of this PR)
but this PR also fixes same issue with Tag counting which PR #21667 does not even touch

A note
the code here is agnostic in regards to which state / condition exist
thus it can be rather easily ported to J4

Loading

@ggppdk
Copy link
Contributor Author

@ggppdk ggppdk commented Sep 11, 2018

Of course if you would want to do same optimization inside PR
#21667 for J4 then please do, as i lack time to do more work now
thanks

Loading

*
* @return stdClass[]
*
* @since 3.6
Copy link
Contributor

@Quy Quy Sep 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to __DEPLOY_VERSION__

Loading

Copy link
Contributor Author

@ggppdk ggppdk Sep 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, lol, i remember adding DEPLOY_VERSION for 2 methods,
but then the 2 methods became 1 because 80% of their code was the same

Loading

@alikon
Copy link
Contributor

@alikon alikon commented Sep 11, 2018

please, can I ask you if can we concentrate our effort on 3.x series for a while !!
if we can improve performance in the current staging....it will'be a real gain
4.x ..... it will come later......

Loading

@ggppdk
Copy link
Contributor Author

@ggppdk ggppdk commented Sep 11, 2018

please, can I ask you if can we concentrate our effort on 3.x series for a while !!
if we can improve performance in the current staging....it will'be a real gain
4.x ..... it will come later......

sure, just i mention that the configuration object of the method countRelations() is structured having in mind to make easy

  • to be used by J3 components with custom states / custom states subsets
  • to be used (after ported) with a custom subset of "conditions" for J4 too

Loading

@alikon
Copy link
Contributor

@alikon alikon commented Sep 11, 2018

sorry for my idiosyncrasy... i just want to plant my foots on earth and not on mars

Loading

@laoneo
Copy link
Member

@laoneo laoneo commented Sep 12, 2018

20, 50, 100 queries to get the assigned items for the categories

This function is used only in the back end category list view. As far as I can see it, it is executed only once. But please prove that I'm wrong.

Loading

@ggppdk
Copy link
Contributor Author

@ggppdk ggppdk commented Sep 12, 2018

This function is used only in the back end category list view. As far as I can see it, it is executed only once. But please prove that I'm wrong.

Sure here is the proof
please notice the existing code inside
helper classes
e.g. for categories of articles ($items contains category objects)

		foreach ($items as $item)
		{
			$item->count_trashed = 0;
			$item->count_archived = 0;
			$item->count_unpublished = 0;
			$item->count_published = 0;
			$query = $db->getQuery(true);
			$query->select('state, count(*) AS count')
				->from($db->qn('#__content'))
				->where('catid = ' . (int) $item->id)
				->group('state');
			$db->setQuery($query);
			$articles = $db->loadObjectList();
		...
		}

and then also i tested to confirm that they are executed, and then spent considerable time for this PR also revising code for tags assignments too

You can confirm the multiple queries by looking at the Debug queries slider,

and then also inside the items loop after:
$articles = $db->loadObjectList();

you can add

echo 'Executed : ' . $query . '<br><br>';

Loading

@ggppdk
Copy link
Contributor Author

@ggppdk ggppdk commented Sep 12, 2018

I speak of the multiple sql queries (20, 50, 100) that a single call to the methods will do

i never said that the method is called multiple times

Loading

@laoneo
Copy link
Member

@laoneo laoneo commented Sep 12, 2018

Now I got you.

Loading

@laoneo
Copy link
Member

@laoneo laoneo commented Sep 12, 2018

What's the idea behind the config variable?

Loading

@HLeithner
Copy link
Member

@HLeithner HLeithner commented Sep 12, 2018

ggppdk

Furthermore this PR not only replaces multiple queries for category assignments with 1 query (this is the purpose of this PR)
but this PR also fixes same issue with Tag counting which PR #21667 does not even touch

I didn't noticed that you changed the complete PR to more then changing the DB query sorry I missed this I talked about this part.

Loading

@ggppdk
Copy link
Contributor Author

@ggppdk ggppdk commented Sep 12, 2018

What's the idea behind the config variable?

basically it makes possible the method to be reusable
we now have 1 method in the parent class and 5 countItems() and 3 counTagItems() methods in descendants helpers classes call it

also copying this from my previous answer

sure, just i mention that the configuration object of the method countRelations() is structured having in mind to make easy

  • to be used by J3 components with custom states / custom states subsets
  • to be used (after ported) with a custom subset of "conditions" for J4 too

e.g. banner has different table name and different column name than content

also the counting for fields in fieldgroups does not use 'catid' column, the column name is 'group_id'

Loading

@alikon
Copy link
Contributor

@alikon alikon commented Sep 16, 2018

I have tested this item successfully on 0028cc8


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

Loading

@Quy
Copy link
Contributor

@Quy Quy commented Sep 27, 2018

I have tested this item successfully on 3d7db8c


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

Loading

@Quy
Copy link
Contributor

@Quy Quy commented Sep 27, 2018

RTC


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

Loading

@csthomas
Copy link
Contributor

@csthomas csthomas commented Sep 29, 2018

@ggppdk Can you explain why you have:

	public static function countItems(&$items, $config = null)
	{
		$config = $config ?: (object) array(
			'related_tbl'   => 'banners',
			'state_col'     => 'state',
			'group_col'     => 'catid',
			'relation_type' => 'category_or_group',
		);
		return parent::countRelations($items, $config);
	}

IMO this would be better:

	public static function countItems(&$items)
	{
		$config = (object) array(
			'related_tbl'   => 'banners',
			'state_col'     => 'state',
			'group_col'     => 'catid',
			'relation_type' => 'category_or_group',
		);
		return parent::countRelations($items, $config);
	}

If someone would like to change some parameter then should use BannersHelper::countRelations($items, $config) directly.

Loading

@ggppdk
Copy link
Contributor Author

@ggppdk ggppdk commented Sep 29, 2018

@csthomas

indeed if someone would want to count relations with a custom table / custom case,
then someone could call somenameHelper::countRelations($items, $config)

which is the method inherited by the parent class

So indeed it seems to me that it can be as you say,
we do not need $config = null variable in methods

public static function countItems(&$items, $config = null)
public static function countTagItems(&$items, $extension, $config = null)

they can be (have same signature like before this)

public static function countItems(&$items)
public static function countTagItems(&$items)

Loading

@ggppdk
Copy link
Contributor Author

@ggppdk ggppdk commented Sep 30, 2018

Please remove RTC

in last CS commit the table aliases are no longer added correctly (i think they are not, i will test)
plus i will remove the unused / redundant parameter as suggested by @csthomas

Loading

@ghost
Copy link

@ghost ghost commented Sep 30, 2018

Status back on Pending as stated above.


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

Loading

@ggppdk ggppdk force-pushed the single_counter_query branch from 3d7db8c to f6591df Oct 12, 2018
@ggppdk
Copy link
Contributor Author

@ggppdk ggppdk commented Oct 12, 2018

Ok
the table aliases were correct
so just rebased, removed unused $config variable for methods (as suggested by @csthomas),
and make a CS fix required by appveyor

i retested, everything seems corrects,
but this needs again testing

Loading

@csthomas
Copy link
Contributor

@csthomas csthomas commented Oct 22, 2018

I have tested this item successfully on 3a580e4ba0f8ba273cf85fa63cfeda8ab8c30a8.

There is a problem with mark this test as success on github.
I got error "Bad credentials" at
https://issues.joomla.org/tracker/joomla-cms/22117

Loading

@csthomas
Copy link
Contributor

@csthomas csthomas commented Oct 22, 2018

I have tested this item successfully but I got error "Bad credentials" at https://issues.joomla.org/tracker/joomla-cms/22117

Loading

@Quy
Copy link
Contributor

@Quy Quy commented Oct 24, 2018

I have tested this item successfully on 23a580e


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

Loading

@Quy
Copy link
Contributor

@Quy Quy commented Oct 24, 2018

RTC


This comment was created with the J!Tracker Application at issues.joomla.org/tracker/joomla-cms/22117.

Loading

@mbabker mbabker added this to the Joomla 3.9.1 milestone Nov 1, 2018
@mbabker mbabker merged commit 8b48522 into joomla:staging Nov 1, 2018
5 checks passed
Loading
@joomla-cms-bot joomla-cms-bot removed the RTC label Nov 1, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

8 participants