StoreAPI: Query parameters via Collection Data endpoint are no longer working

[fabianmarz]

Hey there!

when updating to the latest WooCommerce version my StoreAPI request data doesn't seem to return the data it used to return in WC 7.6.

I'm using the collection-data API to get a pre-filtered result set based on a given categories in a custom ACF block which shows products on any page it is placed on. So the products are not part of the main query.

It doesn't seem to work properly anymore since the this PR seem to dropped support for ProductQuery() which will no longer take the category query parameter into account.

Here's a screenshot of my WP_REST_Request instance but the request returns all attributes instead of the ones filtered for the given category terms

I noticed the Collection API docs still state support for all the parameters

In addition to the above attributes, all product list attributes are supported. This allows you to get data for a certain subset of products. See the products API list products section for the full list.

Is this a bug or am I missing something here? Any help would be appreciated here Thank you! 🙌

[creative-andrew]

@nerrad Do we have news about this issue?
This is breaking a change that should be addressed IMO.

[nerrad]

cc @dinhtungdu is this something you could look into?

[dinhtungdu]

It doesn't seem to work properly anymore since the woocommerce/woocommerce-blocks#8599 PR seem to dropped support for ProductQuery() which will no longer take the category query parameter into account.

I can confirm this. I checked the get_attribute_counts method and it doesn't build the query from request anymore. @nefeline @roykho since your worked on that PR, can you share with us some insights? IMO, this is a bug and should be fixed.

[dinhtungdu]

@nerrad I don't know about the context/reason for the decision made in woocommerce/woocommerce-blocks#8599, but looks like we removed by incident. IMO, we now have some options:

• Revert that change as well as PRs related to Filter data count mismatch #42466
• Refactor the Store API to use the new ProductQueryFiltersServicesProvider, as https://github.com/woocommerce/woocommerce/pull/43811/files. The main PR introduces the new provider is waiting for review. This has bigger risk than the first option btw.

Why reverting instead of fixing? Because to me, the current approach introduced in #42466 is fundamentally different from the former one.

• Formerly, or in other method for price, stock, and rating, we first build and query a set of product first, then we calculate the filter counts from that set of products. That why initially, the filter counts can take category into account. I reused this approach for the new service provider and it's working really well.
• Now, we break the request and manually build the sql for each filter we received in the request. So to support product category, we need to create a sql query for product category. The same story applies to custom taxonomy or meta query. Honestly, I think it's not scale and we shouldn't go forward with it.

[nefeline]

Thanks for your input @dinhtungdu , regarding:

I don't know about the context/reason for the decision made in woocommerce/woocommerce-blocks#8599, but looks like we removed by incident.

No, not really: as you can see in the following discussion https://github.com/woocommerce/woocommerce-blocks/discussions/9023 , by unanimous consensus, we have decided to adopt the implementation path that is aligned with the following rule:

The filtered counts always match the searched results, independent of the combination of filters. They accurately represent what is available in the store for the given criteria.

This removal was a deliberate change to prevent a critical problem: previously, we had an SQL query on the client side to fetch the IDs of all displayed products that was later on used as the value of a param to fetch the filter counts from store API. As you can imagine, this is highly inefficient: ideally, the API should be returning those results to us with all DB requests happening on the backend rather than on the client side. In fact, we have received complaints from external users regarding this client-side query in particular given the huge bottleneck it introduced in our codebase. More details are available on:

woocommerce/woocommerce-blocks#10198

The reported issue has been documented on pdnLyh-3VR-p2
The internal discussion regarding re-enabling support for this feature has been documented internally by @kmanijak on per0F9-Rh-p2

With that said, we are on the same page when it comes to re-enabling this feature, but I strongly disagree that this should be made by reverting all the changes made in the context of the Filter Data count mismatch project: not only because this way we would re-introduce a number of bugs that were effectively solved by that implementation, but also because by re-introducing those queries on the client side we would be, in addition to reintroducing a bug, also negatively impact performance for all stores.

My recommendation is for us to update the parameters received by store API so it is aware of what products are being actively displayed on the frontend and use this information to update the relevant SQL queries and the returned results from the backend, rather than this hybrid approach where the displayed product IDs are fetched on the client side via a highly inefficient SQL query that has already been proven to cause problems. In essence, the backend would do what it is supposed to do, with an API that returns the results we need, given extra params provided.

[dinhtungdu]

@nefeline thanks for the context and links.

This removal was a deliberate change to prevent a critical problem: previously, we had an SQL query on the client side to fetch the IDs of all displayed products that was later on used as the value of a param to fetch the filter counts from store API.

I agree that we should remove the code grabbing all product IDs as you did in woocommerce/woocommerce-blocks#10198. But I don't see it should lead to the removal of query recreation using ProductQuery in the get_attribute_counts method and its general approach. You also mentioned that in the PR:

Its also important to note that these queries are not necessary for maintaining the functionality of the filters or the filter counts; the get_attribute_counts method is.

If we have the performance issue with the former get_attribute_counts, then we should also have an issue with other count methods, as they follow the same methodology.

[dinhtungdu]

Personally, I don't see the method needing to change, but the data passed to the endpoint and the way ProductQuery recreates the query from the param need to be updated. For example, for woocommerce/woocommerce-blocks#7245, instead of passing product IDs, we can pass the on_sale param to the ProductCollectionData endpoint.

That's why I used the nearly identical former method for the new filter blocks, the fundamental change is I passed the whole query arguments, so I can ensure I recreate the same query being displayed on the frontend, resulting in the correct filter count data for any query from simple to complex.

@samueljseay we should profile the new blocks with thousands or even hundreds of thousands of products to see how they keep up with large amounts of data.

[nefeline]

@dinhtungdu regarding:

But I don't see it should lead to the removal of query recreation using ProductQuery in the get_attribute_counts method and its general approach.

Do you mind clarifying what you mean here? What I'm advocating for is to rely exclusively on get_attribute_counts , with the narrowing down of product IDs happening at this stage of the execution (at the ProductCollectionData endpoint - with subsequent trigger of ProductQueryFilters) rather than re-creating a secondary query on ProductQuery (see woocommerce/woocommerce-blocks#10198), which was a poor choice given this query in particular was being triggered for all stores, even the ones that were not using the filters.

If we have the performance issue with the former get_attribute_counts, then we should also have an issue with other count methods, as they follow the same methodology.

Just to be clear, the performance issue was solved with the removal of the queries to fetch the product IDs from ProductQuery and ProductCollection woocommerce/woocommerce-blocks#10198 , it has nothing to do with the ProductQueryFilters->get_attribute_counts, the query on this method was tailor-made, with optimal performance, solving multiple bugs when it comes to attribute counts.

Personally, I don't see the method needing to change, but the data passed to the endpoint [...]. For example, for woocommerce/woocommerce-blocks#7245, instead of passing product IDs, we can pass the on_sale param to the ProductCollectionData endpoint.

Sounds like we are sharing the same point of view here :). This is what I meant with my initial recommendation:

My recommendation is for us to update the parameters received by store API so it is aware of what products are being actively displayed on the frontend and use this information to update the relevant SQL queries and the returned results from the backend, rather than this hybrid approach where the displayed product IDs are fetched on the client side via a highly inefficient SQL query that has already been proven to cause problems. In essence, the backend would do what it is supposed to do, with an API that returns the results we need, given extra params provided.

The only point that sounds like we are still not on the same page is this one:

the way ProductQuery recreates the query from the param need to be updated

I don't think ProductQuery should be "recreating" any queries, but instead, handing over any information needed so the ProductCollectionData endpoint can handle this (with ProductQueryFilters returning the results with the correct range of products already). In practice, this would require a minor update to the relevant queries to narrow down the results from ProductQueryFilters & of course, an update to the params provided to the API. The reason why I believe this is better than re-introducing queries at the ProductQuery and ProductCollection levels is because, this way, we would be jeopardizing performance for all stores again, even the ones that have no intention to use any of our filters. Does it make sense?

[dinhtungdu]

Do you mind clarifying what you mean here?

I meant this query recreation

woocommerce/plugins/woocommerce/src/Internal/ProductQueryFilters/FilterDataProvider.php

Lines 158 to 169 in 76e6d82

	add_filter( 'posts_clauses', array( $this->filter_clauses_generator, 'add_query_clauses' ), 10, 2 );
	add_filter( 'posts_pre_query', '__return_empty_array' );
	$query_vars['no_found_rows'] = true;
	$query_vars['posts_per_page'] = -1;
	$query_vars['fields'] = 'ids';
	$query = new \WP_Query();
	$result = $query->query( $query_vars );
	$product_query_sql = $query->request;
	remove_filter( 'posts_clauses', array( $this->filter_clauses_generator, 'add_query_clauses' ), 10 );
	remove_filter( 'posts_pre_query', '__return_empty_array' );

IMO, recreating the query is cheap because we don't execute the query, we recreate the query to get the SQL request, and then we feed that SQL request as the WHERE condition for the single count query. There aren't any additional SQL queries to be made.

If the former get_attribute_counts method doesn't have good performance, then other methods in ProductQueryFilters should have the same issue, because they all recreate the product query from the API request using ProductQuery, and pass the SQL query of the product query to the count query.

which was a poor choice given this query in particular was being triggered for all stores, even the ones that were not using the filters.

As I mentioned before, I agree with this. I also mentioned that we could solve this differently by passing additional parameters to StoreAPI, like on_sale.

What I'm advocating for is to rely exclusively on get_attribute_counts , with the narrowing down of product IDs happening at this stage of the execution (at the ProductCollectionData endpoint - with subsequent trigger of ProductQueryFilters) rather than re-creating a secondary query on ProductQuery (see woocommerce/woocommerce-blocks#10198),

Just to be clear, the performance issue was solved with the removal of the queries to fetch the product IDs from ProductQuery and ProductCollection woocommerce/woocommerce-blocks#10198 , it has nothing to do with the ProductQueryFilters->get_attribute_counts, the query on this method was tailor-made, with optimal performance, solving multiple bugs when it comes to attribute counts.

I was also talking about the get_attribute_counts only, I didn't acknowledge that PR until now.

I think we can still use the former method to benefit from context-aware ability (like taking category into account like OP uses) without worrying about the performance as the reason I explained above. The only concern I have with the former method is this line:

woocommerce/plugins/woocommerce/src/Internal/ProductQueryFilters/FilterDataProvider.php

Line 178 in 76e6d82

WHERE posts.ID IN ( {$product_query_sql} )

But does it hurt the performance, I'm not sure. So I went ahead to some (very basic) profiling.

My setup:

• My current WP instance I'm using for development.
• I use the WC Smoosh Generator to generate 10000 products: wp wc generate products 10000.
• I use profiling/get_attributes_count-methods which bases on my [Experimental] Refactor: replace QueryFilters by ProductQueryFiltersServiceProvider #43772. The branch contains the current StoreAPI ProductQueryFilters::get_attribute_count and a new one FilterDataProvider::get_attribute_count that follows the former approach recreating product query.
• I disabled the cache for ProductQueryFilters::get_attribute_count to measure raw SQL execution time.
• I added a simple timer to measure the execution time of the whole get_attributes_count method.
• I added the Filter by Attribute block (powered by StoreAPI) and Product Filter: Attribute (powered by new service provider) to the Product Catalog template along with the Product Collection block. Two blocks use the same settings and the same attribute.
• I reloaded this URL http://woo.test/shop/?filter_color=ad-cum&query_type_color=or several times in incognito mode to trigger the execution for both methods, and here is the result:

[02-Feb-2024 10:43:26 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0066521167755127
[02-Feb-2024 10:43:26 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.029701948165894
[02-Feb-2024 10:43:29 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0039560794830322
[02-Feb-2024 10:43:30 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.032266139984131
[02-Feb-2024 10:43:31 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.011115074157715
[02-Feb-2024 10:43:32 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.028652906417847
[02-Feb-2024 10:43:32 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0071561336517334
[02-Feb-2024 10:43:33 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.035531044006348
[02-Feb-2024 10:43:34 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0092430114746094
[02-Feb-2024 10:43:34 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.029453992843628
[02-Feb-2024 10:43:35 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.010504961013794
[02-Feb-2024 10:43:35 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.02213978767395
[02-Feb-2024 10:43:36 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0090651512145996
[02-Feb-2024 10:43:36 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.033707857131958
[02-Feb-2024 10:43:37 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.010114908218384
[02-Feb-2024 10:43:37 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.030858039855957
[02-Feb-2024 10:43:38 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0070819854736328
[02-Feb-2024 10:43:38 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.033290863037109
[02-Feb-2024 10:43:39 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0065598487854004
[02-Feb-2024 10:43:39 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.027875185012817
[02-Feb-2024 10:43:40 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.01097297668457
[02-Feb-2024 10:43:41 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.028970003128052

• After that, I added the timer to two early returns of ProductQueryFilters::get_attribute_counts to measure the execution time of the initial page with filter load. Here is the result of reloading the http://woo.test/shop/ several times, that the same shop page as step above, with Filter by Attribute and Product Filter: Attribute blocks.

[02-Feb-2024 10:49:30 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0021979808807373
[02-Feb-2024 10:49:30 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.02341103553772
[02-Feb-2024 10:49:45 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0041239261627197
[02-Feb-2024 10:49:45 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.024367809295654
[02-Feb-2024 10:49:47 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0026090145111084
[02-Feb-2024 10:49:47 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.022763967514038
[02-Feb-2024 10:49:49 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.002795934677124
[02-Feb-2024 10:49:50 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.02579402923584
[02-Feb-2024 10:49:52 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0027499198913574
[02-Feb-2024 10:49:52 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.023179054260254
[02-Feb-2024 10:49:54 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0028059482574463
[02-Feb-2024 10:49:54 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.024451017379761
[02-Feb-2024 10:49:55 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0025620460510254
[02-Feb-2024 10:49:55 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.024876117706299
[02-Feb-2024 10:49:56 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0025289058685303
[02-Feb-2024 10:49:57 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.023113012313843
[02-Feb-2024 10:49:57 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0026040077209473
[02-Feb-2024 10:49:57 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.024165868759155
[02-Feb-2024 10:49:58 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0024509429931641
[02-Feb-2024 10:49:59 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.025151014328003
[02-Feb-2024 10:49:59 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0025229454040527
[02-Feb-2024 10:50:00 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.023794889450073

From the results, I can see the former method performs better than the current one consistently. I don't have much experience in profiling MySQL performance, so I'm open to learning any better profiling methods.

[dinhtungdu]

I tried another test with min price: http://woo.test/shop/?min_price=200&filter_color=white&query_type_color=or

[02-Feb-2024 11:31:00 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.016616106033325
[02-Feb-2024 11:31:01 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.020287036895752
[02-Feb-2024 11:31:01 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.029187917709351
[02-Feb-2024 11:31:05 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0089359283447266
[02-Feb-2024 11:31:05 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.031203031539917
[02-Feb-2024 11:31:07 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0046279430389404
[02-Feb-2024 11:31:07 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.031788110733032
[02-Feb-2024 11:31:09 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0050640106201172
[02-Feb-2024 11:31:09 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.033957004547119
[02-Feb-2024 11:31:11 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.005220890045166
[02-Feb-2024 11:31:12 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.032218933105469
[02-Feb-2024 11:31:13 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0072529315948486
[02-Feb-2024 11:31:14 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.038492202758789
[02-Feb-2024 11:31:16 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.012527942657471
[02-Feb-2024 11:31:16 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.02892804145813
[02-Feb-2024 11:31:17 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.007800817489624
[02-Feb-2024 11:31:18 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.038853168487549
[02-Feb-2024 11:31:19 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0046548843383789
[02-Feb-2024 11:31:19 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.029859066009521
[02-Feb-2024 11:31:21 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0045139789581299
[02-Feb-2024 11:31:21 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.034087896347046
[02-Feb-2024 11:31:24 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.005836009979248
[02-Feb-2024 11:31:24 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.033492803573608

[dinhtungdu]

There was a mistake in my test branch that misses the last array map call of FilterDataProvider::get_attribute_counts that I fixed here, however, it doesn't affect the test results. Here is the log reloading http://woo.test/shop/ for several times:

[02-Feb-2024 17:45:02 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0024888515472412
[02-Feb-2024 17:45:03 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.030502080917358
[02-Feb-2024 17:45:21 UTC] -----------------------------------------------------------
[02-Feb-2024 17:45:21 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0026588439941406
[02-Feb-2024 17:45:21 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.029579877853394
[02-Feb-2024 17:45:22 UTC] -----------------------------------------------------------
[02-Feb-2024 17:45:22 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0023541450500488
[02-Feb-2024 17:45:22 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.0066170692443848
[02-Feb-2024 17:45:22 UTC] -----------------------------------------------------------
[02-Feb-2024 17:45:22 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0021340847015381
[02-Feb-2024 17:45:23 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.0072751045227051
[02-Feb-2024 17:45:23 UTC] -----------------------------------------------------------
[02-Feb-2024 17:45:23 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0025079250335693
[02-Feb-2024 17:45:23 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.006472110748291
[02-Feb-2024 17:45:23 UTC] -----------------------------------------------------------
[02-Feb-2024 17:45:23 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0023601055145264
[02-Feb-2024 17:45:24 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.0063610076904297
[02-Feb-2024 17:45:24 UTC] -----------------------------------------------------------
[02-Feb-2024 17:45:24 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0024890899658203
[02-Feb-2024 17:45:24 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.022815942764282
[02-Feb-2024 17:45:47 UTC] -----------------------------------------------------------
[02-Feb-2024 17:45:47 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0023648738861084
[02-Feb-2024 17:45:47 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.025309085845947
[02-Feb-2024 17:45:48 UTC] -----------------------------------------------------------
[02-Feb-2024 17:45:48 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.002471923828125
[02-Feb-2024 17:45:48 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.043020963668823
[02-Feb-2024 17:46:16 UTC] -----------------------------------------------------------
[02-Feb-2024 17:46:16 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0023560523986816
[02-Feb-2024 17:46:16 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.032316923141479
[02-Feb-2024 17:46:17 UTC] -----------------------------------------------------------
[02-Feb-2024 17:46:17 UTC] FilterDataProvider::get_attribute_counts  excecution time: 0.0023689270019531
[02-Feb-2024 17:46:18 UTC] ProductQueryFilters::get_attribute_counts excecution time: 0.025528192520142

[dinhtungdu]

FYI, I realized I misunderstood some of Patricia's comments, so I edited my response #42157 (comment), please give it another look.

[nefeline]

Hey @dinhtungdu, thanks.

I have a few questions here:

• If we roll back to the previous get_attribute_counts method, how do you plan to solve all the bugs that @roykho and I fixed with the work made on the new query? Can you please list each one of them individually? Note: rolling back get_attribute_counts method to its previous state doesn't solve the problem on this issue: rolling back Product Filters > Fix Performance issue and Fatal error on stores with a high volume of products woocommerce-blocks#10198 does (although not recommended, the plan to pass over different params to the API for narrowing down the products at that stage of execution is a way better approach). In essence, get_attribute_counts doesn't have any influence on the issue described here.

• Unfortunately, the comparison in performance between old get_attribute_counts and new get_attribute_counts is not a valid one because these methods are not equivalent: the previous method doesn't return the correct counts.

I'm happy to help open a PR specifically to solve the problem described on this issue without going through this major refactor & risking re-introducing regressions for the filters.

[dinhtungdu]

@nefeline thanks. Regarding your questions:

Note: rolling back get_attribute_counts method to its previous state doesn't solve the problem on this issue:
In essence, get_attribute_counts doesn't have any influence on the issue described here.

I don't think so. OP uses Store API directly to get the filter data, not through a filter block, so woocommerce/woocommerce-blocks#10198 isn't related.
If you say that reverting 10198 fixes the count as described in woocommerce/woocommerce-blocks#7245, then I agree. But as I mentioned above, that issue could be solved differently without hurting performance by passing additional parameters to product-collection-data.

If we roll back to the previous get_attribute_counts method, how do you plan to solve all the bugs that @roykho and I fixed with the work made on the new query? Can you please list each one of them individually?

We have already fixed data mismatch issues for the new filter blocks in #42811 (pdnLyh-530-p2). We didn't change the main logic of those count methods, but we changed the data passed to them. We won't fix the data mismatch issue for the current filter blocks, because:

• We fixed them in the new filter blocks, which will replace the current ones.
• Store API was working as expected by its design. The product-collection-data, apart from its unique params, accepts params of the products endpoint to narrow down the products for filter data calculation.
• After spending a lot of time transforming product queries to StoreAPI params, I think StoreAPI isn't a good solution for filter blocks. See #10919 and #11218. That's why I worked on #42811 and #43772. Because we have new filter blocks, we can accept that limitation. It isn't worth solving for current filter blocks at this point IMO.

Unfortunately, the comparison in performance between old get_attribute_counts and new get_attribute_counts is not a valid one because these methods are not equivalent: the previous method doesn't return the correct counts.

The previous method used by the current attribute filter block doesn't return correct counts, but the new attribute filter block using a similar method does. In addition, in my test, the current (trunk) attribute filter block using StoreAPI also doesn't return the correct counts. I made a quick comparison of New and Current Attribute Filters using my test store with 10000 products, the new attribute filter block always shows the correct count that matches with the query being displayed, while the current attribute filter block sometimes shows the incorrect data.

I used this branch for testing, but trunk should give us the same result. I'm inviting you also everyone in this thread to test to ensure this isn't a work-on-my-machine situation.

As I mentioned in the P2:

• Refactor Automattic\WooCommerce\StoreApi\Routes\V1\ProductCollectionData to use our new QueryFilters class.

I want to refactor the StoreAPI to use the new class (now being refactored to a service provider in #43772), I made a draft PR to demonstrate how it looks here. So we can consider another option, which is waiting for the new service provider to be merged.

[nefeline]

I don't think so. OP uses Store API directly to get the filter data, not through a filter block, so woocommerce/woocommerce-blocks#10198 isn't related.
If you say that reverting 10198 fixes the count as described in woocommerce/woocommerce-blocks#7245, then I agree. But as I mentioned above, that issue could be solved differently without hurting performance by passing additional parameters to product-collection-data.

Unfortunately it looks like you accidentally removed an important piece of information from my previous response here, so I'm posting it here again to make sure we are on the same page:

(although not recommended, the plan to pass over different params to the API for narrowing down the products at that stage of execution is a way better approach)

We have already fixed data mismatch issues for the new filter blocks in #42811 (pdnLyh-530-p2).

Sounds like a lot of work has already been made without the involvement of folks who worked on the attribute counts early in the process (@roykho and myself), that's unfortunate as we missed an invaluable opportunity to collaborate early.

Regarding your video:

On Minute 1:57 you are showing the product "Sleek Silk Plate", which is a variable product:

Does this product have two variations? Do both have the "At" attribute? If the answer to both questions is yes, then the bug is on your proposed implementation, not the current one: all counts were tailor-made to account for product variations as well as parent products, that was one of the key changes made as part of the Filter Data Count attribute project.

On Minute 4:40 you enable a combination of filters of products with the attribute equal to "Atque" + a price range: can you please check if there are really two products with the "Atque" attribute AND with a price between the range of 559 and 587?

I strongly recommend you carefully read the details of the implementation on https://github.com/woocommerce/woocommerce-blocks/discussions/9023 as it explains how each one of the decisions were made and why. This also includes the explanation why you are seeing the "zeros" with certain filter combinations: this was all by design, not a regression or a bug.

[dinhtungdu]

Unfortunately it looks like you accidentally removed an important piece of information from my previous response here, so I'm posting it here again to make sure we are on the same page:

(although not recommended, the plan to pass over different params to the API for narrowing down the products at that stage of execution is a way better approach)

@nefeline I didn't miss that. I agree with your comment about passing over additional parameters. I was saying that we could solve woocommerce/woocommerce-blocks#7245 without modifying the original get_attribute_counts method (and avoid breaking the backward compatibility like this issue)

So, we're still not on the same page. I wonder how can we move this forward. Maybe a call could be a better way to clear up the confusion, what do you think?

Does this product have two variations? Do both have the "At" attribute? If the answer to both questions is yes, then the bug is on your proposed implementation, not the current one: all counts were tailor-made to account for product variations as well as parent products, and that was one of the key changes made as part of the Filter Data Count attribute project.

I'm confused here. As a shopper, when I click on an attribute with 2 as the count, I'd expect to see 2 products. So, are you saying, it's expected to have the count is different from the actual number of filtered products? cc @jarekmorawski @nerrad can you comment on this?

On Minute 4:40 you enable a combination of filters of products with the attribute equal to "Atque" + a price range: can you please check if there are really two products with the "Atque" attribute AND with a price between the range of 559 and 587?

Yes, I can confirm, I also clicked the Atque in the video:

This also includes the explanation why you are seeing the "zeros" with certain filter combinations: this was all by design, not a regression or a bug.

This is another point that makes me confused. In my test, I still got more filtered products if I clicked on those 'zero' attribute filters.
I went through https://github.com/woocommerce/woocommerce-blocks/discussions/9023 and I can see that it's intentional, but as @thealexandrelara commentted, I think it can be a frustrating experience and zero counts should be hidden, for both and and or query types.

[nefeline]

I'm confused here. As a shopper, when I click on an attribute with 2 as the count, I'd expect to see 2 products. So, are you saying, it's expected to have the count is different from the actual number of filtered products? cc @jarekmorawski @nerrad can you comment on this?

I'm sorry, you didn't answer my previous question so I'll paste it here:

Does this product have two variations? Do both have the "At" attribute? If the answer to both questions is yes, then the bug is on your proposed implementation, not the current one: all counts were tailor-made to account for product variations as well as parent products, and that was one of the key changes made as part of the Filter Data Count attribute project.

If the user is looking for blue products and a given product has 3 different blue variations, they are all unique/individual blue products, that the user can purchase, aren't they? We have discussed this with design prior to implementing the change, but I'll let @jarekmorawski chime in here for more context.

Yes, I can confirm, I also clicked the Atque in the video:

Sorry, I was probably not very clear: can you please open these two products individually, open their individual variations, and then share the screenshots of their prices and attributes here? what I'm aiming for is a comparison between the product variations, not the parent products.

This is another point that makes me confused. In my test, I still got more filtered products if I clicked on those 'zero' attribute filters.

This also has been discussed with design at the time of implementation, so I'll again defer to @jarekmorawski here. For reference, I'm copy-pasting the context from the original discussion:

Filter by attribute, rating, and price

Example: ?min_price=10&max_price=30&filter_color=green&query_type_color=or&rating_filter=5 :

• Notice how the attribute count matches precisely the search results, which corresponds to a shirt with the color green, rating = 5, and max price = 30.
• Notice how the counts for the size attributes are also updated to match the criteria: the displayed product doesn't have any of the sizes listed.

I was saying that we could solve woocommerce/woocommerce-blocks#7245 without modifying the original get_attribute_counts method (and avoid breaking the backward compatibility like this issue)

Gotcha: based on our conversation so far, everything indicates that there's a misunderstanding of how filter counts should work and what is a bug. All the changes made in the context of the Filter data count mismatch project were reviewed and approved by the design team and the leadership: you didn't share the details for each one of the individual product variations yet, but if my suspicions are accurate, what you are seeing is correct: we have made this decision very clear with the following statement:

The filtered counts always match the searched results, independent of the combination of filters. They accurately represent what is available in the store for the given criteria.

I wonder how can we move this forward. Maybe a call could be a better way to clear up the confusion, what do you think?

I agree, especially because there seems to be a fundamental disagreement with what is and what isn't the expected functionality here.

[samueljseay]

@nefeline I don't want to sidetrack too much but I just wanted to zero in on this, and forgive me I might be misunderstanding some nuance but its just how I see it at first glance.

If the user is looking for blue products and a given product has 3 different blue variations, they are all unique/individual blue products, that the user can purchase, aren't they? We have discussed this with design prior to implementing the change, but I'll let @jarekmorawski chime in here for more context.

I think this is confusing for variations? I can't think of a store where I wanted to purchase a blue t-shirt for example and there is one blue tshirt with variations for small/medium/large would I really want to see the count including those variations? not to mention when the product list is filtered in the UI its showing me one tshirt.

that the user can purchase, aren't they?

Sure, variations are purchaseable, but users don't care about that, variations are somewhat hidden from them because we don't show them every tshirt size for blue tshirt in the product collection UI.

I can understand a lot of thought has gone into all of this, I read through a lot of the previous issues and such, but I think its ok, if we challenge some of this thinking again to make sure we're on the right track about what expected behaviour should be, because every time I encountered this behaviour I felt its not correct for shopper UI personally.

[dinhtungdu]

I'm sorry, you didn't answer my previous question so I'll paste it here:

Does this product have two variations? Do both have the "At" attribute? If the answer to both questions is yes, then the bug is on your proposed implementation, not the current one: all counts were tailor-made to account for product variations as well as parent products, and that was one of the key changes made as part of the Filter Data Count attribute project.

@nefeline I'm sorry I missed this.

The Sleek Silk Plate product is a variable product, It has 3 variation base on only one attribute which is Color. There is only one variation has At attribute.

Screenshots

On Minute 4:40 you enable a combination of filters of products with the attribute equal to "Atque" + a price range: can you please check if there are really two products with the "Atque" attribute AND with a price between the range of 559 and 587?

Yes, I can confirm, I also clicked the Atque in the video:

Sorry, I was probably not very clear: can you please open these two products individually, open their individual variations, and then share the screenshots of their prices and attributes here? what I'm aiming for is a comparison between the product variations, not the parent products.

Here are screenshots I took for the filtered products of this combination. Both products have multiple variations, and both have more than two variations that have the Atque attribute.

AND with a price between the range of 559 and 587

However, I don't see any variation with Atque that has the price in the range of 559 and 587. I took screenshots of every variation with Atque below.

Screenshots

**The filtered products:**

The first product:

The second product:

[sunyatasattva]

Just to quickly chime in regarding the UX issue, I also agree with @samueljseay that counting each variation as a separate product in the UX is confusing and potentially misleading to the users. The fact that they are separate purchasable entities (and I've dealt with some WooCommerce integrations which require to mark them as separate products, to be honest: so, speaking strictly technically, there is a case to be made for this), I think from both the user and merchant perspective, it makes sense to consider them as one product.

Think more about “models” rather than purchasable entities.

I also have tried to follow all the links, but I didn't manage to find where this decision was made. I might be missing it, but, to me, it seems that actually @jarekmorawski was, if anything, basically supporting the opposite of this: per0F9-Rh-p2#comment-899 in a related issue. But then, we proceeded with a different solution in the interest of time, more conversation pending.

[nefeline]

@sunyatasattva @dinhtungdu @samueljseay how about we move your concerns to https://github.com/woocommerce/woocommerce-blocks/discussions/9023 so we can re-open the discussion with the design team and the leadership? WDYT?

[creative-andrew]

@nefeline I just want to chime in and say that this API is not only used in the block editor, but also used as an independent API in decoupled WooCommerce sites. Removing the product query from the count broke the contract of the API and its documentation and should be fixed as soon as possible.

[nefeline]

I just want to chime in and say that this API is not only used in the block editor, but also used as an independent API in decoupled WooCommerce sites. Removing the product query from the count broke the contract of the API and its documentation and should be fixed as soon as possible.

@creative-andrew thanks for chiming in! We are on the same page here: this ended up being a bigger discussion as other aspects of the filter counts are being questioned that are not at all related to this issue in particular.

Fixing the bug as described on this ticket with the current implementation is fairly simple, I'm glad to open a PR with the solution if we verify that the discussion that is now happening on https://github.com/woocommerce/woocommerce-blocks/discussions/9023 will take longer to conclude and/or if we the outcome of that conversation is that the design decision made for the current approach is the one we wanna keep.

[sunyatasattva]

@creative-andrew @fabianmarz just a heads up that this issue was fixed in #45247, and it will be available in WooCommerce 8.9. Thank you for your patience with us while discussing this issue and sorry for the inconvenience for your stores!

[creative-andrew]

Thanks y'all for your hard work!

[fabianmarz]

Thank you guys for the work and the heads up! 💪