MC-23881: When category URL page param is larger than the pagination, the page will break with elastic search 2 #31594

engcom-Golf · 2021-01-08T08:27:34Z

Description (*)

Related Pull Requests

Fixed Issues (if relevant)

Fixes When category URL page param is larger than the pagination, the page will break with elastic search 2 #23843

Manual testing scenarios (*)

Create a category with products
Enable elasticsearch
Reindex all
Go to the category and add this param: ?p=999999999999999
Expected result:
No out of range exception in logs with status 500

Note:
There still can be log entity with 400 status.
Redirect to the last page of the category is not expected.

Questions or comments

Contribution checklist (*)

Pull request has a meaningful description of its purpose
All commits are accompanied by meaningful commit messages
All new or changed code is covered with unit/integration tests (if applicable)
All automated tests passed successfully (all builds are green)

m2-assistant · 2021-01-08T08:27:36Z

Hi @engcom-Golf. Thank you for your contribution
Here is some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

@magento give me test instance - deploy test instance based on PR changes
@magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

@magento run all tests - run or re-run all required tests against the PR changes
@magento run <test-build(s)> - run or re-run specific test build(s)
For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names. Allowed build names are:

Database Compare
Functional Tests CE
Functional Tests EE,
Functional Tests B2B
Integration Tests
Magento Health Index
Sample Data Tests CE
Sample Data Tests EE
Sample Data Tests B2B
Static Tests
Unit Tests
WebAPI Tests
Semantic Version Checker

You can find more information about the builds here

ℹ️ Please run only needed test builds instead of all when developing. Please run all test builds before sending your PR for review.

For more details, please, review the Magento Contributor Guide documentation.

⚠️ According to the Magento Contribution requirements, all Pull Requests must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of Pull Requests happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

engcom-Golf · 2021-01-08T08:27:49Z

@magento run all tests

ihor-sviziev · 2021-01-08T13:39:04Z

app/code/Magento/Elasticsearch/Elasticsearch5/SearchAdapter/Query/Builder.php

        $searchQuery = [
            'index' => $this->searchIndexNameResolver->getIndexName($storeId, $request->getIndex()),
            'type' => $this->clientConfig->getEntityType(),
            'body' => [
-                'from' => $request->getFrom(),
+                'from' => $this->getFrom($request),


I think inline looks a bit cleaner. What do you thing?

Suggested change

'from' => $this->getFrom($request),

'from' => min((int)$request->getFrom(), self::ELASTIC_MAX_INT),

PS What about negative values? Should we also handle it?

Agree. Will update. But don't think that we need to handle negative values. In that case, elastic returns exception with message numHits must be > 0

ihor-sviziev · 2021-01-08T13:40:45Z

app/code/Magento/Elasticsearch/SearchAdapter/Query/Builder.php

@@ -61,7 +63,7 @@ public function initQuery(RequestInterface $request)
            'index' => $this->searchIndexNameResolver->getIndexName($storeId, $request->getIndex()),
            'type' => $this->clientConfig->getEntityType(),
            'body' => [
-                'from' => $request->getFrom(),
+                'from' => $this->getFrom($request),


ihor-sviziev · 2021-01-08T13:41:19Z

app/code/Magento/Elasticsearch/SearchAdapter/Query/Builder.php

@@ -23,6 +23,8 @@
 */
 class Builder extends Elasticsearch5Builder
 {
+    private const ELASTIC_INT_MAX = 2147483647;


Suggested change

private const ELASTIC_INT_MAX = 2147483647;

private const ELASTIC_MAX_INT = 2147483647;

ihor-sviziev · 2021-01-08T13:41:34Z

app/code/Magento/Elasticsearch/Elasticsearch5/SearchAdapter/Query/Builder.php

@@ -22,6 +22,8 @@
 */
 class Builder
 {
+    private const ELASTIC_INT_MAX = 2147483647;


Suggested change

private const ELASTIC_INT_MAX = 2147483647;

private const ELASTIC_MAX_INT = 2147483647;

Agree that max_int sounds better but there is defined PHP_INT_MAX constant. I think it is better to keep the same format here.

Ok, that make sense

ihor-sviziev · 2021-01-08T13:51:14Z

one more question - on elastic 5+ it works fine?

… the page will break with elastic search 2

engcom-Golf · 2021-01-11T18:19:10Z

@ihor-sviziev fix tested for elastcsearch 6 and 7 versions

engcom-Golf · 2021-01-11T18:19:19Z

@magento run all tests

magento-engcom-team · 2021-01-12T07:02:18Z

Hi @ihor-sviziev, thank you for the review.
ENGCOM-8618 has been created to process this Pull Request

engcom-Delta · 2021-01-12T15:12:22Z

✔️ QA passed
Was checked case from manual testing scenario and checked quick search and advanced search work as before
Before:
❌ Exception error in logs:
[2021-01-12 08:33:52] main.CRITICAL: {"error":{"root_cause":[{"type":"input_coercion_exception","reason":"Numeric value (11999999999999976) out of range of int (-2147483648 - 2147483647) at [Source: (org.elasticsearch.common.io.stream.InputStreamStreamInput); line: 1, column: 26]"}],"type":"input_coercion_exception","reason":"Numeric value (11999999999999976) out of range of int (-2147483648 - 2147483647) at [Source: (org.elasticsearch.common.io.stream.InputStreamStreamInput); line: 1, column: 26]"},"status":500} {"exception":"[object] (Elasticsearch\\Common\\Exceptions\\ServerErrorResponseException(code: 500): {\"error\":{\"root_cause\":[{\"type\":\"input_coercion_exception\",\"reason\":\"Numeric value (11999999999999976) out of range of int (-2147483648 - 2147483647)\ at [Source: (org.elasticsearch.common.io.stream.InputStreamStreamInput); line: 1, column: 26]\"}],\"type\":\"input_coercion_exception\",\"reason\":\"Numeric value (11999999999999976) out of range of int (-2147483648 - 2147483647)\ at [Source: (org.elasticsearch.common.io.stream.InputStreamStreamInput); line: 1, column: 26]\"},\"status\":500} at /var/www/html/magento2dev/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:665)"} []

After:
✔️ Error with status code 400 that is acceptable
[2021-01-12 09:10:00] main.CRITICAL: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [2147483659]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"magento2dev_product_1_v109","node":"YDjpqEllRKOGLQ-KYNHWcQ","reason":{"type":"illegal_argument_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [2147483659]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}}],"caused_by":{"type":"illegal_argument_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [2147483659]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.","caused_by":{"type":"illegal_argument_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [2147483659]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}}},"status":400} {"exception":"[object] (Elasticsearch\\Common\\Exceptions\\BadRequest400Exception(code: 400): {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Result window is too large, from + size must be less than or equal to: [10000] but was [2147483659]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\"magento2dev_product_1_v109\",\"node\":\"YDjpqEllRKOGLQ-KYNHWcQ\",\"reason\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Result window is too large, from + size must be less than or equal to: [10000] but was [2147483659]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.\"}}],\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Result window is too large, from + size must be less than or equal to: [10000] but was [2147483659]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Result window is too large, from + size must be less than or equal to: [10000] but was [2147483659]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.\"}}},\"status\":400} at /var/www/html/magento2dev/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:632)"} []

engcom-Delta · 2021-01-12T15:16:58Z

Note: Automation tests are passed

zakdma · 2021-01-13T09:30:54Z

@magento import code to https://github.com/magento-tsg/magento2ce

magento-engcom-team · 2021-01-13T09:34:05Z

@zakdma the branch with code successfully imported intomagento-tsg/magento2ce repository. Branch name: imported-magento-magento2-31594.

m2-assistant · 2021-01-13T20:25:59Z

Hi @engcom-Golf, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.

m2-community-project bot added Priority: P2 A defect with this priority could have functionality issues which are not to expectations. Severity: S1 Affects critical data or functionality and forces users to employ a workaround. labels Jan 8, 2021

magento-engcom-team added Component: Elasticsearch Release Line: 2.4 labels Jan 8, 2021

m2-community-project bot added this to Pending Review in High Priority Pull Requests Dashboard Jan 8, 2021

m2-community-project bot added the Progress: pending review label Jan 8, 2021

ihor-sviziev requested changes Jan 8, 2021

View reviewed changes

m2-community-project bot assigned ihor-sviziev Jan 8, 2021

m2-community-project bot moved this from Pending Review to Changes Requested in High Priority Pull Requests Dashboard Jan 8, 2021

m2-community-project bot added Progress: needs update and removed Progress: pending review labels Jan 8, 2021

ihor-sviziev added Auto-Tests: Covered All changes in Pull Request is covered by auto-tests Award: bug fix Award: test coverage labels Jan 8, 2021

m2-community-project bot moved this from Changes Requested to Review in Progress in High Priority Pull Requests Dashboard Jan 11, 2021

m2-community-project bot added Progress: review and removed Progress: needs update labels Jan 11, 2021

MC-23881: When category URL page param is larger than the pagination,…

512f4bd

… the page will break with elastic search 2

engcom-Golf force-pushed the MC-23881 branch from c1812b9 to 512f4bd Compare January 11, 2021 18:16

engcom-Golf requested a review from ihor-sviziev January 11, 2021 18:19

ihor-sviziev approved these changes Jan 12, 2021

View reviewed changes

m2-community-project bot moved this from Review in Progress to Ready for Testing in High Priority Pull Requests Dashboard Jan 12, 2021

m2-community-project bot added Progress: ready for testing and removed Progress: review labels Jan 12, 2021

zakdma approved these changes Jan 12, 2021

View reviewed changes

m2-community-project bot assigned zakdma Jan 12, 2021

m2-community-project bot moved this from Testing in Progress to Ready for Testing in High Priority Pull Requests Dashboard Jan 12, 2021

m2-community-project bot added Progress: ready for testing and removed Progress: testing in progress labels Jan 12, 2021

engcom-Delta moved this from Ready for Testing to Testing in Progress in High Priority Pull Requests Dashboard Jan 12, 2021

m2-community-project bot added Progress: testing in progress and removed Progress: ready for testing labels Jan 12, 2021

engcom-Delta moved this from Testing in Progress to Merge in Progress in High Priority Pull Requests Dashboard Jan 12, 2021

m2-community-project bot added Progress: accept and removed Progress: testing in progress labels Jan 12, 2021

engcom-Delta added the QA: Added to Regression Scope Scenario was analysed and added to Regression Testing Scope label Jan 12, 2021

m2-community-project bot moved this from Merge in Progress to Ready for Testing in High Priority Pull Requests Dashboard Jan 12, 2021

m2-community-project bot added Progress: ready for testing and removed Progress: accept labels Jan 12, 2021

engcom-Delta moved this from Ready for Testing to Testing in Progress in High Priority Pull Requests Dashboard Jan 12, 2021

engcom-Delta moved this from Testing in Progress to Merge in Progress in High Priority Pull Requests Dashboard Jan 12, 2021

m2-community-project bot added Progress: testing in progress Progress: accept and removed Progress: ready for testing Progress: testing in progress labels Jan 12, 2021

magento-engcom-team merged commit 07b1289 into magento:2.4-develop Jan 13, 2021

gabrieldagama moved this from Merge in Progress to Recently Merged in High Priority Pull Requests Dashboard Jan 25, 2021

gabrieldagama added the Type: Bug Fix label Mar 2, 2021

magento-engcom-team removed this from Recently Merged in High Priority Pull Requests Dashboard Oct 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MC-23881: When category URL page param is larger than the pagination, the page will break with elastic search 2 #31594

MC-23881: When category URL page param is larger than the pagination, the page will break with elastic search 2 #31594

engcom-Golf commented Jan 8, 2021

m2-assistant bot commented Jan 8, 2021

engcom-Golf commented Jan 8, 2021

ihor-sviziev Jan 8, 2021

engcom-Golf Jan 11, 2021

ihor-sviziev Jan 8, 2021

ihor-sviziev Jan 8, 2021

ihor-sviziev Jan 8, 2021

engcom-Golf Jan 11, 2021

ihor-sviziev Jan 11, 2021

ihor-sviziev commented Jan 8, 2021

engcom-Golf commented Jan 11, 2021

engcom-Golf commented Jan 11, 2021

magento-engcom-team commented Jan 12, 2021

engcom-Delta commented Jan 12, 2021

engcom-Delta commented Jan 12, 2021

zakdma commented Jan 13, 2021

magento-engcom-team commented Jan 13, 2021

m2-assistant bot commented Jan 13, 2021

	'from' => $this->getFrom($request),
	'from' => min((int)$request->getFrom(), self::ELASTIC_MAX_INT),

	private const ELASTIC_INT_MAX = 2147483647;
	private const ELASTIC_MAX_INT = 2147483647;

MC-23881: When category URL page param is larger than the pagination, the page will break with elastic search 2 #31594

MC-23881: When category URL page param is larger than the pagination, the page will break with elastic search 2 #31594

Conversation

engcom-Golf commented Jan 8, 2021

Description (*)

Related Pull Requests

Fixed Issues (if relevant)

Manual testing scenarios (*)

Questions or comments

Contribution checklist (*)

m2-assistant bot commented Jan 8, 2021

engcom-Golf commented Jan 8, 2021

ihor-sviziev Jan 8, 2021

Choose a reason for hiding this comment

engcom-Golf Jan 11, 2021

Choose a reason for hiding this comment

ihor-sviziev Jan 8, 2021

Choose a reason for hiding this comment

ihor-sviziev Jan 8, 2021

Choose a reason for hiding this comment

ihor-sviziev Jan 8, 2021

Choose a reason for hiding this comment

engcom-Golf Jan 11, 2021

Choose a reason for hiding this comment

ihor-sviziev Jan 11, 2021

Choose a reason for hiding this comment

ihor-sviziev commented Jan 8, 2021

engcom-Golf commented Jan 11, 2021

engcom-Golf commented Jan 11, 2021

magento-engcom-team commented Jan 12, 2021

engcom-Delta commented Jan 12, 2021

engcom-Delta commented Jan 12, 2021

zakdma commented Jan 13, 2021

magento-engcom-team commented Jan 13, 2021

m2-assistant bot commented Jan 13, 2021