Skip to content

[Issue] Remove limit in IdListBuilder #39621

Closed
@m2-assistant

Description

@m2-assistant

This issue is automatically created based on existing pull request: #39603: Remove limit in IdListBuilder


There should be no limit on the IdListBuilder since it is chunked in: vendor/magento/module-sales/Model/ResourceModel/Grid.php:

    public function refreshBySchedule()
    {
        $lastUpdatedAt = null;
        $notSyncedIds = $this->notSyncedDataProvider->getIds($this->mainTableName, $this->gridTableName);
        foreach (array_chunk($notSyncedIds, self::BATCH_SIZE) as $bunch) {
            $select = $this->getGridOriginSelect()->where($this->mainTableName . '.entity_id IN (?)', $bunch);

Description (*)

The limit in the select is removed so it will fetch all results and later it will be chuncked.

Related Pull Requests

Manual testing scenarios (*)

  1. Make sure you have more than 100 orders
  2. Make sure your sales_order_grid is also filled with this orders
  3. Truncate sales_order_grid
  4. Make sure async indexing of grids is true
  5. run magerun2 sys:cron:run sales_grid_order_async_insert

vendor/magento/module-sales/Model/ResourceModel/Grid.php

    public function refreshBySchedule()
    {
        $lastUpdatedAt = null;
        $notSyncedIds = $this->notSyncedDataProvider->getIds($this->mainTableName, $this->gridTableName);
        foreach (array_chunk($notSyncedIds, self::BATCH_SIZE) as $bunch) {
            $select = $this->getGridOriginSelect()->where($this->mainTableName . '.entity_id IN (?)', $bunch);

The code above will be run and fetch $notSyncedIds and chunck it in batches. The provider used is: vendor/magento/module-sales/Model/ResourceModel/Provider/UpdatedIdListProvider.php and does this:

    public function getIds($mainTableName, $gridTableName)
    {
        $mainTableName = $this->resourceConnection->getTableName($mainTableName);
        $gridTableName = $this->resourceConnection->getTableName($gridTableName);
        $select = $this->idListQueryBuilder->build($mainTableName, $gridTableName);
        return $this->getConnection()->fetchAll($select, [], \Zend_Db::FETCH_COLUMN);

and builds a select with the idListQueryBuilder:

public function build(string $mainTableName, string $gridTableName): Select
    {
        $select = $this->getConnection()->select()
            ->from(['main_table' => $mainTableName], ['main_table.entity_id'])
            ->joinLeft(
                ['grid_table' => $this->resourceConnection->getTableName($gridTableName)],
                'main_table.entity_id = grid_table.entity_id',
                []
            );

        $select->where('grid_table.entity_id IS NULL');
        $select->limit(Grid::BATCH_SIZE); // should be removed
        foreach ($this->additionalGridTables as $table) {
            $select->joinLeft(
                [$table => $table],
                sprintf(
                    '%s.%s = %s.%s',
                    'main_table',
                    'entity_id',
                    $table,
                    'entity_id'
                ),
                []
            )
                ->where($table . '.entity_id IS NULL');
        }
        return $select;
    }

In here there is limit: $select->limit(Grid::BATCH_SIZE); the same as the chunk batch size in refreshBySchedule. This looks like a bug to me, because you will never get more items than 100. Now this will probably only occur when the sales_order_grid is truncated or more than 100 orders are placed between the cron execution.

Questions or comments

Eventually the grid will be filled, because the cron is run every minute. But it takes a lot of time if you have a lot of orders.

Contribution checklist (*)

  • Pull request has a meaningful description of its purpose
    All commits are accompanied by meaningful commit messages
    All new or changed code is covered with unit/integration tests (if applicable)
    README.md files for modified modules are updated and included in the pull request if any README.md predefined sections require an update
    All automated tests passed successfully (all builds are green)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Issue: ready for confirmationReported on 2.4.xIndicates original Magento version for the Issue report.Triage: Dev.ExperienceIssue related to Developer Experience and needs help with Triage to Confirm or Reject it

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      Participants

      @hostep@engcom-Bravo@engcom-Hotel

      Issue actions

        [Issue] Remove limit in IdListBuilder · Issue #39621 · magento/magento2