Skip to content
This repository was archived by the owner on Nov 19, 2024. It is now read-only.

Added info about mview #4683

Merged
merged 4 commits into from
Jun 14, 2019
Merged

Added info about mview #4683

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c22f70a
Added info about mview
Jun 5, 2019
9759135
Added suggestions
Jun 5, 2019
c01ef50
Formatting.
dobooth Jun 7, 2019
544ab95
Merge branch 'master' into mview
dobooth Jun 14, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
264 changes: 164 additions & 100 deletions guides/v2.2/extension-dev-guide/indexing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,17 +11,14 @@ Without indexing, Magento would have to calculate the price of every product on

## Indexing terminology

<dl><dt>Dictionary</dt>
Dictionary
: Original data entered to the system. Dictionaries are organized in <a href="http://en.wikipedia.org/wiki/Database_normalization" target="_blank">normal form</a> to facilitate maintenance (updating the data).

<dd>Original data entered to the system. Dictionaries are organized in <a href="http://en.wikipedia.org/wiki/Database_normalization" target="_blank">normal form</a> to facilitate maintenance (updating the data).</dd>
Index
: Representation of the original data for optimized reading and searching. Indexes can contain results of aggregations and various calculations. Index data can be always re-created from a dictionary using a certain algorithm.

<dt>Index</dt>

<dd>Representation of the original data for optimized reading and searching. Indexes can contain results of aggregations and various calculations. Index data can be always re-created from a dictionary using a certain algorithm.</dd>

<dt>Indexer</dt>

<dd>Object that creates an index.</dd></dl>
Indexer
: Object that creates an index.

### Create custom indexers

Expand All @@ -34,14 +31,14 @@ This topic provides a high level description of how indexing is implemented from
The following components are involved in the indexing process:

<table>
<tbody>
<tr>
<th>Component</th>
<th>Description</th>
</tr>
<tr>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/Indexer" target="_blank">Magento_Indexer</a></td>
<td>Implements:<ul>
<tbody>
<tr>
<th>Component</th>
<th>Description</th>
</tr>
<tr>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/Indexer" target="_blank">Magento_Indexer</a></td>
<td>Implements:<ul>
<li>indexer declaration</li>
<li>indexer running</li>
<li>indexer running mode configuration</li>
Expand All @@ -55,20 +52,18 @@ The following components are involved in the indexing process:
</tr>
</tbody></table>

{: .bs-callout .bs-callout-warning }
{: .bs-callout .bs-callout-warning }
`Magento_Indexer` replaces the Magento 1.x `Magento_Index` module. Use `Magento_Indexer` for all new development.

### Indexing types

Each index can perform the following types of reindex operations:

* Full reindex, which means rebuilding all the indexing-related database tables
* Full reindex, which means rebuilding all the indexing-related database tables
Full reindexing can be caused by a variety of things, including creating a new web store or new customer group.
You can optionally fully reindex at any time using the [command line]({{ page.baseurl }}/config-guide/cli/config-cli-subcommands-index.html).

Full reindexing can be caused by a variety of things, including creating a new web store or new customer group.

You can optionally fully reindex at any time using the [command line]({{ page.baseurl }}/config-guide/cli/config-cli-subcommands-index.html).

* Partial reindex, which means rebuilding the database tables only for the things that changed (like changing a single product attribute or price)
* Partial reindex, which means rebuilding the database tables only for the things that changed (like changing a single product attribute or price)

The type of reindex performed in each particular case depends on the type of changes made in the dictionary or in the system. This dependency is specific for [each indexer](#m2devgde-indexing-outofbox).

Expand All @@ -80,18 +75,18 @@ The following figure shows the logic for partial reindexing.

Depending on whether an index data is up to date, an indexer status value is one of the following:

* valid - data is synchronized, no reindex required
* invalid - the original data was changed, the index should be updated
* working - indexing is in progress
* valid - data is synchronized, no reindex required
* invalid - the original data was changed, the index should be updated
* working - indexing is in progress

The Magento indexing mechanism uses the status value in reindex triggering process. You can check the status of an indexer in the [Admin](https://glossary.magento.com/admin) panel in **System >** Tools **> Index Management** or manually using the [command line]({{ page.baseurl }}/config-guide/cli/config-cli-subcommands-index.html#view-indexer-status).

### Indexing modes {#m2devgde-indexing-modes}

Reindexing can be performed in two modes:

* Update on Save - index tables are updated immediately after the dictionary data is changed.
* Update by Schedule - index tables are updated by cron job according to the configured schedule.
* Update on Save - index tables are updated immediately after the dictionary data is changed.
* Update by Schedule - index tables are updated by cron job according to the configured schedule.

{:.bs-callout .bs-callout-info}
**Update by Schedule** does not support the `customer_grid` indexer. You must either use **Update on Save** or reindex the customer grid manually (`bin/magento indexer:reindex customer_grid`). See the [Help Center article](https://support.magento.com/hc/en-us/articles/360025481892-New-customer-records-are-not-displayed-in-the-Customers-grid-after-importing-them-from-CSV).
Expand All @@ -110,85 +105,154 @@ The following figure shows an example of setting indexers to Update by Schedule:

![Changing indexer modes]({{ site.baseurl }}/common/images/index_index-modes.png){:width="600px"}

### Mview {#m2devgde-mview}

The `mview.xml` file is used to track database changes for a certain entity.

For example part of `Magento/Catalog/etc/mview.xml` is tracking category to product relation described in the following record:

```xml
<!-- ... -->
<view id="catalog_category_product" class="Magento\Catalog\Model\Indexer\Category\Product" group="indexer">
<subscriptions>
<table name="catalog_category_entity" entity_column="entity_id" />
<table name="catalog_category_entity_int" entity_column="entity_id" />
</subscriptions>
</view>
<!-- ... -->
```

Explanation of nodes:

* The `view` node defines an indexer. The `id` attribute is a name of the indexer table, the `class` attribute is indexer executor, the `group` attribute defines
the indexer group.
* The `subscriptions` node is a list of tables for tracking changes.
* The `table` node defines the certain table to observe and track changes. The attribute `name` is a name of an observable table, the attribute `entity_column`
is an identifier column of entity to be re-indexed. So, in case of `catalog_category_product`, whenever one or more categories is saved, updated or deleted in `catalog_category_entity`
the `execute` method of `Magento\Catalog\Model\Indexer\Category\Product` will be called with argument `ids` containing ids of entities from column defined
under `entity_column` attribute. If indexer type is set to "Update on Save" the method is called right away after the operation. If it set to "Update by Schedule"
the mechanism creates a record in the change log table using MYSQL triggers.

A change log table is created according to the naming rule - INDEXER_TABLE_NAME + '_cl', in case of `catalog_category_product` it will be `catalog_category_product_cl`.
The table contains the `version_id` auto-increment column and `entity_id` column that contains identifiers of entities to be re-indexed.
For each `table` node the framework automatically creates MYSQL AFTER triggers for each possible event (INSERT, UPDATE, DELETE).

For the table `catalog_category_entity` triggers will be created with the following statements.
INSERT operation:

```mysql
BEGIN
INSERT IGNORE INTO `catalog_category_product_cl` (`entity_id`) VALUES (NEW.`entity_id`);
END
```

UPDATE operation:

```mysql
BEGIN
IF (NEW.`entity_id` <=> OLD.`entity_id`
OR NEW.`attribute_set_id` <=> OLD.`attribute_set_id`
OR NEW.`parent_id` <=> OLD.`parent_id`
OR NEW.`created_at` <=> OLD.`created_at`
OR NEW.`path` <=> OLD.`path`
OR NEW.`position` <=> OLD.`position`
OR NEW.`level` <=> OLD.`level`
OR NEW.`children_count` <=> OLD.`children_count`)
THEN INSERT IGNORE INTO `catalog_category_product_cl` (`entity_id`) VALUES (NEW.`entity_id`);
END IF;
END
```

DELETE operation:

```mysql
BEGIN
INSERT IGNORE INTO `catalog_category_product_cl` (`entity_id`) VALUES (OLD.`entity_id`);
END
```

The method `Magento\Framework\Mview\ViewInterface::update` is responsible for handling records in the changelog. The method is being called by CRON and
it defines IDs to be re-indexed from the change log by last applied `version_id` and calls the `execute` method for each particular indexer with IDs as an argument.

### How to reindex

You can reindex by:

* Using a [cron job]({{ page.baseurl }}/config-guide/cli/config-cli-subcommands-cron.html), which is preferred because indexing runs every minute.
* Using the [`magento indexer:reindex [indexer]`]({{ page.baseurl }}/config-guide/cli/config-cli-subcommands-index.html#config-cli-subcommands-index-reindex) command, which reindexes selected indexers, or all indexers, one time only.
* Using a [cron job]({{ page.baseurl }}/config-guide/cli/config-cli-subcommands-cron.html), which is preferred because indexing runs every minute.
* Using the [`magento indexer:reindex [indexer]`]({{ page.baseurl }}/config-guide/cli/config-cli-subcommands-index.html#config-cli-subcommands-index-reindex) command, which reindexes selected indexers, or all indexers, one time only.

## Magento indexers {#m2devgde-indexing-outofbox}

The Magento application implements the following indexers:

<table>
<tbody>
<tr>
<th>Indexer name</th>
<th>Indexer method name</th>
<th>Indexer class</th>
<th>Description</th>
</tr>
<tr>
<td>Design Config Grid</td>
<td><code>design_config_grid</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/Theme/Model/Indexer/Design/Config.php" target="_blank">Magento\Theme\Model\Indexer\Design\Config</a></td>
<td />
</tr>
<tr>
<td>Customer Grid</td>
<td><code>customer_grid</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/lib/internal/Magento/Framework/Indexer/Action/Entity.php" target="_blank">Magento\Framework\Indexer\Action\Entity</a></td>
<td>Rebuilds the customer grid index.<br><br>Not supported by the <strong>Update by Schedule</strong> indexing mode. See the <a href="https://support.magento.com/hc/en-us/articles/360025481892-New-customer-records-are-not-displayed-in-the-Customers-grid-after-importing-them-from-CSV)">Help Center article</a>.</td>
<td />
</tr>
<tr>
<td>Category products</td>
<td><code>catalog_category_product</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/Catalog/Model/Indexer/Category/Product.php" target="_blank">Magento\Catalog\Model\Indexer\Category\Product</a></td>
<td>Creates category/products association</td>
</tr>
<tr>
<td>Product categories</td>
<td><code>catalog_product_category</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/Catalog/Model/Indexer/Product/Category.php" target="_blank">Magento\Catalog\Model\Indexer\Product\Category</a></td>
<td>Creates category/products association</td>
</tr>
<tr>
<td>Product price</td>
<td><code>catalog_product_price</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/Catalog/Model/Indexer/Product/Price.php" target="_blank">Magento\Catalog\Model\Indexer\Product\Price</a></td>
<td>Pre-calculates product prices</td>
</tr>
<tr>
<td>Product entity attribute value</td>
<td><code>catalog_product_attribute</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/Catalog/Model/Indexer/Product/Eav.php" target="_blank">Magento\Catalog\Model\Indexer\Product\Eav</a></td>
<td>Reorganizes the EAV product structure to flat structure</td>
</tr>
<tr>
<td>Stock</td>
<td><code>cataloginventory_stock</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/CatalogInventory/Model/Indexer/Stock.php" target="_blank">Magento\CatalogInventory\Model\Indexer\Stock</a></td>
<td />
</tr>
<tr>
<td>Catalog rule product</td>
<td><code>catalogrule_rule</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/CatalogRule/Model/Indexer/Rule/RuleProductIndexer.php" target="_blank">Magento\CatalogRule\Model\Indexer\Rule\RuleProductIndexer</a></td>
<td />
</tr>
<tr>
<td>Catalog product rule</td>
<td><code>catalogrule_product</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/CatalogRule/Model/Indexer/Product/ProductRuleIndexer.php" target="_blank">Magento\CatalogRule\Model\Indexer\Product\ProductRuleIndexer</a></td>
<td />
</tr>
<tr>
<td>Catalog search</td>
<td><code>catalogsearch_fulltext</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/CatalogSearch/Model/Indexer/Fulltext.php" target="_blank">Magento\CatalogSearch\Model\Indexer\Fulltext</a></td>
<td />
</tr>
</tbody>
<tbody>
<tr>
<th>Indexer name</th>
<th>Indexer method name</th>
<th>Indexer class</th>
<th>Description</th>
</tr>
<tr>
<td>Design Config Grid</td>
<td><code>design_config_grid</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/Theme/Model/Indexer/Design/Config.php" target="_blank">Magento\Theme\Model\Indexer\Design\Config</a></td>
<td />
</tr>
<tr>
<td>Customer Grid</td>
<td><code>customer_grid</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/lib/internal/Magento/Framework/Indexer/Action/Entity.php" target="_blank">Magento\Framework\Indexer\Action\Entity</a></td>
<td>Rebuilds the customer grid index.<br><br>Not supported by the <strong>Update by Schedule</strong> indexing mode. See the <a href="https://support.magento.com/hc/en-us/articles/360025481892-New-customer-records-are-not-displayed-in-the-Customers-grid-after-importing-them-from-CSV)">Help Center article</a>.</td>
<td />
</tr>
<tr>
<td>Category products</td>
<td><code>catalog_category_product</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/Catalog/Model/Indexer/Category/Product.php" target="_blank">Magento\Catalog\Model\Indexer\Category\Product</a></td>
<td>Creates category/products association</td>
</tr>
<tr>
<td>Product categories</td>
<td><code>catalog_product_category</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/Catalog/Model/Indexer/Product/Category.php" target="_blank">Magento\Catalog\Model\Indexer\Product\Category</a></td>
<td>Creates category/products association</td>
</tr>
<tr>
<td>Product price</td>
<td><code>catalog_product_price</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/Catalog/Model/Indexer/Product/Price.php" target="_blank">Magento\Catalog\Model\Indexer\Product\Price</a></td>
<td>Pre-calculates product prices</td>
</tr>
<tr>
<td>Product entity attribute value</td>
<td><code>catalog_product_attribute</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/Catalog/Model/Indexer/Product/Eav.php" target="_blank">Magento\Catalog\Model\Indexer\Product\Eav</a></td>
<td>Reorganizes the EAV product structure to flat structure</td>
</tr>
<tr>
<td>Stock</td>
<td><code>cataloginventory_stock</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/CatalogInventory/Model/Indexer/Stock.php" target="_blank">Magento\CatalogInventory\Model\Indexer\Stock</a></td>
<td />
</tr>
<tr>
<td>Catalog rule product</td>
<td><code>catalogrule_rule</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/CatalogRule/Model/Indexer/Rule/RuleProductIndexer.php" target="_blank">Magento\CatalogRule\Model\Indexer\Rule\RuleProductIndexer</a></td>
<td />
</tr>
<tr>
<td>Catalog product rule</td>
<td><code>catalogrule_product</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/CatalogRule/Model/Indexer/Product/ProductRuleIndexer.php" target="_blank">Magento\CatalogRule\Model\Indexer\Product\ProductRuleIndexer</a></td>
<td />
</tr>
<tr>
<td>Catalog search</td>
<td><code>catalogsearch_fulltext</code></td>
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/CatalogSearch/Model/Indexer/Fulltext.php" target="_blank">Magento\CatalogSearch\Model\Indexer\Fulltext</a></td>
<td />
</tr>
</tbody>
</table>