AUTO_INCREMENT grows on every INSERT ... ON DUPLICATE on InnoDB tables

[ilnytskyi]

Preconditions (*)

1. any magento version, mysql or mariaDB database
2. Magento 2.4-develop
3. InnoDB tables
4. Magento uses queries like this to save new values or update existing in tables
   see method \Magento\Eav\Model\Entity\AbstractEntity::_processAttributeValues

INSERT INTO `catalog_product_entity_int` (`attribute_id`, `store_id`, `entity_id`, `value`)
VALUES (97, 1, 123, 1)
ON DUPLICATE KEY UPDATE `value` = VALUES(`value`);

The problem causes that the table reaches max autoincrement value too fast that leads to errors.
Changing column type to bigint just temporalny resolves the problem but autoincrement still grows to infinity
Problem is noticeable when products or some attributes are updated often and new products/attribute values (e.g. in store) added. New attributes have bigger autoincrement than expected.
Moreover, the method \Magento\Eav\Model\Entity\AbstractEntity::_processAttributeValues always uses statements INSERT INTO ... ON DUPLICATE KEY UPDATE to insert new value or update existing. So updating existing values this way when innodb_autoinc_lock_mode > 0 causes the issue.

Steps to reproduce (*)

1. Take auto_increment value from table catalog_product_entity_int
2. Remember that value (e.g. 1000)
3. Save one new product attribute (e.g. int type).

$this->productResourceModel->saveAttribute($product, 'status');

it generates query like above. INSERT INTO ... ON DUPLICATE KEY UPDATE
4. New value inserted and auto_increment is 1001
5. Run the same query, or save the same attribute 4 more times
6. Save the same attribute under store (important that value not exist before insertion)
7. Check the auto_increment value of the table or MAX(value_id)

Expected result (*)

1. The auto_increment value or MAX(value_id) is 1002
2. Only one new record inserted that increased auto_increment by 1

Actual result (*)

1. The auto_increment value or MAX(value_id) is 1006
2. Only one new record inserted but auto_increment increased by 5

Workaround: (*)

Set parametr in my.cnf

innodb_autoinc_lock_mode=0

or recreate column

ALTER TABLE catalog_product_entity_int
drop column value_id;

ALTER TABLE catalog_product_entity_int
ADD value_id BIGINT(11) AUTO_INCREMENT NOT NULL FIRST,
ADD PRIMARY KEY (value_id);

---

Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.

• Severity: S0 - Affects critical data or functionality and leaves users without workaround.
• Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

[m2-assistant]

Hi @ilnytskyi. Thank you for your report.
To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[ilnytskyi]

At first growing auto_increment will cause the issue

SQLSTATE[22003]: Numeric value out of range: 167 Out of range value for column 'value_id' at row 1, query was: INSERT INTO catalog_product_entity_int

Then devs might consider changing the column type to allocate bigger ints

ALTER TABLE catalog_product_entity_int MODIFY value_id bigint(20) unsigned NOT NULL auto_increment COMMENT 'Value ID';

but the value will be growing

It seems to be DB issue or db configuration issue, however It is not mentioned or documented by Magento.

Additionally, magento does not chose between INSERT or UPDATE when add or modify values so the issue happens.

https://dev.mysql.com/doc/refman/8.0/en/innodb-auto-increment-handling.html

Related issue in other repo:
dweeves/magmi-git#320

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 14 days if no further activity occurs. Thank you for your contributions.

[hostep]

Hmm, let's remove the stale issue label, this sounds pretty important.

@sidolov: can somebody verify this and if verified add some priorities?

[m2-assistant]

Hi @engcom-Delta. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• 4. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• 5. Add label Issue: Confirmed once verification is complete.

• 6. Make sure that automatic system confirms that report has been added to the backlog.

[magento-engcom-team]

✅ Confirmed by @engcom-Delta
Thank you for verifying the issue. Based on the provided information internal tickets MC-40808 were created

Issue Available: @engcom-Delta, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

[m2-assistant]

✅ Confirmed by @engcom-Hotel. Thank you for verifying the issue.
Issue Available: @engcom-Hotel, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

[convenient]

Just hit this on a staging instance (thankfully)

[pmonosolo]

Just hit this on a staging instance (thankfully)

@convenient What version of M2 are you running?

[convenient]

@pmonosolo It was a 2.4.2 instance

[pmonosolo]

@convenient hmm. Since 2021, I've been using lock_mode = 0 and its been holding up. Its not perfect - I'm having issues with DB locking - possibly because of having to use lock_mode = 0.

[convenient]

Thanks @pmonosolo

We're just building a module that will fire an alert when we are over 90% capacity on auto increment IDs, to allow us to do maintenance.

It's only every few years we've encountered it

[pmonosolo]

Thanks @pmonosolo

We're just building a module that will fire an alert when we are over 90% capacity on auto increment IDs, to allow us to do maintenance.

It's only every few years we've encountered it

Whats the plan when you do hit the limit?

Change to differen field type or there is some way of purging the table?

[convenient]

@pmonosolo #21890

The "workaround" section on that is largely valid

Due to declarative schema we need to drop the constraints on the existing table, and add properly named constraints on the new table before swapping the table names around.

But this way we end up back at auto_increment=1 with the original Magento schema intact

[TomJ1588]

Any update on this issue? Just ran into this in production. Any timeframe on a fix?

[tuey30716]

Any update on this issue? Just ran into this in production. Any timeframe on a fix?

change column type on value_id to bigint(20)
or
following this step https://gist.github.com/JesKingDev/af70e92740215009ab1bebe5a2cdc17f#2-get-the-current-max-value_id-used-in-the-table

[engcom-Bravo]

Hello,

As I can see this issue got fixed in the scope of the internal Jira ticket ACP2E-1358 by the internal team
Related commits:https://github.com/search?q=repo%3Amagento%2Fmagento2+ACP2E-1358&type=commits

Based on the Jira ticket, the target version is 2.4.7-beta1.

Thanks