Replaced `CREATE TEMPORARY TABLE ... LIKE`with different approach #36022

michalbiarda · 2022-08-25T10:39:40Z

Description (*)

Magento's MySQL adapter uses CREATE TEMPORARY TABLE ... LIKE command in \Magento\Framework\DB\Adapter\Pdo\Mysql::createTemporaryTableLike.

Newer versions of MySQL (8+) have problems with CREATE TEMPORARY TABLE ... LIKE, as you can read in the official documentation (https://dev.mysql.com/doc/refman/8.0/en/create-temporary-table.html):

You cannot use CREATE TEMPORARY TABLE ... LIKE to create an empty table based on the definition of a table that resides in the mysql tablespace, InnoDB system tablespace (innodb_system), or a general tablespace.

The documentation suggests to use CREATE TEMPORARY TABLE ... SELECT * FROM ... LIMIT 0 instead, but such command IS NOT an exact equivalent. It will create temporary table with the same fields, but will not create indexes. Without indexes performance of reindexing big amount of data is absurdly slow.

I suggest to build CREATE TEMPORARY TABLE without LIKE and without SELECT * FROM. Here's my idea:

Firstly use SHOW CREATE TABLE to get query for creation of source table (including indexes creation), and then modify it with PHP: change CREATE TABLE to CREATE TEMPORARY TABLE, add IF NOT EXISTS if necessary, remove TABLESPACE definition if it exists, and replace source table name with temporary table name.

Manual testing scenarios (*)

Install Magento 2.4 on a server with MySQL 8.
Apply sample data.
Run bin/magento index:reindex.
Check data on frontend.

Contribution checklist (*)

Pull request has a meaningful description of its purpose
All commits are accompanied by meaningful commit messages
All new or changed code is covered with unit/integration tests (if applicable)
README.md files for modified modules are updated and included in the pull request if any README.md predefined sections require an update
All automated tests passed successfully (all builds are green)

Resolved issues:

resolves [Issue] Replaced CREATE TEMPORARY TABLE ... LIKEwith different approach #37926: Replaced CREATE TEMPORARY TABLE ... LIKEwith different approach

…at doesn't conflict with MySQL 8

m2-assistant · 2022-08-25T10:39:42Z

Hi @michalbiarda. Thank you for your contribution
Here are some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

@magento give me test instance - deploy test instance based on PR changes
@magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

@magento run all tests - run or re-run all required tests against the PR changes
@magento run <test-build(s)> - run or re-run specific test build(s)
For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names. Allowed build names are:

Database Compare
Functional Tests CE
Functional Tests EE,
Functional Tests B2B
Integration Tests
Magento Health Index
Sample Data Tests CE
Sample Data Tests EE
Sample Data Tests B2B
Static Tests
Unit Tests
WebAPI Tests
Semantic Version Checker

You can find more information about the builds here

ℹ️ Run only required test builds during development. Run all test builds before sending your pull request for review.

For more details, review the Magento Contributor Guide documentation.

⚠️ According to the Magento Contribution requirements, all Pull Requests must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of Pull Requests happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

michalbiarda · 2022-08-25T10:44:27Z

@magento give me test instance

magento-deployment-service · 2022-08-25T10:44:31Z

Hi @michalbiarda. Thank you for your request. I'm working on Magento instance for you.

magento-deployment-service · 2022-08-25T10:48:29Z

Hi @michalbiarda, here is your Magento Instance: https://eb2c38a2f68947025bfccecf2d856c46.instances.magento-community.engineering
Admin access: https://eb2c38a2f68947025bfccecf2d856c46.instances.magento-community.engineering/admin_7f99
Login: 8e47b66d
Password: fd846693815e

michalbiarda · 2022-08-25T10:51:03Z

@magento run all tests

magento-automated-testing · 2022-08-25T10:51:09Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

kandy · 2022-08-25T14:25:53Z

lib/internal/Magento/Framework/DB/Adapter/Pdo/Mysql.php

        $temporaryTable = $this->quoteIdentifier($this->_getTableName($temporaryTableName));
        $originTable = $this->quoteIdentifier($this->_getTableName($originTableName));
-        $sql = sprintf('CREATE TEMPORARY TABLE %s %s LIKE %s', $ifNotExistsSql, $temporaryTable, $originTable);
+        $originCreate = $this->fetchPairs(sprintf('SHOW CREATE TABLE %s', $originTable));


Suggested change

$originCreate = $this->fetchPairs(sprintf('SHOW CREATE TABLE %s', $originTable));

$originCreate = $this->fetchOne("SHOW CREATE TABLE {$originTable}");

You can use fetchOne to get result
Also sprintf is unnecessary

@kandy Unfortunately I can't use fetchOne because SHOW CREATE TABLE returns two fields. First is table name, the second is SQL query.

I like you suggestion about sprintf, though 👍.

michalbiarda · 2022-08-25T14:38:36Z

@kandy I'm converting this PR to draft. I'll apply your suggestion about sprintf, but I will also fix one more thing reported by someone else in the meantime.

…nflict with MySQL 8

michalbiarda · 2022-08-26T13:58:46Z

@magento run all tests

magento-automated-testing · 2022-08-26T13:58:51Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

engcom-Hotel · 2023-06-08T12:50:31Z

@magento run Static Tests, Functional Tests EE

magento-automated-testing · 2023-06-08T12:50:39Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

engcom-Hotel · 2023-07-14T10:02:34Z

✔️ QA Passed

Preconditions:

SET innodb_file_per_table to OFF.

Manual testing scenario:

Install Magento 2.4 on a server with MySQL 8.
Generate data using the performance toolkit provided by Magento. Please refer to this link for this same.
Run bin/magento index:reindex.
Check data on front end.

Actual Result: ✔️

Reindex command should run properly without any errors and frontend is accessible with categories and products.

After: ✔️

Before: ✖️

Some functional tests are still failing hence moving this PR into extended testing.

Thanks

engcom-Echo · 2023-07-17T04:34:22Z

@magento run Functional Tests EE

magento-automated-testing · 2023-07-17T04:34:31Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

engcom-Echo · 2023-07-17T06:34:39Z

Functional Tests EE failures are different on last two run on same commit.
Run1

Run2

Hence moving to Merge In progress

engcom-Charlie · 2023-08-25T09:51:21Z

@magento run Functional Tests EE

magento-automated-testing · 2023-08-25T09:51:26Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

engcom-Hotel · 2023-08-28T10:26:04Z

@magento create issue

engcom-Hotel · 2023-08-28T11:31:56Z

Moving back this PR to Merge in progress

Replaced CREATE TEMPORARY TABLE ... LIKE with different approach th…

c47f5c5

…at doesn't conflict with MySQL 8

kandy reviewed Aug 25, 2022

View reviewed changes

m2-community-project bot assigned kandy Aug 25, 2022

m2-community-project bot added this to Review in Progress in Pull Requests Dashboard Aug 25, 2022

m2-community-project bot added the Progress: review label Aug 25, 2022

michalbiarda marked this pull request as draft August 25, 2022 14:37

Removed TABLESPACE definition from CREATE TEMPORARY TABLE to not co…

a1f1ee3

…nflict with MySQL 8

michalbiarda marked this pull request as ready for review August 26, 2022 13:55

engcom-Hotel added the Priority: P1 Once P0 defects have been fixed, a defect having this priority is the next candidate for fixing. label Aug 26, 2022

m2-community-project bot added this to Review in Progress in High Priority Pull Requests Dashboard Aug 26, 2022

m2-community-project bot removed this from Review in Progress in Pull Requests Dashboard Aug 26, 2022

ishakhsuvarov removed this from Review in Progress in High Priority Pull Requests Dashboard Dec 8, 2022

ishakhsuvarov added this to Pending Review in Pull Requests Dashboard Dec 8, 2022

m2-community-project bot added Progress: pending review and removed Progress: review labels Dec 8, 2022

nrkapoor assigned chernenm May 28, 2023

m2-community-project bot moved this from Pending Review to Review in Progress in Pull Requests Dashboard May 28, 2023

m2-community-project bot added Progress: review and removed Progress: pending review labels May 28, 2023

m2-community-project bot added Progress: ready for testing and removed Progress: review labels Jul 13, 2023

engcom-Hotel moved this from Ready for Testing to Testing in Progress in Community Dashboard Jul 14, 2023

m2-community-project bot added Progress: testing in progress and removed Progress: ready for testing labels Jul 14, 2023

engcom-Hotel moved this from Testing in Progress to Extended testing (optional) in Community Dashboard Jul 14, 2023

m2-community-project bot added Progress: extended testing and removed Progress: testing in progress labels Jul 14, 2023

engcom-Echo moved this from Extended testing (optional) to Merge in Progress in Community Dashboard Jul 17, 2023

m2-community-project bot added Progress: accept and removed Progress: extended testing labels Jul 17, 2023

m2-assistant bot mentioned this pull request Aug 28, 2023

[Issue] Replaced CREATE TEMPORARY TABLE ... LIKEwith different approach #37926

Closed

5 tasks

m2-community-project bot moved this from Merge in Progress to Ready for Testing in Community Dashboard Aug 28, 2023

m2-community-project bot added Progress: ready for testing and removed Progress: accept labels Aug 28, 2023

engcom-Hotel moved this from Ready for Testing to Merge in Progress in Community Dashboard Aug 28, 2023

m2-community-project bot added Progress: accept and removed Progress: ready for testing labels Aug 28, 2023

magento-devops-reposync-svc merged commit e947b85 into magento:2.4-develop Aug 31, 2023
14 checks passed

engcom-Echo moved this from Merge in Progress to Recently Merged in Community Dashboard Aug 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replaced `CREATE TEMPORARY TABLE ... LIKE`with different approach #36022

Replaced `CREATE TEMPORARY TABLE ... LIKE`with different approach #36022

michalbiarda commented Aug 25, 2022 •

edited by m2-assistant bot

m2-assistant bot commented Aug 25, 2022

michalbiarda commented Aug 25, 2022

magento-deployment-service bot commented Aug 25, 2022

magento-deployment-service bot commented Aug 25, 2022

michalbiarda commented Aug 25, 2022

magento-automated-testing bot commented Aug 25, 2022

kandy Aug 25, 2022

michalbiarda Aug 25, 2022

michalbiarda commented Aug 25, 2022

michalbiarda commented Aug 26, 2022

magento-automated-testing bot commented Aug 26, 2022

engcom-Hotel commented Jun 8, 2023

magento-automated-testing bot commented Jun 8, 2023

engcom-Hotel commented Jul 14, 2023

engcom-Echo commented Jul 17, 2023

magento-automated-testing bot commented Jul 17, 2023

engcom-Echo commented Jul 17, 2023

engcom-Charlie commented Aug 25, 2023

magento-automated-testing bot commented Aug 25, 2023

engcom-Hotel commented Aug 28, 2023

engcom-Hotel commented Aug 28, 2023

	$originCreate = $this->fetchPairs(sprintf('SHOW CREATE TABLE %s', $originTable));
	$originCreate = $this->fetchOne("SHOW CREATE TABLE {$originTable}");

Replaced CREATE TEMPORARY TABLE ... LIKEwith different approach #36022

Replaced CREATE TEMPORARY TABLE ... LIKEwith different approach #36022

Conversation

michalbiarda commented Aug 25, 2022 • edited by m2-assistant bot

Description (*)

Manual testing scenarios (*)

Contribution checklist (*)

Resolved issues:

m2-assistant bot commented Aug 25, 2022

michalbiarda commented Aug 25, 2022

magento-deployment-service bot commented Aug 25, 2022

magento-deployment-service bot commented Aug 25, 2022

michalbiarda commented Aug 25, 2022

magento-automated-testing bot commented Aug 25, 2022

kandy Aug 25, 2022

Choose a reason for hiding this comment

michalbiarda Aug 25, 2022

Choose a reason for hiding this comment

michalbiarda commented Aug 25, 2022

michalbiarda commented Aug 26, 2022

magento-automated-testing bot commented Aug 26, 2022

engcom-Hotel commented Jun 8, 2023

magento-automated-testing bot commented Jun 8, 2023

engcom-Hotel commented Jul 14, 2023

engcom-Echo commented Jul 17, 2023

magento-automated-testing bot commented Jul 17, 2023

engcom-Echo commented Jul 17, 2023

engcom-Charlie commented Aug 25, 2023

magento-automated-testing bot commented Aug 25, 2023

engcom-Hotel commented Aug 28, 2023

engcom-Hotel commented Aug 28, 2023

Replaced `CREATE TEMPORARY TABLE ... LIKE`with different approach #36022

Replaced `CREATE TEMPORARY TABLE ... LIKE`with different approach #36022

michalbiarda commented Aug 25, 2022 •

edited by m2-assistant bot