Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Product EAV index throws error with GTID consistency #12124

Closed
scrivvles opened this issue Nov 9, 2017 · 32 comments · Fixed by magento/devdocs#9111
Closed

Product EAV index throws error with GTID consistency #12124

scrivvles opened this issue Nov 9, 2017 · 32 comments · Fixed by magento/devdocs#9111
Assignees
Labels
Area: Catalog Component: CatalogRule Issue: Confirmed Gate 3 Passed. Manual verification of the issue completed. Issue is confirmed Priority: P1 Once P0 defects have been fixed, a defect having this priority is the next candidate for fixing. Progress: done Reproduced on 2.1.x The issue has been reproduced on latest 2.1 release Reproduced on 2.2.x The issue has been reproduced on latest 2.2 release Reproduced on 2.3.x The issue has been reproduced on latest 2.3 release Severity: S1 Affects critical data or functionality and forces users to employ a workaround. Triage: Dev.Experience Issue related to Developer Experience and needs help with Triage to Confirm or Reject it

Comments

@scrivvles
Copy link

Attempting to reindex the Product EAV when binary logging or replication is enabled on MySQL 5.7throws a SQL error. This same type of error is thrown when attempting to create a new category.

Preconditions

  1. Environment is NGINX 1.10.3 on Ubuntu 16.04.3 LTS running PHP7.0-fpm on a Google Cloud instance.
  2. MySQL version is 5.7 running on Google Cloud Platform SQL

Steps to reproduce

  1. Make sure that your Google SQL db has binary logging enabled AND/OR you have replication enabled.
  2. go to your magento installation and run php bin/magento indexer:reindex

Expected result

All indexes run correctly and display that they have rebuilt successfully

Actual result

SQLSTATE[HY000]: General error: 1787 Statement violates GTID consistency: CREATE TEMPORARY TABLE and DROP TEMPORARY TABLE can only be executed outside transactional context. These statements are also not allowed in a function or trigger because functions and triggers are also considered to be multi-statement transactions., query was: CREATE TEMPORARY TABLE IF NOT EXISTS catalog_product_index_eav_temp LIKE catalog_product_index_eav_tmp

This error indicates that there is a transaction that violates GTID consistency somewhere in the Product EAV indexing SQL. One suggestion that was made to avoid this problem would be a CREATE VIEW instead of CREATE TEMPORARY TABLE in the indexer module.

@magento-engcom-team magento-engcom-team added the Issue: Format is valid Gate 1 Passed. Automatic verification of issue format passed label Nov 9, 2017
@gilles6
Copy link

gilles6 commented Nov 13, 2017

The problem was fixed for me after installing this module:
https://github.com/bangerkuwranger/Magento-2-GTID-Safe-URL-Rewrite-Tables

@scrivvles
Copy link
Author

@Gvigner Thanks for the tip! It seems like this is one of those things that belongs in core.

@magento-engcom-team magento-engcom-team added Issue: Clear Description Gate 2 Passed. Manual verification of the issue description passed 2.1.x Issue: Ready for Work Gate 4. Acknowledged. Issue is added to backlog and ready for development Issue: Confirmed Gate 3 Passed. Manual verification of the issue completed. Issue is confirmed Reproduced on 2.1.x The issue has been reproduced on latest 2.1 release Reproduced on 2.2.x The issue has been reproduced on latest 2.2 release Reproduced on 2.3.x The issue has been reproduced on latest 2.3 release labels Nov 20, 2017
@magento-engcom-team
Copy link
Contributor

@scrivvles, thank you for your report.
We've created internal ticket(s) MAGETWO-82828 to track progress on the issue.

@gilles6
Copy link

gilles6 commented Nov 21, 2017

Great news!

@avra911
Copy link

avra911 commented Dec 4, 2017

@Gvigner for us didnt worked in all projects, it depends if root category is assigned to a store or not, which makes create sub-category failing under some circumstances

@stormbyte
Copy link

Linking to #11055

The Module
https://github.com/bangerkuwranger/Magento-2-GTID-Safe-URL-Rewrite-Tables
only solves part of the problem with the queries

@lstellway
Copy link

lstellway commented Jun 27, 2018

Issue 15209 reports the same MySQL error while creating a category with MySQL binary logging enabled.

@pyyick
Copy link

pyyick commented Oct 31, 2018

Any update for this?

@gwharton
Copy link
Contributor

gwharton commented Dec 31, 2018

Still broken on 2.2 and 2.3 over a year after being reported :( Magento should be able to run on a MySQL installation with binary logging enabled.

@loid345
Copy link

loid345 commented Jan 20, 2019

Magento 2.3.0 don't run on a MySQL installation with binary logging enabled.

@robolmos
Copy link

Is there any update?

At the very least the docs should be updated to indicate this issue and a workaround if one is available, such as ignoring temporary tables.

@dharake
Copy link

dharake commented Feb 19, 2019

FOR GOOGLE CLOUD SQL USERS

I'm trying to migrate a 2.3 CE database to Google Cloud SQL on a 2nd gen server. Upon reindexing I get the "violates GTID consistency" error for the Product EAV index.

There's little info on the forums so far, so thought I'd contribute my findings.

  1. Forget about the module (https://github.com/bangerkuwranger/Magento-2-GTID-Safe-URL-Rewrite-Tables), the guy made it for a project on 2.1 and has no plans to maintain it. Won't install on M2.3

  2. Downgrading to a legacy SQL server on Google presented other issues with VM connectivity (ipv4 only, so latency issues / also being discontinued in around a years time)

SOLUTION:

I'm on M2.3 CE and not using db replication, so I just tried to disable "Binary Logging" on the SQL Server. Go to instance > click Edit > go to Enable auto-backups > Uncheck "Enable binary logging (required for replication and earlier position point-in-time recovery)".

Test bin/magento indexer:reindex
My reindexation of Magento completed without the previous error "violates GTID consistency".

I understand the limitations to this approach, but until Magento gets their act together (don't hold your breath, issue ongoing since 2017), this is the only way I can find to run M2.3 on Google SQL 2nd Gen Servers.

@dferdoille
Copy link

Hello,

Any update about this ? Are you working on it and do you have an idea when it will be fixed ?

  • Current version : Magento 3.0 - I can't create products (it was ok with Magento 2.2.2 and Bangerkuwranger_GtidSafeUrlRewriteFallback module)

  • I can't disable binary logging on my MySQL instance.

  • I tried to reconfigure table strategy (Something went wrong while saving the category in Magento 2.1.9 #11055) but it didn't work...

  • I'm currently trying to disable the use of temporary tables patching Magento lib (maybe not a good idea but I need a workaround to move forward...)

Thanks

@alexgoodey
Copy link

@dferdoille you are welcome to try my solution at https://github.com/alexgoodey/magento_gtid_fix

No guarantees.

@bradjones1
Copy link

I believe this can be marked as a duplicate of #15209, as that is scheduled for remediation.

@danemacmillan
Copy link

@dharake Are you still hosting 2.3 on GCP's cloud SQL service?

Disabling binary logging does not disable the GTID settings. Magento2 still throws exceptions in various places, like updating stock, reorganizing categories, indexing.

@ghost ghost removed this from Ready for Dev in Community Backlog Oct 20, 2020
@ghost ghost removed this from Ready for Development in Low Priority Backlog Oct 20, 2020
@ghost ghost removed Issue: Clear Description Gate 2 Passed. Manual verification of the issue description passed Issue: Format is valid Gate 1 Passed. Automatic verification of issue format passed Issue: Ready for Work Gate 4. Acknowledged. Issue is added to backlog and ready for development labels Oct 20, 2020
@sdzhepa sdzhepa added the Triage: Dev.Experience Issue related to Developer Experience and needs help with Triage to Confirm or Reject it label Nov 4, 2020
@magento-engcom-team magento-engcom-team added Priority: P1 Once P0 defects have been fixed, a defect having this priority is the next candidate for fixing. Severity: S1 Affects critical data or functionality and forces users to employ a workaround. labels Nov 30, 2020
@m2-community-project m2-community-project bot added this to Ready for Development in High Priority Backlog Nov 30, 2020
@jorgb90
Copy link

jorgb90 commented Jan 14, 2021

We need this solved..

@ihor-sviziev
Copy link
Contributor

ihor-sviziev commented Aug 4, 2021

Hi @scrivvles,
From the documentation https://cloud.google.com/sql/docs/features#unsupported-statements, it seems like

The following statement is unsupported in MySQL for MySQL for Cloud SQL 5.6 and 5.7, but is supported in MySQL for Cloud SQL 8.0.

CREATE TEMPORARY TABLE statements inside transactions

image

Currently, Magento 2.4 supports MySQL 8 https://devdocs.magento.com/guides/v2.4/install-gde/system-requirements.html.

Could you confirm that this issue is fixed now?

@m2-assistant
Copy link

m2-assistant bot commented Aug 26, 2021

Hi @ihor-sviziev. Thank you for working on this issue.
Looks like this issue is already verified and confirmed. But if you want to validate it one more time, please, go though the following instruction:

    1. Add/Edit Component: XXXXX label(s) to the ticket, indicating the components it may be related to.
    1. Verify that the issue is reproducible on 2.4-develop branch
      Details- Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
      - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
      - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!
    1. If the issue is not relevant or is not reproducible any more, feel free to close it.

@m2-community-project m2-community-project bot moved this from Ready for Development to Dev In Progress in High Priority Backlog Aug 26, 2021
@m2-community-project m2-community-project bot moved this from Dev In Progress to Pull Request In Progress in High Priority Backlog Aug 26, 2021
@ihor-sviziev
Copy link
Contributor

Note I created PR that adds note to dev docs. magento/devdocs#9111

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: Catalog Component: CatalogRule Issue: Confirmed Gate 3 Passed. Manual verification of the issue completed. Issue is confirmed Priority: P1 Once P0 defects have been fixed, a defect having this priority is the next candidate for fixing. Progress: done Reproduced on 2.1.x The issue has been reproduced on latest 2.1 release Reproduced on 2.2.x The issue has been reproduced on latest 2.2 release Reproduced on 2.3.x The issue has been reproduced on latest 2.3 release Severity: S1 Affects critical data or functionality and forces users to employ a workaround. Triage: Dev.Experience Issue related to Developer Experience and needs help with Triage to Confirm or Reject it
Projects
Development

Successfully merging a pull request may close this issue.