Catalog search index process error indexation process

[dariovranjkovic]

Preconditions and environment

• Magento version 2.4.3.
• Elasticsearch 7.10

Steps to reproduce

1. Run re-index command

Expected result

Reindex without error messages

Actual result

Catalog Search index process error during indexation process:
Processed schema file: public_html/vendor/magento/module-elasticsearch/etc/esconfig.xsd
complex type 'mixedDataType': The content model is not determinist.

Additional information

Site was working fine long time, and after run re-index I got error message:

Catalog Search index process error during indexation process:
Processed schema file: public_html/vendor/magento/module-elasticsearch/etc/esconfig.xsd
complex type 'mixedDataType': The content model is not determinist.

Release note

No response

Triage and priority

• Severity: S0 - Affects critical data or functionality and leaves users without workaround.
• Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

[m2-assistant]

Hi @dariovranjkovic. Thank you for your report.
To speed up processing of this issue, make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, Add a comment to the issue:

• @magento give me 2.4-develop instance - upcoming 2.4.x release
• For more details, review the Magento Contributor Assistant documentation.
• Add a comment to assign the issue: @magento I am working on this
• To learn more about issue processing workflow, refer to the Code Contributions.

---

Join Magento Community Engineering Slack and ask your questions in #github channel.
⚠️ According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.
🕙 You can find the schedule on the Magento Community Calendar page.
📞 The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.

[devbikash07]

Hi @dariovranjkovic , It is an issue related to updated ea-libxml2 package. Please downgrade that package using command

yum downgrade ea-libxml2-2.11.*

and retry running reindex.

[m2-assistant]

Hi @engcom-Bravo. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue.
• 3. Add Area: XXXXX label to the ticket, indicating the functional areas it may be related to.
• 4. Verify that the issue is reproducible on 2.4-develop branch
  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!
• 5. Add label Issue: Confirmed once verification is complete.
• 6. Make sure that automatic system confirms that report has been added to the backlog.

[engcom-Bravo]

@devbikash07 Thanks for your contribution & collaboration over here.

Hi @dariovranjkovic,

Thank you for reporting and collaboration.

Verified the issue on Magento 2.4-develop instance and the issue is not reproducible.Kindly refer the screenshots.

Could you please refer this comment #38254 (comment)

Thanks.

[ainigma32]

It is true that downgrading xmllib to a version below 2.12.0 will resolve the problem but only because those versions of libxml allowed invalid XML schemas. See this issue for more info.
If we check the files vendor/magento/module-elasticsearch/etc/esconfig.xml and vendor/magento/module-elasticsearch/etc/esconfig.xsd through xerces using this online service we get the error: [Error] esconfig.xsd:18:38:cos-nonambig: default and WC[##any] (or elements from their substitution group) violate "Unique Particle Attribution". During validation against this schema, ambiguity would be created for those two particles.

I am not an expert on XSD's so I could be wrong but this would suggest the XSD is actually invalid and should be corrected.

[engcom-Bravo]

@ainigma32 Thanks for your contribution & collaboration over here.

@dariovranjkovic Could you please refer this document #38254 (comment) and please let us know if you are still facing any issue.

Thanks.

[Kishorchhugani]

I am facing the same issue , @dariovranjkovic can you find out any solution ?

[dariovranjkovic]

@Kishorchhugani yes we found solution

It is an issue related to updated ea-libxml2 package. Please downgrade that package using command
yum downgrade ea-libxml2-2.11.*

[CalamityJames]

@Kishorchhugani yes we found solution

It is an issue related to updated ea-libxml2 package. Please downgrade that package using command yum downgrade ea-libxml2-2.11.*

Thanks for that temporary fix. Hopefully the details @ainigma32 provided will let the Magento team provide a more permanent fix. Currently this issue reoccurs daily due to our server host forcing WHM to update daily with no apparent way to disable this any more.

[Kishorchhugani]

This Works For me ,
I don't know it is safe or not but it works !!

remove one line check it
vendor/magento/module-elasticsearch/etc/esconfig.xsd
<xs:element type="xs:string" name="default" minOccurs="1" maxOccurs="1" />

[ainigma32]

I don't know it is safe or not but it works !!

Yes that also works and as long as you don't add any invalid esconfig.xml files it should be fine.
Removing that line will make the XSD valid in the eyes of libxml but it will no longer check if your esconfig.xml files contain a "default" element.

[engcom-Bravo]

As per this #38254 (comment) we are closing this issue.As issue has been resolved.

Thanks.

[CalamityJames]

Sorry, but downgrading a built in library that actually implements a correct schema checking for XML is not a fix. This will break any magento installs for anyone with automatic updates enabled who runs EasyApache on WHM and also anyone else who updates libxml.

Can this be reopened please?

[CalamityJames]

@engcom-Bravo please see my above comment re the "solution" that was accepted.

[hostep]

Issue makes sense, seems like core bug, @engcom-Bravo or @ihor-sviziev, can we get this reopened?

If I understand it correctly, it's because this code in the xsd file:

        <xs:element type="xs:string" name="default" minOccurs="1" maxOccurs="1" />
        <xs:any processContents="lax" minOccurs="0" maxOccurs="unbounded" />

It first says: we need an element <default> that needs to exists and it also says, next to that we can have "any" element (so including <default>) 0, 1 or multiple times. Both statements conflict with each other.
(I maybe have misunderstood it, I'm not an expert on this matter, so forgive me if this is a wrong statement)

Out of curiosity, does changing minOccurs="0" to minOccurs="1" on that xs:any also fix the issue? I currently can't install libxml 2.12.0 so can't test it out myself I'm afraid.

[ainigma32]

does changing minOccurs="0" to minOccurs="1" on that xs:any also fix the issue?

Alas, no, it does not.

[dossy]

It is true that downgrading xmllib to a version below 2.12.0 will resolve the problem but only because those versions of libxml allowed invalid XML schemas. See this issue for more info. If we check the files vendor/magento/module-elasticsearch/etc/esconfig.xml and vendor/magento/module-elasticsearch/etc/esconfig.xsd through xerces using this online service we get the error: [Error] esconfig.xsd:18:38:cos-nonambig: default and WC[##any] (or elements from their substitution group) violate "Unique Particle Attribution". During validation against this schema, ambiguity would be created for those two particles.

I am not an expert on XSD's so I could be wrong but this would suggest the XSD is actually invalid and should be corrected.

See also: https://stackoverflow.com/questions/37486498/xsdany-other-elements-inside-choice

In particular, it says that using xs:choice with xs:element and xs:any will work with XSD 1.1, but not XSD 1.0, where it fails the Unique Particle Attribution constraint. And, sure enough, using that online schema validator service and selecting Select an XSD version to use, during validation = 1.1 yields the following output:

You selected XSD 1.1 validation.

XML document is assessed as valid with the XSD document(s) you have provided.

Given that:

1. Magento is using DOMDocument::schemaValidate(),
2. and the PHP documentation for DOMDocument::schemaValidate() indicates that "Only XML Schema 1.0 is supported"
3. and the PHP bug about this points out that PHP is just using libxml2
4. and that libxml2 only implements XML Schema 1.0

It looks like using an XSD that requires XML Schema 1.1 is out of the question, at least in the short term.

I think the we should comment out <xs:element type="xs:string" name="default" minOccurs="1" maxOccurs="1" /> and add a comment before it indicating that uncommenting it would require a XML Schema 1.1 validator, and if PHP ever gets a XML Schema 1.1 validator, and Magento implements whatever changes are necessary to use it, then uncomment the line then.

[m2-assistant]

Hi @engcom-November. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: 👇

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue.
• 3. Add Area: XXXXX label to the ticket, indicating the functional areas it may be related to.
• 4. Verify that the issue is reproducible on 2.4-develop branch
  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!
• 5. Add label Issue: Confirmed once verification is complete.
• 6. Make sure that automatic system confirms that report has been added to the backlog.

[engcom-November]

Thank you @dossy for detailed breakdown of the issue.

As mentioned here #38254 (comment), magento is using DOMDocument::schemaValidate() to validate XML, and this method in turn is using libxml2.

xs:choice with xs:element and xs:any are only compatible with XSD 1.1, whereas libxml2 only supports XSD 1.0.
In the older version of libxml2, invalid XML Schemas(1.1) were not handled, but in the recent release they have addressed this bug. This is what causing the issue.
This can be seen here.

Hence confirming this issue.

Thank you.

[github-jira-sync-bot]

✅ Jira issue https://jira.corp.adobe.com/browse/AC-10838 is successfully created for this GitHub issue.

[m2-assistant]

✅ Confirmed by @engcom-November. Thank you for verifying the issue.
Issue Available: @engcom-November, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

[hostep]

I've just send in this PR: #38553 which seems to fix this problem. Feedback would be appreciated :)