Filter URLs not discoverable by Google crawlers

[attilanetlogiq]

We are currently utilizing the Elastic Suite module within our Magento stores and have encountered an issue with the visibility of filter URLs for Google's crawlers.

Specific Issue: Our website utilizes various attribute types for product filtering (Visual Swatch, Dropdown, Text Swatch). We expect all filter URLs generated through these attributes to be discoverable by Google crawlers. Currently, only filters set as "Visual Swatch" are visible to Googlebot.

Technical Analysis: Our internal development team has investigated this issue further. Their analysis suggests that filter URLs generated by non-Visual Swatch attributes might be rendered through JavaScript, potentially preventing them from being seen by Google.

Example:
Page: Pants & Shorts - Bottoms

Expected Behavior: When inspecting the page and navigating to a filter using any attribute type (e.g., Fashion Material -> Spandex), the corresponding filter URL (e.g., Pants & Shorts - Bottoms ) should be accessible and crawlable by Google.

Actual Behavior: For the test we used Google Rich Results Tool and these are the results for Material filter URLs versus Fashion Color filter URLs. Searching for Pants & Shorts - Bottoms yields no results. In contrast, filter URLs generated by the "Fashion Color" attribute (set as Visual Swatch) are discoverable.

Impact: This limited filter URL visibility will block Google's ability to comprehensively index our product pages based on filter selections and this can negatively impact our search engine ranking as Google may not be able to surface relevant product variations based on user filter selections (e.g., "Venia Pants + Material").

Contrasting Example: For comparison, another popular Magento filtering module, Amasty Layered Navigation appears to handle filter URLs differently. On a page like this one Women Pants, which utilizes various filter attributes, including "Pattern" (as seen in the screenshot), all filter URLs seem to be accessible and crawlable by Google, regardless of the attribute type (you can test it with the tool here).

Request: We kindly request your assistance in resolving this issue. Please advise on the necessary settings within Elastic Suite to ensure that filter URLs generated by all attribute types (Visual Swatch, Dropdown, Text Swatch) are visible and crawlable by Google bots.

If you need additional information please let me know. Thank you for your time and support.

Kind regards,

[romainruaud]

Hi @attilanetlogiq

if you just set the attribute to "is_display_rel_nofollow=0" in the back-office, this should prevent it to have "rel='nofollow'", could it be sufficient for it to be crawled ?

regards

[attilanetlogiq]

@romainruaud
What I noticed is that if I copy the rendered html code from developer toolbar into Rich Results Tests , then the filter links are visible to google. So it seems like the crawler does not execute all the javascript and basically it does not render the filter navigation side menu.

#if you just set the attribute to "is_display_rel_nofollow=0" in the back-office, this should prevent it to have "rel='nofollow'", could it be sufficient for it to be crawled ?

We will make a test on one single category. But our main concern is that the filters are not rendered by google's bot.

[attilanetlogiq]

Hi @romainruaud

Could you please take a look at this issue on demo.magento-elastic-suite.io ?

1. Navigate to https://search.google.com/test/rich-results/result?id=GPhSNkrqRbh7gYc74M6orQ
2. Click View Tested Page
3. In the right side, there is a magnifier/search icon, click and search for fashion_material=
   You will notice that none of the filter options link shows up