Robots.txt SEO improvements

[abTec]

nopCommerce version: 4.20

Steps to reproduce the problem:
I believe all AJAX URLs (changecurrency, changetaxtype, changelanguage...) should be Dissalowed in robots.txt because it makes mess in Google Search Console (indexing)

How to fix it:
All AJAX URLs from src/Presentation/Nop.Web/Infrastructure/RouteProvider.cs
should be added into localizableDisallowPaths at
/src/Presentation/Nop.Web/Factories/CommonModelFactory.cs line 766

[cnEUL]

Yes, also for me it seems strange.
In Google searchconsole i get warnings and also soft errors 401 like : "

"en/search?q={search_term_string}",
"en/benutzer-91ef1ba8-6b70-4bab-86de-98190dcfd19a?orderby=15"
"en/ausstattungspakete-4e78996c-ad51-4f48-94f6-4ec8221c2e1d?pagesize=6"
also changelanguage in searchresults.

[RuS984]

Closed #3230

[RomanovM]

From the discussion here https://www.nopcommerce.com/boards/topic/90841/googlebot-picking-up-incorrect-currency :

I managed to correct the issue by adding some additional settings in the CommonModelFactory.cs robot file generator.

"/changecurrency/*",
"/changecurrency/",

When I ran the Structured data testing tool it was still scraping other currencies, but mostly SEK, I unpublished the SEK currency and all products now show the correct currency on Google search, the problem was either the default /Changecurrency setting in the robots file was not working and needed the /changecurrency/* as this is now used to block /changecurrency/7 or an issue with the Swedish currency.

Do we need to change these patterns?

[exileDev]

I close it because it looks like the topic describes a custom feature.

Closed #3230