feat: update robots.txt

[floyd-li]

Thank you for your contribution to the KodaDot - One Stop Shop for Polkadot NFTs.

👇 __ Let's make a quick check before the contribution.

PR Type

• Bugfix
• Feature
• Refactoring

Context

• Closes Disallow indexing few pages for crawling robots #6283
• Requires deployment <snek/rubick/worker>

Before submitting pull request, please make sure:

• My contribution builds clean without any errors or warnings
• I've merged recent default branch -- main and I've no conflicts
• I've tried to respect high code quality standards
• I've didn't break any original functionality

Optional

• I've tested it at </ksm/collection>
• I've tested PR on mobile
• I've written unit tests 🧪
• I've found edge cases

Did your issue had any of the "$" label on it?

• Fill up your DOT address: Payout

Community participation

• Are you at KodaDot Discord?

Screenshot 📸

• My fix has changed something on UI; a screenshot is best to understand changes for others.

Copilot Summary

🤖 Generated by Copilot at e0dff1c

Updated robots.txt to block unwanted crawlers and optimize site visibility.

🤖 Generated by Copilot at e0dff1c

robots.txt blocks
paths and subdomains to hide
autumn leaves falling

[kodabot]

SUCCESS @floyd-li PR for issue #6283 which is assigned to you. Please wait for review and don't hesitate to grab another issue in the meantime!

[netlify]

✅ Deploy Preview for koda-canary ready!

Name	Link
🔨 Latest commit	2e870ee
🔍 Latest deploy log	https://app.netlify.com/sites/koda-canary/deploys/6492a202fdc5e50008796e80
😎 Deploy Preview	https://deploy-preview-6286--koda-canary.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[reviewpad]

AI-Generated Summary: This pull request updates the robots.txt file by adding 9 new disallowed path patterns, resulting in 1 file changed with 9 insertions and 1 deletion.

[Jarsen136]

robots.txt guide
ref: https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt#syntax

[yangwao]

what happened with tests

[floyd-li]

what happened with tests

hmm not sure what's happened. it should not be related with this pr 🤔️

[Jarsen136]

I guess it should be related to this change
https://github.com/kodadot/nft-gallery/pull/6265/files

Could u pls take a look? @preschian

[roiLeo]

Missing:

• */collection/
• sitemap

otherwise lgtm

[preschian]

I guess it should be related to this change #6265 (files)

Could u pls take a look? @preschian

if still error I think we need to downgrade the version again in the meantime
Wandalen/wretry.action#93

[codeclimate]

Code Climate has analyzed commit 2e870ee and detected 0 issues on this pull request.

View more on Code Climate.

[floyd-li]

Missing:

* `*/collection/`

* sitemap

otherwise lgtm

updated

[Jarsen136]

Missing:

• */collection/

I'm not sure if we are going to disallow the collection details page. @yangwao

[yangwao]

I'm not sure if we are going to disallow the collection details page. @yangwao

for now yes probably, later we will use allow option to list top10 which has liquidity tbh, time to play gods game as still learning what is best tactic as "e-shop" what to index and what to not.

The current goal is to have top10 great pages to show, rather 1000 sub-optimal ones.

We will be selecting pages to put online up

• https://github.com/kodadot/ops-internal/issues/587

[yangwao]

pay 30 usd

[yangwao]

😍 Perfect, I’ve sent the payout
💵 $30 @ 4.66 USD/DOT ~ 6.438 $DOT
🧗 12zQP2JviSyZzAX2j31F2TR5uSvDNerruSJBWxEZXjW3KMaN
🔗 0x75ab994221c585009cbc1cff103beedde4ae02f49197259cb36362868b7582f1

🪅 Let’s grab another issue and get rewarded!
🪄 github.com/kodadot/nft-gallery/issues