Skip to content

fix: filter noindex posts from sitemaps and URL-encode tag slugs#3787

Merged
idoshamun merged 1 commit intomainfrom
fix/sitemap-noindex-dedup-encoding
Apr 13, 2026
Merged

fix: filter noindex posts from sitemaps and URL-encode tag slugs#3787
idoshamun merged 1 commit intomainfrom
fix/sitemap-noindex-dedup-encoding

Conversation

@idoshamun
Copy link
Copy Markdown
Member

Summary

  • Exclude Brief, SocialTwitter, Share, and Collection post types from posts/evergreen sitemaps — matches the webapp's shouldNoindexPost logic and eliminates ~1,238 noindex-in-sitemap errors
  • Filter out non-visible posts (visible = true) from posts/evergreen sitemaps
  • URL-encode tag values in sitemap URLs — fixes canonical mismatch for tags like c++c%2B%2B
  • Collection type exclusion also eliminates ~685 duplicate URLs between posts-*.xml and collections.xml

Test plan

  • Existing sitemap tests pass (36/36)
  • New tests verify Brief, SocialTwitter, Share, and Collection posts are excluded
  • New test verifies non-visible posts are excluded
  • Tag URL encoding test updated for web%26ai
  • Lint passes with 0 warnings

Exclude Brief, SocialTwitter, Share, and Collection post types from
posts/evergreen sitemaps to match the webapp noindex logic and prevent
duplication with collections.xml. Also filter non-visible posts and
URL-encode tag values to fix canonical mismatches (e.g. c++ → c%2B%2B).
@pulumi
Copy link
Copy Markdown

pulumi Bot commented Apr 13, 2026

🍹 The Update (preview) for dailydotdev/api/prod (at 338b402) was successful.

✨ Neo Explanation

Routine image bump deploying commit `a7e900e3` to all services and cronjobs, with fresh DB and ClickHouse migration jobs created for the new release. ✅ Low Risk

This is a standard application deployment rolling commit 807173d2a7e900e3 across the entire workload fleet. Every deployment and cron job receives the new container image tag and matching version labels. The two migration Jobs (Postgres via TypeORM and ClickHouse) are cycled by design — the old commit-stamped Jobs are deleted and new ones for the incoming commit are created, which will run the migrations against both databases before traffic shifts.

The GCP credentials warning on the default_9_12_0 provider is cosmetic and does not affect the Kubernetes resources being deployed.

Resource Changes

    Name                                                       Type                           Operation
-   vpc-native-api-db-migration-807173d2                       kubernetes:batch/v1:Job        delete
~   vpc-native-sync-subscription-with-cio-cron                 kubernetes:batch/v1:CronJob    update
~   vpc-native-hourly-notification-cron                        kubernetes:batch/v1:CronJob    update
~   vpc-native-update-current-streak-cron                      kubernetes:batch/v1:CronJob    update
~   vpc-native-user-profile-updated-sync-cron                  kubernetes:batch/v1:CronJob    update
~   vpc-native-user-profile-analytics-clickhouse-cron          kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-zombie-user-companies-cron                kubernetes:batch/v1:CronJob    update
~   vpc-native-materialize-monthly-best-post-archives-cron     kubernetes:batch/v1:CronJob    update
~   vpc-native-update-highlighted-views-cron                   kubernetes:batch/v1:CronJob    update
~   vpc-native-post-analytics-history-day-clickhouse-cron      kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-gifted-plus-cron                          kubernetes:batch/v1:CronJob    update
~   vpc-native-squad-posts-analytics-refresh-cron              kubernetes:batch/v1:CronJob    update
~   vpc-native-check-analytics-report-cron                     kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-zombie-users-cron                         kubernetes:batch/v1:CronJob    update
~   vpc-native-daily-digest-cron                               kubernetes:batch/v1:CronJob    update
~   vpc-native-channel-digests-cron                            kubernetes:batch/v1:CronJob    update
~   vpc-native-user-profile-analytics-history-clickhouse-cron  kubernetes:batch/v1:CronJob    update
+   vpc-native-api-clickhouse-migration-a7e900e3               kubernetes:batch/v1:Job        create
~   vpc-native-update-trending-cron                            kubernetes:batch/v1:CronJob    update
~   vpc-native-deployment                                      kubernetes:apps/v1:Deployment  update
+   vpc-native-api-db-migration-a7e900e3                       kubernetes:batch/v1:Job        create
~   vpc-native-clean-stale-user-transactions-cron              kubernetes:batch/v1:CronJob    update
~   vpc-native-user-posts-analytics-refresh-cron               kubernetes:batch/v1:CronJob    update
~   vpc-native-temporal-deployment                             kubernetes:apps/v1:Deployment  update
~   vpc-native-calculate-top-readers-cron                      kubernetes:batch/v1:CronJob    update
~   vpc-native-update-source-public-threshold-cron             kubernetes:batch/v1:CronJob    update
~   vpc-native-validate-active-users-cron                      kubernetes:batch/v1:CronJob    update
~   vpc-native-personalized-digest-deployment                  kubernetes:apps/v1:Deployment  update
~   vpc-native-post-analytics-clickhouse-cron                  kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-zombie-opportunities-cron                 kubernetes:batch/v1:CronJob    update
~   vpc-native-private-deployment                              kubernetes:apps/v1:Deployment  update
~   vpc-native-channel-highlights-cron                         kubernetes:batch/v1:CronJob    update
~   vpc-native-worker-job-deployment                           kubernetes:apps/v1:Deployment  update
~   vpc-native-ws-deployment                                   kubernetes:apps/v1:Deployment  update
~   vpc-native-generic-referral-reminder-cron                  kubernetes:batch/v1:CronJob    update
~   vpc-native-rotate-daily-quests-cron                        kubernetes:batch/v1:CronJob    update
~   vpc-native-clean-expired-better-auth-sessions-cron         kubernetes:batch/v1:CronJob    update
~   vpc-native-update-tags-str-cron                            kubernetes:batch/v1:CronJob    update
~   vpc-native-personalized-digest-cron                        kubernetes:batch/v1:CronJob    update
-   vpc-native-api-clickhouse-migration-807173d2               kubernetes:batch/v1:Job        delete
~   vpc-native-materialize-yearly-best-post-archives-cron      kubernetes:batch/v1:CronJob    update
... and 11 other changes

@idoshamun idoshamun merged commit ae58aa5 into main Apr 13, 2026
10 checks passed
@idoshamun idoshamun deleted the fix/sitemap-noindex-dedup-encoding branch April 13, 2026 07:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant