From 87883a1963236e919854914b0a02ab1496bb3334 Mon Sep 17 00:00:00 2001 From: Osama Sayegh Date: Wed, 20 Dec 2023 23:00:37 +0300 Subject: [PATCH] FIX: Show true content of robots.txt after restoring to default (#24980) Meta topic: https://meta.discourse.org/t/reseting-robots-txt-override-doesnt-seem-to-work-as-expected/287880?u=osama Discourse provides a default version for `/robots.txt` which can be customized by admins in `/admin/customize/robots`. In that page, there's a button to reset back to the default version that Discourse provides. However, there's currently a bug with the reset button where the content appears to change to some HTML document instead of the default `robots.txt` version when clicking the button. Refreshing the page shows the true/correct content of `robots.txt` which is the default version, so the reset button actually works but there's a display problem. What causes this display problem is that we use Rails' `render_to_string` method to generate the default content for `robots.txt` from the template, and what we get from that method is the `robots.txt` content wrapped in the application layout. To fix this issue, we need to pass `layout: false` to the `render_to_string` method so that it renders the template without any layouts. --- app/controllers/admin/robots_txt_controller.rb | 4 ++-- spec/requests/admin/robots_txt_controller_spec.rb | 8 +++++++- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/app/controllers/admin/robots_txt_controller.rb b/app/controllers/admin/robots_txt_controller.rb index d2912f06927651..6e1e788d9e5121 100644 --- a/app/controllers/admin/robots_txt_controller.rb +++ b/app/controllers/admin/robots_txt_controller.rb @@ -29,9 +29,9 @@ def current_robots_txt def original_robots_txt if SiteSetting.allow_index_in_robots_txt? @robots_info = ::RobotsTxtController.fetch_default_robots_info - render_to_string "robots_txt/index" + render_to_string "robots_txt/index", layout: false else - render_to_string "robots_txt/no_index" + render_to_string "robots_txt/no_index", layout: false end end end diff --git a/spec/requests/admin/robots_txt_controller_spec.rb b/spec/requests/admin/robots_txt_controller_spec.rb index 095491d15343e9..1c4a396750121a 100644 --- a/spec/requests/admin/robots_txt_controller_spec.rb +++ b/spec/requests/admin/robots_txt_controller_spec.rb @@ -101,11 +101,17 @@ it "resets robots.txt file to the default version" do SiteSetting.overridden_robots_txt = "overridden_content" - delete "/admin/customize/robots.json" + SiteSetting.allowed_crawler_user_agents = "test-user-agent-154" + + delete "/admin/customize/robots.json", xhr: true expect(response.status).to eq(200) + json = response.parsed_body expect(json["robots_txt"]).not_to include("overridden_content") + expect(json["robots_txt"]).not_to include("") + expect(json["robots_txt"]).to include("User-agent: test-user-agent-154\n") expect(json["overridden"]).to eq(false) + expect(SiteSetting.overridden_robots_txt).to eq("") end end