FIX: Show true content of robots.txt after restoring to default (#24980)

Meta topic: https://meta.discourse.org/t/reseting-robots-txt-override-doesnt-seem-to-work-as-expected/287880?u=osama

Discourse provides a default version for `/robots.txt` which can be customized by admins in `/admin/customize/robots`. In that page, there's a button to reset back to the default version that Discourse provides. However, there's currently a bug with the reset button where the content appears to change to some HTML document instead of the default `robots.txt` version when clicking the button. Refreshing the page shows the true/correct content of `robots.txt` which is the default version, so the reset button actually works but there's a display problem.

What causes this display problem is that we use Rails' `render_to_string` method to generate the default content for `robots.txt` from the template, and what we get from that method is the `robots.txt` content wrapped in the application layout. To fix this issue, we need to pass `layout: false` to the `render_to_string` method so that it renders the template without any layouts.

app/controllers/admin/robots_txt_controller.rb

@@ -29,9 +29,9 @@ def current_robots_txt
   def original_robots_txt
     if SiteSetting.allow_index_in_robots_txt?
       @robots_info = ::RobotsTxtController.fetch_default_robots_info
-      render_to_string "robots_txt/index"
+      render_to_string "robots_txt/index", layout: false
     else
-      render_to_string "robots_txt/no_index"
+      render_to_string "robots_txt/no_index", layout: false
     end
   end
 end

spec/requests/admin/robots_txt_controller_spec.rb

@@ -101,11 +101,17 @@
 
       it "resets robots.txt file to the default version" do
         SiteSetting.overridden_robots_txt = "overridden_content"
-        delete "/admin/customize/robots.json"
+        SiteSetting.allowed_crawler_user_agents = "test-user-agent-154"
+
+        delete "/admin/customize/robots.json", xhr: true
         expect(response.status).to eq(200)
+
         json = response.parsed_body
         expect(json["robots_txt"]).not_to include("overridden_content")
+        expect(json["robots_txt"]).not_to include("</html>")
+        expect(json["robots_txt"]).to include("User-agent: test-user-agent-154\n")
         expect(json["overridden"]).to eq(false)
+
         expect(SiteSetting.overridden_robots_txt).to eq("")
       end
     end