Overwriting existing robots.txt

[justinperkins]

It appears that this gem assumes that a robots.txt file doesn't already exist? Or maybe it does but it's being overwritten? Or perhaps I'm including the gem/plugin in the wrong order?

I have a robots.txt file defined in my _pages directory, so that I can have a bit of control over what type of robots are allowed into my various environments, but all of that is being blown away once that I've added this gem.

[pathawks]

Yup, you've found a bug.

jekyll-sitemap/lib/jekyll/jekyll-sitemap.rb

Line 12 in 2e2bb3c

@site.pages << robots unless file_exists?("robots.txt")

jekyll-sitemap/lib/jekyll/jekyll-sitemap.rb

Lines 62 to 68 in 2e2bb3c

	def file_exists?(file_path)
	if @site.respond_to?(:in_source_dir)
	File.exist? @site.in_source_dir(file_path)
	else
	File.exist? Jekyll.sanitized_path(@site.source, file_path)
	end
	end

We are only checking if robots.txt exists in the source. If a user creates that file some other way, we will overwrite it.

[benbalter]

I think this should be something like (site.pages + site.static_files).any? { |p| p.url = "robots.txt" }

[benbalter]

Still would like help with this issue if anyone is interested.

[gsantner]

This is still happening.

[jveillet]

Hello, I've made a PR, I am not sure it covers the original author problems, but if someone would care to have a look and test it, it would be much appreciated! feedback welcomed 😊

[brandonb927]

Hi folks! I love the plugin, great work! Any plan to publish a new release with the merged PR to fix the original issue?

[coliff]

v1.4.0 is available now so I think this issue can be closed.
https://github.com/jekyll/jekyll-sitemap/releases

Do you know when this will be available on GitHub pages though? The version there is still on v1.2.0.
https://pages.github.com/versions/

[DirtyF]

@coliff Can't say, apparently dependabot is not active on https://github.com/github/pages-gem but feel free to open an issue there.

[coliff]

Ah, found that there is currently an open PR to update a bunch of Jekyll plugins for GitHub Pages - including this one. :-)
github/pages-gem#664