Join GitHub today
robots.txt not deployed #148
I tried to figure out why this did not happen, but failed. My suggestion would be to modify the build script to generate one similar to https://url.spec.whatwg.org/robots.txt but without the branch-snapshots/ line as HTML doesn't have that. After that we can remove the file from whatwg/html.
I see, so this could be fixed somewhere in https://github.com/whatwg/html-build/blob/master/build.sh#L455?
Do we want to fix this by writing out a robots.txt resource there or do we want to keep a static one in whatwg/html? My preference is to write out one in the build script as this matches what we do for other standards.