New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

robots.txt not deployed #148

Closed
annevk opened this Issue Jan 11, 2018 · 3 comments

Comments

2 participants
@annevk
Member

annevk commented Jan 11, 2018

I tried to figure out why this did not happen, but failed. My suggestion would be to modify the build script to generate one similar to https://url.spec.whatwg.org/robots.txt but without the branch-snapshots/ line as HTML doesn't have that. After that we can remove the file from whatwg/html.

@foolip

This comment has been minimized.

Show comment
Hide comment
@foolip

foolip Jan 11, 2018

Member

It looks like there isn't anything that copies robots.txt into html-build/output/

Member

foolip commented Jan 11, 2018

It looks like there isn't anything that copies robots.txt into html-build/output/

@annevk

This comment has been minimized.

Show comment
Hide comment
@annevk

annevk Jan 11, 2018

Member

I see, so this could be fixed somewhere in https://github.com/whatwg/html-build/blob/master/build.sh#L455?

Do we want to fix this by writing out a robots.txt resource there or do we want to keep a static one in whatwg/html? My preference is to write out one in the build script as this matches what we do for other standards.

Member

annevk commented Jan 11, 2018

I see, so this could be fixed somewhere in https://github.com/whatwg/html-build/blob/master/build.sh#L455?

Do we want to fix this by writing out a robots.txt resource there or do we want to keep a static one in whatwg/html? My preference is to write out one in the build script as this matches what we do for other standards.

@foolip

This comment has been minimized.

Show comment
Hide comment
@foolip

foolip Jan 11, 2018

Member

My preference is to write out one in the build script as this matches what we do for other standards.

Me too.

Member

foolip commented Jan 11, 2018

My preference is to write out one in the build script as this matches what we do for other standards.

Me too.

annevk added a commit that referenced this issue Jan 11, 2018

annevk added a commit to whatwg/html that referenced this issue Jan 11, 2018

Meta: remove robots.txt (let the build script handle it)
This was not picked up by the build script and it seems better to let the build script generate it anyway. See whatwg/html-build#148.

@annevk annevk closed this in #149 Jan 11, 2018

annevk added a commit that referenced this issue Jan 11, 2018

annevk added a commit to whatwg/html that referenced this issue Jan 11, 2018

Meta: remove robots.txt (let the build script handle it)
This was not picked up by the build script and it seems better to let the build script generate it anyway. See whatwg/html-build#148.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment