Pull request to add robots.txt and sitemap.xml files (v1) #643
Conversation
examples/official-site/robots.txt
Outdated
| User-agent: * | ||
| Disallow: /examples/authentification/ # Block the folder "authentification" to avoid blocking by password | ||
| Disallow: /examples/hash_password.sql # Block access to specific sensitive SQL files like this | ||
| Disallow: /Dockerfile # Block access to Dockerfile |
There was a problem hiding this comment.
There was a problem hiding this comment.
I was told to limit the number of files that need to be crawled so I figure this one is not useful for Google
There was a problem hiding this comment.
Look at the link. The file does not exist
examples/official-site/robots.txt
Outdated
| @@ -0,0 +1,8 @@ | |||
| User-agent: * | |||
| Disallow: /examples/authentification/ # Block the folder "authentification" to avoid blocking by password | |||
| Disallow: /examples/hash_password.sql # Block access to specific sensitive SQL files like this | |||
There was a problem hiding this comment.
Why ? This is a useful page, it is not password protected: https://sql.datapage.app/examples/hash_password.sql
examples/official-site/robots.txt
Outdated
| @@ -0,0 +1,8 @@ | |||
| User-agent: * | |||
| Disallow: /examples/authentification/ # Block the folder "authentification" to avoid blocking by password | |||
There was a problem hiding this comment.
Maybe do not block the entire authentication example ? We still want it referenced. Ideally change the auth example so the password protected page has a different url, that you can block
| <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> | ||
| <url> | ||
| <loc>https://sql.datapage.app/</loc> | ||
| <lastmod>2024-10-10</lastmod> |
There was a problem hiding this comment.
Is it ok to hardcode the date like this ? Won't the prevent re-crawling when we update the page ?
There was a problem hiding this comment.
if it remains hard coded for too long it is an issue, for this first try it is ok
| <loc>https://sql.datapage.app/your-first-sql-website/</loc> | ||
| <lastmod>2024-10-10</lastmod> | ||
| <priority>0.80</priority> | ||
| </url> |
There was a problem hiding this comment.
Is it ok to have such a little subset of the site in the sitemap ? Won't it penalize other pages ?
We should probably either not have a sitemap, or have one that is generated dynamically in sql
2 participants
No description provided.