Serve sitemap.xml and robots.txt from "/"

[dawsbot]

The Issue

• Known as an important aspect of having great SEO, a sitemap.xml in / should be possible with next but currently is not.

• Known as an important aspect of privacy and security, a robots.txt in / should be possible with next but currently is not.

Proposal

These two files could be mapped to / if they exist in /static. I notice that the zeit website does not have a robots.txt nor a sitemap.xml so I assume this issue simply has not presented itself yet. Let us know what you folks think about the feature.

[mikecardwell]

Personally I stick a proper web server in front of my node apps. In the Next app I'm currently developing I stick Nginx in front with the following bit of config in my vhost:

root /path/to/my/next/apps/static/directory;

try_files $uri @dynamic;

location @dynamic {
    proxy_pass http://localhost:3000;
    proxy_buffering off;
    proxy_set_header Host $http_host;
}

That way when you request "/robots.txt" or any other URL, nginx first looks to see if there is a matching file in the "static" directory, and if it's not there, it proxies the request through to my local Next app.

That being said, I don't know why Next requires you to have "/static/" at the beginning of the URL for any static content. It seems unnecessary. Any production system will have a proper web server at the front serving up the static content anyway.

[dawsbot]

Although that method seems straightforward @mikecardwell, when pairing next with now, we do not have that luxury. In the interest of staying inside the Zeit ecosystem and using zero-config tools, I'd love for this functionality to be possible when using these two tools together.

[mikecardwell]

I agree. However, I don't think it should be specific to robots.txt / sitemap.xml, I think it should be for all content. Requests for "/foo" should look for content in the static directory first, and only then fall back to the dynamic route if it doesn't exist.

[dawsbot]

That's an excellent point @mikecardwell, that solution seems the most robust given thus-far. How about some chime-in from the Zeit folks?

[nkzawa]

As @mikecardwell said, I also wonder if we really need /static.
Falling back to dynamic routes sounds nice but confusing 🤔
Any thoughts ? @rauchg

[rauchg]

For now I'd like to preserve /static. It's really a huge time saver in some situations with very little added code or surface for problems for us.

The dynamic mapping solution I really don't like. I think the solution to this problem is #25.

You would do something like this to match your example:

if ('/robots.txt' === req.url) {
  return sendFile(res, './static/robots.txt')
}

[nexdrew]

I recently ran into this same problem trying to add a root-scoped service worker script to my next.js-based site/app. Until #25 is implemented in some way, I'm using this hacky workaround:

pages/sw.js

import React from 'react'

export default class extends React.Component {
  static async getInitialProps ({ res }) {
    if (!res) return {}
    res.setHeader('Content-Type', 'application/javascript')
    res.end(`// custom service worker
// PUT YOUR SERVICE WORKER JS LOGIC HERE
`, 'utf8')
    return {}
  }

  render () {
    return null
  }
}

And then I register the service worker in componentDidMount() (which only runs on the client) of my index/home page component, like so:

componentDidMount () {
  if (navigator && navigator.serviceWorker) {
    navigator.serviceWorker.register('/sw.js')
  }
}

Note that my experience with and understanding of next.js and React is very superficial, so there may be a better way of doing this (and I'm sure this violates some best practices), but at least it works.

Theoretically this would work for /robots.txt and /sitemap.xml too (via pages/robots.txt.js and pages/sitemap.xml.js respectively).

[rauchg]

@nextdrew sorry about having had to use a nasty workaround. #25 is coming very soon :)

[rogvold]

That helped me https://github.com/zeit/next.js/blob/canary/examples/root-static-files/server.js

[goloroden]

@rauchg In November 2016 you said

" #25 is coming very soon :)"

AFAICS it's not there yet. Any timeframe for this?

[timneutkens]

AFAICS it's not there yet. Any timeframe for this?

🤔

https://github.com/zeit/next.js#custom-server-and-routing

[goloroden]

Maybe I got this wrong, but the -s flag does not exist, does it?

[timneutkens]

You can already read in the issue it was superseded by another proposal.

[ThatGuySam]

This worked for me using express https://github.com/ekalinin/sitemap.js#example-of-using-sitemapjs-with-express

It also supports other servers and there may be a way to create an automatically generated sitemap based on defined routes.

[embiem]

Hey! I'm building my new static site using Next.js and stumbled upon this as well.

In this post I explain how I automatically generate the sitemap after I yarn build && yarn export my site. Basically, I look at the files under pages/ and create a very basic sitemap.

Hope this helps anybody stumbling upon this as well!