Unexpected token when parsing robots.txt file

[thomaslombart]

I'm building my personal blog with Next and I had an error while adding the robots.txt file in the public directory:

If I remove the colon since it seems problematic, it doesn't fail anymore. I'm not sure why it happens since I thought that serving these static assets should work out of the box.

Note that I don't have any next.config.js and I have the same problem with a file such as site.webmanifest, still in the public folder.

I'm using Next 11.0.1 with Webpack 5 enabled. Does someone have a clue of what's going on and how to fix it? Am I missing something obvious?

[thomaslombart]

After investigating it for quite some time, I found what was causing the bug. It's not related to Webpack or any configuration.

I was adding some attributes to the head element, especially the cover URL of my blog post through a shared variable called imagePath:

export default function Blog({ post, slug }) {
  const imagePath = `images/${slug}/cover.jpg`;

  return (
    <Layout>
      {/* The SEO component adds a bunch of meta properties to the `Head` component from Next */}
      <SEO
        title={title}
        description={description}
        cover={`${config.url}/${imagePath}`}
      /> 
      
      {/* ... */}
      
      <Image
        priority
        src={require(`../public/${imagePath}`)}
        alt=""
      />
      
      {/* ... */}
      
  )
}     

It seems that having such a variable in the require statement make it fail, so I inlined it like so:

<Image
  priority
  src={require(`../public/images/${slug}/cover.jpg`)}
  alt=""
  className="mt-4 rounded-lg"
/>

...and now, it works fine.

I'm not exactly sure why it happened and why I got an error which seemed totally unrelated, but at least the bug is fixed 🙂

[CMCDragonkai]

We see this error happening in docusaurus as well. Seems like dynamic requires causes webpack to try to load all static assets?

[sporritt]

I saw this in docusaurus too, with a robots.txt file in the /static directory. You can write a plugin to get webpack to not fail - create a directory in your project root to hold your plugin, let's say robots-plugin. Then in that directory create index.js and make this the contents:

export default async function RobotsPlugin(context, options) {
    return {
        name: 'robots-plugin',
        configureWebpack() {
            return {                
                module: {
                    rules: [
                        {
                            test: /\.txt$/,
                            use: [
                                {
                                    loader: 'file-loader'
                                },
                            ],
                        },
                    ],
                }
            };
        },
    };
}

Then in your docusaurus.config.js add this to the plugins:

plugins:[
  ...
  "./robots-plugin"
  ...
]

In this example I have of course used a regex that targets all .txt files despite calling it "robots plugin".