File operations in functions using fs fail

[mbifulco]

(note, this is carried over from a discussion in #706)

edit: related code is in this PR

Hey there - I'm trying to implement a sitemap.xml for my site, and running into problems when my site gets deployed to netlify. It seems to have to do with my use of path.join() in my code to iterate over the files that comprise content on my site.

My current strategy is to look through the /pages directory in my code to create an XML entry for each, using syntax like this:

path.join(process.cwd(), 'src', 'pages');

This runs fine locally in dev, and when deployed on Vercel, but when I deploy to netlify, I'm seeing an error in functions/___netlify-handler when I try to visit the url generating my sitemap:

11:21:47 AM: 92378b15 ERROR  Error: ENOENT: no such file or directory, scandir '/var/task/src/pages'
    at Object.readdirSync (fs.js:1043:3)
    at getStaticPageUrls (/var/task/.next/server/pages/api/sitemap.xml.js:56:47)
    at handler (/var/task/.next/server/pages/api/sitemap.xml.js:212:28)
    at Object.apiResolver (/var/task/node_modules/next/dist/server/api-utils.js:102:15)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async Server.handleApiRequest (/var/task/node_modules/next/dist/server/next-server.js:1014:9)
    at async Object.fn (/var/task/node_modules/next/dist/server/next-server.js:901:37)
    at async applyCheckTrue (/var/task/node_modules/next/dist/server/router.js:93:32)
    at async Router.execute (/var/task/node_modules/next/dist/server/router.js:231:25)
    at async Server.run (/var/task/node_modules/next/dist/server/next-server.js:1085:29) {
  errno: -2,
  syscall: 'scandir',
  code: 'ENOENT',
  path: '/var/task/src/pages'
}

In my case, I'm using an API Route at /pages/api/sitemap.xml.js to generate the sitemap, and pairing that with a redirect in next.config.js to send the output of that function to /sitemap.xml:

module.exports = {
    // ...
    rewrites: async () => [
      {
        source: '/sitemap.xml',
        destination: '/api/sitemap.xml',
      },
    ],
};

I'm wondering if there's something tricky going on here because of how functions are deployed. Any guesses if this might be related to this plugin? Note that this appears to be broken even if I visit the non-redirected page /api/sitemap.xml

Originally posted by @mbifulco in #706 (comment)

[mbifulco]

Hi gang, calling attention to this again because I think there's something flawed with netlify's approach here. I've rewritten this sitemap generator a few times now with different strategies, and I haven't found an approach that will satisfy Netlify's deploy process (though I have bumped into several which work on Vercel).

[ascorbic]

Hi,
This is because we don't by default include the source files when deploying. I would suggest that you try parsing .next/routes-manifest.json. This includes all dynamic and static routes, and will be faster than walking the filesystem. I'm a bit confused that you said that .next/server/pages-manifest.json didn't work for you. Surely src/pages doesn't include all dynamically-generated pages either? Could you share your pages-manifest.json and routes-manifest.json and identify which routes are missing?

[mbifulco]

Hey @ascorbic - thanks for responding. Fundamentally, the problem here seems to be that I'm using next's dynamic routing features (pages with [slug].js as their name). In both of the files below, you'll see /posts/[slug] and /tags/[tag], which don't give me enough to create a sitemap from all of the posts on my site, like a recent article https://mikebifulco.com/posts/product-marketing-defy-expectations. Additionally, I'd like to use a similar approach to create an RSS feeds for my site, which will require access to more than just the URL path (like article titles, excerpts, cover image paths, etc).

Here's the files you asked for:

page-manifest.json

{
  "/_app": "pages/_app.js",
  "/_document": "pages/_document.js",
  "/api/rss.xml": "pages/api/rss.xml.js",
  "/api/sitemap.xml": "pages/api/sitemap.xml.js",
  "/": "pages/index.js",
  "/posts/[slug]": "pages/posts/[slug].js",
  "/tags/[tag]": "pages/tags/[tag].js",
  "/tags": "pages/tags.js",
  "/work": "pages/work.js",
  "/_error": "pages/_error.js",
  "/en/500": "pages/en/500.html",
  "/en/404": "pages/en/404.html",
  "/en/about": "pages/en/about.html",
  "/en/shop": "pages/en/shop.html",
  "/en/newsletter": "pages/en/newsletter.html"
}

routes-manifest.json

{
   "version":3,
   "pages404":true,
   "basePath":"",
   "redirects":[
      {
         "source":"/:path+/",
         "destination":"/:path+",
         "locale":false,
         "internal":true,
         "statusCode":308,
         "regex":"^(?:/((?:[^/]+?)(?:/(?:[^/]+?))*))/$"
      }
   ],
   "headers":[
      
   ],
   "dynamicRoutes":[
      {
         "page":"/posts/[slug]",
         "regex":"^/posts/([^/]+?)(?:/)?$",
         "routeKeys":{
            "slug":"slug"
         },
         "namedRegex":"^/posts/(?<slug>[^/]+?)(?:/)?$"
      },
      {
         "page":"/tags/[tag]",
         "regex":"^/tags/([^/]+?)(?:/)?$",
         "routeKeys":{
            "tag":"tag"
         },
         "namedRegex":"^/tags/(?<tag>[^/]+?)(?:/)?$"
      }
   ],
   "staticRoutes":[
      {
         "page":"/",
         "regex":"^/(?:/)?$",
         "routeKeys":{
            
         },
         "namedRegex":"^/(?:/)?$"
      },
      {
         "page":"/404",
         "regex":"^/404(?:/)?$",
         "routeKeys":{
            
         },
         "namedRegex":"^/404(?:/)?$"
      },
      {
         "page":"/about",
         "regex":"^/about(?:/)?$",
         "routeKeys":{
            
         },
         "namedRegex":"^/about(?:/)?$"
      },
      {
         "page":"/newsletter",
         "regex":"^/newsletter(?:/)?$",
         "routeKeys":{
            
         },
         "namedRegex":"^/newsletter(?:/)?$"
      },
      {
         "page":"/shop",
         "regex":"^/shop(?:/)?$",
         "routeKeys":{
            
         },
         "namedRegex":"^/shop(?:/)?$"
      },
      {
         "page":"/tags",
         "regex":"^/tags(?:/)?$",
         "routeKeys":{
            
         },
         "namedRegex":"^/tags(?:/)?$"
      },
      {
         "page":"/work",
         "regex":"^/work(?:/)?$",
         "routeKeys":{
            
         },
         "namedRegex":"^/work(?:/)?$"
      }
   ],
   "dataRoutes":[
      {
         "page":"/",
         "dataRouteRegex":"^/_next/data/2jIZI\\-rNQAgyLCot\\-QS72/index.json$"
      },
      {
         "page":"/posts/[slug]",
         "routeKeys":{
            "slug":"slug"
         },
         "dataRouteRegex":"^/_next/data/2jIZI\\-rNQAgyLCot\\-QS72/posts/([^/]+?)\\.json$",
         "namedDataRouteRegex":"^/_next/data/2jIZI\\-rNQAgyLCot\\-QS72/posts/(?<slug>[^/]+?)\\.json$"
      },
      {
         "page":"/tags",
         "dataRouteRegex":"^/_next/data/2jIZI\\-rNQAgyLCot\\-QS72/tags.json$"
      },
      {
         "page":"/tags/[tag]",
         "routeKeys":{
            "tag":"tag"
         },
         "dataRouteRegex":"^/_next/data/2jIZI\\-rNQAgyLCot\\-QS72/tags/([^/]+?)\\.json$",
         "namedDataRouteRegex":"^/_next/data/2jIZI\\-rNQAgyLCot\\-QS72/tags/(?<tag>[^/]+?)\\.json$"
      },
      {
         "page":"/work",
         "dataRouteRegex":"^/_next/data/2jIZI\\-rNQAgyLCot\\-QS72/work.json$"
      }
   ],
   "i18n":{
      "locales":[
         "en"
      ],
      "defaultLocale":"en"
   },
   "rewrites":[
      {
         "source":"/:nextInternalLocale(en)/rss.xml",
         "destination":"/:nextInternalLocale/api/rss.xml",
         "regex":"^(?:/(en))/rss\\.xml(?:/)?$"
      },
      {
         "source":"/:nextInternalLocale(en)/sitemap.xml",
         "destination":"/:nextInternalLocale/api/sitemap.xml",
         "regex":"^(?:/(en))/sitemap\\.xml(?:/)?$"
      }
   ]
}

[ascorbic]

Don't you have the same problem in src/pages though? That will only have [slug].js in it too.

[mbifulco]

That's true - I'm not doing a great job of explaining myself here, I apologize. For my site, I'm sourcing content from MDX files which are also in my repo, in /src/data/posts, rather than an external CMS.

https://github.com/mbifulco/blog/tree/main/src/data/posts

All the data I need is in my repo, so I use that to cobble together the list of URLs which should appear in my sitemap, RSS, etc:

https://github.com/mbifulco/blog/pull/543/files#diff-c194bb9360c1180b7f56704b2d832adb6cf3b7e33b2bffcb149b3d7cab3a8e49R20

[ascorbic]

Aah, that makes sense now! So the info you need is in prerender-manifest.json. Next sure likes its manifests! Can you work with that?

[mbifulco]

Call me crazy, but I don't see prerender-manifest.json when I run a build. Which folder should it be in?

[ascorbic]

It should be in .next after you've run the build

[mbifulco]

Ah, thanks. There's some useful information in there, but I'm not sure it's a great approach for me. It looks like the prerender-manifest contains URLs I'm interested in, prefixed with a locale string after the domain (/en/posts/lorem-ipsum-dolor), which isn't the URL scheme I've been using for my site (/posts/lorem-ipsum). That may be a misstep on my part, I'm honestly not sure.

I'm also a bit worried about next changing their build output format/location and breaking my sitemap and RSS feed. It feels like an antipattern to parse the output of a build command from an SSG, but again - I may be wrong here, too.

Seems like I've got some thinking to do 🤔

[ascorbic]

I think using the manifest is actually the most robust way of doing it. Internally Next always includes the locale in URLs, but it's fine to strip the locale off as you only have one.

[ascorbic]

It's not the most efficient solution, but your other option is to add the markdown files to the fucnitons bundle:

[functions]
included_files = ["src/pages/data/*.md"]

..or somehting like that (I've not tested it).

I'm going to close this for now, as we have a couple of possible workarounds.