Fix sitemap generation if next/headers is used

[jlubawy]

I was looking into issue #692 that I've also seen, if a page uses next/headers then the generated sitemap files are empty.

Admittedly I don't understand the NextJS internal files that are created, but I did find that:

1. When headers() is not included (normal behavior): page routes seem to show up in the .next/prerender-manifest.json file
2. When headers() is included: prerender-manifest.json is mostly empty, but there is another file .next/trace that seems to include the missing paths. trace is a line-delimited JSON file, each line contains an array of objects with an optional field tags.path containing the missing paths

This PR adds parsing of the trace file and addition to the manifest. It then uses this as input when building the URL set. I tested against the repro mentioned in #692 (https://github.com/bastienrobert/next-sitemap-generation-with-headers-repro) and it appears to fix the problem.

The parsing of the trace file could probably use some work (scanning each line instead of splitting on \n), but I wanted to get this PR up to get some feedback to make sure I'm on the right track.

[vercel]

Someone is attempting to deploy a commit to a Personal Account owned by @iamvishnusankar on Vercel.

@iamvishnusankar first needs to authorize it.

[github-actions]

Closing this PR due to inactivity.

[jlubawy]

FWIW this is no longer an issue for us, I'm not exactly sure why, but either it was fixed upstream in NextJS, something like this https://github.com/pacocoursey/next-themes?tab=readme-ov-file#avoid-hydration-mismatch, or our code removed the use of next/headers that was causing problems.

I can no longer reproduce this: #692

Closing this PR.

[tiste]

@jlubawy on my side, I still have this issue with last version of Next.js (14.2.1), et next-sitemap (4.2.3), with headers next/headers.

[jlubawy]

Basically what we did was wrap the code that caused the issue in a client-side only hook similar to what is mentioned here vercel/next.js#2473 (comment)