Error "Javascript heap out of memory"

[ShubhamDigole]

Bug report

Describe the bug

can't see any notification of sitemap is generating when pressing the button there is no info that how much time it will take or its generating the sitemap

Steps to reproduce the behavior

1. Go to 'sitemap plugin'
2. add URL bundle and save
3. and generate sitemap
4. you will see no notification is there that sitemap is generating

Expected behavior

It should show that sitemap is generating

System

• Node.js version: V14.17
• NPM version: 8.5
• Strapi version: 4.1.7
• Plugin version:2.0.7
• Database:postgress
• Operating system: ubuntu

[boazpoolman]

@webuilderkrish thanks for reporting this.

I can see the value in this.
How long does it generally take for you ti generate the sitemap?

[ShubhamDigole]

@webuilderkrish thanks for reporting this.

I can see the value in this. How long does it generally take for you ti generate the sitemap?

if there are custom URLs usually it takes few seconds but for dynamic URL I don't know because I am generating one but its not generating. i did wait for whole night but no sitemap

[boazpoolman]

Thats not right.

How many pages do you have? Is it like 1000+?

Also; did you get any errors in the Strapi logs?

[ShubhamDigole]

Thats not right.

How many pages do you have? Is it like 1000+?

Also; did you get any errors in the Strapi logs?

i am working on a e-commerce website where I have like 4 lac merchants and 2 million products but i am was generating pages for only merchants still cannot create sitemap

and there is no error on strapi console

[boazpoolman]

Allright. So that’s interesting.

I would like to keep in touch with you as I think you are the first to use this plugin with that many URLs. Let’s see how that works out.

For the current issue;
Do you see any javascript errors when you open up the dev tools in your browser?

[ShubhamDigole]

Sure I would be in touch with you.
and about the developer console, I am not getting any errors and one more thing when I was using an old version of strapi it was 3.6.8 and was working fine on that version but I have to update the new version and getting this issue

[boazpoolman]

Allright, thanks!

And can you confirm if the sitemap file isn't silently written?
In the public/sitemap/index.xml location.

[ShubhamDigole]

yes i did check that it is showing old XML which i created with custom urls

[boazpoolman]

Well, then I'm fresh out of idea's.

I feel like there should be some kind of error, somewehere.

[ShubhamDigole]

I am seeing no error in the console or on the browser console. how can i find out the error

[boazpoolman]

Can you confirm (in browser network tab) that the /sitemap call finishes when you click generate?

The sitemap is also generated when you update a page of the collection type you added to the sitemap.
Can you try editing a page through the content manager?

[ShubhamDigole]

yes, the sitemap call is getting fired but no response is generating.

and when I am updating the entity it sends http://localhost:1337/sitemap/settings/ request

[boazpoolman]

Hi @webuilderkrish

Wanted to quickly follow up on this.
I think we've gone through the default steps-to-reproduce questions.
Yet I've not been able to pin-point the source of the problem.

So the logical next step would be to do some debugging in the source code.

As you mentioned the /sitemap API call is fired but does not give you a response.
That API call excecutes this service.

You could alter that file in your project (node_modules/strapi-plugin-sitemap/server/services/core.js) and put some console.log's to try to find the place where the service hangs.

Hopefully that will give us a bit more context as to what might be going on here.

[ShubhamDigole]

Okay I will debug the code and let you know

[ShubhamDigole]

hi @boazpoolman

I did debug all code I am getting the below error and my server is restarting.

[boazpoolman]

So a Javascript heap out of memory issue it's then.

Also I see the log fetched405390records.
So I guess you're trying to build a really large sitemap?
Then it would make sense we get a memory issue.

BTW: were you able to pin-point the exact location in the core.js service where the memory issue occurs?
Because fetching the records seems to have worked, looking at your log.
So did the memory issue occur when trying to write the sitemap file?

Thanks for investing your time in this 🚀

[ShubhamDigole]

Hi @boazpoolman
I deeply apologize for the late reply as I was busy with some work.
as above you see that I am fetching 4 lac records and these are just merchants i have another table of products where records are around 2.5 million for now and it's going to increase after some time to around 7.5 million

and about error I think i am getting this error in below code at core,js in services in createSitemapEntries function

await Promise.all(pages.map(async (page) => { const pageData = await getSitemapPageData(page, contentType, excludeDrafts); if (pageData) sitemapEntries.push(pageData); }));

thanks..

[boazpoolman]

Hey @webuilderkrish

I feel like you are fetching more records then you think. As your log from the error screenshot stated fetched405390records.

For now I can ask you to try the beta version of the plugin. In the beta version the plugin will create a paginated sitemap index of sitemaps containing more then 45.000 records. It’s worth a try.

You can install beta like this yarn add strapi-plugin-sitemap@beta.

[ShubhamDigole]

thanks for the update i will use the beta version and let you know the response

[ShubhamDigole]

Hi,
Tried Beta Version but still getting the same error

[boazpoolman]

Allright. Thanks for trying at least.

All I can do for you now is create a test Strapi app with 500k records and trying to add them to the sitemap, hoping I will get the same error as you so I can do some debugging.

Sadly I don’t have time for that anytime soon. So until then I can only encourage you to debug the plugin yourself. Try to optimize the code so you won’t get this error.

Sorry I can’t do more for you at this time.

[ShubhamDigole]

Thanks for the support I will debug the code and try to find the solution thanks for your time.

but please try to fix this bug as you get the time.

Thanks for the support...

[boazpoolman]

I will, once I have the time.

If you make any progress on this feel free to reach out!

Btw; for future reference; do you have localizations enabled in your Strapi app?

[ShubhamDigole]

Okay i will reach out when i will find something
no localization enabled in strapi

[ShubhamDigole]

after increasing the memory size i am getting below error

[boazpoolman]

Hey @webuilderkrish

Allright, this is allready more verbose.
I've asked for a solution in the Strapi Discord and somebody pointed me to this issue:
strapi/strapi#11860

Not sure if it fixes your issue.
But it's worth taking a look!

[boazpoolman]

Performance fixes have been made in version 3.0.0 of the Sitemap plugin.

From now on, by default, the sitemap will be generated through CRON, not through lifecycles. That means a lot less sitemap generations. Apart from that tons of small optimisations have been made to make the plugin perform better with large URL sets.