Processes getting killed because of memory usage

[royduin]

I've got a command to generate a sitemap manually, something like: https://github.com/spatie/laravel-sitemap/issues/258#issuecomment-573346191 but with a lot of urls: 1.000.000 and the process is eating memory. I've put this in the foreach loop:

echo (memory_get_usage(true)/1024/1024).PHP_EOL;

With the result:

204.015625
338.01953125
468.0234375
602.02734375
736.03125
866.03515625
1006.0390625
1136.04296875
1274.046875
1408.05078125
1540.0546875
1626.046875
etc...

So it's not cleaning up afterwards. Tried to debug it but I think it's a Laravel thing with views. When I just "return" in here: https://github.com/spatie/laravel-sitemap/blob/fced5910f1a80399115387557487e0bf261f37ab/src/Sitemap.php#L56:L65 instead of rendering the view it's not eating any memory anymore. Does Laravel keep the rendered view in the container or something?

[syclone]

Same here. Even after doing something like $sitemap = Sitemap::create(); between each chunk to try clearing it. After about the 6th chuck it runs out of memory.

[Dimimo]

@royduin @syclone
After many trials and errors I found a solution. It's more complex than I hoped for but it works. If you want code snippets, I'm willing to share. Just @ me.

In short: I created a laravel command and queue every xml request out.This way I can schedule it.
In long: I don't know what is consuming so much memory. It could be PDO or even PHP itself. The building of the xml pages is an unusual PHP request. In one request we demand the server to crawl through every table that we want to sitemap.

These are the consequent actions I've taken to minimize memory use:

• limit every xml page to 7.500 lines only, my server ran out of memory adding the links to Sitemap
• I extended Spatie\Sitemap\Sitemap and did indeed create my own method called writeLinesToFile($path)
• got the ids from db to a array (pluck('id')), and chunk them in groups of 100
• 75 cycles later the Sitemap has all data needed to create the xml file, which I send to writeLinesToFile($path)
• if a table has more than 7.500 records, I split them in different jobs. Each table has it's own job denomination. Files are consequently created, for example articles_0.xml etc.
• before closing the job class, all data is __destruct()ed. Queues are long running process, it's necessary to manually destruct.

All this needs at it's peak moment is 60M of memory. My (shared) server can handle it now and I can schedule the task. Great! Job done 😎 (3 days later, hmmmm)

[royduin]

I ended up just writing it myself which is way faster and no memory issues:

$sitemapIndex = SitemapIndex::create();

Product::select(['shop_id', 'slug', 'updated_at'])
    ->with('shop')
    ->whereNotNull('price')
    ->chunk(40000, function ($products, $chunk) use ($sitemapIndex) {
        $sitemapName = 'products_sitemap_'.$chunk.'.xml';
        $sitemap = '<?xml version="1.0" encoding="UTF-8"?>';
        $sitemap .= '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml">';

        foreach ($products as $product) {
            $sitemap .= '<url>';
            $sitemap .= '<loc>'.url($product->shop->slug.'/'.$product->slug).'</loc>';
            $sitemap .= '<lastmod>'.$product->updated_at->format('Y-m-d').'</lastmod>';
            $sitemap .= '</url>';
        }

        $sitemap .= '</urlset>';

        file_put_contents(public_path($sitemapName), $sitemap);

        $sitemapIndex->add($sitemapName);
    });

$sitemapIndex->writeToFile(public_path('sitemap.xml'));

[ctf0]

same issue here too but with much lower count of pages, < 500 or something.

Allowed memory size of 536870912 bytes exhausted (tried to allocate 2097184 bytes)

[ven0ms99]

same here

[syclone]

I have returned to this sitemap package since Laravelium/laravel-sitemap does not support Laravel 9. Unfortunately, this issue still exists. I think I will just do @royduin's solution for now.

[j-dexx]

If running with ignition it might be that:
laravel/framework#31505

[j-dexx]

I've found manually requesting a garbage collection per chunk fixed this for me.

Item::chunk(2500, function ($items) {
  $sitemap = Sitemap::create();
  foreach ($items as $item) {
     // add to a sitemap
  }
  gc_collect_cycles();
});

[polopolaw]

Hello there, i used lazycollections.

$count = Product::query()
            ->where('publish', true)
            ->count();
        $chunk = 40000;
        $index = 1;
        for ($i = 0; $i < $count;) {
            $lazy = Product::query()
                ->where('publish', true)
                ->lazy()
                ->skip($i)
                ->take($chunk);

            $i += $chunk;
            Sitemap::create()
                ->add($lazy)
                ->writeToFile(public_path("products_$index.xml"));
            $index++;
        }

Maybe it will help to someone.