Multisite SSG is taking an exponential amount of time to generate #116
Comments
|
What version of Statamic are you on? Are you using S3? |
I'm trying to debug it more now and see if I can pin down where the delay is coming from. |
|
We're using S3 for Assets & Glide but these specific pages didn't include images. These routes mentioned before |
|
I removed all the custom routes and ran into the same problem, it just took a little longer and I managed to generate ~115 pages this time. |
|
Try upgrading to 3.3.44 and see if there's any difference. |
|
I did try updating but had no luck. It might not be a multisite issue. I've done a couple hours of debugging and I think it's these I have some routes like this for pagination: Route::statamic('{locale}/channeling/{year}/{current_page}', 'channeling.years.show');And even if the layout file is totally blank it causes a slow down... If I switch that same function to a normal route it works without any delay. Route::get('{locale}/channeling/{year}/{current_page?}', function() {
return 'test';
});But this <!DOCTYPE html>
<html lang="fr" dir="ltr">
<head>
<meta charset="UTF-8">
</head>
<body>
<!-- body -->
</body>
</html> |
|
I'm going to work on this a bit more this weekend and see if I can replicate it in a new site or see if it's something strange in my own code that's causing the issue. |
|
@jasonvarga I spent the last two days debugging this. It's related to the Statamic update from foreach ($finder->getHints() as $namespace => $paths) {
foreach ($paths as $path) {
$finder->prependNamespace($namespace, $path.'/'.$site);
}
}@edalzell Do you have any ideas on how we could fix this loop in your pull request? 🤷🏻♂️ You can see it goes from around 3s at line [12], to 10s to 65s to 309s ... I'm using the exact same pagination system as you've shown in this example so there's nothing strange going on with these routes themselves.
When I uncomment this new code it's fine.
If I update to Statamic When I added some logging it started doing this kind of stuff: foreach ($finder->getHints() as $namespace => $paths) {
\Log::info('namespace: ' . $namespace);
foreach ($paths as $path) {
\Log::info(' path: ' . $path);
$finder->prependNamespace($namespace, $path.'/'.$site);
}
}It started like this: And ended up doing this after the second loop: And eventually started getting out of control: I also tracked the number of lines growing as it generated pages, you can see it's doubling on each new page and grows quickly. |
|
Good find! We can figure something out. Maybe we can reset the paths to the original state in the middleware after the request ends. |
|
It seems like this middleware is the largest issue but it looks like there's something else causing a slow down recently too. It seems like it takes longer after each new page. The first few pages take less than 1s to generate... So I'm still struggling to generate my ~6000 page site. Besides this exponential time increase with the |
|
Hi @tao ! Did you try to reproduce performance problems without the multi-site feature? I recently locally upgraded two single-site instances (one with approx. 40 pages, one with 267 pages). While the former is working nicely after the updates, the latter shows a similar regression in performance coming to a full stop after approx. half of the pages (tried 1 and 4 workers, does not seem to be related to the concurrency mode). The following is a page per minute (ppm) summary during the process:
This was not happening before the updates.
Edit: The missing addons are my own (not public). |
I can confirm this with my instance. |
|
@globalexport If you use Statamic 3.3.31, it's back to normal? |
|
@jasonvarga Yes, I just downgraded to 3.3.31 and the process completed. I tested 3.3.32 afterwards and it got stuck again. |
|
Would you be able to give me a copy of the site? Either zip it and send it to support@statamic.com or grant access to the github repo? |
|
This is not possible for me, as the sites are customer projects. I am sorry. |
|
Can you show your custom routes and ssg.php config file, at least? |
|
Hi @jasonvarga! Did you mean this? ssg.php: Routes: web.php: |
|
@jasonvarga Here's a demo based on the Cool Runnings starter-kit without multiple sites and ~5000 articles. https://github.com/tao/ssg-slow-mo Just do a composer install then replace the SSG Generator class with the one attached, so you can see the time each page takes.
|
|
What's the best way to try approach fixing this? I need to get my website updating again... Is this a pull request needed on the cms or ssg code to fix things? Is there a strategy that might work that I could try? |
|
It looks like this was an issue from long ago but this method to reset the view paths isn't working correctly anymore? 34255f0 |
|
It looks like it's similar but we need to reset the hints / namespace after each generated page... |
|
@jasonvarga Thank you very much! It is working and I am back to 40 ppm. :) |
I'm almost done working on an update to my website with more languages and I noticed SSG started taking an exponential amount of time to generate the site. At the moment I can get to 14 or 15 pages before it freezes. I also tried this on my linux desktop and had the same problem.
I'm trying to figure out what's going on, the update itself didn't include much change except adding more translated content, using the language files more and the
{{ trans }}tag, and using Statamic navigation... however I tried caching the nav tags and that didn't seem to help.If I only generate English files it seems fine:
I thought it might be an antlers bug by using too many partials but even if I use an empty layout it still slows down.
And here's the result, which is the same... 😕
The text was updated successfully, but these errors were encountered: