Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reduce memory consumption during sitemap generation #37

Merged
merged 1 commit into from
Jun 13, 2018
Merged

reduce memory consumption during sitemap generation #37

merged 1 commit into from
Jun 13, 2018

Conversation

fritzmg
Copy link
Contributor

@fritzmg fritzmg commented Jun 9, 2018

See contao/core-bundle#1157

The issue is the following:

  • During sitemap generation, all news articles of one archive are loaded at once via Models.
  • Changing this to pure database queries requires more work or code duplication, since the functions to generate the news or event URL need a Model.
  • Due to 'load'=>'eager' on the author and pid of tl_news and tl_calendar_events this results in a large memory consumption, since the author and archive/calendar is duplicated in memory for each news entry.
  • Since the advantages of eagerly loading these related objects are negligible and the disadvantages are potentially huge with growing numbers of news entries, the easiest solution is to just set these relations to lazy.
  • Then, in the worst case, if you display 100 news entries in your newslist module for example and each of these news entries have a different author you would have 100 additional queries (assuming those authors haven't been loaded elsewhere before). If you only have 5 different authors in total, there would be only 5 addition queries (due to the Model registry).

Comparison: when using 10.000 news entries the peak memory usage would be around 1.4 GiB when loading them all. After these changes the peak memory usage is only around 60 MiB.

@leofeyer leofeyer merged commit 2beef43 into contao:4.4 Jun 13, 2018
@leofeyer
Copy link
Member

Thank you @fritzmg.

leofeyer pushed a commit to contao/faq-bundle that referenced this pull request Jun 13, 2018
Description
-----------

See contao/news-bundle#37

Commits
-------

5309dac reduce memory consumption during sitemap generation
leofeyer pushed a commit to contao/comments-bundle that referenced this pull request Jun 13, 2018
Description
-----------

If a page shows a lot of comments with a lot of back end user replies, the memory consumption could potentiallly be higher than needed.

However, this problem should not be as severe as contao/news-bundle#37

Commits
-------

77a36be reduce memory consumption
leofeyer pushed a commit to contao/calendar-bundle that referenced this pull request Jun 13, 2018
Description
-----------

See contao/news-bundle#37

Commits
-------

e5ada9e reduce memory consumption during sitemap generation (contao/core-bundle#1157)
leofeyer pushed a commit to contao/core-bundle that referenced this pull request Jun 13, 2018
Description
-----------

See contao/news-bundle#37

Commits
-------

c80024b reduce memory consumption during sitemap generation
@fritzmg fritzmg deleted the fix-memory-consumption branch June 13, 2018 08:14
@leofeyer leofeyer modified the milestones: 4.4.19, 4.4 May 14, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants