New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Media in /content kills performance #1181
Comments
I tested it really causes performance degradation on my Gentoo Linux machine (AMD Athlon II Neo N36L 1.3GHz CPU and 8GB RAM with HDD). This is less CPU power, no SSD drive so I thought it would show the difference clearer than my MacBook. I made 500 1MB dummy media files for the test. First I put them into Here is the result of them. 500 1MB files in
|
They are not used later, so that would be a good fix. |
@tatsushid could you take a stab at fixing this? If we could get this one solved, we could maybe think about asking @spf13 to do a release soon ... I'm not really sure what it contains, but there are some fairly important bug fixes in the mix. |
BTW: When I looked at this a while ago, my first thought was to do a lazy init of the File content (delay read to when needed). |
@bep, I explored the codes but Hmm... I think the easiest way is adding new lazy reader and using it instead of So in my opinion, just adding that functionality isn't much difficult but if we prefer a clear code for making maintenance it easy, first we should clean up them. Hugo already has However, this refactoring would take much time. Hmm, is it better first adding a lazy reader and next improving a file access? |
@tatsushid I think the short term fix is the smallest change to fix this particular problem -- then we can create another issue to fix it in a more clean fashion. |
I wrote a rough lazy reader code and tested it. Here is the result of the new code in the same condition I mentioned. 500 1MB files in
|
But it is still 252 ms |
No, it isn't related to the actual total execution time. As I wrote in my first post, the process time measurement starts after copying The displayed execution time problem should be filed as the other issue. |
Ah, OK -- now I understand. That makes sense. Thanks for looking into this; I can test it myself once you create a pull request. I think the parallel file read could be a big win for really big sites. And I will create a couple of follow-up issues, one of them being getting the "execution time" more accurate. |
Simple ioutil.ReadFile is used for reading file contents but it reads all of the file contents and copies them into the memory and is run in a single goroutine. It causes much memory consumption at copying media files in content directory to publish directory and it is not good at performance. This improves the both issue by replacing ReadFile with LazyFileReader. It postpones reading the file contents until it is really needed. As the result, actual file read is run in parallelized goroutine. It improves performance especially in a really big site. In addition, if this reader is called from io.Copy, it does not copy the file contents into the memory but just copies them into destination file. It improves much memory consumption issue when the site has many media files. Fix gohugoio#1181
Simple ioutil.ReadFile is used for reading file contents but it reads all of the file contents and copies them into the memory and is run in a single goroutine. It causes much memory consumption at copying media files in content directory to publish directory and it is not good at performance. This improves the both issue by replacing ReadFile with LazyFileReader. It postpones reading the file contents until it is really needed. As the result, actual file read is run in parallelized goroutine. It improves performance especially in a really big site. In addition, if this reader is called from io.Copy, it does not copy the file contents into the memory but just copies them into destination file. It improves much memory consumption issue when the site has many media files. Fix gohugoio#1181
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
See http://discuss.gohugo.io/t/special-characters-in-taxonomy/1247/8
/cc @spf13
The text was updated successfully, but these errors were encountered: