Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hexo eats all the memory when generating #3886

Open
steveww opened this issue Nov 27, 2019 · 7 comments
Open

Hexo eats all the memory when generating #3886

steveww opened this issue Nov 27, 2019 · 7 comments

Comments

@steveww
Copy link

@steveww steveww commented Nov 27, 2019

I am building image gallery site. It will have many large images, at the moment with just a small percentage of the images loaded Hexo uses all computer memory when generating the site.

Machine is Linux quad core with 8GB of memory. Hexo eats it all and brings the machine to a standstill due to excessive swapping.

There are 5.7GB of image files. About 1k images.

Looks like Hexo loads everything into memory to generate the site. This is insane!

@YoshinoriN

This comment has been minimized.

Copy link
Member

@YoshinoriN YoshinoriN commented Nov 28, 2019

Please try concurrency option.

https://hexo.io/docs/commands#generate

@SukkaW

This comment has been minimized.

Copy link
Member

@SukkaW SukkaW commented Nov 29, 2019

Hexo doesn't know what files inside source folder should be used during generation, thus Hexo loads everything under source folder.

Try to configure skip_render in your _config.yml to exclude your images to see if it helps.

@steveww

This comment has been minimized.

Copy link
Author

@steveww steveww commented Nov 29, 2019

None of the suggestions made any difference. About 1.5GB of memory is used for generate which is about the size of the directory including all the images. I have my suspicions about the filter https://github.com/hexojs/hexo-filter-responsive-images I think it may be the culprit. I think it may be loading every image into memory to resize them.

@curbengh

This comment has been minimized.

Copy link
Contributor

@curbengh curbengh commented Dec 6, 2019

https://github.com/hexojs/hexo-filter-responsive-images I think it may be the culprit. I think it may be loading every image into memory to resize them.

Does disabling the plugin helped?

@steveww

This comment has been minimized.

Copy link
Author

@steveww steveww commented Dec 6, 2019

I disabled the plugin but it make very little difference.

@curbengh

This comment has been minimized.

Copy link
Contributor

@curbengh curbengh commented Dec 7, 2019

I would like to reproduce this. May I know your folder structure, specifically where you put the images (source/, themes/foo/source/, etc)?
How do you organize the posts? Do you enable post_asset_folder?

@steveww

This comment has been minimized.

Copy link
Author

@steveww steveww commented Dec 7, 2019

I did not use post_asset_folder. Each post represents a different gallery, which is dynamically built using my script. This just lists the target folder and creates a justified-gallery DIV. I've attached a recursive list of the folder structure.

ls.txt.gz

Here is my script.
tag-imgdir.js.gz

The more images I add the more memory hexo takes when generating.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.