New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hexo generate slow with 400 posts #1769

Closed
iMilnb opened this Issue Feb 21, 2016 · 23 comments

Comments

7 participants
@iMilnb

iMilnb commented Feb 21, 2016

$ hexo version
hexo: 3.1.1
os: NetBSD 7.0_STABLE netbsd x64
http_parser: 2.5.0
node: 4.1.1
v8: 4.5.103.33
uv: 1.7.4
zlib: 1.2.3
ares: 1.10.1-DEV
modules: 46
openssl: 1.0.2d
$ hexo clean
INFO  Deleted database.
INFO  Deleted public folder.
$ hexo g
[...]
INFO  1071 files generated in 3.68 min
# without cleaning
$ hexo g
INFO  Files loaded in 4.14 s
INFO  0 files generated in 4 min

Is this a normal generation time for 400 posts? seems awfully slow to me. Every change I make on a layout or a css takes ages to be generated. hexo g -w is barely unusable.

FWIW:

highlight:
  enable: true
  line_number: true
  auto_detect: false
  tab_replace:

Similar results with enable: false.

Theme is light.

@leesei leesei added the enhancement label Feb 22, 2016

@leesei

This comment has been minimized.

Show comment
Hide comment
@leesei

leesei Feb 22, 2016

Member

Did you tried hexo server?

Member

leesei commented Feb 22, 2016

Did you tried hexo server?

@iMilnb

This comment has been minimized.

Show comment
Hide comment
@iMilnb

iMilnb Feb 22, 2016

I did, it is indeed more usable and faster, but only for pre-visualization / debugging purposes

iMilnb commented Feb 22, 2016

I did, it is indeed more usable and faster, but only for pre-visualization / debugging purposes

@leesei

This comment has been minimized.

Show comment
Hide comment
@leesei

leesei Feb 22, 2016

Member

I see. I was trying to suggest you to use it during development, so that generate is used sparingly.
What about not cleaning before every generate? It should be able to pick up changes in posts (a theme change will need to be cleaned though).

Member

leesei commented Feb 22, 2016

I see. I was trying to suggest you to use it during development, so that generate is used sparingly.
What about not cleaning before every generate? It should be able to pick up changes in posts (a theme change will need to be cleaned though).

@iMilnb

This comment has been minimized.

Show comment
Hide comment
@iMilnb

iMilnb Feb 22, 2016

Sure, I don't run clean everytime, just for testing, but TBH, even hexo g without a previous clean takes ages...

$ hexo g
INFO  Files loaded in 4.14 s
INFO  0 files generated in 4 min

iMilnb commented Feb 22, 2016

Sure, I don't run clean everytime, just for testing, but TBH, even hexo g without a previous clean takes ages...

$ hexo g
INFO  Files loaded in 4.14 s
INFO  0 files generated in 4 min

@leesei leesei added the #perfmatters label Feb 22, 2016

@iMilnb

This comment has been minimized.

Show comment
Hide comment
@iMilnb

iMilnb Feb 23, 2016

For the sake of it, I've tried to hexo g on a tmpfs mounted directory, and the result is pretty much the same. Meaning it's not about filesystem access but really processing time.

iMilnb commented Feb 23, 2016

For the sake of it, I've tried to hexo g on a tmpfs mounted directory, and the result is pretty much the same. Meaning it's not about filesystem access but really processing time.

@Xuanwo

This comment has been minimized.

Show comment
Hide comment
@Xuanwo

Xuanwo Feb 25, 2016

Contributor

I don't think it's normal.
As in my blog: https://travis-ci.org/Xuanwo/xuanwo.github.io/builds/111696580 ,

INFO  Files loaded in 4.46 s
INFO  718 files generated in 22 s

The curve of time/files is strange from 22/718(about 0.03) to 240/1071(about 0.224).

Maybe you can try set line_number to false to disable highlight.js totally.

Contributor

Xuanwo commented Feb 25, 2016

I don't think it's normal.
As in my blog: https://travis-ci.org/Xuanwo/xuanwo.github.io/builds/111696580 ,

INFO  Files loaded in 4.46 s
INFO  718 files generated in 22 s

The curve of time/files is strange from 22/718(about 0.03) to 240/1071(about 0.224).

Maybe you can try set line_number to false to disable highlight.js totally.

@iMilnb

This comment has been minimized.

Show comment
Hide comment
@iMilnb

iMilnb Feb 25, 2016

Well:

highlight:
  enable: false
  line_number: false
  auto_detect: false
  tab_replace:
$ hexo g 
INFO  Files loaded in 4.35 s
[...]
INFO  36 files generated in 4.08 min

Doesn't seem to run faster. Any hint on how to help debug this?

iMilnb commented Feb 25, 2016

Well:

highlight:
  enable: false
  line_number: false
  auto_detect: false
  tab_replace:
$ hexo g 
INFO  Files loaded in 4.35 s
[...]
INFO  36 files generated in 4.08 min

Doesn't seem to run faster. Any hint on how to help debug this?

@leesei

This comment has been minimized.

Show comment
Hide comment
@leesei

leesei Feb 25, 2016

Member

Using genstubs to generate 500 posts: ./genstubs.js -N500 -P20

Symlinking source to https://github.com/leesei/hexo-vanilla-site

hexo clean
hexo g --debug

00:10:36.697 DEBUG Hexo version: 3.1.1
...
00:10:38.561 INFO  Files loaded in 1.7 s
...
00:10:42.014 INFO  723 files generated in 3.45 s

(Disclaimer: I'm on SSD, but since you've tried with tmpfs, it shouldn't matter)
Here's the full log: hexo-stubs.txt

Please check where most of the time is spent on your run.

Member

leesei commented Feb 25, 2016

Using genstubs to generate 500 posts: ./genstubs.js -N500 -P20

Symlinking source to https://github.com/leesei/hexo-vanilla-site

hexo clean
hexo g --debug

00:10:36.697 DEBUG Hexo version: 3.1.1
...
00:10:38.561 INFO  Files loaded in 1.7 s
...
00:10:42.014 INFO  723 files generated in 3.45 s

(Disclaimer: I'm on SSD, but since you've tried with tmpfs, it shouldn't matter)
Here's the full log: hexo-stubs.txt

Please check where most of the time is spent on your run.

@leesei leesei referenced this issue Feb 25, 2016

Open

Speed up generating speed #550

2 of 3 tasks complete
@iMilnb

This comment has been minimized.

Show comment
Hide comment
@iMilnb

iMilnb Feb 26, 2016

--debug seems to indicate the rendering is the slowest process, ~0.3s by post. Is there any way to dig deeper on the rendering process to know which part is the slowest?

iMilnb commented Feb 26, 2016

--debug seems to indicate the rendering is the slowest process, ~0.3s by post. Is there any way to dig deeper on the rendering process to know which part is the slowest?

@leesei

This comment has been minimized.

Show comment
Hide comment
@leesei

leesei Feb 26, 2016

Member

@tl3shi I cloned your repo
It took 16mins to run hexo g --debug, however if I changed the theme from maupassant to landscape, it took 16 seconds.
maupassant used 0.6s to render a post vs 0.1s to that of landsacape. Maybe you can reflect this to the author.

@iMilnb what theme are you using? Can you give landscape a try and report.

Member

leesei commented Feb 26, 2016

@tl3shi I cloned your repo
It took 16mins to run hexo g --debug, however if I changed the theme from maupassant to landscape, it took 16 seconds.
maupassant used 0.6s to render a post vs 0.1s to that of landsacape. Maybe you can reflect this to the author.

@iMilnb what theme are you using? Can you give landscape a try and report.

@iMilnb

This comment has been minimized.

Show comment
Hide comment
@iMilnb

iMilnb Feb 26, 2016

theme: landscape
INFO  1074 files generated in 15 s

Boom! that's it, it took only 15 seconds to render the 400 posts with landscape

iMilnb commented Feb 26, 2016

theme: landscape
INFO  1074 files generated in 15 s

Boom! that's it, it took only 15 seconds to render the 400 posts with landscape

@leesei

This comment has been minimized.

Show comment
Hide comment
@leesei

leesei Feb 26, 2016

Member

Alas, though this is not a Hexo problem, it seems easy to write an inefficient theme.

Member

leesei commented Feb 26, 2016

Alas, though this is not a Hexo problem, it seems easy to write an inefficient theme.

@iMilnb

This comment has been minimized.

Show comment
Hide comment
@iMilnb

iMilnb Feb 26, 2016

Sad thing is that I used the light theme which has been forked by many themes and so is spreading around a lot; so the question remains, how to dig the debug deeper to understand what makes the rendering so slow in comparison to landscape. Meanwhile I'll open an issue at https://github.com/hexojs/hexo-theme-light

iMilnb commented Feb 26, 2016

Sad thing is that I used the light theme which has been forked by many themes and so is spreading around a lot; so the question remains, how to dig the debug deeper to understand what makes the rendering so slow in comparison to landscape. Meanwhile I'll open an issue at https://github.com/hexojs/hexo-theme-light

@iMilnb

This comment has been minimized.

Show comment
Hide comment
@iMilnb

iMilnb Feb 27, 2016

Ok I found that it's the tagcloud widget that slows the pages generation. Without it I get:

INFO  840 files generated in 3.92 s

Even faster than landscape.

Edit

The tag widget also slows down page generation. FWIW we're talking about 276 tags.

iMilnb commented Feb 27, 2016

Ok I found that it's the tagcloud widget that slows the pages generation. Without it I get:

INFO  840 files generated in 3.92 s

Even faster than landscape.

Edit

The tag widget also slows down page generation. FWIW we're talking about 276 tags.

@leesei

This comment has been minimized.

Show comment
Hide comment
@leesei

leesei Feb 27, 2016

Member

Themes are supposed to make use of fragment_cache (cached render result) to improve performance.
I think theme-light didn't use it.

Templates Optimization| Hexo
Fragment cache by tommy351 · Pull Request #637 · hexojs/hexo

Member

leesei commented Feb 27, 2016

Themes are supposed to make use of fragment_cache (cached render result) to improve performance.
I think theme-light didn't use it.

Templates Optimization| Hexo
Fragment cache by tommy351 · Pull Request #637 · hexojs/hexo

@iMilnb

This comment has been minimized.

Show comment
Hide comment
@iMilnb

iMilnb Feb 27, 2016

Perfect! here's the fix for light, in layout.ejs, replace:

<aside id="sidebar" class="alignright"><%- partial('_partial/sidebar') %></aside>

with

<aside id="sidebar" class="alignright"><%- partial('_partial/sidebar', {}, {cache: true}) %></aside>

And then:

INFO  1072 files generated in 6.44 s

Thanks a lot @leesei !

iMilnb commented Feb 27, 2016

Perfect! here's the fix for light, in layout.ejs, replace:

<aside id="sidebar" class="alignright"><%- partial('_partial/sidebar') %></aside>

with

<aside id="sidebar" class="alignright"><%- partial('_partial/sidebar', {}, {cache: true}) %></aside>

And then:

INFO  1072 files generated in 6.44 s

Thanks a lot @leesei !

@iMilnb iMilnb closed this Feb 27, 2016

@leesei

This comment has been minimized.

Show comment
Hide comment
@leesei

leesei Feb 27, 2016

Member

Thanks for your testing and helping us to get to fragment_cache.
Care to submit PR to theme-light?

Member

leesei commented Feb 27, 2016

Thanks for your testing and helping us to get to fragment_cache.
Care to submit PR to theme-light?

@iMilnb

This comment has been minimized.

Show comment
Hide comment
@iMilnb

iMilnb Feb 27, 2016

Will definitely do

iMilnb commented Feb 27, 2016

Will definitely do

@iMilnb

This comment has been minimized.

Show comment
Hide comment
@iMilnb

iMilnb commented Feb 28, 2016

@leesei PR merged hexojs/hexo-theme-light#58 (comment)
thanks for your help!

zklhp added a commit to zklhp/maupassant-hexo that referenced this issue Jan 1, 2017

Use cache.
With this change, the generation time shorts from 3 min to 3 s for about 800 files. 

Reference: hexojs/hexo#1769

@zklhp zklhp referenced this issue Jan 1, 2017

Merged

Use cache. #213

@rahil471

This comment has been minimized.

Show comment
Hide comment
@rahil471

rahil471 Apr 13, 2017

Can we only generate/deploy single file?
Eg:

  • I run hexo deploy/generate once. (This renders everything)
  • Now I'm only changing the _posts (creating a new article or editing it).
  • Now instead of having me to do hexo generate again it would be great if I could just place that new/edited article.

rahil471 commented Apr 13, 2017

Can we only generate/deploy single file?
Eg:

  • I run hexo deploy/generate once. (This renders everything)
  • Now I'm only changing the _posts (creating a new article or editing it).
  • Now instead of having me to do hexo generate again it would be great if I could just place that new/edited article.
@mnlbox

This comment has been minimized.

Show comment
Hide comment
@mnlbox

mnlbox May 24, 2017

Hi guys,
I have same issue with 10000 posts. Can you make me some suggestion in related issue:
#2579

mnlbox commented May 24, 2017

Hi guys,
I have same issue with 10000 posts. Can you make me some suggestion in related issue:
#2579

@chilly

This comment has been minimized.

Show comment
Hide comment
@chilly

chilly Jun 11, 2017

git clone -b source git@github.com:chilly/chilly.github.io.git
git submodules update
hexo generate

hexo generate is too slow. only 600+ posts. And cost more then 10 mins.

chilly commented Jun 11, 2017

git clone -b source git@github.com:chilly/chilly.github.io.git
git submodules update
hexo generate

hexo generate is too slow. only 600+ posts. And cost more then 10 mins.

@qixiaobo

This comment has been minimized.

Show comment
Hide comment
@qixiaobo

qixiaobo Sep 18, 2017

git clone -b source git@github.com:chilly/chilly.github.io.git
git submodules update
hexo generate

hexo generate is too slow. only 600+ posts. And cost more then 10 mins.

Me 2

qixiaobo commented Sep 18, 2017

git clone -b source git@github.com:chilly/chilly.github.io.git
git submodules update
hexo generate

hexo generate is too slow. only 600+ posts. And cost more then 10 mins.

Me 2

@ivan-nginx ivan-nginx referenced this issue Sep 29, 2017

Open

标签云显示不完整 #1908

3 of 3 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment