New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up generating speed #550

Open
wants to merge 6 commits into
base: master
from

Conversation

@tommy351
Member

tommy351 commented Mar 22, 2014

Generating speed is slow when we have mass files. It may take 10 minutes to build a blog with 300+ posts.

Most time is consuming by file rendering. Every file needs 500ms to render on average (depends on the quantity of posts).

  • Generate files right after routes are updated
  • Optimize rendering process #637
  • Optimize database

@tommy351 tommy351 added bug labels Mar 22, 2014

@tommy351 tommy351 added this to the 2.6 milestone Mar 22, 2014

@chadkoh

This comment has been minimized.

Show comment
Hide comment
@chadkoh

chadkoh Mar 24, 2014

Yep, this is pretty brutal. My blog has 397 posts. It takes 2.5 hours to render!

839 files generated in 8767.078s

chadkoh commented Mar 24, 2014

Yep, this is pretty brutal. My blog has 397 posts. It takes 2.5 hours to render!

839 files generated in 8767.078s

@wzpan

This comment has been minimized.

Show comment
Hide comment
@wzpan

wzpan Mar 24, 2014

Member

FYI, have you ever tried other renderers? Do they render files slowly, too?

Member

wzpan commented Mar 24, 2014

FYI, have you ever tried other renderers? Do they render files slowly, too?

@chadkoh

This comment has been minimized.

Show comment
Hide comment
@chadkoh

chadkoh Mar 24, 2014

I am currently using Octopress and just converted over to hexo this weekend. Octopress takes like 15 seconds to render my site. I started testing hexo on 2.3 and it was fast, so I was excited. I finally got around to doing the conversion this weekend and it was a brutal experience. I hope this issue gets solved... it is a bit of a barrier to entry.

chadkoh commented Mar 24, 2014

I am currently using Octopress and just converted over to hexo this weekend. Octopress takes like 15 seconds to render my site. I started testing hexo on 2.3 and it was fast, so I was excited. I finally got around to doing the conversion this weekend and it was a brutal experience. I hope this issue gets solved... it is a bit of a barrier to entry.

@tommy351

This comment has been minimized.

Show comment
Hide comment
@tommy351

tommy351 Apr 18, 2014

Member

In commit 80da498, I moved rendering process to workers, which can make use of all cores of CPU. But it wastes lots of memory.

Plus, I found the elapsed time decreased a lot if we don't use categories/tags. The bottleneck may be the data model?

Member

tommy351 commented Apr 18, 2014

In commit 80da498, I moved rendering process to workers, which can make use of all cores of CPU. But it wastes lots of memory.

Plus, I found the elapsed time decreased a lot if we don't use categories/tags. The bottleneck may be the data model?

@kamenitxan

This comment has been minimized.

Show comment
Hide comment
@kamenitxan

kamenitxan May 21, 2014

I tried jade&less theme (https://github.com/widatama/hexo-jade-barebone) and it generates my blog in 8 seconds, down from 3 minutes with default theme.

I guess that hexo-renderer-ejs needs optimalization, not Hexo.

kamenitxan commented May 21, 2014

I tried jade&less theme (https://github.com/widatama/hexo-jade-barebone) and it generates my blog in 8 seconds, down from 3 minutes with default theme.

I guess that hexo-renderer-ejs needs optimalization, not Hexo.

@tommy351

This comment has been minimized.

Show comment
Hide comment
@tommy351

tommy351 May 21, 2014

Member

@kamenitxan I think the problem is not the template engine. It's because this theme doesn't have widgets. The default theme has many widgets enabled by default which takes lots of time to query.

Member

tommy351 commented May 21, 2014

@kamenitxan I think the problem is not the template engine. It's because this theme doesn't have widgets. The default theme has many widgets enabled by default which takes lots of time to query.

@tommy351

This comment has been minimized.

Show comment
Hide comment
@tommy351

tommy351 May 23, 2014

Member

Because it's more difficult to optimize data model, I came up with another idea: Fragment Cache #637

Member

tommy351 commented May 23, 2014

Because it's more difficult to optimize data model, I came up with another idea: Fragment Cache #637

@yimt

This comment has been minimized.

Show comment
Hide comment
@yimt

yimt Sep 14, 2014

在hexo建立稳定之后,都是往里面添加新的文章,有没有可能在generate的时候只generate一些必要的页面,比如新的文章、archives、sitemap之类的?

yimt commented Sep 14, 2014

在hexo建立稳定之后,都是往里面添加新的文章,有没有可能在generate的时候只generate一些必要的页面,比如新的文章、archives、sitemap之类的?

@Xuanwo

This comment has been minimized.

Show comment
Hide comment
@Xuanwo

Xuanwo Apr 6, 2015

Contributor

每次运行的时候都diff一下,然后生成有变动的网页?

不知道这样搞会不会反而速度更慢= =

Contributor

Xuanwo commented Apr 6, 2015

每次运行的时候都diff一下,然后生成有变动的网页?

不知道这样搞会不会反而速度更慢= =

@rose1988c

This comment has been minimized.

Show comment
Hide comment
@rose1988c

rose1988c Nov 19, 2015

hexo server & 在后台运行的时候,hexo g 非常卡

rose1988c commented Nov 19, 2015

hexo server & 在后台运行的时候,hexo g 非常卡

@Xuanwo

This comment has been minimized.

Show comment
Hide comment
@Xuanwo

Xuanwo Nov 19, 2015

Contributor

@rose1988c 不用hexo server的时候卡不卡呢?

Contributor

Xuanwo commented Nov 19, 2015

@rose1988c 不用hexo server的时候卡不卡呢?

@rose1988c

This comment has been minimized.

Show comment
Hide comment
@rose1988c

rose1988c Nov 20, 2015

@Xuanwo

// 在 hexo server在后台运行的情况,hexo g 卡主了,cpu爆满,直接被系统Killed
root@hexo:/opt/www/rose1988c.github.io# hexo g

INFO  Generated: archives/2014/03/index.html
INFO  Generated: archives/2014/09/index.html
INFO  Generated: archives/2014/10/index.html
......
INFO  Generated: index.html
INFO  Generated: page/2/index.html
Killed

//  下面kill hexo server 再 hexo g
root@hexo:/opt/www/rose1988c.github.io# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  20260     0 ?        Ss+  Nov19   0:00 /bin/bash
root        20  0.0  0.1  20268  1588 ?        Ss   Nov19   0:00 /bin/bash
root       352  5.3 28.9 1295944 293088 ?      Sl   09:42   0:47 hexo                           
root       396  0.0  0.1  17492  1964 ?        R+   09:56   0:00 ps aux

root@hexo:/opt/www/rose1988c.github.io# kill 352

root@hexo:/opt/www/rose1988c.github.io# hexo g

INFO  Files loaded in 24 s
INFO  Generated: tags/index.html
INFO  Generated: link/index.html
......
INFO  Generated: tags/java/index.html
INFO  Generated: tags/maven/index.html
INFO  220 files generated in 20 s
[1]+  Terminated              hexo server
// 成功正常跑完

rose1988c commented Nov 20, 2015

@Xuanwo

// 在 hexo server在后台运行的情况,hexo g 卡主了,cpu爆满,直接被系统Killed
root@hexo:/opt/www/rose1988c.github.io# hexo g

INFO  Generated: archives/2014/03/index.html
INFO  Generated: archives/2014/09/index.html
INFO  Generated: archives/2014/10/index.html
......
INFO  Generated: index.html
INFO  Generated: page/2/index.html
Killed

//  下面kill hexo server 再 hexo g
root@hexo:/opt/www/rose1988c.github.io# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  20260     0 ?        Ss+  Nov19   0:00 /bin/bash
root        20  0.0  0.1  20268  1588 ?        Ss   Nov19   0:00 /bin/bash
root       352  5.3 28.9 1295944 293088 ?      Sl   09:42   0:47 hexo                           
root       396  0.0  0.1  17492  1964 ?        R+   09:56   0:00 ps aux

root@hexo:/opt/www/rose1988c.github.io# kill 352

root@hexo:/opt/www/rose1988c.github.io# hexo g

INFO  Files loaded in 24 s
INFO  Generated: tags/index.html
INFO  Generated: link/index.html
......
INFO  Generated: tags/java/index.html
INFO  Generated: tags/maven/index.html
INFO  220 files generated in 20 s
[1]+  Terminated              hexo server
// 成功正常跑完
@tommy351

This comment has been minimized.

Show comment
Hide comment
@tommy351

tommy351 Nov 20, 2015

Member

Why you run hexo generate and hexo server at the same time?

Member

tommy351 commented Nov 20, 2015

Why you run hexo generate and hexo server at the same time?

@rose1988c

This comment has been minimized.

Show comment
Hide comment
@rose1988c

rose1988c Nov 20, 2015

@tommy351 我相信有部分人因为这个hexo server,卡死在hexo g上面。

因为我是 127.0.0.1:4000github gh-pages 是一个负载均衡,所以hexo server 是一直运行着的

rose1988c commented Nov 20, 2015

@tommy351 我相信有部分人因为这个hexo server,卡死在hexo g上面。

因为我是 127.0.0.1:4000github gh-pages 是一个负载均衡,所以hexo server 是一直运行着的

@Xuanwo

This comment has been minimized.

Show comment
Hide comment
@Xuanwo

Xuanwo Nov 20, 2015

Contributor

@rose1988c 我还是认为各司其职比较好,测试用hexo server,拿来访问&作负载均衡还是用apache或者nginx

Contributor

Xuanwo commented Nov 20, 2015

@rose1988c 我还是认为各司其职比较好,测试用hexo server,拿来访问&作负载均衡还是用apache或者nginx

@rose1988c

This comment has been minimized.

Show comment
Hide comment
@rose1988c

rose1988c Nov 20, 2015

@Xuanwo 是nginx做的......

俺是看到上面的说hexo g 卡死,我发了这个我碰到的,并提醒类似的小伙伴~

rose1988c commented Nov 20, 2015

@Xuanwo 是nginx做的......

俺是看到上面的说hexo g 卡死,我发了这个我碰到的,并提醒类似的小伙伴~

@Xuanwo

This comment has been minimized.

Show comment
Hide comment
@Xuanwo

Xuanwo Nov 20, 2015

Contributor

@rose1988c 哎?跑nginx的话,就不用跑hexo sever了吧?。。

不过你的意思我明白了~

Contributor

Xuanwo commented Nov 20, 2015

@rose1988c 哎?跑nginx的话,就不用跑hexo sever了吧?。。

不过你的意思我明白了~

@tl3shi

This comment has been minimized.

Show comment
Hide comment
@tl3shi

tl3shi Feb 18, 2016

20:18:47.107 INFO  3384 files generated in 25 min
20:18:47.335 DEBUG Database saved

387篇 post, 有很多tag/category, 哭了~ 发布一篇 花25min?

tl3shi commented Feb 18, 2016

20:18:47.107 INFO  3384 files generated in 25 min
20:18:47.335 DEBUG Database saved

387篇 post, 有很多tag/category, 哭了~ 发布一篇 花25min?

@leesei leesei added the #perfmatters label Feb 22, 2016

@Xuanwo

This comment has been minimized.

Show comment
Hide comment
Contributor

Xuanwo commented Feb 25, 2016

@tl3shi

This comment has been minimized.

Show comment
Hide comment
@tl3shi

tl3shi Feb 25, 2016

@Xuanwo 求帮忙看看啊, 为啥我的这么慢~
github repo

tl3shi commented Feb 25, 2016

@Xuanwo 求帮忙看看啊, 为啥我的这么慢~
github repo

@Xuanwo

This comment has been minimized.

Show comment
Hide comment
@Xuanwo

Xuanwo Feb 25, 2016

Contributor

@t3shi 我也不知道- -,或许你可以把分页关掉试试。。

Contributor

Xuanwo commented Feb 25, 2016

@t3shi 我也不知道- -,或许你可以把分页关掉试试。。

@tl3shi

This comment has been minimized.

Show comment
Hide comment
@tl3shi

tl3shi Feb 25, 2016

@Xuanwo 你那个travis-ci提供了编译环境?免费帮忙编译部署到github?

tl3shi commented Feb 25, 2016

@Xuanwo 你那个travis-ci提供了编译环境?免费帮忙编译部署到github?

@Xuanwo

This comment has been minimized.

Show comment
Hide comment
@Xuanwo

Xuanwo Feb 25, 2016

Contributor

@tl3shi 持续集成,你可以自行google了解一下~

Contributor

Xuanwo commented Feb 25, 2016

@tl3shi 持续集成,你可以自行google了解一下~

@leesei

This comment has been minimized.

Show comment
Hide comment
@leesei

leesei Feb 25, 2016

Member

@tl3shi This PR has been hung for a long time.
Please join #1769 for the latest thread.

Member

leesei commented Feb 25, 2016

@tl3shi This PR has been hung for a long time.
Please join #1769 for the latest thread.

@NoahDragon NoahDragon added this to the 4.0 milestone Apr 9, 2017

@mnlbox

This comment has been minimized.

Show comment
Hide comment
@mnlbox

mnlbox May 24, 2017

+1 for this improvement. 👍

mnlbox commented May 24, 2017

+1 for this improvement. 👍

@rbarilani

This comment has been minimized.

Show comment
Hide comment
@rbarilani

rbarilani Jun 29, 2017

@tommy351 is this project still alive?

rbarilani commented Jun 29, 2017

@tommy351 is this project still alive?

@jhabdas

This comment has been minimized.

Show comment
Hide comment
@jhabdas

jhabdas Jun 29, 2017

@rnarilani When you say "project" are you referring to Hexo, or this issue specifically?

jhabdas commented Jun 29, 2017

@rnarilani When you say "project" are you referring to Hexo, or this issue specifically?

@rbarilani

This comment has been minimized.

Show comment
Hide comment
@rbarilani

rbarilani Jun 29, 2017

@jhabdas The hexo project, I just post it here since will probably open PR to enhance the project but the amount of open PR and their dates are making us doubt a little bit.

rbarilani commented Jun 29, 2017

@jhabdas The hexo project, I just post it here since will probably open PR to enhance the project but the amount of open PR and their dates are making us doubt a little bit.

@NoahDragon

This comment has been minimized.

Show comment
Hide comment
@NoahDragon

NoahDragon Jun 29, 2017

Member

@rbarilani Yes, it is alive. @tommy351 is busy, and I'm the current maintainer for the project. Feel free to submit any issues and PRs. 😄

Member

NoahDragon commented Jun 29, 2017

@rbarilani Yes, it is alive. @tommy351 is busy, and I'm the current maintainer for the project. Feel free to submit any issues and PRs. 😄

@jhabdas

This comment has been minimized.

Show comment
Hide comment
@jhabdas

jhabdas Jun 29, 2017

@rbarilani if ever in doubt just look at the GitHub pulse and other metrics. Or just look at the date of the last closed issue(s) for bigger projects. @NoahDragon has been doing a bang up job triaging. Lucky for him this isn't the Yarn repo! xD

jhabdas commented Jun 29, 2017

@rbarilani if ever in doubt just look at the GitHub pulse and other metrics. Or just look at the date of the last closed issue(s) for bigger projects. @NoahDragon has been doing a bang up job triaging. Lucky for him this isn't the Yarn repo! xD

@rbarilani

This comment has been minimized.

Show comment
Hide comment
@rbarilani

rbarilani Jun 29, 2017

@jhabdas thanks and good to know.

rbarilani commented Jun 29, 2017

@jhabdas thanks and good to know.

@jhabdas

This comment has been minimized.

Show comment
Hide comment
@jhabdas

jhabdas Jun 30, 2017

@rbarilani My pleasure. Just for you my friend (kidding, it's actually for a blog post) I've put together frequency charts for Hugo, Hexo and Jekyll. And here they are:

screen shot 2017-06-30 at 6 13 55 pm

jhabdas commented Jun 30, 2017

@rbarilani My pleasure. Just for you my friend (kidding, it's actually for a blog post) I've put together frequency charts for Hugo, Hexo and Jekyll. And here they are:

screen shot 2017-06-30 at 6 13 55 pm

@rbarilani

This comment has been minimized.

Show comment
Hide comment
@rbarilani

rbarilani Jun 30, 2017

@jhabdas Can we use gitter chat for questions? I'm building a documentation theme on top of hexo and I have and I will have some questions.

rbarilani commented Jun 30, 2017

@jhabdas Can we use gitter chat for questions? I'm building a documentation theme on top of hexo and I have and I will have some questions.

@JLHwung JLHwung closed this Jul 8, 2017

@JLHwung JLHwung changed the base branch from dev to master Jul 8, 2017

@JLHwung

This comment has been minimized.

Show comment
Hide comment
@JLHwung

JLHwung Jul 8, 2017

Collaborator

@jhabdas I don't mean to close this PR. I apologize if it seems confusing.

What happens is that this PR's original base is dev branch, which is a stale merged branch for a long time. So I deleted this dev branch and Github notice that the dev branch is deleted and closesd this PR.

I just change the base to master and reopen this PR. Thank you for your contributions.

Collaborator

JLHwung commented Jul 8, 2017

@jhabdas I don't mean to close this PR. I apologize if it seems confusing.

What happens is that this PR's original base is dev branch, which is a stale merged branch for a long time. So I deleted this dev branch and Github notice that the dev branch is deleted and closesd this PR.

I just change the base to master and reopen this PR. Thank you for your contributions.

@JLHwung JLHwung reopened this Jul 8, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment