Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up generating speed #550

Open
wants to merge 6 commits into
base: master
from
Open

Speed up generating speed #550

wants to merge 6 commits into from

Conversation

@tommy351
Copy link
Member

@tommy351 tommy351 commented Mar 22, 2014

Generating speed is slow when we have mass files. It may take 10 minutes to build a blog with 300+ posts.

Most time is consuming by file rendering. Every file needs 500ms to render on average (depends on the quantity of posts).

  • Generate files right after routes are updated
  • Optimize rendering process #637
  • Optimize database
@tommy351 tommy351 added bug labels Mar 22, 2014
@tommy351 tommy351 added this to the 2.6 milestone Mar 22, 2014
@chadkoh
Copy link

@chadkoh chadkoh commented Mar 24, 2014

Yep, this is pretty brutal. My blog has 397 posts. It takes 2.5 hours to render!

839 files generated in 8767.078s

@wzpan
Copy link
Member

@wzpan wzpan commented Mar 24, 2014

FYI, have you ever tried other renderers? Do they render files slowly, too?

@chadkoh
Copy link

@chadkoh chadkoh commented Mar 24, 2014

I am currently using Octopress and just converted over to hexo this weekend. Octopress takes like 15 seconds to render my site. I started testing hexo on 2.3 and it was fast, so I was excited. I finally got around to doing the conversion this weekend and it was a brutal experience. I hope this issue gets solved... it is a bit of a barrier to entry.

@tommy351
Copy link
Member Author

@tommy351 tommy351 commented Apr 18, 2014

In commit 80da498, I moved rendering process to workers, which can make use of all cores of CPU. But it wastes lots of memory.

Plus, I found the elapsed time decreased a lot if we don't use categories/tags. The bottleneck may be the data model?

@kamenitxan
Copy link

@kamenitxan kamenitxan commented May 21, 2014

I tried jade&less theme (https://github.com/widatama/hexo-jade-barebone) and it generates my blog in 8 seconds, down from 3 minutes with default theme.

I guess that hexo-renderer-ejs needs optimalization, not Hexo.

@tommy351
Copy link
Member Author

@tommy351 tommy351 commented May 21, 2014

@kamenitxan I think the problem is not the template engine. It's because this theme doesn't have widgets. The default theme has many widgets enabled by default which takes lots of time to query.

@tommy351
Copy link
Member Author

@tommy351 tommy351 commented May 23, 2014

Because it's more difficult to optimize data model, I came up with another idea: Fragment Cache #637

@yimt
Copy link

@yimt yimt commented Sep 14, 2014

在hexo建立稳定之后,都是往里面添加新的文章,有没有可能在generate的时候只generate一些必要的页面,比如新的文章、archives、sitemap之类的?

@Xuanwo
Copy link
Contributor

@Xuanwo Xuanwo commented Apr 6, 2015

每次运行的时候都diff一下,然后生成有变动的网页?

不知道这样搞会不会反而速度更慢= =

@rose1988c
Copy link

@rose1988c rose1988c commented Nov 19, 2015

hexo server & 在后台运行的时候,hexo g 非常卡

@Xuanwo
Copy link
Contributor

@Xuanwo Xuanwo commented Nov 19, 2015

@rose1988c 不用hexo server的时候卡不卡呢?

@rose1988c
Copy link

@rose1988c rose1988c commented Nov 20, 2015

@Xuanwo

// 在 hexo server在后台运行的情况,hexo g 卡主了,cpu爆满,直接被系统Killed
root@hexo:/opt/www/rose1988c.github.io# hexo g

INFO  Generated: archives/2014/03/index.html
INFO  Generated: archives/2014/09/index.html
INFO  Generated: archives/2014/10/index.html
......
INFO  Generated: index.html
INFO  Generated: page/2/index.html
Killed

//  下面kill hexo server 再 hexo g
root@hexo:/opt/www/rose1988c.github.io# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  20260     0 ?        Ss+  Nov19   0:00 /bin/bash
root        20  0.0  0.1  20268  1588 ?        Ss   Nov19   0:00 /bin/bash
root       352  5.3 28.9 1295944 293088 ?      Sl   09:42   0:47 hexo                           
root       396  0.0  0.1  17492  1964 ?        R+   09:56   0:00 ps aux

root@hexo:/opt/www/rose1988c.github.io# kill 352

root@hexo:/opt/www/rose1988c.github.io# hexo g

INFO  Files loaded in 24 s
INFO  Generated: tags/index.html
INFO  Generated: link/index.html
......
INFO  Generated: tags/java/index.html
INFO  Generated: tags/maven/index.html
INFO  220 files generated in 20 s
[1]+  Terminated              hexo server
// 成功正常跑完
@tommy351
Copy link
Member Author

@tommy351 tommy351 commented Nov 20, 2015

Why you run hexo generate and hexo server at the same time?

@rose1988c
Copy link

@rose1988c rose1988c commented Nov 20, 2015

@tommy351 我相信有部分人因为这个hexo server,卡死在hexo g上面。

因为我是 127.0.0.1:4000github gh-pages 是一个负载均衡,所以hexo server 是一直运行着的

@Xuanwo
Copy link
Contributor

@Xuanwo Xuanwo commented Nov 20, 2015

@rose1988c 我还是认为各司其职比较好,测试用hexo server,拿来访问&作负载均衡还是用apache或者nginx

@rose1988c
Copy link

@rose1988c rose1988c commented Nov 20, 2015

@Xuanwo 是nginx做的......

俺是看到上面的说hexo g 卡死,我发了这个我碰到的,并提醒类似的小伙伴~

@Xuanwo
Copy link
Contributor

@Xuanwo Xuanwo commented Nov 20, 2015

@rose1988c 哎?跑nginx的话,就不用跑hexo sever了吧?。。

不过你的意思我明白了~

@tl3shi
Copy link

@tl3shi tl3shi commented Feb 18, 2016

20:18:47.107 INFO  3384 files generated in 25 min
20:18:47.335 DEBUG Database saved

387篇 post, 有很多tag/category, 哭了~ 发布一篇 花25min?

@leesei leesei added the #perfmatters label Feb 22, 2016
@Xuanwo
Copy link
Contributor

@Xuanwo Xuanwo commented Feb 25, 2016

@tl3shi
Copy link

@tl3shi tl3shi commented Feb 25, 2016

@Xuanwo 求帮忙看看啊, 为啥我的这么慢~
github repo

@Xuanwo
Copy link
Contributor

@Xuanwo Xuanwo commented Feb 25, 2016

@t3shi 我也不知道- -,或许你可以把分页关掉试试。。

@tl3shi
Copy link

@tl3shi tl3shi commented Feb 25, 2016

@Xuanwo 你那个travis-ci提供了编译环境?免费帮忙编译部署到github?

@Xuanwo
Copy link
Contributor

@Xuanwo Xuanwo commented Feb 25, 2016

@tl3shi 持续集成,你可以自行google了解一下~

@leesei
Copy link
Member

@leesei leesei commented Feb 25, 2016

@tl3shi This PR has been hung for a long time.
Please join #1769 for the latest thread.

@mnlbox
Copy link

@mnlbox mnlbox commented May 24, 2017

+1 for this improvement. 👍

@rbarilani
Copy link

@rbarilani rbarilani commented Jun 29, 2017

@tommy351 is this project still alive?

@ghost
Copy link

@ghost ghost commented Jun 29, 2017

@rnarilani When you say "project" are you referring to Hexo, or this issue specifically?

@rbarilani
Copy link

@rbarilani rbarilani commented Jun 29, 2017

@jhabdas The hexo project, I just post it here since will probably open PR to enhance the project but the amount of open PR and their dates are making us doubt a little bit.

@NoahDragon
Copy link
Member

@NoahDragon NoahDragon commented Jun 29, 2017

@rbarilani Yes, it is alive. @tommy351 is busy, and I'm the current maintainer for the project. Feel free to submit any issues and PRs. 😄

@ghost
Copy link

@ghost ghost commented Jun 29, 2017

@rbarilani if ever in doubt just look at the GitHub pulse and other metrics. Or just look at the date of the last closed issue(s) for bigger projects. @NoahDragon has been doing a bang up job triaging. Lucky for him this isn't the Yarn repo! xD

@rbarilani
Copy link

@rbarilani rbarilani commented Jun 29, 2017

@jhabdas thanks and good to know.

@ghost
Copy link

@ghost ghost commented Jun 30, 2017

@rbarilani My pleasure. Just for you my friend (kidding, it's actually for a blog post) I've put together frequency charts for Hugo, Hexo and Jekyll. And here they are:

screen shot 2017-06-30 at 6 13 55 pm

@rbarilani
Copy link

@rbarilani rbarilani commented Jun 30, 2017

@jhabdas Can we use gitter chat for questions? I'm building a documentation theme on top of hexo and I have and I will have some questions.

@JLHwung JLHwung closed this Jul 8, 2017
@JLHwung JLHwung changed the base branch from dev to master Jul 8, 2017
@JLHwung
Copy link
Collaborator

@JLHwung JLHwung commented Jul 8, 2017

@jhabdas I don't mean to close this PR. I apologize if it seems confusing.

What happens is that this PR's original base is dev branch, which is a stale merged branch for a long time. So I deleted this dev branch and Github notice that the dev branch is deleted and closesd this PR.

I just change the base to master and reopen this PR. Thank you for your contributions.

@JLHwung JLHwung reopened this Jul 8, 2017
@Zhang21
Copy link

@Zhang21 Zhang21 commented Dec 5, 2018

我也有这个问题,随着文章数量和文章字数的不断增加,hexo g所需的时间便不断增加。

有没有办法让hexo只生成新的post,因为老的文章已经生成和发布,便不需要再生成,这样也可以节约时间。如加入一个diff之类的命令?

@ghost
ghost approved these changes Dec 22, 2020
Copy link

@ghost ghost left a comment

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet