Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(cache): Improve cache, Reduce Memory Usages #3756

Merged
merged 5 commits into from Oct 10, 2019

Conversation

@dailyrandomphoto
Copy link
Contributor

dailyrandomphoto commented Oct 9, 2019

What does it do?

Hexo uses a lot of memory to make a huge site.
This PR will reduce memory usages by modifying the usage of the unnecessary cache.
It can help with the "out of memory" related issues. (e.g., #3350 #3311 #1769 #1763 #1735 #1529)

There are two kinds of caches that hexo using

I've found something that can be improved a little bit to reduce memory usages.

Before

Mode Use Cache Of Rendered HTML Use Fragment Cache
generate true true
generate + watch false false
server(watch) false false
  1. In generate mode, Use Cache Of Rendered HTML should be false, since each route just be called once. The unnecessary cache uses a lot of memory.
    On the other hand, in server mode, Use Cache Of Rendered HTML should be true, since each route will be called several times. After enabling the cache, the route will respond more quickly from second requests.

  2. In generate + watch mode, generating a huge site will be slower than generate mode since Use Fragment Cache is disabled. Use Fragment Cache should always be enabled to speed up generating.

After

This PR will change the usages of caches to

Mode Use Cache Of Rendered HTML Use Fragment Cache
generate false true
generate + watch false true
server(watch) true true

How to test

git clone -b improve-cache https://github.com/dailyrandomphoto/hexo.git
cd hexo
npm install
npm test

Compare: Before vs. After

Before

git clone https://github.com/dailyrandomphoto/hexo-huge-site-test.git
cd hexo-huge-site-test
npm install
npm run g

Environment & Result

  • hexo: hexojs/hexo
  • theme: landscape
  • concurrency: 1 (Introduced in #3665)
  • 3059 files generated in 2.13 min
  • Memory usage: 944MB
  • Rendered HTML strings hold a lot of memory. see screenshots.

After

change the hexo dependency to dailyrandomphoto/hexo#improve-cache in package.json

npm install dailyrandomphoto/hexo#improve-cache
npm run g

Environment & Result

  • hexo: dailyrandomphoto/hexo#improve-cache
  • theme: landscape
  • concurrency: 1 (Introduced in #3665)
  • 3059 files generated in 2.03 min
  • Memory usage: 445MB
  • Rendered HTML strings were garbage collected. see screenshots.

Screenshots

Before

2019100901

After

2019100902

Pull request tasks

  • Add test cases for the changes.
  • Passed the CI test.
@coveralls

This comment has been minimized.

Copy link

coveralls commented Oct 9, 2019

Coverage Status

Coverage decreased (-0.02%) to 97.273% when pulling 1ff8265 on dailyrandomphoto:improve-cache into d2662d4 on hexojs:master.

@SukkaW

This comment has been minimized.

Copy link
Member

SukkaW commented Oct 9, 2019

LGTM. I will run a benchmark on this.

@SukkaW

This comment has been minimized.

Copy link
Member

SukkaW commented Oct 9, 2019

With fragment cache enabled:

NodeJS 8 NodeJS 10 NodeJS 12
Hexo (master) 10s 9.39s 9.4s
This PR 11s 9.33s 9.38s

With fragment cache disabled:

NodeJS 8 NodeJS 10 NodeJS 12
Hexo (master) 15.5s 15s 15s
This PR 16s 15s 14.5s

I can see this PR has its generation speed more or less as same as the current master branch. I am wonder why @tommy351 implement Cache Of Rendered HTML at the beginning.

@SukkaW SukkaW added this to the v4.0.0 milestone Oct 9, 2019
@curbengh

This comment has been minimized.

Copy link
Contributor

curbengh commented Oct 10, 2019

Fragment cache reminds me of this issue #2503, specifically this comment (I highlighted the last line):

Can you confirm whether you have a layout file that is using the {cache:true} option set on the partial that loads the open_graph tag? I had the same issue, and then realized I was caching the partial that was loading my header which then caused this issue. Take a look, that might also have been your issue! hexo s must not respect the cache:true?

By enabling Fragment Cache for every mode, I believe this PR is also an improvement to the Different result between hexo g and hexo s of roadmap. If Fragment Cache was the sole cause, this PR can even fully fix it; though I suspect there might be other cause.

const cache = {};
let cache = {};

ctx.on('generateBefore', function() {

This comment has been minimized.

Copy link
@curbengh

curbengh Oct 10, 2019

Contributor

I think can use arrow function?

@dailyrandomphoto

This comment has been minimized.

Copy link
Contributor Author

dailyrandomphoto commented Oct 10, 2019

@SukkaW

I am wonder why @tommy351 implement Cache Of Rendered HTML at the beginning.

For server mode, Cache Of Rendered HTML can make the route respond more quickly from second requests.

At the beginning, Cache Of Rendered HTML was enabled for server mode. commit

hexo/lib/hexo/index.js

Lines 362 to 367 in e8e45ed

return self.execFilter('template_locals', locals, {context: self})
.then(function(locals){
route.set(path, function(){
if (cache != null) return cache;
var view, name;

But from this commit it was disabled by options.cache option value.

hexo/lib/hexo/index.js

Lines 362 to 368 in c313b5f

return self.execFilter('template_locals', locals, {context: self})
.then(function(locals){
route.set(path, function(){
if (options.cache && cache != null) return cache;
var view, name;

I think @tommy351 wants to disable it for watch mode, however it was disabled for server mode also.

If this feature useful for server mode only(if my guess is not wrong), I think it is better to refactoring as implement cache on the hexo-server repository as a filter or on a separate plugin.

@dailyrandomphoto

This comment has been minimized.

Copy link
Contributor Author

dailyrandomphoto commented Oct 10, 2019

Fragment Cache(introduced in #637) can improve performance a lot.
as @tommy351 said,

The effect of fragment cache is remarkable. I used fragment_cache helper in sidebar to reduce database queries. It only took less than 10 seconds to generate 1000+ static files. The original elapsed time is 10 minutes, which means is at least 60x faster.

However, some partial which output is different between pages should turn it off.
e.g.

  • head: title, og tags, meta tags should diffrent
  • toc
  • etc.

Like these partial(widget) can turn it on:

  • footer
  • tag, tagcloud
  • category
  • archives
  • etc.
@SukkaW
SukkaW approved these changes Oct 10, 2019
Copy link
Member

SukkaW left a comment

LGTM!

@SukkaW SukkaW merged commit 6f6084c into hexojs:master Oct 10, 2019
3 of 4 checks passed
3 of 4 checks passed
coverage/coveralls Coverage decreased (-0.02%) to 97.273%
Details
Travis CI - Pull Request Build Passed
Details
codeclimate All good!
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
@NoahDragon NoahDragon mentioned this pull request Oct 14, 2019
17 of 53 tasks complete
@SukkaW SukkaW added the #perfmatters label Nov 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.