Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Solve "Out of memory" during generation by limiting the number of concurrent tasks #3665

merged 1 commit into from Aug 20, 2019


Copy link

@ppoffice ppoffice commented Aug 15, 2019

What does it do?

Allow users to set the number of files to be generated in parallel.
Users who are struggling with the "out of memory" problem (e.g., #3350 #3311) can limit Hexo's memory usage by hexo g --concurrency <number>.

How to test

  1. hexo init a new blog.
  2. change the hexo dependency to ppoffice/hexo#master in package.json
  3. npm install
  4. Add some posts git clone source/_posts/tests
  5. hexo g -c <number of concurrent tasks>



Test 1: hexo clean && hexo g

  • No concurrency limit
  • 659 files generated in 24 s
  • Memory usage: 500 ~ 660 MB

Test 2: hexo clean && hexo g -c 20

  • 20 concurrent tasks at most
  • 659 files generated in 30 s
  • Memory usage: 450 ~ 550 MB

Test 3: hexo clean && hexo g -c 10

  • 10 concurrent tasks at most
  • 659 files generated in 34 s
  • Memory usage: 480 ~ 550 MB

Test 4: hexo clean && hexo g -c 5

  • 5 concurrent tasks at most
  • 659 files generated in 34 s
  • Memory usage: 420 ~ 550 MB

Pull request tasks

  • Add test cases for the changes.
  • Passed the CI test.

Copy link

@coveralls coveralls commented Aug 15, 2019

Coverage Status

Coverage increased (+0.004%) to 97.157% when pulling 7a51bb4 on ppoffice:master into 48b506d on hexojs:master.

Copy link

@curbengh curbengh commented Aug 15, 2019

LGTM, I don't mind --concurrency since it reflects the option in, but I figure some might prefer -p, --parallel.

Edit: -p, --parallel could be confused with -p, --path.

Copy link

@tcrowe tcrowe commented Aug 15, 2019

I have reviewed and tested it myself. Thank you @ppoffice and @curbengh

--concurrency -c is appropriate.

mkdir concurrency-test
cd concurrency-test
hexo init
npm install 'ppoffice/hexo#master'
git clone source/_posts/tests
hexo gen
hexo clean
hexo gen -c 8
hexo clean
hexo gen --concurrency 1
hexo clean
hexo gen -c 200

@yoshinorin yoshinorin added the enhancement label Aug 20, 2019
@yoshinorin yoshinorin added this to the v4.0.0 milestone Aug 20, 2019
@yoshinorin yoshinorin merged commit ec99001 into hexojs:master Aug 20, 2019
3 of 4 checks passed
Copy link

@curbengh curbengh commented Sep 2, 2019

Related issues: #1763 #1735 #1529

Copy link

@chenshiforever chenshiforever commented Sep 22, 2019

but I donn't kown what i can do ,this step2. change the hexo dependency to ppoffice/hexo#master in package.json is not understand . i think i have nothing to do except wait for it to update to 4.0.

Copy link

@curbengh curbengh commented Sep 22, 2019

That is using ppoffice's fork, you can use this repository instead.

-  "hexo": "^3.9.0",
+  "hexo": "hexojs/hexo",

Modify the line with "hexo" in your "package.json" file, change the value from "^3.9.0", to "hexojs/hexo",


This is akin to using bleeding-edge release.

Copy link

@chenshiforever chenshiforever commented Sep 22, 2019

But this approach doesn't seem to be of much use. It does add more than a hundred posts, but exceeding that number will crash, and here's my error message

INFO  1968 files generated in 6.87 min
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node_module_register
 2: v8::internal::FatalProcessOutOfMemory
 3: v8::internal::FatalProcessOutOfMemory
 4: v8::internal::Factory::NewRawTwoByteString
 5: v8::internal::Smi::SmiPrint
 6: v8::internal::StackGuard::HandleInterrupts
 7: v8::String::WriteUtf8
 8: v8_inspector::V8InspectorClient::currentTimeMS
 9: node::Buffer::New
10: node::Buffer::New
11: 00000199789F17C6
npm ERR! errno 3
npm ERR! hexo@0.0.1 push: `hexo cl && hexo g -c 30 && gulp && gulp mv && hexo s`
npm ERR! Exit status 3
npm ERR!
npm ERR! Failed at the hexo@0.0.1 push script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

Copy link
Contributor Author

@ppoffice ppoffice commented Sep 22, 2019

Please try to use a smaller concurrency value such as 10. By the way, what theme are you using? It may also have something to do with its optimization.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

Successfully merging this pull request may close these issues.

None yet

6 participants