Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Solve "Out of memory" during generation by limiting the number of concurrent tasks #3665

Merged
merged 1 commit into from Aug 20, 2019

Conversation

ppoffice
Copy link
Contributor

@ppoffice ppoffice commented Aug 15, 2019

What does it do?

Allow users to set the number of files to be generated in parallel.
Users who are struggling with the "out of memory" problem (e.g., #3350 #3311) can limit Hexo's memory usage by hexo g --concurrency <number>.

How to test

  1. hexo init a new blog.
  2. change the hexo dependency to ppoffice/hexo#master in package.json
  3. npm install
  4. Add some posts git clone https://github.com/SukkaLab/hexo-many-posts.git source/_posts/tests
  5. hexo g -c <number of concurrent tasks>

Testings

Environment

Test 1: hexo clean && hexo g

  • No concurrency limit
  • 659 files generated in 24 s
  • Memory usage: 500 ~ 660 MB
    mem-graph

Test 2: hexo clean && hexo g -c 20

  • 20 concurrent tasks at most
  • 659 files generated in 30 s
  • Memory usage: 450 ~ 550 MB
    mem-graph

Test 3: hexo clean && hexo g -c 10

  • 10 concurrent tasks at most
  • 659 files generated in 34 s
  • Memory usage: 480 ~ 550 MB
    mem-graph

Test 4: hexo clean && hexo g -c 5

  • 5 concurrent tasks at most
  • 659 files generated in 34 s
  • Memory usage: 420 ~ 550 MB
    mem-graph

Pull request tasks

  • Add test cases for the changes.
  • Passed the CI test.

@coveralls
Copy link

@coveralls coveralls commented Aug 15, 2019

Coverage Status

Coverage increased (+0.004%) to 97.157% when pulling 7a51bb4 on ppoffice:master into 48b506d on hexojs:master.

@curbengh
Copy link
Contributor

@curbengh curbengh commented Aug 15, 2019

LGTM, I don't mind --concurrency since it reflects the option in Promise.map(), but I figure some might prefer -p, --parallel.

Edit: -p, --parallel could be confused with -p, --path.

@tcrowe
Copy link
Contributor

@tcrowe tcrowe commented Aug 15, 2019

I have reviewed and tested it myself. Thank you @ppoffice and @curbengh

--concurrency -c is appropriate.

mkdir concurrency-test
cd concurrency-test
hexo init
npm install 'ppoffice/hexo#master'
git clone https://github.com/SukkaLab/hexo-many-posts.git source/_posts/tests
hexo gen
hexo clean
hexo gen -c 8
hexo clean
hexo gen --concurrency 1
hexo clean
hexo gen -c 200

@yoshinorin yoshinorin added the enhancement label Aug 20, 2019
@yoshinorin yoshinorin added this to the v4.0.0 milestone Aug 20, 2019
@yoshinorin yoshinorin merged commit ec99001 into hexojs:master Aug 20, 2019
3 of 4 checks passed
@curbengh
Copy link
Contributor

@curbengh curbengh commented Sep 2, 2019

Related issues: #1763 #1735 #1529

@chenshiforever
Copy link

@chenshiforever chenshiforever commented Sep 22, 2019

but I donn't kown what i can do ,this step2. change the hexo dependency to ppoffice/hexo#master in package.json is not understand . i think i have nothing to do except wait for it to update to 4.0.

@curbengh
Copy link
Contributor

@curbengh curbengh commented Sep 22, 2019

That is using ppoffice's fork, you can use this repository instead.

package.json
-  "hexo": "^3.9.0",
+  "hexo": "hexojs/hexo",

Modify the line with "hexo" in your "package.json" file, change the value from "^3.9.0", to "hexojs/hexo",

Example

This is akin to using bleeding-edge release.

@chenshiforever
Copy link

@chenshiforever chenshiforever commented Sep 22, 2019

But this approach doesn't seem to be of much use. It does add more than a hundred posts, but exceeding that number will crash, and here's my error message
中文版:但是这个方法对我来说似乎没有起太大的作用。确实可以增加一百多的post数量,但是超过了这个数字就会崩溃,下面是我的报错信息

INFO  1968 files generated in 6.87 min
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node_module_register
 2: v8::internal::FatalProcessOutOfMemory
 3: v8::internal::FatalProcessOutOfMemory
 4: v8::internal::Factory::NewRawTwoByteString
 5: v8::internal::Smi::SmiPrint
 6: v8::internal::StackGuard::HandleInterrupts
 7: v8::String::WriteUtf8
 8: v8_inspector::V8InspectorClient::currentTimeMS
 9: node::Buffer::New
10: node::Buffer::New
11: 00000199789F17C6
npm ERR! code ELIFECYCLE
npm ERR! errno 3
npm ERR! hexo@0.0.1 push: `hexo cl && hexo g -c 30 && gulp && gulp mv && hexo s`
npm ERR! Exit status 3
npm ERR!
npm ERR! Failed at the hexo@0.0.1 push script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

@ppoffice
Copy link
Contributor Author

@ppoffice ppoffice commented Sep 22, 2019

@chenshiforever
Please try to use a smaller concurrency value such as 10. By the way, what theme are you using? It may also have something to do with its optimization.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants