Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Solve "Out of memory" during generation by limiting the number of concurrent tasks #3665

Merged
merged 1 commit into from Aug 20, 2019

Conversation

@ppoffice
Copy link
Contributor

ppoffice commented Aug 15, 2019

What does it do?

Allow users to set the number of files to be generated in parallel.
Users who are struggling with the "out of memory" problem (e.g., #3350 #3311) can limit Hexo's memory usage by hexo g --concurrency <number>.

How to test

  1. hexo init a new blog.
  2. change the hexo dependency to ppoffice/hexo#master in package.json
  3. npm install
  4. Add some posts git clone https://github.com/SukkaLab/hexo-many-posts.git source/_posts/tests
  5. hexo g -c <number of concurrent tasks>

Testings

Environment

Test 1: hexo clean && hexo g

  • No concurrency limit
  • 659 files generated in 24 s
  • Memory usage: 500 ~ 660 MB
    mem-graph

Test 2: hexo clean && hexo g -c 20

  • 20 concurrent tasks at most
  • 659 files generated in 30 s
  • Memory usage: 450 ~ 550 MB
    mem-graph

Test 3: hexo clean && hexo g -c 10

  • 10 concurrent tasks at most
  • 659 files generated in 34 s
  • Memory usage: 480 ~ 550 MB
    mem-graph

Test 4: hexo clean && hexo g -c 5

  • 5 concurrent tasks at most
  • 659 files generated in 34 s
  • Memory usage: 420 ~ 550 MB
    mem-graph

Pull request tasks

  • Add test cases for the changes.
  • Passed the CI test.
@coveralls

This comment has been minimized.

Copy link

coveralls commented Aug 15, 2019

Coverage Status

Coverage increased (+0.004%) to 97.157% when pulling 7a51bb4 on ppoffice:master into 48b506d on hexojs:master.

@curbengh

This comment has been minimized.

Copy link
Contributor

curbengh commented Aug 15, 2019

LGTM, I don't mind --concurrency since it reflects the option in Promise.map(), but I figure some might prefer -p, --parallel.

Edit: -p, --parallel could be confused with -p, --path.

@tcrowe

This comment has been minimized.

Copy link
Contributor

tcrowe commented Aug 15, 2019

I have reviewed and tested it myself. Thank you @ppoffice and @curbengh

--concurrency -c is appropriate.

mkdir concurrency-test
cd concurrency-test
hexo init
npm install 'ppoffice/hexo#master'
git clone https://github.com/SukkaLab/hexo-many-posts.git source/_posts/tests
hexo gen
hexo clean
hexo gen -c 8
hexo clean
hexo gen --concurrency 1
hexo clean
hexo gen -c 200
@YoshinoriN YoshinoriN added this to the v4.0.0 milestone Aug 20, 2019
@YoshinoriN YoshinoriN merged commit ec99001 into hexojs:master Aug 20, 2019
3 of 4 checks passed
3 of 4 checks passed
codeclimate 1 issue to fix
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls Coverage increased (+0.004%) to 97.157%
Details
@curbengh

This comment has been minimized.

Copy link
Contributor

curbengh commented Sep 2, 2019

Related issues: #1763 #1735 #1529

@chenshiforever

This comment has been minimized.

Copy link

chenshiforever commented Sep 22, 2019

but I donn't kown what i can do ,this step2. change the hexo dependency to ppoffice/hexo#master in package.json is not understand . i think i have nothing to do except wait for it to update to 4.0.

@curbengh

This comment has been minimized.

Copy link
Contributor

curbengh commented Sep 22, 2019

That is using ppoffice's fork, you can use this repository instead.

package.json
-  "hexo": "^3.9.0",
+  "hexo": "hexojs/hexo",

Modify the line with "hexo" in your "package.json" file, change the value from "^3.9.0", to "hexojs/hexo",

Example

This is akin to using bleeding-edge release.

@chenshiforever

This comment has been minimized.

Copy link

chenshiforever commented Sep 22, 2019

But this approach doesn't seem to be of much use. It does add more than a hundred posts, but exceeding that number will crash, and here's my error message
中文版:但是这个方法对我来说似乎没有起太大的作用。确实可以增加一百多的post数量,但是超过了这个数字就会崩溃,下面是我的报错信息

INFO  1968 files generated in 6.87 min
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node_module_register
 2: v8::internal::FatalProcessOutOfMemory
 3: v8::internal::FatalProcessOutOfMemory
 4: v8::internal::Factory::NewRawTwoByteString
 5: v8::internal::Smi::SmiPrint
 6: v8::internal::StackGuard::HandleInterrupts
 7: v8::String::WriteUtf8
 8: v8_inspector::V8InspectorClient::currentTimeMS
 9: node::Buffer::New
10: node::Buffer::New
11: 00000199789F17C6
npm ERR! code ELIFECYCLE
npm ERR! errno 3
npm ERR! hexo@0.0.1 push: `hexo cl && hexo g -c 30 && gulp && gulp mv && hexo s`
npm ERR! Exit status 3
npm ERR!
npm ERR! Failed at the hexo@0.0.1 push script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

@ppoffice

This comment has been minimized.

Copy link
Contributor Author

ppoffice commented Sep 22, 2019

@chenshiforever
Please try to use a smaller concurrency value such as 10. By the way, what theme are you using? It may also have something to do with its optimization.

@dailyrandomphoto dailyrandomphoto mentioned this pull request Oct 9, 2019
2 of 2 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.