Running bundle install --jobs 4 --retry 3 intermittently throws [ruby] WARNING OutOfMemoryError
#2127
Comments
|
The truffleruby/src/main/c/zlib/zlib.c Lines 2409 to 2411 in d84da65 and truffleruby/src/main/c/cext/exception.c Lines 79 to 91 in d84da65 and truffleruby/lib/truffle/truffle/cext.rb Line 1503 in d84da65 Probably we should filter out |
|
@eregon - I have no idea, just the messenger... Thanks. |
|
Yeah I know, just making notes of what I found and how to help fix this :) |
|
The actual I'd like to close this issue, but feel free to reopen or comment if an OutOfMemoryError happens again, the |
|
Thanks. Oddly, I just saw the error in nio4r CI yesterday. I knew you'd get to it sometime. If I see it again, I'll reopen. Question - would you suggest running CI with truffleruby-head as a normal job (not 'allow-failure')? The question is somewhat clouded since 'allow-failure' doesn't really work in Actions... |
|
I saw it in https://github.com/ruby/setup-ruby/runs/1373979628?check_suite_focus=true And also https://github.com/ruby/setup-ruby/runs/1379242481 https://github.com/ruby/setup-ruby/runs/1380279025?check_suite_focus=true https://github.com/eregon/setup-ruby-test/runs/1399749440?check_suite_focus=true https://github.com/eregon/setup-ruby-test/runs/1399903396?check_suite_focus=true |
|
It was quite tricky to reproduce, because this OOM actually happens when downloading the full index with all gem versions, and that gets cached under truffleruby/lib/mri/bundler/compact_index_client/updater.rb Lines 56 to 58 in 69dc4f1 So something in that seems particularly inefficient in memory footprint. This reproduces it reliably: # Gemfile
source 'https://rubygems.org'
gem "rake"
gem "path"
gem "json", "2.2.0"# once is enough
bundle config --local path vendor/bundle
rm -rf Gemfile.lock vendor/bundle ~/.bundle
TRUFFLERUBYOPT="--vm.Xmx1g" DEBUG=true bundle install --jobs 4With |
|
Simpler reproduction: require 'net/http'
# https://index.rubygems.org/versions
Net::HTTP.start('index.rubygems.org', 443, use_ssl: true) do |http|
headers = {"Accept-Encoding"=>"gzip"}
request = Net::HTTP::Get.new "/versions", headers
response = http.request request
content = response.body
p content.encoding
p content.bytesize
raise unless response["Content-Encoding"] == "gzip"
content = Zlib::GzipReader.new(StringIO.new(content)).read
p content.bytesize
end |
|
@norswap found out that this results in computing many SubstringRope's of a large ConcatRope (of ConcatRopes)*. I've also found that Net::HTTP can decode gzip-encoded contents itself, and it does so in a streaming fashion which seems to consume much less memory than the naive approach of reading the whole String in memory, wrapping it in a StringIO and passing that to Zlib. That's trivial to change, just removing that line work so I'll merge that as a short-term workaround for this issue. Repro: require 'net/http'
# https://index.rubygems.org/versions
Net::HTTP.start('index.rubygems.org', 443, use_ssl: true) do |http|
headers = {} # {"Accept-Encoding"=>"gzip"}
request = Net::HTTP::Get.new "/versions", headers
response = http.request request
content = response.body
p content.encoding
p content.bytesize
end |
graalvmbot
pushed a commit
that referenced
this issue
Nov 19, 2020
…index * That is more memory efficient than reading the whole String in memory, wrapping in a StringIO and then using Zlib on it. * See #2127 (comment)
eregon
added a commit
to eregon/rubygems
that referenced
this issue
Nov 23, 2020
* Setting the Accept-Encoding header actually disables the automatic decompression as documented in Net::HTTP. https://github.com/ruby/ruby/blob/79a8ed07650dcbb36ec4b49a22596275e6c0fe23/lib/net/http/response.rb#L105-L107 https://github.com/ruby/ruby/blob/c5eb24349a4535948514fe765c3ddb0628d81004/lib/net/http/request.rb#L11-L12 * This is also more memory efficient as Net::HTTP decompresses chunk by chunk instead of having the entire compressed String in memory and also wrapped in a StringIO which is particularly inefficient. * See oracle/truffleruby#2127 (comment) for more details.
4 tasks
eregon
added a commit
to eregon/rubygems
that referenced
this issue
Dec 4, 2020
* Setting the Accept-Encoding header actually disables the automatic decompression as documented in Net::HTTP. https://github.com/ruby/ruby/blob/79a8ed07650dcbb36ec4b49a22596275e6c0fe23/lib/net/http/response.rb#L105-L107 https://github.com/ruby/ruby/blob/c5eb24349a4535948514fe765c3ddb0628d81004/lib/net/http/request.rb#L11-L12 * This is also more memory efficient as Net::HTTP decompresses chunk by chunk instead of having the entire compressed String in memory and also wrapped in a StringIO which is particularly inefficient. * See oracle/truffleruby#2127 (comment) for more details.
rbotafogo
pushed a commit
to rbotafogo/truffleruby
that referenced
this issue
Dec 8, 2020
…index * That is more memory efficient than reading the whole String in memory, wrapping in a StringIO and then using Zlib on it. * See oracle#2127 (comment)
deivid-rodriguez
pushed a commit
to eregon/rubygems
that referenced
this issue
Jan 7, 2021
* Setting the Accept-Encoding header actually disables the automatic decompression as documented in Net::HTTP. https://github.com/ruby/ruby/blob/79a8ed07650dcbb36ec4b49a22596275e6c0fe23/lib/net/http/response.rb#L105-L107 https://github.com/ruby/ruby/blob/c5eb24349a4535948514fe765c3ddb0628d81004/lib/net/http/request.rb#L11-L12 * This is also more memory efficient as Net::HTTP decompresses chunk by chunk instead of having the entire compressed String in memory and also wrapped in a StringIO which is particularly inefficient. * See oracle/truffleruby#2127 (comment) for more details.
deivid-rodriguez
pushed a commit
to eregon/rubygems
that referenced
this issue
Jan 7, 2021
* Setting the Accept-Encoding header actually disables the automatic decompression as documented in Net::HTTP. https://github.com/ruby/ruby/blob/79a8ed07650dcbb36ec4b49a22596275e6c0fe23/lib/net/http/response.rb#L105-L107 https://github.com/ruby/ruby/blob/c5eb24349a4535948514fe765c3ddb0628d81004/lib/net/http/request.rb#L11-L12 * This is also more memory efficient as Net::HTTP decompresses chunk by chunk instead of having the entire compressed String in memory and also wrapped in a StringIO which is particularly inefficient. * See oracle/truffleruby#2127 (comment) for more details.
|
This should be properly fixed now with ec62fae. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Using truffleruby-head with GitHub Actions, see:
https://github.com/MSP-Greg/puma/runs/1288146881?check_suite_focus=true#step:5:9
Maybe something with zlib? Some of the log:
Reminder - this is an intermittent error...
The text was updated successfully, but these errors were encountered: