-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
mini_racer 0.12.0 Segmentation fault on x86_64-linux with libv8-node 21.7.2.0 #300
Comments
Hey @D-system. A couple of question hopefully helping to narrow the issue down
|
I don't think jemalloc is enabled: $ ruby -r rbconfig -e "puts RbConfig::CONFIG['MAINLIBS']"
-lz -lrt -lrt -ldl -lcrypt -lm -lpthread Even by using the $ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.2 ruby -r rbconfig -e "puts RbConfig::CONFIG['MAINLIBS']"
-lz -lrt -lrt -ldl -lcrypt -lm -lpthread
# Checking the directory
$ ls -l /usr/lib/x86_64-linux-gnu/libjemalloc*
-rw-r--r-- 1 root root 1119814 Jan 24 2022 /usr/lib/x86_64-linux-gnu/libjemalloc.a
-rw-r--r-- 1 root root 1119970 Jan 24 2022 /usr/lib/x86_64-linux-gnu/libjemalloc_pic.a
lrwxrwxrwx 1 root root 16 Jan 24 2022 /usr/lib/x86_64-linux-gnu/libjemalloc.so -> libjemalloc.so.2
-rw-r--r-- 1 root root 744440 Jan 24 2022 /usr/lib/x86_64-linux-gnu/libjemalloc.so.2 Let me see if I can make a new app and get the same error. I didn't realized there was a more debug output:
```
-- Ruby level backtrace information ---------------------------------------- -- Machine register context ------------------------------------------------ -- C level backtrace information -------------------------------------------
|
I made a brand new Rails 6.1.7.7 app I got the same error with those changes:
Note: it does precompile the files then seg fault.
```
$ bundle ; ./bin/rails assets:clobber assets:precompile
Bundle complete! 18 Gemfile dependencies, 84 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
I, [2024-04-23T18:59:24.147478 #10018] INFO -- : Removed /home/circleci/new_rails_app/public/assets
yarn install v1.22.19
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.42s.
I, [2024-04-23T18:59:25.739888 #10018] INFO -- : Writing /home/circleci/new_rails_app/public/assets/manifest-b4bf6e57a53c2bdb55b8998cc94cd00883793c1c37c5e5aea3ef6749b4f6d92b.js
I, [2024-04-23T18:59:25.740896 #10018] INFO -- : Writing /home/circleci/new_rails_app/public/assets/manifest-b4bf6e57a53c2bdb55b8998cc94cd00883793c1c37c5e5aea3ef6749b4f6d92b.js.gz
I, [2024-04-23T18:59:25.741133 #10018] INFO -- : Writing /home/circleci/new_rails_app/public/assets/application-04024382391bb910584145d8113cf35ef376b55d125bb4516cebeb14ce788597.css
I, [2024-04-23T18:59:25.742186 #10018] INFO -- : Writing /home/circleci/new_rails_app/public/assets/application-04024382391bb910584145d8113cf35ef376b55d125bb4516cebeb14ce788597.css.gz
./bin/rails: [BUG] Segmentation fault at 0x00007fb3f7901008
ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [x86_64-linux]
-- Control frame information ----------------------------------------------- -- Machine register context ------------------------------------------------ -- C level backtrace information ------------------------------------------- -- Other runtime information -----------------------------------------------
[...]
|
I also tried the same project with mini_racer 0.9. It works.
|
It seems sprocket is used by default if no other assets pipeline exists. The Gemfile without the comments source 'https://rubygems.org'
git_source(:github) { |repo| "https://github.com/#{repo}.git" }
ruby '3.2.3'
gem 'rails', '~> 6.1.7', '>= 6.1.7.7'
gem 'sqlite3', '~> 1.4'
gem 'puma', '~> 5.0'
gem 'sass-rails', '>= 6'
gem 'turbolinks', '~> 5'
gem 'jbuilder', '~> 2.7'
gem "bootstrap", "~> 4.3" # <- the problem
gem "mini_racer", "~> 0.9.0" # <- added
gem 'bootsnap', '>= 1.4.4', require: false
group :development, :test do
gem 'byebug', platforms: [:mri, :mingw, :x64_mingw]
end
group :development do
gem 'web-console', '>= 4.1.0'
gem 'rack-mini-profiler', '~> 2.0'
gem 'listen', '~> 3.3'
gem 'spring'
end
group :test do
gem 'capybara', '>= 3.26'
gem 'selenium-webdriver', '>= 4.0.0.rc1'
gem 'webdrivers'
end
gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby] |
@lloeki this looks urgent , should we roll out a release that puts us back at an earlier version while we work this out ? |
@D-system does this repro in Linux as well |
@SamSaffron it's all tested inside the CircleCI image linked above. So, it's an Ubuntu based linux on x86_64. |
So the idea would be to have 0.12.1 be 0.9.0? That would be odd. |
I'll try a repro. |
The official |
I'm not able to reproduce yet, can't spend much time on it right now though. The product I'm working on in my day job is using docker images based on the 🤔 |
I tried with I'm not able to reproduce the issue in a local docker either. |
In any case it appears that
I couldn't reproduce it either inside that |
@D-system any chance you can create a container that repros this consistently. Then we can |
I just wanted to add another case of this issue. I tried to update yesterday from 0.8 to 0.12 and started getting segfaults. I don't use docker or any version managers, just the system's 3.0.2p107 (the default ruby) on Ubuntu 22.04. This is on rails 7.0.8.1 with libv8 21.7.2.0. I've reverted to 0.8 and libv8 18.16.0.0 for now. I'm not using jemalloc (I have no idea what it is, and the mentioned libraries are not on my system). If you need any info or need anything tested let me know, but I don't want to figure out how to build or use docker images (have no interest in that). |
I also saw it yesterday when upgrading from 0.9 to 0.12 both on Fedora 40 with ruby 3.3.0 and also on Ubuntu 22.04 with ruby 3.0.2p107. |
Thanks for the additional reports. Is it Sprockets-related as well? |
It was using rails with sprockets yes, specifically using https://rubygems.org/gems/rtlcss in the sprockets chain which uses mini_racer to run https://www.npmjs.com/package/rtlcss on CSS files. |
I am using sprockets, but I had not checked whether it's involved or not. I first got these errors when rendering a js response to an ajax request, and also saw them when shutting down the server (running puma). |
@SamSaffron I was not able to reproduce on my local host or local docker yesterday. I will give a new try tomorrow. |
We were getting this on 0.9.0 -> 0.12.0 (production only) on Heroku on the '20' image with Ruby 3.3.0 and jemalloc and node I did see this in a prod console, though may not be related
and looking at prod crash logs
we're just using this to run embedded javascript
|
Are you using puma in clustered mode? forking is known to cause some troubles with mini_racer/v8, see https://github.com/rubyjs/mini_racer#fork-safety. |
I am indeed using puma in clustered mode, but it was working fine on 0.8.0, and starting puma in single mode gives me the same segfault in the same place. I have the suggested fork safety line in an initializer, and removing this makes no difference in any scenario. |
I'm still not able to reproduce it in a local Docker even when I use the Docker image of the CI. I continue to investigate on the CI directly. SSH and vi for the win! Anyway, I track down the issue. On a brand new Rails 6.1 app:
I don't know what else I could search. |
@D-system I wish I had more to add here, but so much depends on a standalone repro. In theory most people use mini_racer in a very simple way, if you can get a crash then you can monkey patch eval to save what it is evaling into a file and then we can use that to cause the crash. Once we have one we will very likely be able to sort this out quickly There is a chance here as well that libraries were using mini racer in a fork unsafe way and somehow got away with it in earlier versions due to luck. |
The same happened to me after upgrading to the latest libv8-node (21.7.2.0) and mini_racer (0.12.0) during running specs
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-linux] |
After some lengthy monkey patching session, I'm 99% sure it's due to this file: https://github.com/ai/autoprefixer-rails/blob/master/vendor/autoprefixer.js The file is 6.29MB which isn't the simplest to debug. |
Oh wow, does an eval of this file cause the crash?
…On Wed, 1 May 2024 at 6:28 PM, Thomas ***@***.***> wrote:
After some lengthy monkey patching session, I'm 99% sure it's due to this
file:
https://github.com/ai/autoprefixer-rails/blob/master/vendor/autoprefixer.js
I removed the source map at the last line, it does not change anything.
The file is 6.29MB which isn't the simplest to debug.
—
Reply to this email directly, view it on GitHub
<#300 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAABIXPXZDI2KX724Z3GTFTZACRTFAVCNFSM6AAAAABGUEE3QSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBYGE2DQNZUHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
From Chrome 92 (and Firefox too), https://web.dev/articles/cross-origin-isolation-guide https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer |
Thanks for diving this deep into it @D-system, IIUC this still not reproduced locally, only in CircleCI? |
I think what Sam meant is mimicking https://github.com/ai/autoprefixer-rails/blob/cf0af2807108e43f4858574b5d22dfaec0630d80/lib/autoprefixer-rails/processor.rb#L133C9-L133C23 i.e does the following cause a crash?
If that's not enough, maybe it can be reproduced by mimicking https://github.com/ai/autoprefixer-rails/blob/cf0af2807108e43f4858574b5d22dfaec0630d80/lib/autoprefixer-rails/processor.rb#L35 with a specific input file?
Notes:
|
I don't know if it's the same bug, but recently I added a new plugin to the Gemfile of a template repository, which caused the Docker image to be rebuilt with the latest versions of the libraries. The build failed in GitHub actions, with the message:
|
@george-gca entirely unrelated. see rubyjs/libv8-node#36 (comment) |
Thank you for dealing.
|
I don't have time to investigate further at the moment. I'll be AFK from tomorrow until after the RubyKaigi (2024-05-15 to 17). |
I'm back. After adding (the Then I copied the The error certainly happens after the file is loaded and when the CSS is being process. Another time, I'll look at how https://github.com/ai/autoprefixer-rails works to debug. |
Interesting, @D-system. Would be great to have a working reproduction. Maybe all that's left is some CSS, if I'm reading your findings correctly. Thank you for your efforts! |
I'm using a brand new app. The CSS is non existent. Just the /*
* This is a manifest file that'll be compiled into application.css, which will include all the files
* listed below.
*
* Any CSS and SCSS file within this directory, lib/assets/stylesheets, or any plugin's
* vendor/assets/stylesheets directory can be referenced here using a relative path.
*
* You're free to add application-wide styles to this file and they'll appear at the bottom of the
* compiled file so the styles you add here take precedence over styles defined in any other CSS/SCSS
* files in this directory. Styles in this file should be added after the last require_* statement.
* It is generally better to create a new file per style scope.
*
*= require_tree .
*= require_self
*/ I also would love to have the bug reproduced on my local. I'm still SSH-ing to CircleCI. Here's the command that I use to reproduce the bug on CircleCI: gem install rails -v 6.1.7.7
rails new --skip-webpack-install app
cd app
echo "gem 'mini_racer', '~> 0.12.0'" >> Gemfile
echo 'gem "autoprefixer-rails", "10.4.16.0"' >> Gemfile
bundle install
./bin/rails assets:clobber assets:precompile |
This does reproduce it in docker:
I looked at the files and there is nothing special about them, the backtrace for the segfault is on cleanup. specifically miniracer calling deallocate from the finalize_list in ruby and @lloeki is this enough for you to debug, or do you need more help here? One possibility here is that somehow autoprefixer is forking processes, I looked at the code and it does not look like it. The crash looks like is is from if (isolate_info) { |
@cataphract any ideas about ⏫ ? We tried to fix a premature cleanup previously. |
jFYI: The example does not crash under |
In our app we encountered segfaults with mini_racer and turf.js when doing the bump from 0.9.0 to 0.12.0. It didn't occur in our CI but it did in our prod instance. CI is running on buildjet ubuntu 22.04 with ruby 3.2.2 and prod is running on heroku on ubuntu 20.04 with ruby 3.2.2 and YJIT and jemalloc enabled |
This is most probably jemalloc-related, which is known to trigger core dumps unrelated to this very one. |
@SamSaffron if I manage to reliably reproduce it, possibly so. |
Sadly, no repro (neither on M1 hardware nor on Intel hardware) :/ |
If this is any help, @SamSaffron's repro did work for me and I traced the segfault a bit further down to this line:
I forced libv8-node down to 20.12.1.0 and the issue went away, so perhaps it's possible that it's either a bug in 21.7.2.0 or a breaking change requiring an update? Edit: for those who don't know (like me before yesterday!) that |
Can confirm a downgrade of libv8-node fixed the issue. |
I'm having the exact same issues and have three crash reports I can share if that will help. Ruby 3.3.2 on heroku with Rails 6.1.7.7. Segmentation fault is always here:
I can provide more if needed. |
@huda-kh what did you downgrade libv8-node to and did you do that by adding it directly to your gemfile? |
I can also confirm that downgrading to |
Yes, |
Hello the team
Thank you for the update to libv8-node 21.7.2.0 yesterday.
I wanted to report an error.
Setup:
RAILS_ENV=production SECRET_KEY_BASE=1 bin/rails assets:clobber assets:precompile
It does work on Darwin Kernel Version 23.2.0 arm64 (Mac on arm) that compile from source.
The error is from eval -> eval_unsafe, so I suspect it is an issue with the
libv8-node
mini_racer/lib/mini_racer.rb
Line 228 in 5b2a25c
The text was updated successfully, but these errors were encountered: