macOS 13.x Ventura, daemon crashes on start

[paulatbipsync]

When starting the daemon on macOS 13.0 Beta Ventura the daemon appears to crash on startup,

The following log lines are output:

objc[15801]: +[NSPlaceholderMutableString initialize] may have been in progress in another thread when fork() was called. objc[15801]: +[NSPlaceholderMutableString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

Running the service in the foreground with:
vagrant dns --start --ontop
Runs fine as a workaround, no errors logged.

[fnordfish]

Interesting. Might be an issue in one of our dependencies or ruby.
Which version of vagrant and ruby (/opt/vagrant/embedded/bin/ruby -v) are you running.
Please try to re-install the plugin - vagrant plugin uninstall vagrant-dns, vagrant plugin install vagrant-dns and attach the log of the "install" command. This gives us a list of which dependencies and versions got installed.

[thegcat]

This seems to be something happening every now and then with stuff that usually works well under linux but doesn't like macOS in that regard. If it's the same issue it isn't limited to macOS 13.0 either.

See ansible/ansible#49207 for example for details, but the gist is trying to export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES in your shell before you execute the crashing commands.

[paulatbipsync]

Interesting. Might be an issue in one of our dependencies or ruby.
Which version of vagrant and ruby (/opt/vagrant/embedded/bin/ruby -v) are you running.
Please try to re-install the plugin - vagrant plugin uninstall vagrant-dns, vagrant plugin install vagrant-dns and attach the log of the "install" command. This gives us a list of which dependencies and versions got installed.

Yes it looks like a dependency issue,

My ruby version:
ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-darwin19]

I've attached the install log output. Issues persists after reinstall.

vagrant-plugin-install-vagrant-dns.log

[paulatbipsync]

This seems to be something happening every now and then with stuff that usually works well under linux but doesn't like macOS in that regard. If it's the same issue it isn't limited to macOS 13.0 either.

See ansible/ansible#49207 for example for details, but the gist is trying to export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES in your shell before you execute the crashing commands.

Thank you! This workaround prevented the issue.

[fnordfish]

A bit weird though, that Python and Ruby (I even found a Nix issue, mentioning the same env var) all sharing the same issue.

[fredngo]

Just updated to Ventura and found this problem... so no longer a Ventura Beta problem. 😢

[mpdude]

Any chance Vagrant or this plugin or ... could take care of exporting that env var?

I don't know what OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES does, but it does not sound as if it were a smart choice to set that in your .profile, right?

[janopae]

From https://stackoverflow.com/questions/73638290/python-on-mac-is-it-safe-to-set-objc-disable-initialize-fork-safety-yes-globall:

Apple changed the way fork() behaves in High Sierra (>= 10.13).

If enabled, the OBJC_DISABLE_INITIALIZE_FORK_SAFETY variable turns off the immediate crash behaviour that their newer ObjectiveC framework otherwise enforces by default, as part of this change.

Your question "is it safe to set it as a global environment variable" depends on your definition of "safe" in this context.

It's safe in the sense your computer won't burst into flames.

It's unsafe in the sense it may mask crash information that would be otherwise presented by an app that goes awry, and may allow a fork-bomb type process to crash your computer.

So if you only have one use case where setting the flag is strictly necessary, then it's best to localise its setting to just that script/scenario.

I'm not sure if this means that OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES creates the same behaviour for fork() as it had before the change?

If yes: my computer indeed did not burst into flames using older MacOS versions (and I had no fork-bomb type processes crashing my computer either). So I consider this a workaround?

According to https://stackoverflow.com/a/46593856/20727119, this issue should have been fixed in Ruby 2.4.4 and higher, but I have Ruby 2.6 and the problem still occurs.

This highly-upvoted StackOverflow reply even recommends putting the environment variable in your .profile. However, only because it is highly upvoted doesn't mean that the author knows what they're doing. I certainly still don't ¯_(ツ)_/¯

[fredngo]

If the behavior of fork() changed in High Sierra, I wonder why this problem is appearing only now in Ventura?

[mpdude]

Might be a different version of Ruby that is bundled with Ventura?

[fnordfish]

AFAIK, Vagrant ships it's own Ruby.

[fnordfish]

Finally updated to Ventura. Tested in development with a fresh install of ruby 2.7.6 (ruby-install ruby 2.7.6 -- --enable-shared) and that one works. So I believe the ruby Vagrants ships is not good on Ventura:

/opt/vagrant/embedded/bin/ruby -v
ruby 2.7.6p219 (2022-04-12 revision c9c2245c0a) [x86_64-darwin19]

versus

ruby -v
ruby 2.7.6p219 (2022-04-12 revision c9c2245c0a) [x86_64-darwin22]

[mpdude]

@fnordfish wow, thank you for the effort and investigation! 😍

Do you think it would make sense to @-mention folks from hashicorp/vagrant and/or raise an issue over there?

[fnordfish]

It's all pretty weird. It also seems to have something to do with XCode 14.

Here's a small sample script that runs just fine with a fresh install of ruby 2.7.7:

gem install daemons

# try_fork_daemons.rb
require 'daemons'
opts = {
  ARGV: ARGV,
  dir_mode: :normal,
  dir: __dir__,
  log_output: true,
  log_dir: __dir__,
  backtrace: true,
  multiple: false
}

puts "script start"
Daemons.run_proc("try_fork_daemons", opts) do
  begin
    require "not-there"
  rescue LoadError
    puts "Ignore LoadError"
  end
  puts "new pid #{Process.pid}"
  loop { sleep 1 }
end

$ ruby -v try_fork_daemons.rb start
ruby 2.7.7p221 (2022-11-24 revision 36cadad643) [x86_64-darwin22]
script start
try_fork_daemons: process with pid 73355 started.

$ ruby -v try_fork_daemons.rb status
ruby 2.7.7p221 (2022-11-24 revision 36cadad643) [x86_64-darwin22]
script start
try_fork_daemons: running [pid 73355]

Runing with Vagrants Ruby

Using the daemons gem installed in vagrants gemdir (as a dependency of vagrant-dns):

$LOAD_PATH.unshift(File.join(ENV["HOME"], ".vagrant.d/gems/2.7.6/gems/daemons-1.4.1/lib"))

# ... the rest of the original try_fork_daemons.rb

$ GEMRC=/opt/vagrant/embedded/etc/gemrc GEM_HOME=/opt/vagrant/embedded/gems/2.3.4 RUBYOPT= GEM_PATH=/opt/vagrant/embedded/gems/2.3.4 RUBYLIB= /opt/vagrant/embedded/bin/ruby try_fork_daemons.rb start
script start
try_fork_daemons: process with pid 74269 started.

No luck:

$ ps -p 74269
  PID TTY           TIME CMD

Using inline bundler:

require 'bundler/inline'
gemfile(true) do
  source 'https://rubygems.org'
  gem 'daemons'
end

# ... the rest of the original try_fork_daemons.rb

$ GEMRC=/opt/vagrant/embedded/etc/gemrc GEM_HOME=/opt/vagrant/embedded/gems/2.3.4 RUBYOPT= GEM_PATH=/opt/vagrant/embedded/gems/2.3.4 RUBYLIB= /opt/vagrant/embedded/bin/ruby try_fork_daemons.rb start
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Using bundler 2.1.4
Using daemons 1.4.1
Following files may not be writable, so sudo is needed:
  /opt/vagrant/embedded/gems/2.3.4
  /opt/vagrant/embedded/gems/2.3.4/bin
  /opt/vagrant/embedded/gems/2.3.4/bin
  /opt/vagrant/embedded/gems/2.3.4/build_info
  /opt/vagrant/embedded/gems/2.3.4/cache
  /opt/vagrant/embedded/gems/2.3.4/doc
  /opt/vagrant/embedded/gems/2.3.4/extensions
  /opt/vagrant/embedded/gems/2.3.4/gems
  /opt/vagrant/embedded/gems/2.3.4/specifications
script start
try_fork_daemons: process with pid 74530 started.

This works:

$ ps -p 74530
  PID TTY           TIME CMD
74530 ??         0:00.05 try_fork_daemons

---

I've no idea what's really going on and what's the best approach to actually fixing it.
I still believe, that it would be nice if Vagrant could update ruby to 2.7.7 (see https://bugs.ruby-lang.org/issues/19005 and https://bugs.ruby-lang.org/issues/18912) and maybe even ship a different ruby for Ventura/XCode 14?

There's one hot-fix we could do in vagrant-dns:

In https://github.com/BerlinVagrant/vagrant-dns/blob/master/lib/vagrant-dns/service.rb#L30-L32, when we move the require statements before the daemonized block, so that everything is loaded in the parent process already.

@mpdude Yes, I was working on an issue but wanted to get a testable script first.

[fnordfish]

I'm a bit hesitant to release it yet, but you might want to give #73 a try.

[mpdude]

How could we best try this?

/cc @janopae

[fnordfish]

The quick and dirty way would be to directly apply the patch:
curl -s https://patch-diff.githubusercontent.com/raw/BerlinVagrant/vagrant-dns/pull/73.patch | (cd ~/.vagrant.d/gems/2.7.6/gems/vagrant-dns-2.2.2; git apply)

Or, you could clone this repo, checkout the 72-workaround-darwin22-crash branch, and build it (requires a working ruby 2.7 installation):

$ bundle install
$ bundle exec rake build
$ vagrant plugin install ./pkg/vagrant-dns-2.2.3.pre.dev1.gem

[fnordfish]

.. or download it from here: https://dotless-downloads.s3.amazonaws.com/vagrant-dns-2.2.3.pre.dev1.gem

[janopae]

Thanks a lot, @fnordfish, I can confirm that this fixes the problem on my machine, and it didn't cause any problems yet. 👍

[fnordfish]

Merged the work-around and released as 2.2.3

[fnordfish]

vagrant-dns 2.2.2 works on Vagrant 2.3.5.dev