Nokogiri 1.8.4 installation challenges on macOS Mojave

[deepj]

If you're having trouble installing Nokogiri ...

Have you tried following [the installation tutorial][tutorial]?
yes

What is the output of gem install?

Building native extensions with: '--use-system-libraries --with-xslt-dir=/usr/local/opt/libxslt --with-xslt-dir=/usr/local/opt/libxml2'
This could take a while...
ERROR:  Error installing nokogiri:
	ERROR: Failed to build gem native extension.

    current directory: ~/.gem/ruby/2.5.1/gems/nokogiri-1.8.4/ext/nokogiri
~/.rubies/ruby-2.5.1/bin/ruby -r ./siteconf20180926-34637-1628dkt.rb extconf.rb --use-system-libraries --with-xslt-dir=/usr/local/opt/libxslt --with-xslt-dir=/usr/local/opt/libxml2
checking if the C compiler accepts ... yes
checking if the C compiler accepts -Wno-error=unused-command-line-argument-hard-error-in-future... no
Building nokogiri using system libraries.
ERROR: cannot discover where libxml2 is located on your system. please make sure `pkg-config` is installed.
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=~/.rubies/ruby-2.5.1/bin/$(RUBY_BASE_NAME)
	--help
	--clean
	--use-system-libraries=true
	--with-zlib-dir
	--without-zlib-dir
	--with-zlib-include
	--without-zlib-include=${zlib-dir}/include
	--with-zlib-lib
	--without-zlib-lib=${zlib-dir}/lib
	--with-xml2-dir
	--without-xml2-dir
	--with-xml2-include
	--without-xml2-include=${xml2-dir}/include
	--with-xml2-lib
	--without-xml2-lib=${xml2-dir}/lib
	--with-libxml-2.0-config
	--without-libxml-2.0-config
	--with-pkg-config
	--without-pkg-config
	--with-xslt-dir
	--with-xslt-include
	--without-xslt-include=${xslt-dir}/include
	--with-xslt-lib
	--without-xslt-lib=${xslt-dir}/lib
	--with-exslt-dir
	--without-exslt-dir
	--with-exslt-include
	--without-exslt-include=${exslt-dir}/include
	--with-exslt-lib
	--without-exslt-lib=${exslt-dir}/lib
	--with-libexslt-config
	--without-libexslt-config

To see why this extension failed to compile, please check the mkmf.log which can be found here:

  ~/.gem/ruby/2.5.1/extensions/x86_64-darwin-18/2.5.0-static/nokogiri-1.8.4/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in ~/.gem/ruby/2.5.1/gems/nokogiri-1.8.4 for inspection.
Results logged to ~/.gem/ruby/2.5.1/extensions/x86_64-darwin-18/2.5.0-static/nokogiri-1.8.4/gem_make.out

What are the contents of the mkmf.log file?

"clang -o conftest -I~/.rubies/ruby-2.5.1/include/ruby-2.5.0/x86_64-darwin18 -I~/.rubies/ruby-2.5.1/include/ruby-2.5.0/ruby/backward -I~/.rubies/ruby-2.5.1/include/ruby-2.5.0 -I. -I~/.rubies/ruby-2.5.1/include  -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT    -O3 -Wno-error=shorten-64-to-32  -pipe  conftest.c  -L. -L~/.rubies/ruby-2.5.1/lib -L. -L~/.rubies/ruby-2.5.1/lib  -fstack-protector -L/usr/local/lib     -lruby.2.5.1-static -framework Foundation  -lpthread -lgmp -ldl -lobjc  "
checked program was:
/* begin */
1: #include "ruby.h"
2:
3: int main(int argc, char **argv)
4: {
5:   return 0;
6: }
/* end */

"clang -I~/.rubies/ruby-2.5.1/include/ruby-2.5.0/x86_64-darwin18 -I~/.rubies/ruby-2.5.1/include/ruby-2.5.0/ruby/backward -I~/.rubies/ruby-2.5.1/include/ruby-2.5.0 -I. -I~/.rubies/ruby-2.5.1/include  -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT    -O3 -Wno-error=shorten-64-to-32  -pipe    -Werror -c conftest.c"
checked program was:
/* begin */
1: #include "ruby.h"
2:
3: int main() {return 0;}
/* end */

"clang -I~/.rubies/ruby-2.5.1/include/ruby-2.5.0/x86_64-darwin18 -I~/.rubies/ruby-2.5.1/include/ruby-2.5.0/ruby/backward -I~/.rubies/ruby-2.5.1/include/ruby-2.5.0 -I. -I~/.rubies/ruby-2.5.1/include  -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT    -O3 -Wno-error=shorten-64-to-32  -pipe  -Wno-error=unused-command-line-argument-hard-error-in-future   -Werror -c conftest.c"
error: unknown warning option '-Werror=unused-command-line-argument-hard-error-in-future'; did you mean '-Werror=unused-command-line-argument'? [-Werror,-Wunknown-warning-option]
checked program was:
/* begin */
1: #include "ruby.h"
2:
3: int main() {return 0;}
/* end */

"pkg-config --exists libxml-2.0"
| pkg-config --libs libxml-2.0
=> "-lxml2\n"
"clang -o conftest -I~/.rubies/ruby-2.5.1/include/ruby-2.5.0/x86_64-darwin18 -I~/.rubies/ruby-2.5.1/include/ruby-2.5.0/ruby/backward -I~/.rubies/ruby-2.5.1/include/ruby-2.5.0 -I. -I~/.rubies/ruby-2.5.1/include  -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT    -O3 -Wno-error=shorten-64-to-32  -pipe  conftest.c  -L. -L~/.rubies/ruby-2.5.1/lib -L. -L~/.rubies/ruby-2.5.1/lib  -fstack-protector -L/usr/local/lib     -lruby.2.5.1-static -framework Foundation -lxml2 -lpthread -lgmp -ldl -lobjc  "
checked program was:
/* begin */
1: #include "ruby.h"
2:
3: int main(int argc, char **argv)
4: {
5:   return 0;
6: }
/* end */

| pkg-config --cflags-only-I libxml-2.0
=> "-I/usr/include/libxml2\n"
| pkg-config --cflags-only-other libxml-2.0
=> "\n"
| pkg-config --libs-only-l libxml-2.0
=> "-lxml2\n"
package configuration for libxml-2.0
cflags:
ldflags:
libs: -lxml2

"pkg-config --exists libexslt"
| pkg-config --libs libexslt
=> "-lexslt -lxslt -lxml2 -lz -lpthread -licucore -lm -lxml2\n"
"clang -o conftest -I~/.rubies/ruby-2.5.1/include/ruby-2.5.0/x86_64-darwin18 -I~/.rubies/ruby-2.5.1/include/ruby-2.5.0/ruby/backward -I~/.rubies/ruby-2.5.1/include/ruby-2.5.0 -I. -I/usr/include/libxml2 -I/usr/local/opt/libxml2/include -I~/.rubies/ruby-2.5.1/include  -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT    -O3 -Wno-error=shorten-64-to-32  -pipe   conftest.c  -L. -L~/.rubies/ruby-2.5.1/lib -L/usr/local/opt/libxml2/lib -L. -L~/.rubies/ruby-2.5.1/lib  -fstack-protector -L/usr/local/lib      -lxml2 -lruby.2.5.1-static -framework Foundation -lexslt -lxslt -lxml2 -lz -lpthread -licucore -lm -lxml2 -lpthread -lgmp -ldl -lobjc  "
checked program was:
/* begin */
1: #include "ruby.h"
2:
3: int main(int argc, char **argv)
4: {
5:   return 0;
6: }
/* end */

| pkg-config --cflags-only-I libexslt
=> "-I/usr/include/libxml2\n"
| pkg-config --cflags-only-other libexslt
=> "\n"
| pkg-config --libs-only-l libexslt
=> "-lexslt -lxslt -lxml2 -lz -lpthread -licucore -lm -lxml2\n"
package configuration for libexslt
cflags:
ldflags:
libs: -lexslt -lxslt -lxml2 -lz -lpthread -licucore -lm -lxml2

"clang -E -I~/.rubies/ruby-2.5.1/include/ruby-2.5.0/x86_64-darwin18 -I~/.rubies/ruby-2.5.1/include/ruby-2.5.0/ruby/backward -I~/.rubies/ruby-2.5.1/include/ruby-2.5.0 -I. -I/usr/include/libxml2 -I/usr/include/libxml2 -I/usr/local/opt/libxml2/include -I~/.rubies/ruby-2.5.1/include  -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT    -O3 -Wno-error=shorten-64-to-32  -pipe     conftest.c -o conftest.i"
conftest.c:3:10: fatal error: 'libxml/xmlversion.h' file not found
#include <libxml/xmlversion.h>
         ^~~~~~~~~~~~~~~~~~~~~
1 error generated.
checked program was:
/* begin */
1: #include "ruby.h"
2:
3: #include <libxml/xmlversion.h>
/* end */

What operating system are you using?

macOS 10.14 Mojave

I've tried to reinstall everything what I could but without any success. I used pkg-config, libxml2 and libxslt from Homebrew...

[deepj]

I was successful with gem install nokogiri -- --use-system-libraries=true --with-xml2-include="$(xcrun --show-sdk-path)"/usr/include/libxml2

But not with libxml2 from Homebrew.

[flavorjones]

I'm sorry y'all are having trouble.

If someone who is a Mac user would care to verify that this works and submit a PR to the installation tutorial, I would be very grateful. Alternatively, if there's something we can change in the extconf, I'd appreciate a PR.

Also tagging @zenspider as he's been kind enough to offer assistance on behalf of Mac users in the past.

[zenspider]

I don't think the original report makes sense to me (reformatted):

Building native extensions with: '--use-system-libraries \
      --with-xslt-dir=/usr/local/opt/libxslt \
      --with-xslt-dir=/usr/local/opt/libxml2'

This says "build and link against system libraries, but look for their configs in these completely different builds". I would expect that to break. See this for more details:

http://www.nokogiri.org/tutorials/installing_nokogiri.html#using-your-system-libraries

[deepj]

@zenspider I put one of many iterations made by me here. The problem is still same if I just gem install nokogiri. It's not possible to install Nokogiri on Mojave without additional parameters.

You're right. There is a mistake in the command --with-xslt-dir=/usr/local/opt/libxml2' => --with-xml2-dir=/usr/local/opt/libxml2'
But the result is the same even I use the right form. Installation of Nokogiri failed on Mojave.

Only this worked

gem install nokogiri -- --use-system-libraries=true --with-xml2-include="$(xcrun --show-sdk-path)"/usr/include/libxml2

The expectation is I need only run gem install nokogiri on Mojave. There was no problems on the previous High Sierra.

[deepj]

OK. I found the reason why it's happening on Mojave

See https://developer.apple.com/documentation/xcode_release_notes/xcode_10_release_notes#3035624

It's needed to make another step and install this as a temporary workaround on Mojave:

open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

For explanation see the Xcode release notes

[flavorjones]

@deepj Thank you for following up on the thread. Would you be so kind as to submit a PR to this file that includes steps other Mojave users can follow?

https://github.com/sparklemotion/nokogiri.org-tutorials/blob/master/content/installing_nokogiri.md

I unfortunately don't have a Mac and so can't verify the steps that I might write up are correct. I'd really appreciate your help!

[deepj]

@flavorjones I'll do. But what I got from the Xcode release notes, it's needed to change a way for looking headers on macOS 10.14 and later. Can @zenspider help here, please? I don't have no insight how these compilation stuffs work.

[dhh]

I just hit this same problem on a machine upgraded to 10.14.

[wsizoo]

Updating command line tools for Xcode fixed this for me.

Command Line Tools for Xcode 10.14

[flavorjones]

Again, an open request for Mac users to help support the Ruby community: Can someone please submit a PR to the installation tutorial explaining how to update xcode tooling on 10.14?

[cjxonix]

I was successful with gem install nokogiri -- --use-system-libraries=true --with-xml2-include="$(xcrun --show-sdk-path)"/usr/include/libxml2

But not with libxml2 from Homebrew.

This actually worked for on Mojav Mac OS X

[flavorjones]

Once more, this time with gusto: can a Mac user on this issue please help support the Ruby community by submitting a PR to the installation tutorial explaining how to update xcode tooling on 10.14?

[deepj]

@flavorjones the xcode command line tools is updated via app store. @wsizoo note seems to be confusing here. Just read the xcode release notes https://developer.apple.com/documentation/xcode_release_notes/xcode_10_release_notes#3035624

My understanding is that nokogiri needs to change a way where is looking for the headers on Mojave 10.14 with new xcode command line tools 10.x. As Apple noted in the release notes they offer a temporary workaround to install the headers to the old place. This won’t be possible later. Apple recommends to change the place where looking for the headers to new one.

So updating xcode command line tools doesn’t help either. That is my understanding the situation. I may be wrong here:

[grega]

I managed to install it on 10.14 by running:

xcode-select --install
sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -target /

Followed by:

sudo mv /usr/local/include /usr/local/include_old
gem install nokogiri

Happy to submit a doc update PR if this solution appears to help others.

[deepj]

@grega the problem with this, it won't work in the future. It's just a temporary workaround offered by Apple. Just read this https://developer.apple.com/documentation/xcode_release_notes/xcode_10_release_notes#3035624

[flavorjones]

Ok, I better understand the underlying challenge, which is to look for header files and libraries in the right place with the new XCode SDK. Thanks for that.

This is still not something I'm personally equipped to handle, so I'd like to ask that someone who cares about Mac support help me either by submitting a PR or working with me to figure out what an appropriate solution looks like. I don't own a Mac, and and in addition this is something I'd expect someone from the community can help with or own entirely.

[flavorjones]

OK, maybe if I provide a bit more information on how I'd address this if I had access to a Mac running Mojave, somebody can verify that the approach would work.

extconf.rb is a pretty big file at this point, but if someone can find the section, at line 395, which hacks CFLAGS for OpenBSD, we can try something similar for Mojave:

if darwin?
  ENV['CFLAGS'] = "#{ENV['CFLAGS']} -I /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk"
end

Is anyone here able to try patching their extconf.rb in this manner and see if it addresses the underlying problem?

[markoa]

I was able to install Nokogiri on Mojave after running only xcode-select --install.

[bacchir]

For people using homebrew the documentation around macOS and homebrew install helped me figure out the command bellow which worked for me:

gem install nokogiri -- --use-system-libraries --with-iconv-dir=$(brew --prefix libiconv) --with-xml2-include=$(brew --prefix libxml2)/include/libxml2

[flavorjones]

Hey all,

I my comments above I suggested a potential fix, provided a patch, and asked if any macOS users would try it out and let me know if it resolves the underlying issues.

This comment is asking one last time if anyone in Ruby community who's a macOS user can try this three-line change out on their local machine and let me know if it addresses the underlying issue.

If somebody can try it out and respond, I'll provide an emphatic "thank you" in the changelog for your help. If nobody responds, I'm going to assume it's no longer an issue.

[grega]

@flavorjones Afraid I haven't had any luck getting my Mac back to a state in which Nokogiri was failing to install, so I have been unable to test the suggested fix in extconf.rb 😞

[bbonamin]

Hey @flavorjones, I want to let you know that 1.8.5 installs automatically in Mojave computers with the latest updates and without needing to do any changes to nokogiri (or your suggested change to extconf.rb)

[AndreiRailean]

@bbonamin you're probably talking about a clean install where there was no nokogiri in the past. I think half the people come here because it breaks after os or brew upgrade. Definitely was the case for me.

After bashing my head against it for a few hours it turned out that I had a stale libxml path in bundle config. I ended up using bundle config build.nokogiri --use-system-libraries --with-xml2-include=/usr/include/libxml2. previously --with-xml2-include was set to something deep inside an Xcode SDK which was removed and blocked the build.

[kenkuts]

I managed to install it on 10.14 by running:

xcode-select --install
sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -target /

Followed by:

sudo mv /usr/local/include /usr/local/include_old
gem install nokogiri

Happy to submit a doc update PR if this solution appears to help others.

I am so happy this solved my problem I've been trying to fix this for the past few days. Y'all need to fix these installation issues.

[flavorjones]

@kenkuts Did you happen to read any of my comments above where I'm trying to get someone to confirm that a one-line change in extconf.rb works?

Once again: any Mac users arriving at this issue, STOP trying to hack it. Read this comment above:

#1801 (comment)

Please clone the repository, make this small change to extconf.rb, run rake compile and respond if it completes successfully with the change, and fails without the change.

I'm about to cut v1.9.0 and to date nobody who is using a Mac has been able to tell me if that fix works, and so it's not going to be in this release. It won't be in a release until someone can confirm that it works.

Snarky comments will not get the job done; testing a suggested fix will.

[deepj]

@flavorjones OK, I figured out why this happens to someone of us. Someone of us has something like export NOKOGIRI_USE_SYSTEM_LIBRARIES=true in their shell profiles.

I've tried the patch. And that didn't help at all. libxml2, libxslt and pkg-config is installed via Homebrew.

[duranmla]

In my case, the problem was more related to https://stackoverflow.com/questions/14607193/installing-gem-or-updating-rubygems-fails-with-permissions-error

Basically, when I run the bundle install command it wasn't able to "understand" that I had rbenv

Once I make sure that after run which gem I got /Users/justin/.rbenv/shims/gem everything runs smoothly

[dmzza]

Updating to the latest version of Nokogiri 1.9.0 solved the problem for me

git install nokogiri

[rajivshah3]

OK, maybe if I provide a bit more information on how I'd address this if I had access to a Mac running Mojave, somebody can verify that the approach would work.

extconf.rb is a pretty big file at this point, but if someone can find the section, at line 395, which hacks CFLAGS for OpenBSD, we can try something similar for Mojave:

if darwin?
  ENV['CFLAGS'] = "#{ENV['CFLAGS']} -I /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk"
end

Is anyone here able to try patching their extconf.rb in this manner and see if it addresses the underlying problem?

Hey @flavorjones , this seems to work for me (I used ENV['CFLAGS'] = "#{ENV['CFLAGS']} -I /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk" instead though because that's where my SDK is). However, it also seems to compile successfully without this change. I'm using macOS Mojave 10.14.2 Beta (18C52a) with Xcode 10.1 (10B61). Please let me know if you'd like me to try anything else.

Edit: Nokogiri 1.9.1 now works for me when I install it from a Gemfile and use bundle install, but I'm not sure if that's because I reinstalled rvm and bundler

[zenspider]

@rajivshah3 thank you. That last one is bundler... And I thought it was fixed. I haven't seen it for a little while. You might want to update.

[Onkar09]

I was successful with gem install nokogiri -- --use-system-libraries=true --with-xml2-include="$(xcrun --show-sdk-path)"/usr/include/libxml2

But not with libxml2 from Homebrew.

This worked for me. Tried almost all the suggestions available on google, stackoverflow, github but failed. This is like life saver for me.

[flavorjones]

@deepj Thanks for your reply -- there's a lot of noise in this thread and you're providing some clarity.

Can you provide your mkmf.log file from a failed bundle exec rake compile in which you had the suggested patch applied and you saw the "cannot discover where libxml2 is located on your system" error message? It would also be helpful to provide all of the commands you're running so I have that context as well.

[deepj]

@flavorjones Hello, I'll look at it this weekend and then I let you know. I'm sorry for the late answer, holidays here.

[flavorjones]

Please note that @mlj confirmed that the fix suggested in #1801 (comment) works for him, and has submitted the PR in #1851

[flavorjones]

I've merged #1851 into master. I'm considering cutting a v1.10.1 release with this change. Will leave this open until a version is shipped.

[FranklinYu]

@Onkar09 Could you please provide more context?

1. Just to make sure, simply gem install nokogiri didn’t work for you, right? This is the most recommended way.
2. Did you try the second recommended way, Install with system libraries?

[FranklinYu]

@flavorjones I think most of the “solutions” mentioned above are already covered in the documentation; others just cannot reproduce, so we can close this issue. I don’t think your patch is necessary, because it should be covered by the “Install with system libraries” chapter. The pkg-config from Homebrew should be able to find the libraries provided by Apple:

https://github.com/Homebrew/brew/tree/master/Library/Homebrew/os/mac/pkgconfig

Ok I read that macOS Mojave is not adding /usr/include. I will discuss with Homebrew team to look for a solution. This is out of scope for Nokogiri.

[flavorjones]

@FranklinYu A fix has already been commited to master - please see my comment above.