nokogiri 1.6.8.rc3 fails to build in a fresh install of ruby 2.3 on OS X

[cbowns]

• OS X version: 10.11.3
• ruby version: ruby 2.3 installed with ruby-install to ~/.rubies, and activated with chruby
• nokogiri version: 1.6.8.rc3

Build error:

xmlIO.c:1450:52: error: use of undeclared identifier 'LZMA_OK'
    ret =  (__libxml2_xzclose((xzFile) context) == LZMA_OK ) ? 0 : -1;
                                                   ^
1 error generated.
make[2]: *** [xmlIO.lo] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
========================================================================
*** extconf.rb failed ***

Full log is attached at gem_make.out.txt

More configuration info:

I just cloned the repo which calls for nokogiri about an hour ago, and installed ruby 2.3 with ruby-install. I then installed bundler (gem install bundler), then installed my repo's gems, like nokogiri, with bundle install:

% bundle install
Fetching gem metadata from https://rubygems.org/.............
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Fetching https://github.com/dominic/jsonapi-serializers.git
Installing rake 10.5.0
Installing i18n 0.7.0
Using json 1.8.3
Installing minitest 5.8.4
Installing thread_safe 0.3.5
Installing builder 3.2.2
Installing erubis 2.7.0
Installing mini_portile2 2.1.0
Installing pkg-config 1.1.7
Installing rack 1.6.4
Installing mime-types 2.99
Installing arel 6.0.3
Installing unf_ext 0.0.7.1 with native extensions
Installing netrc 0.11.0
Installing rspec-support 3.4.1
Installing diff-lcs 1.2.5
Installing encryptor 1.3.0
Installing bcrypt 3.1.10 with native extensions
Installing debug_inspector 0.0.2 with native extensions
Installing uniform_notifier 1.9.0
Installing byebug 8.2.1 with native extensions
Installing coderay 1.1.0
Installing concurrent-ruby 1.0.0
Installing orm_adapter 0.5.0
Installing thor 0.19.1
Installing docile 1.1.5
Installing dotenv 2.1.0
Installing execjs 2.6.0
Installing multipart-post 2.0.0
Installing ffi 1.9.10 with native extensions
Installing formatador 0.2.5
Installing rb-fsevent 0.9.7
Installing lumberjack 1.0.10
Installing nenv 0.2.0
Installing shellany 0.0.1
Installing method_source 0.8.2
Installing slop 3.6.0
Installing guard-compat 1.2.1
Installing multi_json 1.11.2
Using bundler 1.11.2
Installing mimemagic 0.3.0
Installing pg 0.18.4 with native extensions
Installing puma 2.15.3 with native extensions
Installing rails_serve_static_assets 0.0.4
Installing rails_stdout_logging 0.0.4
Using rdoc 4.2.1
Installing sass 3.4.21
Installing tilt 2.0.2
Installing simplecov-html 0.10.0
Installing spring 1.6.2
Installing swagger-ui_rails 0.1.7
Installing terminal-notifier 1.6.3
Installing terminal-notifier-guard 1.7.0
Installing tzinfo 1.2.2
Installing nokogiri 1.6.8.rc3 with native extensions
…
<error above spits out>

I get the same error when I try gem install nokogiri -v '1.6.8.rc3'.

Let me know what other information I can get you!

[flavorjones]

Does 1.6.7.2 install on this same system? It would be useful to know if it's a behavior change.

If so, please include the output from 'nokogiri -v' in your reply.

[cbowns]

1.6.7.2 successfully installs:

% ± chruby
   ruby-2.2.1
   ruby-2.2.3
 * ruby-2.3.0

% ± gem install nokogiri -v '1.6.7.2'

Fetching: mini_portile2-2.0.0.gem (100%)
Successfully installed mini_portile2-2.0.0
Fetching: nokogiri-1.6.7.2.gem (100%)
Building native extensions.  This could take a while...
Successfully installed nokogiri-1.6.7.2
Parsing documentation for mini_portile2-2.0.0
Installing ri documentation for mini_portile2-2.0.0
Parsing documentation for nokogiri-1.6.7.2
Installing ri documentation for nokogiri-1.6.7.2
Done installing documentation for mini_portile2, nokogiri after 4 seconds
2 gems installed

Version info:

% ± which nokogiri

/usr/local/bin/nokogiri

% ± nokogiri -v

# Nokogiri (1.6.7.2)
    ---
    warnings: []
    nokogiri: 1.6.7.2
    ruby:
      version: 2.3.0
      platform: x86_64-darwin15
      description: ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-darwin15]
      engine: ruby
    libxml:
      binding: extension
      source: packaged
      libxml2_path: "/Users/cbowns/.gem/ruby/2.3.0/gems/nokogiri-1.6.7.2/ports/x86_64-apple-darwin15.3.0/libxml2/2.9.2"
      libxslt_path: "/Users/cbowns/.gem/ruby/2.3.0/gems/nokogiri-1.6.7.2/ports/x86_64-apple-darwin15.3.0/libxslt/1.1.28"
      libxml2_patches:
      - 0001-Revert-Missing-initialization-for-the-catalog-module.patch
      - 0002-Fix-missing-entities-after-CVE-2014-3660-fix.patch
      - 0003-Stop-parsing-on-entities-boundaries-errors.patch
      - 0004-Cleanup-conditional-section-error-handling.patch
      - 0005-CVE-2015-1819-Enforce-the-reader-to-run-in-constant-.patch
      - 0006-Another-variation-of-overflow-in-Conditional-section.patch
      - 0007-Fix-an-error-in-previous-Conditional-section-patch.patch
      - 0008-CVE-2015-8035-Fix-XZ-compression-support-loop.patch
      - 0009-Updated-config.guess.patch
      - 0010-Fix-parsering-short-unclosed-comment-uninitialized-access.patch
      - 0011-Avoid-extra-processing-of-MarkupDecl-when-EOF.patch
      - 0012-Avoid-processing-entities-after-encoding-conversion-.patch
      - 0013-CVE-2015-7497-Avoid-an-heap-buffer-overflow-in-xmlDi.patch
      - 0014-CVE-2015-5312-Another-entity-expansion-issue.patch
      - 0015-Add-xmlHaltParser-to-stop-the-parser.patch
      - 0016-Detect-incoherency-on-GROW.patch
      - 0017-CVE-2015-7500-Fix-memory-access-error-due-to-incorre.patch
      - 0018-CVE-2015-8242-Buffer-overead-with-HTML-parser-in-pus.patch
      - 0019-Do-not-print-error-context-when-there-is-none.patch
      - 0020-xmlStopParser-reset-errNo.patch
      - 0021-Reuse-xmlHaltParser-where-it-makes-sense.patch
      libxslt_patches:
      - 0001-Adding-doc-update-related-to-1.1.28.patch
      - 0002-Fix-a-couple-of-places-where-f-printf-parameters-wer.patch
      - 0003-Initialize-pseudo-random-number-generator-with-curre.patch
      - 0004-EXSLT-function-str-replace-is-broken-as-is.patch
      - 0006-Fix-str-padding-to-work-with-UTF-8-strings.patch
      - 0007-Separate-function-for-predicate-matching-in-patterns.patch
      - 0008-Fix-direct-pattern-matching.patch
      - 0009-Fix-certain-patterns-with-predicates.patch
      - 0010-Fix-handling-of-UTF-8-strings-in-EXSLT-crypto-module.patch
      - 0013-Memory-leak-in-xsltCompileIdKeyPattern-error-path.patch
      - 0014-Fix-for-bug-436589.patch
      - 0015-Fix-mkdir-for-mingw.patch
      - 0016-Fix-for-type-confusion-in-preprocessing-attributes.patch
      - 0017-Updated-config.guess.patch
      compiled: 2.9.2
      loaded: 2.9.2

[cbowns]

I think this is an issue with libxml2's 2.9.3 release. The related diff: https://fossies.org/diffs/libxml2/2.9.2_vs_2.9.3/xzlib.c-diff.html

And other users reporting issues:

• http://qiita.com/south37/items/63c938474bf35b30c988
• http://louistiao.me/posts/installing-lxml-on-mac-osx-1011-inside-a-virtualenv-with-pip/

[mojca]

I was hit by the same problem, see some comments in #1423. I filed this upstream bug report.

I wonder if nokogiri shouldn't perhaps disable some functionality, for example by passing --without-lzma. The libxml2 library version 2.9.3 now comes with a zillion of optional dependencies and I bet that many others might cause random problems if they get picked up randomly:

Optional Packages:
  --with-PACKAGE[=ARG]    use PACKAGE [ARG=yes]
  --without-PACKAGE       do not use PACKAGE (same as --with-PACKAGE=no)
  --with-pic[=PKGS]       try to use only PIC/non-PIC objects [default=use
                          both]
  --with-aix-soname=aix|svr4|both
                          shared library versioning (aka "SONAME") variant to
                          provide on AIX, [default=aix].
  --with-gnu-ld           assume the C compiler uses GNU ld [default=no]
  --with-sysroot[=DIR]    Search for dependent libraries within DIR (or the
                          compiler's sysroot if not specified).
  --with-c14n             add the Canonicalization support (on)
  --with-catalog          add the Catalog support (on)
  --with-debug            add the debugging module (on)
  --with-docbook          add Docbook SGML support (on)
  --with-fexceptions      add GCC flag -fexceptions for C++ exceptions (off)
  --with-ftp              add the FTP support (on)
  --with-history          add history support to xmllint shell(off)
  --with-html             add the HTML support (on)
  --with-html-dir=path    path to base html directory, default
                          $datadir/doc/html
  --with-html-subdir=path directory used under html-dir, default
                          $PACKAGE-$VERSION/html
  --with-http             add the HTTP support (on)
  --with-iconv[=DIR]      add ICONV support (on)
  --with-icu                add ICU support (off)
  --with-iso8859x         add ISO8859X support if no iconv (on)
  --with-legacy           add deprecated APIs for compatibility (on)
  --with-mem-debug        add the memory debugging module (off)
  --with-minimum          build a minimally sized library (off)
  --with-output           add the serialization support (on)
  --with-pattern          add the xmlPattern selection interface (on)
  --with-push             add the PUSH parser interfaces (on)
  --with-python[=DIR]     build Python bindings if found
  --with-python-install-dir=DIR
                          install Python bindings in DIR
  --with-reader           add the xmlReader parsing interface (on)
  --with-readline=DIR     use readline in DIR
  --with-regexps          add Regular Expressions support (on)
  --with-run-debug        add the runtime debugging module (off)
  --with-sax1             add the older SAX1 interface (on)
  --with-schemas          add Relax-NG and Schemas support (on)
  --with-schematron       add Schematron support (on)
  --with-threads          add multithread support(on)
  --with-thread-alloc     add per-thread memory(off)
  --with-tree             add the DOM like tree manipulation APIs (on)
  --with-valid            add the DTD validation support (on)
  --with-writer           add the xmlWriter saving interface (on)
  --with-xinclude         add the XInclude support (on)
  --with-xpath            add the XPATH support (on)
  --with-xptr             add the XPointer support (on)
  --with-modules          add the dynamic modules support (on)
  --with-zlib[=DIR]       use libz in DIR
  --with-lzma[=DIR]       use liblzma in DIR
  --with-coverage         build for code coverage with GCC (off)

[nnc]

I had this same issue today with installing 1.6.8.rc3 on OS X 10.11.4 and spent a lot of time trying different non-bundled libxml2/libxslt combinations, and what did the trick for me in the end was a simple xcode-select --install.

I have no idea why, as I already had Xcode installed, and all other gems with native extensions installed fine before I ran xcode-select.
After xcode-select finished its business, it installed without any additional configuration.

$ gem install nokogiri -v '1.6.8.rc3'
Building native extensions.  This could take a while...
Successfully installed nokogiri-1.6.8.rc3
1 gem installed

Before I ran xcode-select, I also updated RubyGems with gem update --system as instructed here, but that on its own didn't change anything.

Anyway, hope it helps someone.

[cbowns]

@nnc what's your ruby -v?

[nnc]

@cbowns

$ ruby -v
ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin15]

[flavorjones]

@nnc Please note that xcode-select --install is documented at http://www.nokogiri.org/tutorials/installing_nokogiri.html#mac_os_x

How can we make that documentation more visible to developers? Any ideas?

[nnc]

@flavorjones that is where I saw it, but only after spending some time to try and work around the issues myself.

I would have loved to see a reminder like

Hey! Before you dive into these compile errors below, make sure you follow all the steps at http://www.nokogiri.org/tutorials/installing_nokogiri.html#mac_os_x

but I guess thats not really possible, so no I don't really see what else you could do.

And on top of that, it didn't cross my mind to mess around with xcode earlier because all other gems with native extensions installed just fine with whatever xcode setup I had previously.

[cbowns]

Hi! So, I opened this issue, not @nnc, and while it's great that their problems are fixed, mine aren't. Can you reopen this?

[cbowns]

Welp! That worked for me. I'll be damned.

I find this solution really, really weird: I have Xcode 7.3.1 installed (I'm an iOS developer and used to do iOS builds at Apple, so I know this dumb toolchain inside and out). I'm not really sure what the best way is to point people to the install notes, though. I was installing nokogiri via some bundler dependencies, so I didn't necessarily spend much time digging at http://www.nokogiri.org/tutorials/installing_nokogiri.html#mac_os_x

Maybe adding it to a repo issue template here on GitHub, now that they support that? https://github.com/blog/2111-issue-and-pull-request-templates

[jamezening]

I have no idea why, as I already had Xcode installed, and all other gems with native extensions installed fine before I ran xcode-select.

@nnc this solved my issue as well (identical to the issue in the original post)
for some reason my xCode cli tools had not been installed and now my ruby and ror are setup perfectly

[MrMarvin]

for some reason my xCode cli tools had not been installed and now my ruby and ror are setup perfectly

Did you run an (automatic) OSX update recently? This happened to somehow uninstall/break xcode CLI tools for me in the past. xcode-select --install and letting it install again fixed it for me too.

[ghost]

Running xcode-select --install worked for me, this error happened to me after I upgraded from OS X 10.10 to 10.11. I guess I was missing some of the command line tools after the upgrade.

[AntonTrapp]

Same here (10.11.5, upgraded OS X - everything running, upgraded gem - same error, xcode-select --install - error gone). TNX

[MGasztold]

Same solution solved it for me: run xcode-select --install and then repeat sudo gem install rails.

[marco-brandizi]

This might be useful: xcode-select --install worked for me too, but you need to run it after you have logged in OS X with an admin account. Beware, I mean the OS X GUI, not just su - or sudo from the terminal. That's because --install tries to pop-up a confirmation graphical dialog, and OS X apparently prevents an admin user from throwing such dialog into the window manager of a regular user.

This command install a set of additional command line tools that Apple, damn it! doesn't include in the 3.8Gb bundle (!!!) which comes with the installation of XCode from the App Store. In other words, many build files won't see Xcode until you issue this command too, after App Store installation.

[brandon-carag]

Running El Capitan 10.11.6 and xcode-select --install resolved the issue for me as well.

[enclle]

# YOU ARE ALL WRONG.
THE ERROR xmlIO.c:1450:52: error: use of undeclared identifier 'LZMA_OK' , IS BECAUSE THE HEADER FILES FOR LZMA ARE NOT INSTALLED.
SO, YOU HAVE TO INSTALL LZMA LIBRARY. THE EASIEST WAY IS TO ISSUE ON COMMAND LINE:
brew install xz
THEN YOU CAN JUST DO TRY AGAIN TO INSTALL WHATEVER YOU WANTED TO INSTALL. FOR EXAMPLE, RAILS:
gem install rails.

[RobinDaugherty]

In my case, the following fixed the Nokogiri build. This is on Sierra with Xcode 8, and installing Nokogiri through bundle install.

bundle config build.nokogiri --use-system-libraries

This is hinted at in the output from Nokogiri before the build starts, and it also includes an example command line to build Nokogiri directly.

(And note that I did have xz installed through Homebrew, and other gem extensions were building successfully.)

[binarykitchen]

@RobinDaugherty thanks but I dont have the bundle command here?

[AntonTrapp]

@binarykitchen - no bundler

gem install nokogiri -- --use-system-libraries

should do the same (http://www.nokogiri.org/tutorials/installing_nokogiri.html)

[spaquet]

If running OS X just run xcode-select --install in a terminal and your problem should be fixed.

[sinhyeok-seo]

gem install nokogiri
=> build failed

brew install xz
=> already installed
gem install nokogiri
=> build failed

xcode-select --install
gem install nokogiri
=> build failed

brew install libxml2
brew install libxslt
gem install nokogiri -- --use-system-libraries --with-xml2-config=/usr/local/Cellar/libxml2/2.9.4 --with-xslt-config=/usr/local/opt/libxslt
=> build failed

gem install nokogiri -- --with-xml2-dir=/usr/local/Cellar/libxml2/2.9.4 --with-xslt-dir=/usr/local/opt/libxslt
=> build failed

brew uninstall xz
brew uninstall --force xz
gem install nokogiri
=> build succeeded

[AntonTrapp]

Have you tried brew doctor and update homebrew to the latest version (brew update). The latest doctor should tell you you are ready to brew. If not -> fix + homebrew help.

After that, is your xcode up to date? If the xcode-select --install is still failing that would sound like a completely messed up system to me. Would check path settings / uninstall & reinstall xcode / reinstall Mac - but normally you should not come to the point with the reinstall of OS X - never been there :) - maybe some homebrew experts can help here too.

[sinhyeok-seo]

Sorry for not enough informations. I updated my last comment.
Actually, I meant nokogiri build is succeeded after brew uninstall xz.

[Ivor]

OSX Sierra
I had installed xcode 8 via App Store. After that nokogiri failed.
After running xcode-select --install the installation worked.

[kimroen]

How can we make that documentation more visible to developers? Any ideas?

@flavorjones In my case, I didn't try xcode-select --install because that suggestion is listed under the heading "I see error messages about libiconv", and that's not the error I was seeing, so I tried everything else, until I finally found this thread.

[cabo]

A simple brew unlink xz was sufficient for me this time.

[bobbytables]

Thank you @cabo !

[ozdigennaro]

I had the same error with attempts to install multiple versions of nokogiri.
xcode-select --install fixed it.

[tibomogul]

$ brew unlink xz

did it for me. Apparently building the native components is seeing the xz installed with homebrew.

[flavorjones]

Future readers: please note that the installation tutorial provides instructions for working around this issue.