Can't install snappy_ext gem on Mac OS

[murdoch]

I'm having trouble installing the gem on MacOS M1:
I've got Mac OS 13.6
I've installed snappy via homebrew if that helps, but I see the same issue without it.

I get the following output when I try to install the gem:

± gem install snappy_ext
Building native extensions. This could take a while...
ERROR:  Error installing snappy_ext:
    ERROR: Failed to build gem native extension.

    current directory: /Users/smurdoch/.asdf/installs/ruby/3.0.6/lib/ruby/gems/3.0.0/gems/snappy_ext-0.1.2/ext/snappy
/Users/smurdoch/.asdf/installs/ruby/3.0.6/bin/ruby extconf.rb
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /opt/homebrew/bin/gmkdir -p
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking build system type... arm-apple-darwin22.6.0
checking host system type... arm-apple-darwin22.6.0
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking dependency style of gcc... gcc3
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /Library/Developer/CommandLineTools/usr/bin/ld
checking if the linker (/Library/Developer/CommandLineTools/usr/bin/ld) is GNU ld... no
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 786432
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking for /Library/Developer/CommandLineTools/usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for ar... ar
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for dsymutil... dsymutil
checking for nmedit... nmedit
checking for lipo... lipo
checking for otool... otool
checking for otool64... no
checking for -single_module linker flag... no
checking for -exported_symbols_list linker flag... yes
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... yes
checking for gcc option to produce PIC... -fno-common -DPIC
checking if gcc PIC flag -fno-common -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/Library/Developer/CommandLineTools/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... darwin22.6.0 dyld
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... gcc3
checking whether we are using the GNU C++ compiler... (cached) yes
checking whether g++ accepts -g... (cached) yes
checking dependency style of g++... (cached) gcc3
checking how to run the C++ preprocessor... g++ -E
checking for ld used by g++... /Library/Developer/CommandLineTools/usr/bin/ld
checking if the linker (/Library/Developer/CommandLineTools/usr/bin/ld) is GNU ld... no
checking whether the g++ linker (/Library/Developer/CommandLineTools/usr/bin/ld) supports shared libraries... yes
checking for g++ option to produce PIC... -fno-common -DPIC
checking if g++ PIC flag -fno-common -DPIC works... yes
checking if g++ static flag -static works... no
checking if g++ supports -c -o file.o... yes
checking if g++ supports -c -o file.o... (cached) yes
checking whether the g++ linker (/Library/Developer/CommandLineTools/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... darwin22.6.0 dyld
checking how to hardcode library paths into programs... immediate
checking whether byte ordering is bigendian... no
checking for stdint.h... (cached) yes
checking stddef.h usability... yes
checking stddef.h presence... yes
checking for stddef.h... yes
checking for mmap... yes
checking for 'gtest-config'... checking for gtest-config... no
no
checking for pkg-config... no
checking for gflags... checking if the compiler supports __builtin_expect... yes
checking if the compiler supports __builtin_ctzll... yes
checking for zlibVersion in -lz... yes
checking for lzo1x_1_15_compress in -llzo2... no
checking for lzf_compress in -llzf... no
checking for fastlz_compress in -lfastlz... no
checking for qlz_compress in -lquicklz... no
configure: creating ./config.status
config.status: creating Makefile
config.status: creating snappy-stubs-public.h
config.status: creating config.h
config.status: executing depfiles commands
config.status: executing libtool commands
/Library/Developer/CommandLineTools/usr/bin/make  all-am
/bin/sh ./libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I.     -g -O2 -MT snappy.lo -MD -MP -MF .deps/snappy.Tpo -c -o snappy.lo snappy.cc
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -g -O2 -MT snappy.lo -MD -MP -MF .deps/snappy.Tpo -c snappy.cc  -fno-common -DPIC -o .libs/snappy.o
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -g -O2 -MT snappy.lo -MD -MP -MF .deps/snappy.Tpo -c snappy.cc -o snappy.o >/dev/null 2>&1
mv -f .deps/snappy.Tpo .deps/snappy.Plo
/bin/sh ./libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I.     -g -O2 -MT snappy-sinksource.lo -MD -MP -MF .deps/snappy-sinksource.Tpo -c -o snappy-sinksource.lo snappy-sinksource.cc
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -g -O2 -MT snappy-sinksource.lo -MD -MP -MF .deps/snappy-sinksource.Tpo -c snappy-sinksource.cc  -fno-common -DPIC -o .libs/snappy-sinksource.o
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -g -O2 -MT snappy-sinksource.lo -MD -MP -MF .deps/snappy-sinksource.Tpo -c snappy-sinksource.cc -o snappy-sinksource.o >/dev/null 2>&1
mv -f .deps/snappy-sinksource.Tpo .deps/snappy-sinksource.Plo
/bin/sh ./libtool --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I.     -g -O2 -MT snappy-stubs-internal.lo -MD -MP -MF .deps/snappy-stubs-internal.Tpo -c -o snappy-stubs-internal.lo snappy-stubs-internal.cc
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -g -O2 -MT snappy-stubs-internal.lo -MD -MP -MF .deps/snappy-stubs-internal.Tpo -c snappy-stubs-internal.cc  -fno-common -DPIC -o .libs/snappy-stubs-internal.o
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -g -O2 -MT snappy-stubs-internal.lo -MD -MP -MF .deps/snappy-stubs-internal.Tpo -c snappy-stubs-internal.cc -o snappy-stubs-internal.o >/dev/null 2>&1
mv -f .deps/snappy-stubs-internal.Tpo .deps/snappy-stubs-internal.Plo
/bin/sh ./libtool --tag=CXX   --mode=link g++  -g -O2 -version-info 1:0:0  -o libsnappy.la -rpath /usr/local/lib snappy.lo snappy-sinksource.lo snappy-stubs-internal.lo  -lz
libtool: link: g++ -r -keep_private_externs -nostdlib -o .libs/libsnappy.1.dylib-master.o  .libs/snappy.o .libs/snappy-sinksource.o .libs/snappy-stubs-internal.o
libtool: link: g++ -dynamiclib -Wl,-flat_namespace -Wl,-undefined -Wl,suppress -o .libs/libsnappy.1.dylib .libs/libsnappy.1.dylib-master.o  -lz    -install_name  /usr/local/lib/libsnappy.1.dylib -compatibility_version 2 -current_version 2.0
ld: warning: -undefined suppress is deprecated
libtool: link: dsymutil .libs/libsnappy.1.dylib || :
libtool: link: (cd ".libs" && rm -f "libsnappy.dylib" && ln -s "libsnappy.1.dylib" "libsnappy.dylib")
libtool: link: ar cru .libs/libsnappy.a  snappy.o snappy-sinksource.o snappy-stubs-internal.o
libtool: link: ranlib .libs/libsnappy.a
libtool: link: ( cd ".libs" && rm -f "libsnappy.la" && ln -s "../libsnappy.la" "libsnappy.la" )
g++ -DHAVE_CONFIG_H -I.      -g -O2 -MT snappy_unittest-snappy_unittest.o -MD -MP -MF .deps/snappy_unittest-snappy_unittest.Tpo -c -o snappy_unittest-snappy_unittest.o `test -f 'snappy_unittest.cc' || echo './'`snappy_unittest.cc
snappy_unittest.cc:719:24: warning: implicit conversion from 'int' to 'value_type' (aka 'char') changes value from 128 to -128 [-Wconstant-conversion]
  compressed.push_back(128);
             ~~~~~~~~~ ^~~
snappy_unittest.cc:720:24: warning: implicit conversion from 'int' to 'value_type' (aka 'char') changes value from 128 to -128 [-Wconstant-conversion]
  compressed.push_back(128);
             ~~~~~~~~~ ^~~
snappy_unittest.cc:721:24: warning: implicit conversion from 'int' to 'value_type' (aka 'char') changes value from 128 to -128 [-Wconstant-conversion]
  compressed.push_back(128);
             ~~~~~~~~~ ^~~
snappy_unittest.cc:722:24: warning: implicit conversion from 'int' to 'value_type' (aka 'char') changes value from 128 to -128 [-Wconstant-conversion]
  compressed.push_back(128);
             ~~~~~~~~~ ^~~
snappy_unittest.cc:723:24: warning: implicit conversion from 'int' to 'value_type' (aka 'char') changes value from 128 to -128 [-Wconstant-conversion]
  compressed.push_back(128);
             ~~~~~~~~~ ^~~
5 warnings generated.
mv -f .deps/snappy_unittest-snappy_unittest.Tpo .deps/snappy_unittest-snappy_unittest.Po
g++ -DHAVE_CONFIG_H -I.      -g -O2 -MT snappy_unittest-snappy-test.o -MD -MP -MF .deps/snappy_unittest-snappy-test.Tpo -c -o snappy_unittest-snappy-test.o `test -f 'snappy-test.cc' || echo './'`snappy-test.cc
mv -f .deps/snappy_unittest-snappy-test.Tpo .deps/snappy_unittest-snappy-test.Po
/bin/sh ./libtool --tag=CXX   --mode=link g++  -g -O2   -o snappy_unittest snappy_unittest-snappy_unittest.o snappy_unittest-snappy-test.o libsnappy.la   -lz
libtool: link: g++ -g -O2 -o .libs/snappy_unittest snappy_unittest-snappy_unittest.o snappy_unittest-snappy-test.o -Wl,-bind_at_load  ./.libs/libsnappy.dylib -lz
ld: warning: -bind_at_load is deprecated on macOS
checking for ruby/st.h... yes
checking for -lstdc++... no
creating Makefile

ld: 45 duplicate symbols
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [snappy_ext.bundle] Error 1

make failed, exit code 2

This issue started recently, after I reinstalled homebrew. I'm guessing I've lost some package that I previously required.
Any ideas what might be causing this?

[murdoch]

I got it installed!

It turns out the snappy_ext gem pulls the snappy source code from a google code repo that no longer exists.

I'm guessing the people in my office who aren't having this issue have the url cached somewhere in their systems, whereas I don't.

So to fix this, I downloaded the snappy-1.0.0.tar.gz from an ancient fedora archive found here: https://src.fedoraproject.org/repo/pkgs/snappy/snappy-1.0.0.tar.gz/9d83bdcf0c79a8de608fa969c2909204/

I've attached the tar file below fro anyone who runs into this issue:
snappy-1.0.0.tar.gz

Here's a brief summary of what I did

1. Download the snappy tar ball from the fedora project, or use the one I have attached to this comment if you trust strange random people like me on the internet

2. clone the gem

git clone git@github.com:willglynn/snappy-ruby.git

3. CD into the directory and checkout commit 6a12208

cd snappy-ruby

# checkout the specific sha from which version 0.1.2 of this gem was built
git checkout 6a12208  

4. Place the snappy tar ball from step 1 in the specific folder this gem expects it to be in

# place snappy tar ball in vendor directory and untar it
cd snappy_ext/ext/snappy
mkdir vendor # folder doesn't exist yet, we need to create it
cd vendor
cp ~/Downloads/snappy-1.0.0.tar.gz .
tar -xzf snappy-1.0.0.tar.gz

5. CD into the untarred snappy folder, and make the following change to the autogen.sh file

# you need to be at the following path snappy-ruby/snappy_ext/ext/snappy/vendor/snappy-1.0.0
cd snappy-1.0.0 

# I have to use glibtoolize because I installed autoconf with homebrew
sed 's/libtoolize/glibtoolize/' autogen.sh

# if you don't have sed installed, then open the file and change this manually or use something else to substitute the code

6. Run ./configure script manually to generate the Makefile

# Three is a file in snappy-ruby/snappy_ext/ext/snappy/vendor/snappy-1.0.0 called configure,
# which builds the Makefile, we need to run it manually as we need to modify the Makefile 
./cofigure

7. Open up snappy-ruby/snappy_ext/ext/snappy/vendor/snappy-1.0.0/Makefile

You may not need to do the following, but I have installed GCC via homebrew, and needed to use it over MacOS pre-installed GCC. There are also other ways to do this, but I couldn't be bothered messing with zsh env variables

# Change the following lines
CC=gcc
CPP=gcc -E

# To the following
CC=gcc-13
CPP=gcc-13 -E

# replace gcc-13 with whatever version of gcc homebrew installs on your system
# note that I had to do the above because Mac OS gcc was having trouble compiling

8. Open up snappy-ruby/snappy_ext/ext/snappy/extconf.rb and make the following change

# Remove the following code as we have already run the configure script

fork {
  exec("./configure")
}; Process.wait; exit(1) unless $?.success?

9. Open up snappy-ruby/snappy_ext/snappy_ext.gemspec and make the following change

# DELETE or comment out the following code block, we don't need it anymore
# It also happens to be code that causes most of the trouble, notice how it's trying to curl from a googlecode.com url, which no longer exists (kindof)

unless File.exists?(SNAPPY_DIR)
  old_pwd = Dir.pwd
  archive = "#{File.basename SNAPPY_DIR}.tar.gz"
 
  begin
    Dir.chdir(File.dirname(SNAPPY_DIR))
    puts "Downloading and unpacking #{File.basename SNAPPY_DIR}..."
    system "curl", "http://snappy.googlecode.com/files/snappy-#{SNAPPY_VERSION}.tar.gz", "-o", archive
    system "tar", "xzf", archive
 
  ensure
    File.unlink(archive) if File.exists?(archive)
    Dir.chdir(old_pwd)
  end
end

10. go back to the root folder and make the gem

# from ./snappy-ruby folder
make all
gem install snappy_ext

And that's it. The gem should install on Mac M1 with latest homebrew and ruby 3.0.6.

[Shivamcskk]

point no 10.
make all is not running, assuming it should be run on root folder snappy-ruby.
and skiiping this and running gem install snappy_ext is resulting same.

Please help @murdoch

[murdoch]

Hi @Shivamcskk

What happens if you try to use snappy gem https://github.com/miyucy/snappy as a replacement for this one? I think SnappyExt abandoned.