Installation of secp256k1 dependency fails

[xuhcc]

• Populus Version: 1.11.0
• OS: Arch Linux
• OpenSSL version: 1.1.0f 25 May 2017

What was wrong?

Populus depends on secp256k1 package, which throws an error during installation:

$ pip install populus
...
  Running setup.py install for secp256k1 ... error
    Complete output from command venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-rrl2k2go/secp256k1/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-4pke8v3f-record/install-record.txt --single-version-externally-managed --compile --install-headers venv/include/site/python3.6/secp256k1:
    0.29.2
    Using bundled libsecp256k1
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/secp256k1
    copying secp256k1/__main__.py -> build/lib.linux-x86_64-3.6/secp256k1
    copying secp256k1/__init__.py -> build/lib.linux-x86_64-3.6/secp256k1
    running build_clib
    checking build system type... x86_64-pc-linux-gnu
    checking host system type... x86_64-pc-linux-gnu
    checking for a BSD-compatible install... /usr/bin/install -c
    checking whether build environment is sane... yes
    checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
    checking for gawk... gawk
    checking whether make sets $(MAKE)... yes
    checking whether make supports nested variables... yes
    checking how to print strings... printf
    checking for style of include used by make... GNU
    checking for gcc... gcc
    checking whether the C compiler works... yes
    checking for C compiler default output file name... a.out
    checking for suffix of executables...
    checking whether we are cross compiling... no
    checking for suffix of object files... o
    checking whether we are using the GNU C compiler... yes
    checking whether gcc accepts -g... yes
    checking for gcc option to accept ISO C89... none needed
    checking whether gcc understands -c and -o together... yes
    checking dependency style of gcc... none
    checking for a sed that does not truncate output... /usr/bin/sed
    checking for grep that handles long lines and -e... /usr/bin/grep
    checking for egrep... /usr/bin/grep -E
    checking for fgrep... /usr/bin/grep -F
    checking for ld used by gcc... /usr/bin/ld
    checking if the linker (/usr/bin/ld) is GNU ld... yes
    checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
    checking the name lister (/usr/bin/nm -B) interface... BSD nm
    checking whether ln -s works... yes
    checking the maximum length of command line arguments... 1572864
    checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop
    checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
    checking for /usr/bin/ld option to reload object files... -r
    checking for objdump... objdump
    checking how to recognize dependent libraries... pass_all
    checking for dlltool... no
    checking how to associate runtime and link libraries... printf %s\n
    checking for ar... ar
    checking for archiver @FILE support... @
    checking for strip... strip
    checking for ranlib... ranlib
    checking command to parse /usr/bin/nm -B output from gcc object... ok
    checking for sysroot... no
    checking for a working dd... /usr/bin/dd
    checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1
    checking for mt... no
    checking if : is a manifest tool... no
    checking how to run the C preprocessor... gcc -E
    checking for ANSI C header files... yes
    checking for sys/types.h... yes
    checking for sys/stat.h... yes
    checking for stdlib.h... yes
    checking for string.h... yes
    checking for memory.h... yes
    checking for strings.h... yes
    checking for inttypes.h... yes
    checking for stdint.h... yes
    checking for unistd.h... yes
    checking for dlfcn.h... yes
    checking for objdir... .libs
    checking if gcc supports -fno-rtti -fno-exceptions... no
    checking for gcc option to produce PIC... -fPIC -DPIC
    checking if gcc PIC flag -fPIC -DPIC works... yes
    checking if gcc static flag -static works... yes
    checking if gcc supports -c -o file.o... yes
    checking if gcc supports -c -o file.o... (cached) yes
    checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
    checking dynamic linker characteristics... GNU/Linux ld.so
    checking how to hardcode library paths into programs... immediate
    checking whether stripping libraries is possible... yes
    checking if libtool supports shared libraries... yes
    checking whether to build shared libraries... no
    checking whether to build static libraries... yes
    checking whether make supports nested variables... (cached) yes
    checking for pkg-config... /usr/bin/pkg-config
    checking pkg-config is at least version 0.9.0... yes
    checking for ar... /usr/bin/ar
    checking for ranlib... /usr/bin/ranlib
    checking for strip... /usr/bin/strip
    checking for gcc... gcc
    checking whether we are using the GNU C compiler... (cached) yes
    checking whether gcc accepts -g... yes
    checking for gcc option to accept ISO C89... (cached) none needed
    checking whether gcc understands -c and -o together... (cached) yes
    checking dependency style of gcc... (cached) none
    checking how to run the C preprocessor... gcc -E
    checking for gcc option to accept ISO C89... (cached) none needed
    checking dependency style of gcc... none
    checking if gcc supports -std=c89 -pedantic -Wall -Wextra -Wcast-align -Wnested-externs -Wshadow -Wstrict-prototypes -Wno-unused-function -Wno-long-long -Wno-overlength-strings... yes
    checking if gcc supports -fvisibility=hidden... yes
    checking for __int128... yes
    checking for __builtin_expect... yes
    checking native compiler: gcc... ok
    checking for x86_64 assembly availability... yes
    checking for libcrypto... yes
    checking for main in -lcrypto... yes
    checking for EC functions in libcrypto... yes
    checking for javac... no
    configure: WARNING: cannot find JDK; try setting $JAVAC or $JAVA_HOME
    checking jni headers... none
    configure: WARNING: jni headers/dependencies not found. jni support disabled
    checking whether byte ordering is bigendian... no
    configure: Using static precomputation: yes
    configure: Using assembly optimizations: x86_64
    configure: Using field implementation: 64bit
    configure: Using bignum implementation: no
    configure: Using scalar implementation: 64bit
    configure: Using endomorphism optimizations: no
    configure: Building ECDH module: no
    configure: Building Schnorr signatures module: no
    configure: Building ECDSA pubkey recovery module: yes
    configure: Using jni: no
    checking that generated files are newer than configure... done
    configure: creating ./config.status
    config.status: creating Makefile
    config.status: creating libsecp256k1.pc
    config.status: creating src/libsecp256k1-config.h
    config.status: executing depfiles commands
    config.status: executing libtool commands
      CC       src/libsecp256k1_la-secp256k1.lo
    In file included from /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/secp256k1.c:14:0:
    /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/ecmult_impl.h: In function ‘secp256k1_ecmult_context_clone’:
    /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/ecmult_impl.h:186:9: warning: implicit declaration of function ‘memcpy’ [-Wimplicit-function-declaration]
             memcpy(dst->pre_g, src->pre_g, size);
             ^~~~~~
    /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/ecmult_impl.h:186:9: warning: incompatible implicit declaration of built-in function ‘memcpy’
    /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/ecmult_impl.h:186:9: note: include ‘<string.h>’ or provide a declaration of ‘memcpy’
    /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/ecmult_impl.h: In function ‘secp256k1_ecmult_wnaf’:
    /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/ecmult_impl.h:230:5: warning: implicit declaration of function ‘memset’ [-Wimplicit-function-declaration]
         memset(wnaf, 0, len * sizeof(wnaf[0]));
         ^~~~~~
    /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/ecmult_impl.h:230:5: warning: incompatible implicit declaration of built-in function ‘memset’
    /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/ecmult_impl.h:230:5: note: include ‘<string.h>’ or provide a declaration of ‘memset’
      CCLD     libsecp256k1.la
      CC       src/tests-tests.o
    In file included from /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/secp256k1.c:14:0,
                     from /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/tests.c:16:
    /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/ecmult_impl.h: In function ‘secp256k1_ecmult_context_clone’:
    /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/ecmult_impl.h:186:9: warning: implicit declaration of function ‘memcpy’ [-Wimplicit-function-declaration]
             memcpy(dst->pre_g, src->pre_g, size);
             ^~~~~~
    /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/ecmult_impl.h:186:9: warning: incompatible implicit declaration of built-in function ‘memcpy’
    /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/ecmult_impl.h:186:9: note: include ‘<string.h>’ or provide a declaration of ‘memcpy’
    /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/ecmult_impl.h: In function ‘secp256k1_ecmult_wnaf’:
    /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/ecmult_impl.h:230:5: warning: implicit declaration of function ‘memset’ [-Wimplicit-function-declaration]
         memset(wnaf, 0, len * sizeof(wnaf[0]));
         ^~~~~~
    /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/ecmult_impl.h:230:5: warning: incompatible implicit declaration of built-in function ‘memset’
    /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/ecmult_impl.h:230:5: note: include ‘<string.h>’ or provide a declaration of ‘memset’
    /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/tests.c: In function ‘test_ecdsa_der_parse’:
    /tmp/pip-build-rrl2k2go/secp256k1/libsecp256k1/src/tests.c:3702:52: error: dereferencing pointer to incomplete type ‘ECDSA_SIG {aka struct ECDSA_SIG_st}’
             valid_openssl = !BN_is_negative(sig_openssl->r) && !BN_is_negative(sig_openssl->s) && BN_num_bits(sig_openssl->r) > 0 && BN_num_bits(sig_openssl->r) <= 256 && BN_num_bits(sig_openssl->s) > 0 && BN_num_bits(sig_openssl->s) <= 256;
                                                        ^~
    make: *** [Makefile:1048: src/tests-tests.o] Error 1
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-rrl2k2go/secp256k1/setup.py", line 295, in <module>
        "Topic :: Security :: Cryptography"
      File "/usr/lib64/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/usr/lib64/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/home/xuhcc/Documents/coding/ws-contracts/venv/lib/python3.6/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/usr/lib64/python3.6/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "/usr/lib64/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/usr/lib64/python3.6/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/usr/lib64/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/usr/lib64/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/tmp/pip-build-rrl2k2go/secp256k1/setup.py", line 217, in run
        subprocess.check_call(["make"], cwd=build_temp)
      File "/usr/lib64/python3.6/subprocess.py", line 291, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['make']' returned non-zero exit status 2.
    
    ----------------------------------------

Cute Animal Picture

[veox]

EDIT (2018-07-19): libsecp256k1 has bee in Arch's community repo since at least 2017-12-31; mentioned in a later comment.

---

Installing the secp256k1-git package from AUR system-wide helped for me - it conveniently skips building/running tests.

The relevant pip package - secp256k1 - then picks up on the fact that there's a system-wide binary object already available, and uses that.

This is perhaps sub-par as far as "cleanroom isolation" goes, but at least you can use Populus then.

[pipermerriam]

This dependency should be going away entirely in the coming few weeks when the switch to eth-tester happens.

[introom]

also fails on debian sid.

apt install libsecp256k1-dev solves the issue.

[ShivaGuntuku]

Following these steps may be help for your problem, it worked for me.

sudo apt-get install libssl-dev build-essential automake pkg-config libtool libffi-dev libgmp-dev libyaml-cpp-dev
git clone https://github.com/ethereum/pyethereum/
cd pyethereum
python setup.py install

pyethereum

[MarcusJones]

I finally got this working after several days. I had the same error as @xuhcc ;

  subprocess.CalledProcessError: Command '['/tmp/pip-install-tvd6uck6/secp256k1/libsecp256k1/autogen.sh']' returned non-zero exit status 127.
  
  ----------------------------------------
  Failed building wheel for secp256k1
  Running setup.py clean for secp256k1
Successfully built anyconfig
Failed to build secp256k1

Thanks to @ShivaGuntuku, I ran the following;

sudo apt-get install libssl-dev build-essential automake pkg-config libtool libffi-dev libgmp-dev libyaml-cpp-dev

And it installed 🎉

I am using requirements.txt as follows, thanks to @SmokinCaterpillar who I met at a conference!

anyconfig==0.9.4
argh==0.26.2
atomicwrites==1.1.5
attrdict==2.0.0
attrs==18.1.0
bitcoin==1.1.42
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
click==6.7
contextlib2==0.5.5
Cython==0.28.3
cytoolz==0.9.0.1
eth-abi==0.5.0
eth-account==0.2.3
eth-hash==0.1.4
eth-keyfile==0.5.1
eth-keys==0.1.0b4
eth-rlp==0.1.2
eth-tester==0.1.0b11
eth-testrpc==1.3.5
eth-utils==0.7.1
ethereum==1.6.1
hexbytes==0.1.0
idna==2.7
json-rpc==1.11.0
jsonschema==2.6.0
lru-dict==1.1.6
mkl-fft==1.0.0
mkl-random==1.0.1
more-itertools==4.2.0
numpy==1.14.5
parsimonious==0.8.0
pathtools==0.1.2
pbkdf2==1.3
pluggy==0.6.0
populus==2.2.0
py==1.5.3
py-geth==2.0.1
py-solc==3.1.0
pycparser==2.18
pycryptodome==3.6.1
pyethash==0.1.27
pylru==1.1.0
pysha3==1.0.2
pytest==3.6.1
PyYAML==3.12
repoze.lru==0.7
requests==2.19.1
rlp==0.6.0
scrypt==0.8.6
secp256k1==0.13.2
semantic-version==2.6.0
six==1.11.0
toolz==0.9.0
toposort==1.5
urllib3==1.23
watchdog==0.8.3
web3==3.16.5
websockets==4.0.1
Werkzeug==0.14.1

Python 3.6.5

[paulhauner]

I'm on Arch and solved this by installing community/libsecp256k1. I naturally prefer this to secp256k1-git because it comes from the community repo instead of the AUR. Install it with this:

$ pacman -S libsecp256k1

[aashutoshrathi]

Thanks @introom @paulhauner for saving the day.