Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
Add fallback bundled libsecp256k1
This commit adds the ability to install this wrapper without the need
to have a prior installation of libsecp256k1 on one's system.

This works as follows:

- During 'sdist':
  If the directory 'libsecp256k1' doesn't exist in the
  source directory it is downloaded from the location specified
  by the `LIB_TARBALL_URL` constant in setup.py and extracted into
  a directory called 'libsecp256k1'

  To upgrade to a newer version of the bundled libsecp256k1 source
  simply delete the 'libsecp256k1' directory and update the
  `LIB_TARBALL_URL` to point to a newer commit.

- During 'install':
  If an existing (system) installation of libsecp256k1 is found
  (either in the default library locations or in the location pointed
  to by the environment variable `LIB_DIR`) it is used as before.

  Due to the way the way cffi modules are implemented it is necessary
  to perform this detection in the cffi build module
  '_cffi_build/build.py' as well as in 'setup.py'. For that reason
  some utility functions have been moved into a 'setup_support.py'
  module which is imported from both.

  If however no existing installation can be found the bundled
  source code is used to build a library locally that will be
  statically linked into the CFFI extension.

  By default only the 'recovery' module will be enabled in this bundled
  version as it is the only one not considered to be 'experimental' by
  the libsecp256k1 authors. It is possible to override this and enable
  all modules by setting the environment variable
  `SECP_BUNDLED_EXPERIMENTAL`.

Additionally there are some small other optimizations:

- The source code has been moved into a package so that site-packages
  top-level doesn't get cluttered with the cffi generated source files.

- The cffi build script has been moved to a '_cffi_build' directory
  since it is not needed at runtime.

- The C-definitions used by cffi have been extracted into their own
  files inside the '_cffi_build' directory to make the build script more
  readable.
  • Loading branch information
ulope committed Feb 12, 2016
1 parent b9598cc commit a0e9d96
Show file tree
Hide file tree
Showing 18 changed files with 861 additions and 371 deletions.
8 changes: 8 additions & 0 deletions .coveragerc
Expand Up @@ -2,3 +2,11 @@

exclude_lines =
if __name__ == .__main__.:

[paths]
source =
secp256k1
*/site-packages/secp256k1/
*/site-packages/*.egg/secp256k1/
*/dist-packages/secp256k1/
*/dist-packages/*.egg/secp256k1/
1 change: 1 addition & 0 deletions .gitignore
Expand Up @@ -9,3 +9,4 @@ _libsecp256k1*
.cache/
*.egg
htmlcov
libsecp256k1
40 changes: 29 additions & 11 deletions .travis.yml
@@ -1,7 +1,19 @@
language: python

python:
- "2.7"
- "3.3"
- "3.4"

env:
global:
- LD_LIBRARY_PATH=./libsecp256k1_ext/.libs
- LIB_DIR=./libsecp256k1_ext/.libs
- INCLUDE_DIR=./libsecp256k1_ext/include
matrix:
- BUNDLED=0
- BUNDLED=1
- BUNDLED=1 SECP_BUNDLED_EXPERIMENTAL=1

sudo: false

Expand All @@ -12,23 +24,29 @@ addons:
- libtool
- autoconf
- automake
- pkg-config
- libffi-dev
- libgmp-dev

before_install:
- git clone git://github.com/bitcoin/secp256k1.git libsecp256k1
- pushd libsecp256k1
- ./autogen.sh
- ./configure --enable-module-recovery --enable-module-ecdh --enable-module-schnorr
- make
- popd
- virtualenv ENV
- source ENV/bin/activate
- pip install -U pip setuptools cffi coverage coveralls
- ./travis_install.sh
- pip install -U pip setuptools cffi pytest coverage coveralls

install:
- LIB_DIR=./libsecp256k1/.libs INCLUDE_DIR=./libsecp256k1/include python setup.py -q install
- python setup.py install

# This is a bit of a hack:
# We want to ensure that we test the installed version, not the local source.
# For that reason we rename the local source directory before running the
# tests.
# Unfortunately this compilcates using coverage. We use '--parallel' and
# 'combine' to massage it into producing correct paths. For that to work
# we need to re-rename the source back to it's correct name.
script:
- LD_LIBRARY_PATH=./libsecp256k1/.libs LIB_DIR=./libsecp256k1/.libs INCLUDE_DIR=./libsecp256k1/include coverage run --source=secp256k1 setup.py pytest
- mv secp256k1 _secp256k1
- coverage run --parallel --include="*/site-packages/*.egg/secp256k1/*" -m py.test
- mv _secp256k1 secp256k1
- coverage combine

after_success:
- coveralls
3 changes: 3 additions & 0 deletions MANIFEST.in
@@ -0,0 +1,3 @@
include setup_support.py
recursive-include _cffi_build *.py *.h
graft libsecp256k1
86 changes: 82 additions & 4 deletions README.md
@@ -1,19 +1,61 @@
# secp256k1-py [![Build Status](https://travis-ci.org/ludbb/secp256k1-py.svg?branch=master)](https://travis-ci.org/ludbb/secp256k1-py) [![Coverage Status](https://coveralls.io/repos/ludbb/secp256k1-py/badge.svg?branch=master&service=github)](https://coveralls.io/github/ludbb/secp256k1-py?branch=master)

Python FFI bindings for [secp256k1](https://github.com/bitcoin/secp256k1)
Python FFI bindings for [libsecp256k1](https://github.com/bitcoin/secp256k1)
(an experimental and optimized C library for EC operations on curve secp256k1).

## Installation

```
pip install secp256k1
```

In case the headers or lib for secp256k1 are not in your path, it's
possible to specify `INCLUDE_DIR` and `LIB_DIR` as in:
There are two modes of installation depending on whether you already have
libsecp256k1 installed on your system:


###### Using a system installed libsecp256k1

If the library is already installed it should usually be automatically detected
and used.
However if libsecp256k1 is installed in a non standard location you can use the
environment variables `INCLUDE_DIR` and `LIB_DIR` to point the way:

```
INCLUDE_DIR=/opt/somewhere/include LIB_DIR=/opt/somewhere/lib pip install secp256k1
```


###### Using the bundled libsecp256k1

If on the other hand you don't have libsecp256k1 installed on your system, a
bundled version will be built and used. In this case only the `recovery` module
will be enabled since it's the only one not currently considered as
"experimental" by the library authors. This can be overridden by setting the
`SECP_BUNDLED_EXPERIMENTAL` environment variable:

```
INCLUDE_DIR=/usr/local/include LIB_DIR=/usr/local/lib pip install secp256k1
SECP_BUNDLED_EXPERIMENTAL=1 pip install secp256k1
```

For the bundled version to compile successfully you need to have a C compiler
as well as the development headers for `libffi` and `libgmp` installed.

On Debian / Ubuntu for example the necessary packages are:

* `build-essential`
* `automake`
* `pkg-config`
* `libtool`
* `libffi-dev`
* `libgmp-dev`

On OS X the necessary homebrew packages are:

* `automake`
* `pkg-config`
* `libffi`
* `gmp`


## Command line usage

Expand Down Expand Up @@ -287,3 +329,39 @@ pubkey_ser_uncompressed = privkey.pubkey.serialize(compressed=False)
assert pubkey_ser == bytes(bytearray.fromhex(pub_compressed))
assert pubkey_ser_uncompressed == bytes(bytearray.fromhex(pub_uncompressed))
```


## Technical details about the bundled libsecp256k1

The bundling of libsecp256k1 is handled by the various setup.py build phases:

- During 'sdist':
If the directory `libsecp256k1` doesn't exist in the
source directory it is downloaded from the location specified
by the `LIB_TARBALL_URL` constant in `setup.py` and extracted into
a directory called `libsecp256k1`

To upgrade to a newer version of the bundled libsecp256k1 source
simply delete the `libsecp256k1` directory and update the
`LIB_TARBALL_URL` to point to a newer commit.

- During 'install':
If an existing (system) installation of libsecp256k1 is found
(either in the default library locations or in the location pointed
to by the environment variable `LIB_DIR`) it is used as before.

Due to the way the way cffi modules are implemented it is necessary
to perform this detection in the cffi build module
`_cffi_build/build.py` as well as in `setup.py`. For that reason
some utility functions have been moved into a `setup_support.py`
module which is imported from both.

If however no existing installation can be found the bundled
source code is used to build a library locally that will be
statically linked into the CFFI extension.

By default only the `recovery` module will be enabled in this bundled
version as it is the only one not considered to be 'experimental' by
the libsecp256k1 authors. It is possible to override this and enable
all modules by setting the environment variable
`SECP_BUNDLED_EXPERIMENTAL`.
93 changes: 93 additions & 0 deletions _cffi_build/build.py
@@ -0,0 +1,93 @@
import os
import sys
from collections import namedtuple
from itertools import combinations

from cffi import FFI, VerificationError

sys.path.append(os.path.abspath(os.path.dirname(__file__)))
from setup_support import has_system_lib, redirect, workdir, absolute

Source = namedtuple('Source', ('h', 'include'))


class Break(Exception):
pass


def _mk_ffi(sources, name="_libsecp256k1", bundled=True, **kwargs):
ffi = FFI()
code = []
if 'INCLUDE_DIR' in os.environ:
kwargs['include_dirs'] = [absolute(os.environ['INCLUDE_DIR'])]
if 'LIB_DIR' in os.environ:
kwargs['library_dirs'] = [absolute(os.environ['LIB_DIR'])]
for source in sources:
with open(source.h, 'rt') as h:
ffi.cdef(h.read())
code.append(source.include)
if bundled:
code.append("#define PY_USE_BUNDLED")
ffi.set_source(name, "\n".join(code), **kwargs)
return ffi


_base = [Source(absolute("_cffi_build/secp256k1.h"), "#include <secp256k1.h>", )]

_modules = {
'ecdh': Source(absolute("_cffi_build/secp256k1_ecdh.h"), "#include <secp256k1_ecdh.h>", ),
'recovery': Source(absolute("_cffi_build/secp256k1_recovery.h"), "#include <secp256k1_recovery.h>", ),
'schnorr': Source(absolute("_cffi_build/secp256k1_schnorr.h"), "#include <secp256k1_schnorr.h>", ),
}


ffi = None

# The following is used to detect whether the library is already installed on
# the system (and if so which modules are enabled) or if we will use the
# bundled one.
if has_system_lib():
_available = []
try:
# try all combinations of optional modules that could be enabled
# works downwards from most enabled modules to fewest
for l in range(len(_modules), -1, -1):
for combination in combinations(_modules.items(), l):
try:
_test_ffi = _mk_ffi(
_base + [item[1] for item in combination],
name="_testcompile",
bundled=False,
libraries=['secp256k1']
)
with redirect(sys.stderr, os.devnull), workdir():
_test_ffi.compile()
_available = combination
raise Break()
except VerificationError as ex:
pass
except Break:
ffi = _mk_ffi(
_base + [i[1] for i in _available],
bundled=False,
libraries=['secp256k1']
)
print("Using system libsecp256k1 with modules: {}".format(
", ".join(i[0] for i in _available))
)
else:
# We didn't find any functioning combination of modules
# Normally this shouldn't happen but just in case we will fall back
# to the bundled library
print("Installed libsecp256k1 is unusable falling back to bundled version.")

if ffi is None:
# Library is not installed - use bundled one
print("Using bundled libsecp256k1")

# By default we only build with recovery enabled since the other modules
# are experimental
if os.environ.get('SECP_BUNDLED_EXPERIMENTAL'):
ffi = _mk_ffi(_base + list(_modules.values()), libraries=['secp256k1'])
else:
ffi = _mk_ffi(_base + [_modules['recovery']], libraries=['secp256k1'])

0 comments on commit a0e9d96

Please sign in to comment.