New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experimental feature: remote proving service #2120

Open
wants to merge 8 commits into
base: master
from
View
@@ -5,12 +5,14 @@ src/bitcoin
src/zcashd
src/zcash-cli
src/zcash-gtest
src/zcash-proving-service
src/zcash-tx
src/test/test_bitcoin
# Zcash utilities
src/zcash/GenerateParams
src/zcash/CreateJoinSplit
src/zcash/ExampleProvingServiceClient
*zcashTest.pk
*zcashTest.vk
@@ -0,0 +1,111 @@
# ===========================================================================
# https://www.gnu.org/software/autoconf-archive/ax_boost_regex.html
# ===========================================================================
#
# SYNOPSIS
#
# AX_BOOST_REGEX
#
# DESCRIPTION
#
# Test for Regex library from the Boost C++ libraries. The macro requires
# a preceding call to AX_BOOST_BASE. Further documentation is available at
# <http://randspringer.de/boost/index.html>.
#
# This macro calls:
#
# AC_SUBST(BOOST_REGEX_LIB)
#
# And sets:
#
# HAVE_BOOST_REGEX
#
# LICENSE
#
# Copyright (c) 2008 Thomas Porschberg <thomas@randspringer.de>
# Copyright (c) 2008 Michael Tindal
#
# Copying and distribution of this file, with or without modification, are
# permitted in any medium without royalty provided the copyright notice
# and this notice are preserved. This file is offered as-is, without any
# warranty.

This comment has been minimized.

@daira

daira Dec 22, 2017

Contributor

Add this to contrib/debian/copyright:

Files: build-aux/m4/ax_boost_regex.m4
Copyright: 2008 Thomas Porschberg <thomas@randspringer.de>;
 2008 Michael Tindal
License: GNU-All-permissive-License

Actually there should be similar entries for all of the files under build-aux/m4:

Files: build-aux/m4/ax_boost_base.m4
Copyright: 2008 Thomas Porschberg <thomas@randspringer.de>;
 2009 Peter Adolphs
License: GNU-All-permissive-License

Files: build-aux/m4/ax_boost_chrono.m4
Copyright: 2012 Xiyue Deng <manphiz@gmail.com>
License: GNU-All-permissive-License

Files: build-aux/m4/ax_boost_filesystem.m4
Copyright: 2008 Thomas Porschberg <thomas@randspringer.de>;
 2009 Michael Tindal;
 2009 Roman Rybalko <libtorrent@romanr.info>
License: GNU-All-permissive-License

Files: build-aux/m4/ax_boost_program_options.m4;
 build-aux/m4/ax_boost_unit_test_framework.m4
Copyright: 2008 Thomas Porschberg <thomas@randspringer.de>
License: GNU-All-permissive-License

Files: build-aux/m4/ax_boost_system.m4
Copyright: 2008 Thomas Porschberg <thomas@randspringer.de>;
 2008 Michael Tindal;
 2008 Daniel Casimiro <dan.casimiro@gmail.com>
License: GNU-All-permissive-License

Files: build-aux/m4/ax_cxx_compile_stdcxx.m4
Copyright: 2008 Benjamin Kosnik <bkoz@redhat.com>;
 2012 Zack Weinberg <zackw@panix.com>;
 2013 Roy Stogner <roystgnr@ices.utexas.edu>;
 2014, 2015 Google Inc. contributed by Alexey Sokolov <sokolov@google.com>;
 2015 Paul Norman <penorman@mac.com>;
 2015 Moritz Klammler <moritz@klammler.eu>
License: GNU-All-permissive-License

Files: build-aux/m4/ax_gcc_func_attribute.m4
Copyright: 2013 Gabriele Svelto <gabriele.svelto@gmail.com>
License: GNU-All-permissive-License

Files: build-aux/m4/l_atomic.m4
Copyright: 2004-2017 Tim Kosse
License: GPLv2

Files: build-aux/m4/ax_check_compile_flag.m4;
 build-aux/m4/ax_check_link_flag.m4;
 build-aux/m4/ax_check_preproc_flag.m4
Copyright: 2008 Guido U. Draheim <guidod@gmx.de>;
 2011 Maarten Bosmans <mkbosmans@gmail.com>
License: GPLv3-with-Autoconf-Macro-exception

Files: build-aux/m4/ax_openmp.m4
Copyright: 2008 Steven G. Johnson <stevenj@alum.mit.edu>;
 2015 John W. Peterson <jwpeterson@gmail.com>;
 2016 Nick R. Papior <nickpapior@gmail.com>
License: GPLv3-with-Autoconf-Macro-exception

Files: build-aux/m4/ax_pthread.m4
Copyright: 2008 Steven G. Johnson <stevenj@alum.mit.edu>;
 2011 Daniel Richard G. <skunk@iSKUNK.ORG>
License: GPLv3-with-Autoconf-Macro-exception

License: GPLv3-with-Autoconf-Macro-exception
 This program is free software: you can redistribute it and/or modify it
 under the terms of the GNU General Public License as published by the
 Free Software Foundation, either version 3 of the License, or (at your
 option) any later version.
 .
 This program is distributed in the hope that it will be useful, but
 WITHOUT ANY WARRANTY; without even the implied warranty of
 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 See the GNU General Public License for more details.
 .
 You should have received a copy of the GNU General Public License along
 with this program. If not, see <http://www.gnu.org/licenses/>.
 .
 As a special exception, the respective Autoconf Macro's copyright owner
 gives unlimited permission to copy, distribute and modify the configure
 scripts that are the output of Autoconf when processing the Macro. You
 need not follow the terms of the GNU General Public License when using
 or distributing such scripts, even though portions of the text of the
 Macro appear in them. The GNU General Public License (GPL) does govern
 all other use of the material that constitutes the Autoconf Macro.
 .
 This special exception to the GPL applies to versions of the Autoconf
 Macro released by the Autoconf Archive. When you make and distribute a
 modified version of the Autoconf Macro, you may extend this special
 exception to the GPL to apply to your modified version as well.

build-aux/m4/l_atomic.m4 comes from Filezilla which is GPLv2. This is a problem, which I will file another ticket for.

This comment has been minimized.

@daira

daira Dec 22, 2017

Contributor

That other ticket is #2827.

#serial 23
AC_DEFUN([AX_BOOST_REGEX],
[
AC_ARG_WITH([boost-regex],
AS_HELP_STRING([--with-boost-regex@<:@=special-lib@:>@],
[use the Regex library from boost - it is possible to specify a certain library for the linker
e.g. --with-boost-regex=boost_regex-gcc-mt-d-1_33_1 ]),
[
if test "$withval" = "no"; then
want_boost="no"
elif test "$withval" = "yes"; then
want_boost="yes"
ax_boost_user_regex_lib=""
else
want_boost="yes"
ax_boost_user_regex_lib="$withval"
fi
],
[want_boost="yes"]
)
if test "x$want_boost" = "xyes"; then
AC_REQUIRE([AC_PROG_CC])
CPPFLAGS_SAVED="$CPPFLAGS"
CPPFLAGS="$CPPFLAGS $BOOST_CPPFLAGS"
export CPPFLAGS
LDFLAGS_SAVED="$LDFLAGS"
LDFLAGS="$LDFLAGS $BOOST_LDFLAGS"
export LDFLAGS
AC_CACHE_CHECK(whether the Boost::Regex library is available,
ax_cv_boost_regex,
[AC_LANG_PUSH([C++])
AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[@%:@include <boost/regex.hpp>
]],
[[boost::regex r(); return 0;]])],
ax_cv_boost_regex=yes, ax_cv_boost_regex=no)
AC_LANG_POP([C++])
])
if test "x$ax_cv_boost_regex" = "xyes"; then
AC_DEFINE(HAVE_BOOST_REGEX,,[define if the Boost::Regex library is available])
BOOSTLIBDIR=`echo $BOOST_LDFLAGS | sed -e 's/@<:@^\/@:>@*//'`
if test "x$ax_boost_user_regex_lib" = "x"; then
for libextension in `ls $BOOSTLIBDIR/libboost_regex*.so* $BOOSTLIBDIR/libboost_regex*.dylib* $BOOSTLIBDIR/libboost_regex*.a* 2>/dev/null | sed 's,.*/,,' | sed -e 's;^lib\(boost_regex.*\)\.so.*$;\1;' -e 's;^lib\(boost_regex.*\)\.dylib.*;\1;' -e 's;^lib\(boost_regex.*\)\.a.*$;\1;'` ; do
ax_lib=${libextension}
AC_CHECK_LIB($ax_lib, exit,
[BOOST_REGEX_LIB="-l$ax_lib"; AC_SUBST(BOOST_REGEX_LIB) link_regex="yes"; break],
[link_regex="no"])
done
if test "x$link_regex" != "xyes"; then
for libextension in `ls $BOOSTLIBDIR/boost_regex*.dll* $BOOSTLIBDIR/boost_regex*.a* 2>/dev/null | sed 's,.*/,,' | sed -e 's;^\(boost_regex.*\)\.dll.*$;\1;' -e 's;^\(boost_regex.*\)\.a.*$;\1;'` ; do
ax_lib=${libextension}
AC_CHECK_LIB($ax_lib, exit,
[BOOST_REGEX_LIB="-l$ax_lib"; AC_SUBST(BOOST_REGEX_LIB) link_regex="yes"; break],
[link_regex="no"])
done
fi
else
for ax_lib in $ax_boost_user_regex_lib boost_regex-$ax_boost_user_regex_lib; do
AC_CHECK_LIB($ax_lib, main,
[BOOST_REGEX_LIB="-l$ax_lib"; AC_SUBST(BOOST_REGEX_LIB) link_regex="yes"; break],
[link_regex="no"])
done
fi
if test "x$ax_lib" = "x"; then
AC_MSG_ERROR(Could not find a version of the Boost::Regex library!)
fi
if test "x$link_regex" != "xyes"; then
AC_MSG_ERROR(Could not link against $ax_lib !)
fi
fi
CPPFLAGS="$CPPFLAGS_SAVED"
LDFLAGS="$LDFLAGS_SAVED"
fi
])
View
@@ -149,6 +149,12 @@ AC_ARG_ENABLE([zmq],
[use_zmq=$enableval],
[use_zmq=yes])
AC_ARG_ENABLE([proving-service],
[AS_HELP_STRING([--disable-proving-service],
[disable proving service support in daemon])],
[use_proving_service=$enableval],
[use_proving_service=yes])
AC_ARG_WITH([protoc-bindir],[AS_HELP_STRING([--with-protoc-bindir=BIN_DIR],[specify protoc bin path])], [protoc_bin_path=$withval], [])
AC_ARG_ENABLE(man,
@@ -228,6 +234,12 @@ AC_ARG_WITH([daemon],
[build_bitcoind=$withval],
[build_bitcoind=yes])
AC_ARG_WITH([proving-service-daemon],
[AS_HELP_STRING([--with-proving-service-daemon],
[build zcash-proving-service (default=no)])],
[build_zcash_proving_service=$withval],
[build_zcash_proving_service=no])
use_pkgconfig=yes
case $host in
*mingw*)
@@ -574,6 +586,7 @@ AX_BOOST_FILESYSTEM
AX_BOOST_PROGRAM_OPTIONS
AX_BOOST_THREAD
AX_BOOST_CHRONO
AX_BOOST_REGEX
fi
@@ -615,7 +628,7 @@ if test x$use_tests = xyes; then
fi
if test x$use_boost = xyes; then
BOOST_LIBS="$BOOST_LDFLAGS $BOOST_SYSTEM_LIB $BOOST_FILESYSTEM_LIB $BOOST_PROGRAM_OPTIONS_LIB $BOOST_THREAD_LIB $BOOST_CHRONO_LIB"
BOOST_LIBS="$BOOST_LDFLAGS $BOOST_SYSTEM_LIB $BOOST_FILESYSTEM_LIB $BOOST_PROGRAM_OPTIONS_LIB $BOOST_THREAD_LIB $BOOST_CHRONO_LIB $BOOST_REGEX_LIB"
fi
if test x$use_pkgconfig = xyes; then
@@ -693,6 +706,19 @@ AC_CHECK_LIB([gmp],[[__gmpn_sub_n]],GMP_LIBS=-lgmp, [AC_MSG_ERROR(libgmp missing
AC_CHECK_HEADER([gmpxx.h],,AC_MSG_ERROR(libgmpxx headers missing))
AC_CHECK_LIB([gmpxx],[main],GMPXX_LIBS=-lgmpxx, [AC_MSG_ERROR(libgmpxx missing)])
if test x$use_proving_service = xyes; then
if test "x$use_zmq" = "xyes"; then
AC_DEFINE([ENABLE_PROVING_SERVICE],[1],[Define to 1 to enable proving service functions])
else
AC_MSG_WARN([zmq not enabled, disabling proving service support]);
use_proving_service=no
AC_DEFINE([ENABLE_PROVING_SERVICE],[0],[Define to 1 to enable proving service functions])
fi
else
AC_DEFINE_UNQUOTED([ENABLE_PROVING_SERVICE],[0],[Define to 1 to enable proving service functions])
fi
AM_CONDITIONAL([ENABLE_PROVING_SERVICE], [test x$use_proving_service = xyes])
RUST_LIBS=""
if test x$enable_rust != xno; then
RUST_LIBS="-lrustzcash"
@@ -739,6 +765,20 @@ fi
AC_MSG_RESULT($build_bitcoin_libs)
AC_MSG_CHECKING([whether to build zcash-proving-service])
if test x$build_zcash_proving_service = xyes; then
if test "x$use_zmq" = "xyes"; then
AC_MSG_RESULT([yes])
BUILD_ZCASH_PROVING_SERVICE="yes"
else
AC_MSG_ERROR([zcash-proving-service requires ZMQ.]);
fi
else
AC_MSG_RESULT([no])
BUILD_ZCASH_PROVING_SERVICE=""
fi
AM_CONDITIONAL([BUILD_ZCASH_PROVING_SERVICE], [test x$BUILD_ZCASH_PROVING_SERVICE = xyes])
AC_LANG_POP
if test "x$use_ccache" != "xno"; then
View
@@ -0,0 +1,25 @@
package=azmq
$(package)_version=1.0
$(package)_download_path=https://github.com/zeromq/$(package)/archive/
$(package)_file_name=$(package)-$($(package)_version).tar.gz
$(package)_download_file=v$($(package)_version).tar.gz
$(package)_sha256_hash=c204c731bcb7810ca3a2c5515e88974ef2ff8d0589e60a897dc238b369180e7b

This comment has been minimized.

@bitcartel

bitcartel Dec 21, 2017

Contributor

latest version?

This comment has been minimized.

@str4d

str4d Dec 21, 2017

Contributor

Looks like it, although there have been numerous commits to master since then.

This comment has been minimized.

@daira

daira Dec 22, 2017

Contributor

I may be misinterpreting, but this recently fixed ticket looks like a memory safety bug: zeromq/azmq#111

Not to worry, that was before the 1.0 release.

This comment has been minimized.

@daira

daira Dec 22, 2017

Contributor

zeromq/azmq#129 is a reliability bug.

$(package)_dependencies=boost zeromq
$(package)_patches=libsodium.patch
define $(package)_preprocess_cmds
patch -p1 < $($(package)_patch_dir)/libsodium.patch && \
mkdir build
endef
define $(package)_config_cmds
cd build; cmake .. -DCMAKE_INSTALL_PREFIX=/ -DBOOST_ROOT=$(host_prefix) -DZMQ_ROOT=$(host_prefix)
endef
define $(package)_build_cmds
cd build; $(MAKE) VERBOSE=1
endef
define $(package)_stage_cmds
cd build; $(MAKE) VERBOSE=1 DESTDIR=$($(package)_staging_prefix_dir) install
endef
@@ -20,7 +20,7 @@ $(package)_toolset_$(host_os)=gcc
$(package)_archiver_$(host_os)=$($(package)_ar)
$(package)_toolset_darwin=darwin
$(package)_archiver_darwin=$($(package)_libtool)
$(package)_config_libraries=chrono,filesystem,program_options,system,thread,test
$(package)_config_libraries=chrono,filesystem,program_options,system,thread,test,date_time,log,regex,random
$(package)_cxxflags=-std=c++11 -fvisibility=hidden
$(package)_cxxflags_linux=-fPIC
endef
View
@@ -0,0 +1,12 @@
package=cppzmq
$(package)_version=4.2.1

This comment has been minimized.

@bitcartel

This comment has been minimized.

@str4d

str4d Dec 21, 2017

Contributor

ACK, will add a commit to bump this dependency (and zeromq too, which also just release 4.2.3, so I'll try that and see if they are compatible, or otherwise just use 4.2.2).

$(package)_download_path=https://github.com/zeromq/$(package)/archive/
$(package)_file_name=$(package)-$($(package)_version).tar.gz
$(package)_download_file=v$($(package)_version).tar.gz
$(package)_sha256_hash=11c699001659336c7d46779f714f3e9d15d63343cd2ae7c1905e4bf58907cef9
$(package)_dependencies=zeromq
define $(package)_stage_cmds
mkdir $($(package)_staging_dir)$(host_prefix)/include/ && \
cp zmq.hpp $($(package)_staging_dir)$(host_prefix)/include/
endef
@@ -1,6 +1,6 @@
rust_packages := rust librustzcash
proton_packages := proton
zcash_packages := libgmp libsodium
zcash_packages := azmq cppzmq libgmp libsodium
packages := boost openssl libevent zeromq $(zcash_packages) googletest
native_packages := native_ccache
@@ -3,9 +3,10 @@ $(package)_version=4.2.1
$(package)_download_path=https://github.com/zeromq/libzmq/releases/download/v$($(package)_version)/
$(package)_file_name=$(package)-$($(package)_version).tar.gz
$(package)_sha256_hash=27d1e82a099228ee85a7ddb2260f40830212402c605a4a10b5e5498a7e0e9d03
$(package)_dependencies=libsodium
define $(package)_set_vars
$(package)_config_opts=--without-documentation --disable-shared --disable-curve
$(package)_config_opts=--without-documentation --disable-shared --with-libsodium

This comment has been minimized.

@arielgabizon

arielgabizon Dec 7, 2017

Contributor

can you explain this change?

This comment has been minimized.

@str4d

str4d Dec 16, 2017

Contributor

Removing --disable-curve enables CurveMQ, ZeroMQ's transport-layer encryption. It can optionally use libsodium to implement the primitives, which I am choosing to do here given that we already depend on it.

This comment has been minimized.

@arielgabizon

arielgabizon Dec 19, 2017

Contributor

why were we disabling CurveMQ before?

This comment has been minimized.

@str4d

str4d Dec 19, 2017

Contributor

We inherited that from Bitcoin Core when we pulled in the ZMQ notification API, presumably because they weren't using it (and so we'd likely have set the same options, like we do with OpenSSL, to reduce the code we are depending on).

This comment has been minimized.

@daira

daira Dec 19, 2017

Contributor

Does this affect compatibility with any clients that were previously using ZMQ without authentication?

This comment has been minimized.

@str4d

str4d Dec 20, 2017

Contributor

No - the server has to explicitly enable CurveMQ, and this PR does not do that for the ZMQ notifications API.

$(package)_config_opts_linux=--with-pic
$(package)_cxxflags=-std=c++11
endef
@@ -0,0 +1,22 @@
diff -ur azmq-1.0-orig/CMakeLists.txt azmq-1.0/CMakeLists.txt
--- azmq-1.0-orig/CMakeLists.txt 2017-09-25 12:08:31.487969892 +0100
+++ azmq-1.0/CMakeLists.txt 2017-09-25 12:13:45.449318390 +0100
@@ -49,18 +49,5 @@
${ZeroMQ_INCLUDE_DIRS}
${PROJECT_SOURCE_DIR})
-enable_testing()
-
-macro(add_catch_test name)
- if (TEST_REPORT_FORMAT)
- add_test(NAME ${name} COMMAND ${name} -r ${TEST_REPORT_FORMAT} -o "${name}.test_out.xml")
- else()
- add_test(NAME ${name} COMMAND ${name})
- endif()
-endmacro()
-
-add_subdirectory(test)
-add_subdirectory(doc)
-
install(DIRECTORY ${PROJECT_SOURCE_DIR}/azmq
DESTINATION include)
View
@@ -112,6 +112,10 @@ LIBZCASH_H = \
zcash/util.h \
zcash/Zcash.h
if ENABLE_PROVING_SERVICE
LIBZCASH_H += zcash/ProvingServiceClient.hpp
endif
.PHONY: FORCE collate-libsnark check-symbols check-security
# bitcoin core #
BITCOIN_CORE_H = \
@@ -527,6 +531,10 @@ libzcash_a_SOURCES = \
zcash/circuit/prfs.tcc \
zcash/circuit/utils.tcc
if ENABLE_PROVING_SERVICE
libzcash_a_SOURCES += zcash/ProvingServiceClient.cpp
endif
libzcash_a_CPPFLAGS = -DMULTICORE -fopenmp -fPIC -DBINARY_OUTPUT -DCURVE_ALT_BN128 -DBOOST_SPIRIT_THREADSAFE -DHAVE_BUILD_INFO -D__STDC_FORMAT_MACROS $(HARDENED_CPPFLAGS) -pipe -O1 -g -Wstack-protector -fstack-protector-all -fPIE -fvisibility=hidden -DSTATIC $(BITCOIN_INCLUDES)
libzcash_a_CXXFLAGS = $(HARDENED_CXXFLAGS) -fwrapv -fno-strict-aliasing
View
@@ -2,6 +2,42 @@ noinst_PROGRAMS += \
zcash/GenerateParams \
zcash/CreateJoinSplit
if BUILD_ZCASH_PROVING_SERVICE
noinst_PROGRAMS += zcash-proving-service
# proving service daemon
zcash_proving_service_SOURCES = zcash-proving-service.cpp
zcash_proving_service_CPPFLAGS = $(AM_CPPFLAGS) $(BITCOIN_INCLUDES)
zcash_proving_service_CXXFLAGS = $(AM_CXXFLAGS) $(PIE_FLAGS)
zcash_proving_service_LDADD = \
$(LIBBITCOIN_COMMON) \
$(LIBZCASH) \
$(LIBSNARK) \
$(LIBBITCOIN_UTIL) \
$(LIBBITCOIN_CRYPTO) \
$(BOOST_LIBS) \
$(ZMQ_LIBS) \
$(LIBZCASH_LIBS)
endif
if ENABLE_PROVING_SERVICE
noinst_PROGRAMS += zcash/ExampleProvingServiceClient
# example client for proving service
zcash_ExampleProvingServiceClient_SOURCES = zcash/ExampleProvingServiceClient.cpp
zcash_ExampleProvingServiceClient_CPPFLAGS = $(AM_CPPFLAGS) $(BITCOIN_INCLUDES)
zcash_ExampleProvingServiceClient_CXXFLAGS = $(AM_CXXFLAGS) $(PIE_FLAGS)
zcash_ExampleProvingServiceClient_LDADD = \
$(LIBBITCOIN_COMMON) \
$(LIBZCASH) \
$(LIBSNARK) \
$(LIBBITCOIN_UTIL) \
$(LIBBITCOIN_CRYPTO) \
$(BOOST_LIBS) \
$(ZMQ_LIBS) \
$(LIBZCASH_LIBS)
endif
# tool for generating our public parameters
zcash_GenerateParams_SOURCES = zcash/GenerateParams.cpp
zcash_GenerateParams_CPPFLAGS = $(AM_CPPFLAGS)
@@ -22,7 +22,7 @@ JSDescription::JSDescription(ZCJoinSplit& params,
{
boost::array<libzcash::Note, ZC_NUM_JS_OUTPUTS> notes;
auto witness = params.witness(
witness = params.witness(

This comment has been minimized.

@daira

daira Dec 22, 2017

Contributor

Does this change make the witness stay in memory longer (than needed)?

inputs,
outputs,
notes,
@@ -68,6 +68,9 @@ class JSDescription
// This is a zk-SNARK which ensures that this JoinSplit is valid.
libzcash::ZCProof proof;
// In-memory only
ZCJSProofWitness witness;
JSDescription(): vpub_old(0), vpub_new(0) { }
JSDescription(ZCJoinSplit& params,
Oops, something went wrong.
ProTip! Use n and p to navigate between commits in a pull request.