Regression Tests: Migrated rpc-tests.sh to all Python rpc-tests.py #6616

Merged
merged 1 commit into from Oct 1, 2015

Conversation

Projects
None yet
8 participants
@ptschip
Contributor

ptschip commented Sep 1, 2015

  1. created rpc-tests.py
  2. deleted rpc-tests.sh
  3. deleted tests-config.sh.in
  4. travis.yml points to rpc-tests.py
@ptschip

This comment has been minimized.

Show comment
Hide comment
@ptschip

ptschip Sep 1, 2015

Contributor

I'll do that

I was wondering if we should move that README.md up to the /qa folder
...I think it would be more visible there. I didn't even know it
existed until you just mentioned it.

On 01/09/2015 2:47 PM, MarcoFalke wrote:

Don't forget to adjust
https://github.com/ptschip/bitcoin/blob/all_python/qa/rpc-tests/README.md#notes
and https://github.com/ptschip/bitcoin/blob/all_python/README.md as well


Reply to this email directly or view it on GitHub
#6616 (comment).

Contributor

ptschip commented Sep 1, 2015

I'll do that

I was wondering if we should move that README.md up to the /qa folder
...I think it would be more visible there. I didn't even know it
existed until you just mentioned it.

On 01/09/2015 2:47 PM, MarcoFalke wrote:

Don't forget to adjust
https://github.com/ptschip/bitcoin/blob/all_python/qa/rpc-tests/README.md#notes
and https://github.com/ptschip/bitcoin/blob/all_python/README.md as well


Reply to this email directly or view it on GitHub
#6616 (comment).

@theuni

This comment has been minimized.

Show comment
Hide comment
@theuni

theuni Sep 2, 2015

Member

The hard-coded values are a major step backwards. File existence is a very bad metric considering that many of us often cross-compile. Also, anything that requires a certain PWD is inflexible.

Please re-add a pre-processed config file and import it from rpc-tests.py.

Member

theuni commented Sep 2, 2015

The hard-coded values are a major step backwards. File existence is a very bad metric considering that many of us often cross-compile. Also, anything that requires a certain PWD is inflexible.

Please re-add a pre-processed config file and import it from rpc-tests.py.

@theuni

This comment has been minimized.

Show comment
Hide comment
@theuni

theuni Sep 2, 2015

Member

@ptschip like this:

diff --git a/configure.ac b/configure.ac
index a524bde..c0a4e54 100644
--- a/configure.ac
+++ b/configure.ac
@@ -878,7 +878,7 @@ AC_SUBST(MINIUPNPC_CPPFLAGS)
 AC_SUBST(MINIUPNPC_LIBS)
 AC_CONFIG_FILES([Makefile src/Makefile share/setup.nsi share/qt/Info.plist src/test/buildenv.py])
 AC_CONFIG_FILES([qa/pull-tester/run-bitcoind-for-test.sh],[chmod +x qa/pull-tester/run-bitcoind-for-test.sh])
-AC_CONFIG_FILES([qa/pull-tester/tests-config.sh],[chmod +x qa/pull-tester/tests-config.sh])
+AC_CONFIG_FILES([qa/rpc-tests/tests_config.py],[chmod +x qa/rpc-tests/tests_config.py])

 dnl boost's m4 checks do something really nasty: they export these vars. As a
 dnl result, they leak into secp256k1's configure and crazy things happen.
diff --git a/qa/rpc-tests/tests_config.py.in b/qa/rpc-tests/tests_config.py.in
new file mode 100644
index 0000000..adc7860
--- /dev/null
+++ b/qa/rpc-tests/tests_config.py.in
@@ -0,0 +1,11 @@
+#!/usr/bin/env python2
+BUILDDIR="@abs_top_builddir@"
+EXEEXT="@EXEEXT@"
+
+# These will turn into comments if they were disabled when configuring.
+@ENABLE_WALLET_TRUE@ENABLE_WALLET=1
+@BUILD_BITCOIN_UTILS_TRUE@ENABLE_UTILS=1
+@BUILD_BITCOIND_TRUE@ENABLE_BITCOIND=1
+
+REAL_BITCOIND=BUILDDIR + '/src/bitcoind' + EXEEXT
+REAL_BITCOINCLI=BUILDDIR + '/src/bitcoin-cli' + EXEEXT

Then in the migrated script you can use it like:

#!/usr/bin/env python2
import tests_config

print tests_config.REAL_BITCOINCLI
Member

theuni commented Sep 2, 2015

@ptschip like this:

diff --git a/configure.ac b/configure.ac
index a524bde..c0a4e54 100644
--- a/configure.ac
+++ b/configure.ac
@@ -878,7 +878,7 @@ AC_SUBST(MINIUPNPC_CPPFLAGS)
 AC_SUBST(MINIUPNPC_LIBS)
 AC_CONFIG_FILES([Makefile src/Makefile share/setup.nsi share/qt/Info.plist src/test/buildenv.py])
 AC_CONFIG_FILES([qa/pull-tester/run-bitcoind-for-test.sh],[chmod +x qa/pull-tester/run-bitcoind-for-test.sh])
-AC_CONFIG_FILES([qa/pull-tester/tests-config.sh],[chmod +x qa/pull-tester/tests-config.sh])
+AC_CONFIG_FILES([qa/rpc-tests/tests_config.py],[chmod +x qa/rpc-tests/tests_config.py])

 dnl boost's m4 checks do something really nasty: they export these vars. As a
 dnl result, they leak into secp256k1's configure and crazy things happen.
diff --git a/qa/rpc-tests/tests_config.py.in b/qa/rpc-tests/tests_config.py.in
new file mode 100644
index 0000000..adc7860
--- /dev/null
+++ b/qa/rpc-tests/tests_config.py.in
@@ -0,0 +1,11 @@
+#!/usr/bin/env python2
+BUILDDIR="@abs_top_builddir@"
+EXEEXT="@EXEEXT@"
+
+# These will turn into comments if they were disabled when configuring.
+@ENABLE_WALLET_TRUE@ENABLE_WALLET=1
+@BUILD_BITCOIN_UTILS_TRUE@ENABLE_UTILS=1
+@BUILD_BITCOIND_TRUE@ENABLE_BITCOIND=1
+
+REAL_BITCOIND=BUILDDIR + '/src/bitcoind' + EXEEXT
+REAL_BITCOINCLI=BUILDDIR + '/src/bitcoin-cli' + EXEEXT

Then in the migrated script you can use it like:

#!/usr/bin/env python2
import tests_config

print tests_config.REAL_BITCOINCLI
@ptschip

This comment has been minimized.

Show comment
Hide comment
@ptschip

ptschip Sep 2, 2015

Contributor

duh, oh yeah, that's good, i was wondering how to get rid of
tests.config.sh..

On 02/09/2015 10:09 AM, Cory Fields wrote:

@ptschip https://github.com/ptschip like this:

diff --git a/configure.ac b/configure.ac
index a524bde..c0a4e54 100644
--- a/configure.ac
+++ b/configure.ac
@@ -878,7 +878,7 @@ AC_SUBST(MINIUPNPC_CPPFLAGS)
AC_SUBST(MINIUPNPC_LIBS)
AC_CONFIG_FILES([Makefile src/Makefile share/setup.nsi share/qt/Info.plist src/test/buildenv.py])
AC_CONFIG_FILES([qa/pull-tester/run-bitcoind-for-test.sh],[chmod +x qa/pull-tester/run-bitcoind-for-test.sh])
-AC_CONFIG_FILES([qa/pull-tester/tests-config.sh],[chmod +x
qa/pull-tester/tests-config.sh])
+AC_CONFIG_FILES([qa/rpc-tests/tests_config.py],[chmod +x
qa/rpc-tests/tests_config.py])

dnl boost's m4 checks do something really nasty: they export these vars. As a
dnl result, they leak into secp256k1's configure and crazy things happen.
diff --git a/qa/rpc-tests/tests_config.py.in
b/qa/rpc-tests/tests_config.py.in
new file mode 100644
index 0000000..adc7860
--- /dev/null
+++ b/qa/rpc-tests/tests_config.py.in
@@ -0,0 +1,11 @@
+#!/usr/bin/env python2
+BUILDDIR="@abs_top_builddir@"
+EXEEXT="@EXEEXT@"
+
+# These will turn into comments if they were disabled when configuring.
+@ENABLE_WALLET_TRUE@ENABLE_WALLET=1
+@BUILD_BITCOIN_UTILS_TRUE@ENABLE_UTILS=1
+@BUILD_BITCOIND_TRUE@ENABLE_BITCOIND=1
+
+REAL_BITCOIND=BUILDDIR + '/src/bitcoind' + EXEEXT
+REAL_BITCOINCLI=BUILDDIR + '/src/bitcoin-cli' + EXEEXT

Then in the migrated script you can use it like:

#!/usr/bin/env python2
import tests_config

print tests_config.REAL_BITCOINCLI


Reply to this email directly or view it on GitHub
#6616 (comment).

Contributor

ptschip commented Sep 2, 2015

duh, oh yeah, that's good, i was wondering how to get rid of
tests.config.sh..

On 02/09/2015 10:09 AM, Cory Fields wrote:

@ptschip https://github.com/ptschip like this:

diff --git a/configure.ac b/configure.ac
index a524bde..c0a4e54 100644
--- a/configure.ac
+++ b/configure.ac
@@ -878,7 +878,7 @@ AC_SUBST(MINIUPNPC_CPPFLAGS)
AC_SUBST(MINIUPNPC_LIBS)
AC_CONFIG_FILES([Makefile src/Makefile share/setup.nsi share/qt/Info.plist src/test/buildenv.py])
AC_CONFIG_FILES([qa/pull-tester/run-bitcoind-for-test.sh],[chmod +x qa/pull-tester/run-bitcoind-for-test.sh])
-AC_CONFIG_FILES([qa/pull-tester/tests-config.sh],[chmod +x
qa/pull-tester/tests-config.sh])
+AC_CONFIG_FILES([qa/rpc-tests/tests_config.py],[chmod +x
qa/rpc-tests/tests_config.py])

dnl boost's m4 checks do something really nasty: they export these vars. As a
dnl result, they leak into secp256k1's configure and crazy things happen.
diff --git a/qa/rpc-tests/tests_config.py.in
b/qa/rpc-tests/tests_config.py.in
new file mode 100644
index 0000000..adc7860
--- /dev/null
+++ b/qa/rpc-tests/tests_config.py.in
@@ -0,0 +1,11 @@
+#!/usr/bin/env python2
+BUILDDIR="@abs_top_builddir@"
+EXEEXT="@EXEEXT@"
+
+# These will turn into comments if they were disabled when configuring.
+@ENABLE_WALLET_TRUE@ENABLE_WALLET=1
+@BUILD_BITCOIN_UTILS_TRUE@ENABLE_UTILS=1
+@BUILD_BITCOIND_TRUE@ENABLE_BITCOIND=1
+
+REAL_BITCOIND=BUILDDIR + '/src/bitcoind' + EXEEXT
+REAL_BITCOINCLI=BUILDDIR + '/src/bitcoin-cli' + EXEEXT

Then in the migrated script you can use it like:

#!/usr/bin/env python2
import tests_config

print tests_config.REAL_BITCOINCLI


Reply to this email directly or view it on GitHub
#6616 (comment).

@ptschip

This comment has been minimized.

Show comment
Hide comment
@ptschip

ptschip Sep 3, 2015

Contributor

@theuni switched over to tests_config.py and imported the vars directly and the tests run fine on Travis. However the BUILDDIR is not compatible with native windows (ie /c/bitcoin) and is not handled well by python so I had to keep the scripting that finds the builddir. Therefore there are a few options moving forward: 1) Keep the scripting that finds the builddir...I made changes so that it uses the system path rather than the os.path so we don't have to be in the pull-tester folder to run the scripts. (2) I can do an os specific hack to normalize the builddir for native windows (3) if we could somehow find a way to put the native os path into the tests_config.py directly but I don't know how...I'll research this a little more tomorrow, any suggestions are welcome...

Contributor

ptschip commented Sep 3, 2015

@theuni switched over to tests_config.py and imported the vars directly and the tests run fine on Travis. However the BUILDDIR is not compatible with native windows (ie /c/bitcoin) and is not handled well by python so I had to keep the scripting that finds the builddir. Therefore there are a few options moving forward: 1) Keep the scripting that finds the builddir...I made changes so that it uses the system path rather than the os.path so we don't have to be in the pull-tester folder to run the scripts. (2) I can do an os specific hack to normalize the builddir for native windows (3) if we could somehow find a way to put the native os path into the tests_config.py directly but I don't know how...I'll research this a little more tomorrow, any suggestions are welcome...

@ptschip

This comment has been minimized.

Show comment
Hide comment
@ptschip

ptschip Sep 3, 2015

Contributor

@theuni The path for the builddir is now correct for each system. I put a few lines in rpc-tests.py to create the correct path if running in a mingw shell.

I looked into putting it further upstream, in configure.ac when tests_config.py.in gets called, but I don't see we can override the @abs_top_builddir@ with our own function and I'm not sure what we would gain.

Contributor

ptschip commented Sep 3, 2015

@theuni The path for the builddir is now correct for each system. I put a few lines in rpc-tests.py to create the correct path if running in a mingw shell.

I looked into putting it further upstream, in configure.ac when tests_config.py.in gets called, but I don't see we can override the @abs_top_builddir@ with our own function and I'm not sure what we would gain.

@@ -37,9 +37,13 @@ Helper functions for creating blocks and transactions.
Notes
=====
-You can run a single test by calling `qa/pull-tester/rpc-tests.sh <testname>`.
+You can run any single test by calling qa/pull-tester/rpc-tests.py <testname>

This comment has been minimized.

@MarcoFalke

MarcoFalke Sep 3, 2015

Member

Make sure to get the backticks ` right. (L44 as well)

Also, https://github.com/ptschip/bitcoin/blob/all_python/README.md#automated-testing needs update?

@MarcoFalke

MarcoFalke Sep 3, 2015

Member

Make sure to get the backticks ` right. (L44 as well)

Also, https://github.com/ptschip/bitcoin/blob/all_python/README.md#automated-testing needs update?

This comment has been minimized.

@ptschip

ptschip Sep 3, 2015

Contributor

thanks, done both

On 03/09/2015 11:08 AM, MarcoFalke wrote:

In qa/rpc-tests/README.md
#6616 (comment):

@@ -37,9 +37,13 @@ Helper functions for creating blocks and transactions.
Notes

-You can run a single test by calling qa/pull-tester/rpc-tests.sh <testname>.
+You can run any single test by calling qa/pull-tester/rpc-tests.py

Make sure to get the backticks ` right. (L44 as well)

Also,
https://github.com/ptschip/bitcoin/blob/all_python/README.md#automated-testing
needs update?


Reply to this email directly or view it on GitHub
https://github.com/bitcoin/bitcoin/pull/6616/files#r38677104.

@ptschip

ptschip Sep 3, 2015

Contributor

thanks, done both

On 03/09/2015 11:08 AM, MarcoFalke wrote:

In qa/rpc-tests/README.md
#6616 (comment):

@@ -37,9 +37,13 @@ Helper functions for creating blocks and transactions.
Notes

-You can run a single test by calling qa/pull-tester/rpc-tests.sh <testname>.
+You can run any single test by calling qa/pull-tester/rpc-tests.py

Make sure to get the backticks ` right. (L44 as well)

Also,
https://github.com/ptschip/bitcoin/blob/all_python/README.md#automated-testing
needs update?


Reply to this email directly or view it on GitHub
https://github.com/bitcoin/bitcoin/pull/6616/files#r38677104.

@theuni

This comment has been minimized.

Show comment
Hide comment
@theuni

theuni Sep 4, 2015

Member

@ptschip I'll be away for a few days, will review when I'm back. There are a few PRs going on parallel to this though (libevent httpd, zmq, etc) that may make sense before merging this anyway, since those things will conflict.

Member

theuni commented Sep 4, 2015

@ptschip I'll be away for a few days, will review when I'm back. There are a few PRs going on parallel to this though (libevent httpd, zmq, etc) that may make sense before merging this anyway, since those things will conflict.

@ptschip

This comment has been minimized.

Show comment
Hide comment
@ptschip

ptschip Sep 4, 2015

Contributor

@theuni Have a few good days off. I did manage to fix the issue with the path further upstream by adding a script at the end of configure.ac in order to make the correct path edits when running in mingw .

Contributor

ptschip commented Sep 4, 2015

@theuni Have a few good days off. I did manage to fix the issue with the path further upstream by adding a script at the end of configure.ac in order to make the correct path edits when running in mingw .

@MarcoFalke

This comment has been minimized.

Show comment
Hide comment
@MarcoFalke

MarcoFalke Sep 22, 2015

Member

Looks like the last conflict is #6686. I'd say it's safe to wait until #6686 is merged before you rebase.

Member

MarcoFalke commented Sep 22, 2015

Looks like the last conflict is #6686. I'd say it's safe to wait until #6686 is merged before you rebase.

@jgarzik

This comment has been minimized.

Show comment
Hide comment
@jgarzik

jgarzik Sep 22, 2015

Contributor

concept ACK

Contributor

jgarzik commented Sep 22, 2015

concept ACK

@laanwj laanwj added the Tests label Sep 27, 2015

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Sep 27, 2015

Member

tested ACK
(but needs rebase)

Member

laanwj commented Sep 27, 2015

tested ACK
(but needs rebase)

@ptschip

This comment has been minimized.

Show comment
Hide comment
@ptschip

ptschip Oct 1, 2015

Contributor

@laanwj @MarcoFalke I added the zmq tests, rebased and Travis is happy.

Contributor

ptschip commented Oct 1, 2015

@laanwj @MarcoFalke I added the zmq tests, rebased and Travis is happy.

@ptschip

This comment has been minimized.

Show comment
Hide comment
@ptschip

ptschip Oct 1, 2015

Contributor

@laanwj @MarcoFalke just got rid of a little whitespace...should be good after this build finishes.

Contributor

ptschip commented Oct 1, 2015

@laanwj @MarcoFalke just got rid of a little whitespace...should be good after this build finishes.

+ for i in range(len(testScripts)):
+ if (len(opts) == 0 or (len(opts) == 1 and "-win" in opts ) or '-extended' in opts
+ or testScripts[i] in opts or re.sub(".py$", "", testScripts[i]) in opts ):
+ print "Running testscript " + testScripts[i] + "..."

This comment has been minimized.

@laanwj

laanwj Oct 1, 2015

Member

Nit: I think having the testscript name in bold was nice:

print  "Running testscript \033[1m" + testScripts[i] + "\033[0m..." 
@laanwj

laanwj Oct 1, 2015

Member

Nit: I think having the testscript name in bold was nice:

print  "Running testscript \033[1m" + testScripts[i] + "\033[0m..." 

This comment has been minimized.

@ptschip

ptschip Oct 1, 2015

Contributor

I've tried, tried double and triple and quadruple escaping that python
usually needs, and also tried the termcolor module. It doesn't work
from python and in a "mingw" shell. It will probably work on a native
unix shell but I can't test that out as I'm on Windows. Any suggestions
welcome...

On 01/10/2015 11:09 AM, Wladimir J. van der Laan wrote:

In qa/pull-tester/rpc-tests.py
#6616 (comment):

+# 'forknotify.py',

  • 'p2p-acceptblock.py',
  • 'mempool_packages.py',
    +]

+#Enable ZMQ tests
+if ENABLE_ZMQ == 1:

  • testScripts.append('zmq_test.py')

+if(ENABLE_WALLET == 1 and ENABLE_UTILS == 1 and ENABLE_BITCOIND == 1):

  • rpcTestDir = buildDir + '/qa/rpc-tests/'
  • #Run Tests
  • for i in range(len(testScripts)):
  •   if (len(opts) == 0 or (len(opts) == 1 and "-win" in opts ) or '-extended' in opts 
    
  •       or testScripts[i] in opts or  re.sub(".py$", "", testScripts[i]) in opts ):
    
  •        print  "Running testscript " + testScripts[i] + "..." 
    

Nit: I think having the testscript name in bold was nice:

|print "Running testscript \033[1m" + testScripts[i] + "\033[0m..." |


Reply to this email directly or view it on GitHub
https://github.com/bitcoin/bitcoin/pull/6616/files#r40946395.

@ptschip

ptschip Oct 1, 2015

Contributor

I've tried, tried double and triple and quadruple escaping that python
usually needs, and also tried the termcolor module. It doesn't work
from python and in a "mingw" shell. It will probably work on a native
unix shell but I can't test that out as I'm on Windows. Any suggestions
welcome...

On 01/10/2015 11:09 AM, Wladimir J. van der Laan wrote:

In qa/pull-tester/rpc-tests.py
#6616 (comment):

+# 'forknotify.py',

  • 'p2p-acceptblock.py',
  • 'mempool_packages.py',
    +]

+#Enable ZMQ tests
+if ENABLE_ZMQ == 1:

  • testScripts.append('zmq_test.py')

+if(ENABLE_WALLET == 1 and ENABLE_UTILS == 1 and ENABLE_BITCOIND == 1):

  • rpcTestDir = buildDir + '/qa/rpc-tests/'
  • #Run Tests
  • for i in range(len(testScripts)):
  •   if (len(opts) == 0 or (len(opts) == 1 and "-win" in opts ) or '-extended' in opts 
    
  •       or testScripts[i] in opts or  re.sub(".py$", "", testScripts[i]) in opts ):
    
  •        print  "Running testscript " + testScripts[i] + "..." 
    

Nit: I think having the testscript name in bold was nice:

|print "Running testscript \033[1m" + testScripts[i] + "\033[0m..." |


Reply to this email directly or view it on GitHub
https://github.com/bitcoin/bitcoin/pull/6616/files#r40946395.

This comment has been minimized.

@laanwj

laanwj Oct 1, 2015

Member

I've tried, tried double and triple and quadruple escaping that python
usually needs, and also tried the termcolor module. It doesn't work
from python and in a "mingw" shell. It will probably work on a native
unix shell but I can't test that out as I'm on Windows. Any suggestions
welcome...

Ok, no problem, then leave it like this. Escape-code based formatting probably doesn't work on windows at all.
If we want this it needs to be conditional on the terminal type, which is too much bother...

@laanwj

laanwj Oct 1, 2015

Member

I've tried, tried double and triple and quadruple escaping that python
usually needs, and also tried the termcolor module. It doesn't work
from python and in a "mingw" shell. It will probably work on a native
unix shell but I can't test that out as I'm on Windows. Any suggestions
welcome...

Ok, no problem, then leave it like this. Escape-code based formatting probably doesn't work on windows at all.
If we want this it needs to be conditional on the terminal type, which is too much bother...

@laanwj

View changes

qa/pull-tester/rpc-tests.py
+ for i in range(len(testScriptsExt)):
+ if ('-extended' in opts or testScriptsExt[i] in opts
+ or re.sub(".py$", "", testScriptsExt[i]) in opts):
+ print "Running testscript " + testScriptsExt[i] + "..."

This comment has been minimized.

@laanwj

laanwj Oct 1, 2015

Member

Nit: do we still want to call this "2nd level testscript"?

echo -e "Running \033[1m2nd level\033[0m testscript \033[1m${testScriptsExt[$i]}...\033[0m"
@laanwj

laanwj Oct 1, 2015

Member

Nit: do we still want to call this "2nd level testscript"?

echo -e "Running \033[1m2nd level\033[0m testscript \033[1m${testScriptsExt[$i]}...\033[0m"

This comment has been minimized.

@ptschip

ptschip Oct 1, 2015

Contributor

i'll fix that and push.

On 01/10/2015 11:10 AM, Wladimir J. van der Laan wrote:

In qa/pull-tester/rpc-tests.py
#6616 (comment):

  • #Run Tests
  • for i in range(len(testScripts)):
  •   if (len(opts) == 0 or (len(opts) == 1 and "-win" in opts ) or '-extended' in opts 
    
  •       or testScripts[i] in opts or  re.sub(".py$", "", testScripts[i]) in opts ):
    
  •        print  "Running testscript " + testScripts[i] + "..." 
    
  •        subprocess.call(rpcTestDir + testScripts[i] + " --srcdir " + buildDir + '/src ' + passOn,shell=True) 
    
  •   #exit if help is called so we print just one set of instructions
    
  •        p = re.compile(" -h| --help")
    
  •        if p.match(passOn):
    
  •            sys.exit(0)
    
  • #Run Extended Tests
  • for i in range(len(testScriptsExt)):
  •    if ('-extended' in opts or testScriptsExt[i] in opts 
    
  •       or re.sub(".py$", "", testScriptsExt[i]) in opts):
    
  •        print  "Running testscript " + testScriptsExt[i] + "..." 
    

Nit: do we still want to call this "2nd level testscript"?

|echo -e "Running \033[1m2nd level\033[0m testscript
\033[1m${testScriptsExt[$i]}...\033[0m" |


Reply to this email directly or view it on GitHub
https://github.com/bitcoin/bitcoin/pull/6616/files#r40946519.

@ptschip

ptschip Oct 1, 2015

Contributor

i'll fix that and push.

On 01/10/2015 11:10 AM, Wladimir J. van der Laan wrote:

In qa/pull-tester/rpc-tests.py
#6616 (comment):

  • #Run Tests
  • for i in range(len(testScripts)):
  •   if (len(opts) == 0 or (len(opts) == 1 and "-win" in opts ) or '-extended' in opts 
    
  •       or testScripts[i] in opts or  re.sub(".py$", "", testScripts[i]) in opts ):
    
  •        print  "Running testscript " + testScripts[i] + "..." 
    
  •        subprocess.call(rpcTestDir + testScripts[i] + " --srcdir " + buildDir + '/src ' + passOn,shell=True) 
    
  •   #exit if help is called so we print just one set of instructions
    
  •        p = re.compile(" -h| --help")
    
  •        if p.match(passOn):
    
  •            sys.exit(0)
    
  • #Run Extended Tests
  • for i in range(len(testScriptsExt)):
  •    if ('-extended' in opts or testScriptsExt[i] in opts 
    
  •       or re.sub(".py$", "", testScriptsExt[i]) in opts):
    
  •        print  "Running testscript " + testScriptsExt[i] + "..." 
    

Nit: do we still want to call this "2nd level testscript"?

|echo -e "Running \033[1m2nd level\033[0m testscript
\033[1m${testScriptsExt[$i]}...\033[0m" |


Reply to this email directly or view it on GitHub
https://github.com/bitcoin/bitcoin/pull/6616/files#r40946519.

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Oct 1, 2015

Member

No need to fix all the nits in this pull, we can do those later, if you don't get around to them - probably more important to get this merged before any other changes to the test runner.

Member

laanwj commented Oct 1, 2015

No need to fix all the nits in this pull, we can do those later, if you don't get around to them - probably more important to get this merged before any other changes to the test runner.

Migrated rpc-tests.sh to all python rpc-tests.py
1) created rpc-tests.py
2) deleted rpc-tests.sh
3) travis.yml points to rpc-tests.py
4) Modified Makefile.am
5) Updated README.md
6) Added tests_config.py and deleted tests-config.sh
7) Modified configure.ac with script to set correct path in tests_config.py
@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Oct 1, 2015

Member

Concept ACK.
Keeping the bold output would be nice:

This works for me (and should also work on bash mingw?):

old mode 100644
new mode 100755
index f55c5b6..8c7a6b6
--- a/qa/pull-tester/rpc-tests.py
+++ b/qa/pull-tester/rpc-tests.py
@@ -101,7 +101,7 @@ if(ENABLE_WALLET == 1 and ENABLE_UTILS == 1 and ENABLE_BITCOIND == 1):
     for i in range(len(testScripts)):
        if (len(opts) == 0 or (len(opts) == 1 and "-win" in opts ) or '-extended' in opts 
            or testScripts[i] in opts or  re.sub(".py$", "", testScripts[i]) in opts ):
-            print  "Running testscript " + testScripts[i] + "..." 
+            print  "Running testscript \033[1m" + testScripts[i] + "\033[0m..." 
             subprocess.call(rpcTestDir + testScripts[i] + " --srcdir " + buildDir + '/src ' + passOn,shell=True) 
            #exit if help is called so we print just one set of instructions
             p = re.compile(" -h| --help")
Member

jonasschnelli commented Oct 1, 2015

Concept ACK.
Keeping the bold output would be nice:

This works for me (and should also work on bash mingw?):

old mode 100644
new mode 100755
index f55c5b6..8c7a6b6
--- a/qa/pull-tester/rpc-tests.py
+++ b/qa/pull-tester/rpc-tests.py
@@ -101,7 +101,7 @@ if(ENABLE_WALLET == 1 and ENABLE_UTILS == 1 and ENABLE_BITCOIND == 1):
     for i in range(len(testScripts)):
        if (len(opts) == 0 or (len(opts) == 1 and "-win" in opts ) or '-extended' in opts 
            or testScripts[i] in opts or  re.sub(".py$", "", testScripts[i]) in opts ):
-            print  "Running testscript " + testScripts[i] + "..." 
+            print  "Running testscript \033[1m" + testScripts[i] + "\033[0m..." 
             subprocess.call(rpcTestDir + testScripts[i] + " --srcdir " + buildDir + '/src ' + passOn,shell=True) 
            #exit if help is called so we print just one set of instructions
             p = re.compile(" -h| --help")
@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Oct 1, 2015

Member

@jonasschnelli That's what I proposed too, it works on Linux and OpenBSD but it doesn't on Win apparently. (outputing junk) So it would have to be conditional.

Member

laanwj commented Oct 1, 2015

@jonasschnelli That's what I proposed too, it works on Linux and OpenBSD but it doesn't on Win apparently. (outputing junk) So it would have to be conditional.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Oct 1, 2015

Member
+#Set colors
+cOn = cOff = ""
+if (not "-win" in opts):
+    cOn = "\033[1m"
+    cOff = "\033[0m"

Then use something like: print "Running testscript " + cOn + testScripts[i] + cOff + "..."

Member

jonasschnelli commented Oct 1, 2015

+#Set colors
+cOn = cOff = ""
+if (not "-win" in opts):
+    cOn = "\033[1m"
+    cOff = "\033[0m"

Then use something like: print "Running testscript " + cOn + testScripts[i] + cOff + "..."

@ptschip

This comment has been minimized.

Show comment
Hide comment
@ptschip

ptschip Oct 1, 2015

Contributor

@laanwj I fixed the nit regarding the "2nd level" and left the other for another time. Pushed and all is well...

Contributor

ptschip commented Oct 1, 2015

@laanwj I fixed the nit regarding the "2nd level" and left the other for another time. Pushed and all is well...

@laanwj laanwj merged commit 5467820 into bitcoin:master Oct 1, 2015

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

laanwj added a commit that referenced this pull request Oct 1, 2015

Merge pull request #6616
5467820 Migrated rpc-tests.sh to all python rpc-tests.py (ptschip)
@ptschip

This comment has been minimized.

Show comment
Hide comment
@ptschip

ptschip Oct 1, 2015

Contributor

i like the idea, but -win tests will not always be disabled (i hope)...

so it would have to be something operating specific like

+#Set colors
+cOn = cOff = ""
+if (os.name == 'posix'):

  • cOn = '\033[1m'
  • cOff = '\033[0m'

(oddly python needs the double \ for a )

If this is ok, i'll add it and push.

On 01/10/2015 12:32 PM, Jonas Schnelli wrote:

+#Set colors
+cOn = cOff = ""
+if (not "-win" in opts):

  • cOn = "\033[1m"
  • cOff = "\033[0m"

Then use something like: |print "Running testscript " + cOn +
testScripts[i] + cOff + "..."|

Contributor

ptschip commented Oct 1, 2015

i like the idea, but -win tests will not always be disabled (i hope)...

so it would have to be something operating specific like

+#Set colors
+cOn = cOff = ""
+if (os.name == 'posix'):

  • cOn = '\033[1m'
  • cOff = '\033[0m'

(oddly python needs the double \ for a )

If this is ok, i'll add it and push.

On 01/10/2015 12:32 PM, Jonas Schnelli wrote:

+#Set colors
+cOn = cOff = ""
+if (not "-win" in opts):

  • cOn = "\033[1m"
  • cOff = "\033[0m"

Then use something like: |print "Running testscript " + cOn +
testScripts[i] + cOff + "..."|

@btcdrak

This comment has been minimized.

Show comment
Hide comment
@btcdrak

btcdrak Oct 1, 2015

Member

@ptschip This PR is now merged, any new changes would have to be in a new branch + PR

Member

btcdrak commented Oct 1, 2015

@ptschip This PR is now merged, any new changes would have to be in a new branch + PR

@mruddy

This comment has been minimized.

Show comment
Hide comment
@ptschip

This comment has been minimized.

Show comment
Hide comment
@ptschip

ptschip Oct 3, 2015

Contributor

thanks, that one got missed....updated and new PR created.

On 03/10/2015 4:51 AM, mruddy wrote:

might want to update
https://github.com/bitcoin/bitcoin/blob/master/README.md#automated-testing


Reply to this email directly or view it on GitHub
#6616 (comment).

Contributor

ptschip commented Oct 3, 2015

thanks, that one got missed....updated and new PR created.

On 03/10/2015 4:51 AM, mruddy wrote:

might want to update
https://github.com/bitcoin/bitcoin/blob/master/README.md#automated-testing


Reply to this email directly or view it on GitHub
#6616 (comment).

laanwj added a commit to laanwj/bitcoin that referenced this pull request Oct 5, 2015

build: Remove unnecessary chmods after #6616
Don't chmod a repository-included file in the configure script, and
`tests_config.py` is a module that doesn't need to be executable.

laanwj added a commit that referenced this pull request Oct 5, 2015

Merge pull request #6759
ea70997 build: Remove unnecessary chmods after #6616 (Wladimir J. van der Laan)

@ptschip ptschip deleted the ptschip:all_python branch Oct 17, 2015

@dagurval dagurval referenced this pull request in bitcoinxt/bitcoinxt Nov 11, 2016

Merged

Migrate rpc-tests.sh to rpc-tests.py #170

@jnewbery jnewbery referenced this pull request Jan 31, 2017

Merged

Improve rpc-tests.py #9657

@str4d str4d referenced this pull request in zcash/zcash Mar 24, 2017

Merged

libevent-based http server #2176

str4d added a commit to str4d/zcash that referenced this pull request Mar 25, 2017

Revert "Revert "rpc-tests: re-enable rpc-tests for Windows""
This reverts commit 7fd5d4e.

We need to un-revert this before pulling in
bitcoin/bitcoin#6616

joshuayabut added a commit to zencashio/zen that referenced this pull request Apr 24, 2017

V1.0.8-1 (#11)
* Don't assume sizes of unsigned short and unsigned int in GetSizeOfCompactSize and WriteCompactSize.
Fixes #2137

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* Inc num of txs in test mempool

* Closes #1969. Default fee now sufficient for large shielded tx.

Sets default min relay tx fee to 100, which is used to initialize
nSatoshisPerK for min fee calculation.  The default z_sendmany
fee of 10000 is now sufficient to cover a tx of size 100000 bytes,
so txs will not be treated as "free" and hit the free rate limiter.

* Part of #1969. Changing min fee calculation also changes the dust threshold.

* Part of #1969. Update tests to avoid error 'absurdly high fee' from change in min fee calc.

*   Pause mining during joinsplit creation

* add conditional for --enable-man, default is yes

* change help string --enable-man to --disable-man

* regenerated all manpages with commit tag stripped, also add bitcoin-tx

* improved gen-manpages.sh, includes bitcoin-tx and strips commit tag, now also runs binaries from build dir by default, added variables for more control

* Adjust gen-manpages.sh for Zcash, use in Debian builds

* Regenerate and collate Zcash manpages, delete Bitcoin ones

* Update release process with gen-manpages.sh

* rpc: make `gettxoutsettinfo` run lock-free

For leveldb "An iterator operates on a snapshot of the database taken
when the iterator is created". This means that it is unnecessary to
lock out other threads while computing statistics, and neither to hold
cs_main for the whole time. Let the thread run free.

* [REST] add JSON support for /rest/headers/

* fundrawtransaction tests

* Small tweaks to CCoinControl for fundrawtransaction

* Add fundrawtransaction RPC method

* Add FundTransaction method to wallet

Some code stolen from Jonas Schnelli <jonas.schnelli@include7.ch>

* Add DummySignatureCreator which just creates zeroed sigs

* [net] extend core functionallity for ban/unban/listban

* [RPC] add setban/listbanned/clearbanned RPC commands

* [QA] add setban/listbanned/clearbanned tests

* [net] remove unused return type bool from CNode::Ban()

* [RPC] extend setban to allow subnets

* Add getblockheader RPC call

Alternative to getblock that works even when the block itself has been
pruned, returning all available information.

* Return all available information via validateaddress

`"validateaddress"` omits some information, even in cases where is it available.

The primary motivation is to be able to retrieve redeemScripts, after using `"addmultisigaddress"`, when not all keys are available in the keystore, but the redeemScript actually is.

The output of `"validateaddress"` with this commit:

Keys not available:
```js
validateaddress "n4KWZKx349gdMQGgTnZ8W6WfgSwybkGSK3"
{
  "isvalid": true,
  "address": "n4KWZKx349gdMQGgTnZ8W6WfgSwybkGSK3",
  "scriptPubKey": "76a914fa20d564550b105787f7ce3a9ad7fd9a45cd407088ac",
  "ismine": false,
  "iswatchonly": false,
  "isscript": false
}
```
```js
validateaddress "2N2g2H7gjA8a11g1yKBgh5VTqndyvbnWpBK"
{
  "isvalid": true,
  "address": "2N2g2H7gjA8a11g1yKBgh5VTqndyvbnWpBK",
  "scriptPubKey": "a9146769c19a16c9400b908756e19a4d2afb9e9760e187",
  "ismine": false,
  "iswatchonly": false,
  "isscript": true
}
```

After adding the redeemScript:
```js
addmultisigaddress 2 '["02537357B156A33306A7A014A3748631C59DF405B56F11BA4AA4A3CE81501AF095","02F1FB200390E7864EF4450C07B15988179A57C3CF3A878F668E1070CB615749FE"]'
2N2g2H7gjA8a11g1yKBgh5VTqndyvbnWpBK

validateaddress "2N2g2H7gjA8a11g1yKBgh5VTqndyvbnWpBK"
{
  "isvalid": true,
  "address": "2N2g2H7gjA8a11g1yKBgh5VTqndyvbnWpBK",
  "scriptPubKey": "a9146769c19a16c9400b908756e19a4d2afb9e9760e187",
  "ismine": false,
  "iswatchonly": false,
  "isscript": true,
  "script": "multisig",
  "hex": "522102537357b156a33306a7a014a3748631c59df405b56f11ba4aa4a3ce81501af0952102f1fb200390e7864ef4450c07b15988179a57c3cf3a878f668e1070cb615749fe52ae",
  "addresses": [
    "n4KWZKx349gdMQGgTnZ8W6WfgSwybkGSK3",
    "mmSKNtbYYHRrhTLKiok5TuYrGEs4Y2A4k6"
  ],
  "sigsrequired": 2,
  "account": ""
}
```

All keys available:
```js
validateaddress "n4KWZKx349gdMQGgTnZ8W6WfgSwybkGSK3"
{
  "isvalid": true,
  "address": "n4KWZKx349gdMQGgTnZ8W6WfgSwybkGSK3",
  "scriptPubKey": "76a914fa20d564550b105787f7ce3a9ad7fd9a45cd407088ac",
  "ismine": true,
  "iswatchonly": false,
  "isscript": false,
  "pubkey": "02537357b156a33306a7a014a3748631c59df405b56f11ba4aa4a3ce81501af095",
  "iscompressed": true,
  "account": ""
}
```
```js
validateaddress "2N2g2H7gjA8a11g1yKBgh5VTqndyvbnWpBK"
{
  "isvalid": true,
  "address": "2N2g2H7gjA8a11g1yKBgh5VTqndyvbnWpBK",
  "scriptPubKey": "a9146769c19a16c9400b908756e19a4d2afb9e9760e187",
  "ismine": true,
  "iswatchonly": false,
  "isscript": true,
  "script": "multisig",
  "hex": "522102537357b156a33306a7a014a3748631c59df405b56f11ba4aa4a3ce81501af0952102f1fb200390e7864ef4450c07b15988179a57c3cf3a878f668e1070cb615749fe52ae",
  "addresses": [
    "n4KWZKx349gdMQGgTnZ8W6WfgSwybkGSK3",
    "mmSKNtbYYHRrhTLKiok5TuYrGEs4Y2A4k6"
  ],
  "sigsrequired": 2,
  "account": ""
}
```

* Fix univalue handling of \u0000 characters.

Univalue's parsing of \u escape sequences did not handle NUL characters
correctly.  They were, effectively, dropped.  The extended test-case
fails with the old code, and is fixed with this patch.

* Add paytxfee to getwalletinfo, warnings to getnetworkinfo

* New RPC command disconnectnode

* rename json field "bannedtill" to "banned_until"

* setban: rewrite to UniValue, allow absolute bantime

* fix CSubNet comparison operator

* setban: add RPCErrorCode

* add RPC tests for setban & disconnectnode

* test: Move reindex test to standard tests

This test finishes very quickly, so it should be part of the default set
of tests in rpc-tests.

* fix missing lock in CNode::ClearBanned()

* setban: add IPv6 tests

* fix lock issue for QT node diconnect and RPC disconnectnode

Zcash: only RPC disconnectnode

* UniValue: don't escape solidus, keep espacing of reverse solidus

Zcash: The UniValue changes are already merged; this updates the test

* add tests for the decodescript rpc. add mention of the rpc regression tests to the testing seciton of the main readme.

* When processing RPC commands during warmup phase, parse the
request object before returning an error so that id value can
be used in the response.

Prior to this commit, RPC commands sent during Bitcoin's
warmup/startup phase were responded to with a JSON-RPC error
with an id of null, which violated the JSON-RPC 2.0 spec:

id: This member is REQUIRED. It MUST be the same as the value
of the id member in the Request Object. If there was an error
in detecting the id in the Request object (e.g. Parse
error/Invalid Request), it MUST be Null.

* rpc: Remove chain-specific RequireRPCPassword

I've never liked the chain-specific exception to having to set a
password. It gives issues with #6388 which makes it valid to
set no password in every case (as it enables random cookie authentication).

This pull removes the flag, so that all chains are regarded the same.

It also removes the username==password test, which doesn't provide any
substantial extra security.

* [QA] fix possible reorg issue in rawtransaction.py/fundrawtransaction.py RPC test

- added missing mempool sync between block generations

* [QA] remove rawtransactions.py from the extended test list

rawtransactions.py is already in the standard test list

* Adjust blockheaderToJSON() for Zcash block header

* Exempt unspendable transaction outputs from dust checks

Since unspendable outputs can't be spent, there is no threshold at which it would be uneconomic to spend them.

This primarily targets transaction outputs with `OP_RETURN`.

---

Initially based on:

commit 9cf0ae26350033d43d5dd3c95054c0d1b1641eda
Author: zathras-crypto <zathrasc@gmail.com>
Date:   Wed Mar 25 02:04:02 2015 -0700

Changes:

- cherry-picked on top of bitcoin:master
- added RPC test for fundrawtransaction

* rpc: Accept strings in AmountFromValue

Accept strings containing decimal values, in addition to bare values.

Useful from JSON-RPC implementations where it's not possible to have
direct control over the text of numbers (e.g. where numbers are always
doubles), and it's still desired to send an exact value.

This would allow users to post JSON content with numbers encoded like
`{"value": "0.00000001"}` instead of `{"value": 0.00000001}` which some
php/python encoders wrap into 1e-8, or worse.

* [QA] add testcases for parsing strings as values

* univalue: Avoid unnecessary roundtrip through double for numbers

JSON makes no distinction between numbers and reals, and our code
doesn't need to do so either.

This removes VREAL, as well as its specific post-processing in
`UniValue::write`. Non-monetary amounts do not need to be forcibly
formatted with 8 decimals, so the extra roundtrip was unnecessary
(and potentially loses precision).

Zcash: cherry-picked from commit 7650449a6777710cf818d41862626164da0cd412
Left over from 2aee461

* Fix crash in validateaddress with -disablewallet

Fix a null pointer dereference in validateaddress with -disablewallet. Also add a regression testcase.

* Adjust fundrawtransaction RPC test for Zcash

- Enable wallet encryption
- Correct block reward

* BTC -> ZEC in paytxfee RPC docs

Revert before merging bitcoin/bitcoin#6504.

* Update default RPC port in help strings

* Fix typo in listbanned RPC keys

* Re-encode t-addrs in disablewallet.py with Zcash prefixes

* Improve comment explaining purpose of MAX_MONEY constant

* use const references where appropriate

* Reduce checkpoints' effect on consensus.

Instead of only checking height to decide whether to disable script checks,
actually check whether a block is an ancestor of a checkpoint, up to which
headers have been validated. This means that we don't have to prevent
accepting a side branch anymore - it will be safe, just less fast to
do.

We still need to prevent being fed a multitude of low-difficulty headers
filling up our memory. The mechanism for that is unchanged for now: once
a checkpoint is reached with headers, no headers chain branching off before
that point are allowed anymore.

* [init] add -blockversion help and extend -upnp help

* Consensus: Refactor: Separate Consensus::CheckTxInputs and GetSpendHeight in CheckInputs

Zcash:
- Replaces some of the refactoring performed in #1017
- Exposed Consensus::CheckTxInputs for testing

* make CAddrMan::size() return the correct type of size_t

* Improve proxy initialization

Simplify and make the code in AppInit2 more clear.

This provides a straightforward flow, gets rid of .count() (which makes
it possible to override an earlier provided proxy option to nothing), as
well as comments the different cases.

* tests: Extend RPC proxy tests

Also test -noonion

* build: Remove -DBOOST_SPIRIT_THREADSAFE

Now that boost spirit is no longer used, `-DBOOST_SPIRIT_THREADSAFE`
doesn't need to be passed to the compiler anymore.

* Fix various warnings

Found while building on Debian 7

* [bitcoin-cli] improve error output

* Add BITCOIND_SIGTERM_TIMEOUT to OpenRC init scripts

This allows users to specify, e.g. raise, the default timeout of 60
seconds. Some bitcoind instances, especially long running ones on slow
hardware, require a higher timeout for a clean shut down.

Also add a comment to bitcoind.openrc's 'retry=', since it is not
obvious from the variable name what it does.

* configure --enable-debug changes

Three changes to how configure --enable-debug behaves:

1. Preserve user-passed CXXFLAGS/CFLAGS
2. Compile with -DDEBUG_LOCKORDER
3. Add -DDEBUG -DDEBUG_LOCKORDER to CPPFLAGS (since they are preprocessor options)

* Update Linearize tool to support Windows paths

* remove unused inv from ConnectTip()

* Buffer log messages and explicitly open logs

Prevents stomping on debug logs in datadirs that are locked by other
instances and lost parameter interaction messages that can get wiped by
ShrinkDebugFile().

The log is now opened explicitly and all emitted messages are buffered
until this open occurs.  The version message and log cut have also been
moved to the earliest possible sensible location.

* fix and extend CBitcoinExtKeyBase template

- fix Decode call (req. only one param)
- add constructor for base58c->CExtKey

* extend bip32 tests to cover Base58c/CExtKey decode

* don't try to decode invalid encoded ext keys

* Testing infrastructure: mocktime fixes

New, undocumented-on-purpose -mocktime=timestamp command-line
argument to startup with mocktime set. Needed because
time-related blockchain sanity checks are done on startup, before a
test has a chance to make a setmocktime RPC call.

And changed the setmocktime RPC call so calling it will not result in
currently connected peers being disconnected due to inactivity timeouts.

* tests: Fix bitcoin-tx signing testcase

Fixes wrong scriptPubkey problem, which caused the transaction to
not actually be signed.

* Assert on probable deadlocks if the second lock isnt try_lock

* locking: teach Clang's -Wthread-safety to cope with our scoped lock macros

This allows us to use function/variable/class attributes to specify locking
requisites, allowing problems to be detected during static analysis.

This works perfectly with newer Clang versions (tested with 3.3-3.7). For older
versions (tested 3.2), it compiles fine but spews lots of false-positives.

* locking: add a quick example of GUARDED_BY

This was chosen not because it's necessarily helpful, but because its locking
assumptions were already correct.

* Don't share objects between TestInstances

* Remove src/qt.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* Update release to 1.0.7, generate manpages

* Add 1.0.7 release notes and update authors.md

* Update debian changelog for 1.0.7 release process

* Update README for 1.0.7 release

* Document returned results of submitblock

* License updates for removal of src/qt.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* Correct license text for LGPL.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* Remove QT gunk from Makefiles.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* Remove some more QT-related stragglers.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* Update documentation for QT removal.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* Update which libraries are allowed to be linked to zcashd by symbol-check.py.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* Remove NO_QT make option.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* .gitignore cache/ and venv-mnf/

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* Remove unused packages and patches.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* Add -t to git fetch for release-notes.py

* Update version to 1.0.7-1

* Update auto-generated manpages to 1.0.7-1

* Add updated release notes for v1.0.7-1

* Update debian package changelog for 1.0.7+1 (use + instead of - for debian versioning)

* Remove stale Qt comments and dead code

* Delete -rootcertificates from bash completion script.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* Add anchor to output of getblock

Closes #2164.

* Remove QT translation support files

* Remove redundant gui options from build scripts

* Edit release-process.md for clarity

* Implement accurate memory accounting for mempool

* Separate core memory usage computation in core_memusage.h

* Implement REST mempool API, add test and documentation.

* Migrate IncrementalMerkleTree memory usage calls

* Removed markdown from COPYING

Updated the Copyright statement at the top of the file from "2016" to "2016-2017" and removed markdown markup from the main text.

* Add tests for getmempoolinfo

* Usability improvements for z_importkey

- Add height parameter to z_importkey to reduce rescan range
- Change semantics of rescan parameter, so users can explicitly force a rescan
  for existing keys.

Closes #2032

* Updated the Bitcoin Core copyright statement

* Introduce librustzcash and Rust to depends system.

* Allow Rust-language related assets to be disabled with `--disable-rust`.

* Implement an AtomicTimer

* Use AtomicTimer for more accurate local solution rate

* Metrics: Move local solution rate into stats

* Metrics: Improve mining status

* Expand on reasons for mining being paused

* Simplify z_importkey by making rescan a string

Transparently handles older boolean values as well.

* libevent: add depends

* build: build-system changes for libevent

* evhttpd implementation

- *Replace usage of boost::asio with [libevent2](http://libevent.org/)*.
boost::asio is not part of C++11, so unlike other boost there is no
forwards-compatibility reason to stick with it. Together with #4738 (convert
json_spirit to UniValue), this rids Bitcoin Core of the worst offenders with
regard to compile-time slowness.

- *Replace spit-and-duct-tape http server with evhttp*. Front-end http handling
is handled by libevent, a work queue (with configurable depth and parallelism)
is used to handle application requests.

- *Wrap HTTP request in C++ class*; this makes the application code mostly
HTTP-server-neutral

- *Refactor RPC to move all http-specific code to a separate file*.
Theoreticaly this can allow building without HTTP server but with another RPC
backend, e.g. Qt's debug console (currently not implemented) or future RPC
mechanisms people may want to use.

- *HTTP dispatch mechanism*; services (e.g., RPC, REST) register which URL
paths they want to handle.

By using a proven, high-performance asynchronous networking library (also used
by Tor) and HTTP server, problems such as #5674, #5655, #344 should be avoided.

What works? bitcoind, bitcoin-cli, bitcoin-qt. Unit tests and RPC/REST tests
pass. The aim for now is everything but SSL support.

Configuration options:

- `-rpcthreads`: repurposed as "number of  work handler threads". Still
defaults to 4.

- `-rpcworkqueue`: maximum depth of work queue. When this is reached, new
requests will return a 500 Internal Error.

- `-rpctimeout`: inactivity time, in seconds, after which to disconnect a
client.

- `-debug=http`: low-level http activity logging

* doc: remove documentation for rpcssl

* qa: Remove -rpckeepalive tests from httpbasics

This option was a temporary workaround, and is no longer necessary
with the new web server.

* Remove rpc_boostasiotocnetaddr test

Dropping all use of boost::asio.

* tests: GET requests cannot have request body, use POST in rest.py

Sending a request body with GET request is not valid in HTTP spec, and
not compatible with evhttpd.

* Implement RPCTimerHandler for Qt RPC console

Implement RPCTimerHandler for Qt RPC console, so that `walletpassphrase`
works with GUI and `-server=0`.

Also simplify HTTPEvent-related code by using boost::function directly.

Zcash: QT changes ommitted

* Fix race condition between starting HTTP server thread and setting EventBase()

Split StartHTTPServer into InitHTTPServer and StartHTTPServer to give
clients a window to register their handlers without race conditions.

Thanks @ajweiss for figuring this out.

* Document options for new HTTP/RPC server in --help

* libevent: Windows reuseaddr workaround in depends

Make it possible to reuse sockets.
This is necessary to make the RPC tests work in WINE.

* Move windows socket init to utility function

* Revert "rpc-tests: re-enable rpc-tests for Windows"

This reverts commit bd30c3dced21fca869a14c75081f15195762afe1.

Disable windows RPC tests for now. These should be re-enabled once a
suitable Wine version is used on Travis.

* init: Ignore SIGPIPE

Ignore SIGPIPE on all non-win32 OSes, otherwise an unexpectedly disconnecting
RPC client will terminate the application. This problem was introduced
with the libhttp-based RPC server.

Fixes #6660.

* http: Disable libevent debug logging, if not explicitly enabled

Add a option "-debug=libevent" to enable libevent debugging for troubleshooting.
Libevent logging is redirected to our own log.

* rpc: Split option -rpctimeout into -rpcservertimeout and -rpcclienttimeout

The two timeouts for the server and client, are essentially different:

- In the case of the server it should be a lower value to avoid clients
clogging up connection slots

- In the case of the client it should be a high value to accomedate slow
  responses from the server, for example for slow queries or when the
  lock is contended

Split the options into `-rpcservertimeout` and `-rpcclienttimeout` with
respective defaults of 30 and 900.

* Make RPC tests cope with server-side timeout between requests

Python's httplib does not graciously handle disconnections from the http server, resulting in BadStatusLine errors.
See https://bugs.python.org/issue3566 "httplib persistent connections violate MUST in RFC2616 sec 8.1.4."

This was fixed in Python 3.5.

Work around it for now.

* Add explicit shared_ptr constructor due to C++11 error

* Avoid a compile error on hosts with libevent too old for EVENT_LOG_WARN.

This uses _EVENT_LOG_WARN instead, which appears to be defined in the
 old versions of libevent that I have on some systems.

* chain: define enum used as bit field as uint32_t

Bitwise logic combined with `<` with undefined signedness will
potentially results in undefined behavior. Fix this by defining the type
as a c++11 typed enum.

Fixes #6017.

* auto_ptr → unique_ptr

Change the few occurrences of the deprecated `auto_ptr` to c++11 `unique_ptr`.
Silences the deprecation warnings.

Also add a missing `std::` for consistency.

* bitcoin-cli: More detailed error reporting

Register a evhttp error handler to get a more detailed error message
if the HTTP request fails.

* depends: Add libevent compatibility patch for windows

Add a patch that seems to be necessary for compatibilty of libevent
2.0.22 with recent mingw-w64 gcc versions (at least GCC 5.3.1 from Ubuntu
16.04).

Without this patch the Content-Length in the HTTP header ends up as
`Content-Length: zu`, causing communication between the RPC
client and server to break down. See discussion in #8653.

Source: https://sourceforge.net/p/levent/bugs/363/

Thanks to @sstone for the suggestion.

* Add common failure cases for rpc server connection failure

* bitcoin-cli: Make error message less confusing

Sorry for the churn on this, but the current message (introduced in #9073)
isn't acceptable:

    $ src/bitcoin-cli getinfo
    rpc: couldn't connect to server
    (make sure server is running and you are connecting to the correct RPC port: -1 unknown)

Putting the error code after the words "RPC port" made me wonder whether
there was a port configuration issue.

This changes it to:

    $ src/bitcoin-cli getinfo
    error: couldn't connect to server: unknown (code -1)
    (make sure server is running and you are connecting to the correct RPC port)

* Added std::unique_ptr<> wrappers with deleters for libevent modules.

* Switched bitcoin-cli.cpp to use RAII unique pointers with deleters.

* Added some simple tests for the RAII-style events.

* Added EVENT_CFLAGS to test makefile to explicitly include libevent headers.

* [depends] libevent 2.1.7rc

* Skip RAII event tests if libevent is built without event_set_mem_functions

* Revert "Closes #1680, temporary fix for rpc deadlock inherited from upstream."

This reverts commit f4404d7.

* Add libevent to zcash-gtest

* Docs: add details to -rpcclienttimeout doc

* [depends] libevent 2.1.8

* Test boolean fallback in z_importkey

* Add security warning to zcash-cli --help and --version message output

* Closes #2186. RPC getblock now accepts height or hash.

* Require that z_importkey height parameter be in valid range

* Add security warning to zcashd metrics display

* [TRIVIAL] Fix typo: exactmath -> exactmatch

... but not yet in trivial tree

* Fix interrupted HTTP RPC connection workaround for Python 3.5+

* test: Avoid ConnectionResetErrors during RPC tests

This is necessary on FreeBSD and MacOSX, at least.

See bitcoin/bitcoin#8834 (comment)

* Add security message to license text, rm url from translation string

* Update LocalSolPS test

* Add AtomicTimer tests

* Revert "Revert "rpc-tests: re-enable rpc-tests for Windows""

This reverts commit 7fd5d4e.

We need to un-revert this before pulling in
bitcoin/bitcoin#6616

* Wrap error string

* Fix typo

* Consolidate individual references to the current maximum peer connection
value of 125 into a single constant declaration.

* net: Automatically create hidden service, listen on Tor

Starting with Tor version 0.2.7.1 it is possible, through Tor's control socket
API, to create and destroy 'ephemeral' hidden services programmatically.
https://stem.torproject.org/api/control.html#stem.control.Controller.create_ephemeral_hidden_service

This means that if Tor is running (and proper authorization is available),
bitcoin automatically creates a hidden service to listen on, without user
manual configuration. This will positively affect the number of available
.onion nodes.

- When the node is started, connect to Tor through control socket
- Send `ADD_ONION` command
- First time:
    - Make it create a hidden service key
    - Save the key in the data directory for later usage
- Make it redirect port 8333 to the local port 8333 (or whatever port we're listening on).
- Keep control socket connection open for as long node is running. The hidden service will
  (by default) automatically go away when the connection is closed.

* Better error message if Tor version too old

* torcontrol improvements and fixes

- Force AUTHCOOKIE size to be 32 bytes: This provides protection against
  an attack where a process pretends to be Tor and uses the cookie
  authentication method to nab arbitrary files such as the
  wallet
- torcontrol logging
- fix cookie auth
- add HASHEDPASSWORD auth, fix fd leak when fwrite() fails
- better error reporting when cookie file is not ok
- better init/shutdown flow
- stop advertizing service when disconnected from tor control port
- COOKIE->SAFECOOKIE auth

* doc: update docs for Tor listening

- add new data directory files for 0.12 to doc/files.md
- mention torcontrol in doc/tor.md

* [doc] [tor] Clarify when to use bind

c.f. #6585

* Connect to Tor hidden services by default

Adds 127.0.0.1:9050 for the .onion proxy if we can succesfully connect
to the control port.

Natural followup to creating hidden services automatically.

* torcontrol: only output disconnect if -debug=tor

* tests: Disable Tor interaction

This is unnecessary during the current tests (any test for Tor
interaction can explicitly enable it) and interferes with the proxy
test.

* torcontrol debug: Change to a blanket message that covers both cases

* Fixing typos on security-check.py and torcontrol.cpp

* Do not absolutely protect local peers from eviction.

With automatic tor HS support in place we should probably not be providing
 absolute protection for local peers, since HS inbound could be used to
 attack pretty easily.  Instead, this counts on the latency metric inside
 AttemptToEvictConnection to privilege actually local peers.

* Decide eviction group ties based on time.

This corrects a bug the case of tying group size where the code may
 fail to select the group with the newest member. Since newest time
 is the final selection criteria, failing to break ties on it
 on the step before can undermine the final selection.

Tied netgroups are very common.

* Remove vfReachable and modify IsReachable to only use vfLimited.

We do not know that a class of Network is reachable, only that it is not.

* Fix memleak in TorController [rework]

It looks like, TorController::disconnected_cb(TorControlConnection&
conn) gets called multiple times which results in multiple event_new().

Avoid this by creating the event only once in the constructore, and
deleting it only once in the destructor (thanks to Cory Fields for the
idea).

Replaces the fix by Jonas Schnelli in #7610, see discussion there.

* Fix torcontrol.cpp unused private field warning

* [doc] Update port in tor.md

Tor Browser Bundle spawns the Tor process and listens on port 9150, it doesn't randomly pick a port.

[ci skip]
(cherry picked from commit 1b63cf98347b2a62915425576930f55c2126c2ff)

* tor: Change auth order to only use HASHEDPASSWORD if -torpassword

Change authentication order to make it more clear (see #7700).

- If the `-torpassword` option is provided, force use of
  `HASHEDPASSWORD` auth.

- Give error message if `-torpassword` provided, but
  `HASHEDPASSWORD` auth is not available.

- Give error message if only `HASHEDPASSWORD` available, but
  `-torpassword` not given.

* Clarify documentation for running a tor node

Previous wording suggested that no additional setup was required for a
tor hidden service to be created.

* Clarify `listenonion`

> This new feature is enabled by default if Bitcoin Core is listening, and a connection to Tor can be made. It can be configured with the -listenonion, -torcontrol and -torpassword settings. To show verbose debugging information, pass -debug=tor.

But it is correct to say that the feature is enabled *regardless* of whether a connection to Tor can be made.

I propose to clarify that so that users can eliminate these in their logs (when `listen=1` and no Tor).
And I think it's okay to clarify about the `listen` option, because on several occasions when I read this before I always assumed `listening` meant `server=1` which cost me a lot of time in troubleshooting.

```
2016-10-24 06:19:22.551029 tor: Error connecting to Tor control socket
2016-10-24 06:19:22.551700 tor: Not connected to Tor control port 127.0.0.1:9051, trying to reconnect
```

0.12.1

* torcontrol: Explicitly request RSA1024 private key

When generating a new service key, explicitly request a RSA1024 one.

The bitcoin P2P protocol has no support for the longer hidden service names
that will come with ed25519 keys, until it does, we depend on the old
hidden service type so make this explicit.

See #9214.

* Prevent -Wshadow warnings with gcc versions 4.8.5, 5.3.1 and 6.2.1.

Zcash: partial (ignoring src/test/script_tests.cpp due to merge conflicts)

* Make some global variables less-global (static)

* torcontrol: Improve comments

* torcontrol: Add unit tests for Tor reply parsers

* torcontrol: Fix ParseTorReplyMapping

- Ignore remaining input if it is an OptArguments
- Correctly handle escapes

* torcontrol: Check for reading errors in ReadBinaryFile

This ensures that ReadBinaryFile never returns exactly TOR_COOKIE_SIZE bytes if
the file was larger than that.

* Line-wrap privacy notice. Use <> around URL and end sentence with '.'.
Include privacy notice in help text for zcashd -help.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* torcontrol: Log invalid parameters in Tor reply strings where meaningful

* Update version numbers.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* Improvement to release process doc.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* Generate man pages.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* Update authors, release notes, and Debian package metadata.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* Use a larger -rpcclienttimeout for slow performance measurements

Closes #2238.

* Don't rely on a finite upper bound on fee rate or priority.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* Update tests to check actual infinity as well as INF_FEERATE

* Add unit test for security issue 2017-04-11.a

* Simplify JoinSplit priority calculation. refs 1896

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* Add check for JoinSplit priority as calculated by CCoinsViewCache::GetPriority.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

* Update text to Zclassic

* User Zcash repos for depends

* Remove Zclassic references

* Add libevents to gtest makefile
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment