Refactor of BitcoinTestFramework main function. #26

jachiang · 2019-09-02T23:58:25Z

In order to wrap the BitcoinTestFramework class in a user-defined class, which can be started and shutdown, the code in BitcoinTestFramework main has been encapsulated in respective functions. Additionally, test-run specific state in temp-directory and NetworkThread is now reset after shutdown.

Behavioural changes:
Potential exceptions raised from any code in setup section of the code are now handled. Previously, only exceptions raised by code beginning here was handled.

TODO's:

~~Refactor NetworkThread class in mininode.py, so each BitcoinTestFramework object initialises a different network thread.~~
Documentation (markdown file) describing how to run an interactive test with TestWrapper.

jnewbery · 2019-09-03T16:49:44Z

test/functional/test_framework/mininode.py

@@ -480,7 +480,7 @@ def close(self, timeout=10):
        wait_until(lambda: not self.network_event_loop.is_running(), timeout=timeout)
        self.network_event_loop.close()
        self.join(timeout)
-
+        NetworkThread.network_event_loop = None # Safe to remove event loop.


nit: the convention seems to be that comments usually go the line above in the python test code

Fixed in b1f803d.

jnewbery · 2019-09-03T16:51:08Z

test/functional/test_framework/test_framework.py

+            self.run_test()
+        except BaseException as exception:
+            e = exception
+        self.shutdown(e = e, exit = True)


The second parameter (exit) isn't used in this repo. Can you remove it by having self.shutdown return the exit_code and then main() calls sys.exit().

Fixed in e163c98.

jnewbery · 2019-09-03T17:01:50Z

test/functional/test_framework/test_framework.py

+    def shutdown(self, e=None, exit=False):
+        """Call this method to shutdown the test object and optionally handle an exception."""
+
+        if e != None:


I think it makes more logical sense to have the exception handling code in the main() function, and then just start shutdown() at the if self.success == TestStatus.FAILED and ... line

I am not sure I fully understand your suggestion. If we move error handling into main, this is code that needs to be replicated in TestWrapper.

why not:

try: self.setup() self.run_test() except BaseException as e: handle_exception(e) finally: exit_code = self.shutdown() sys.exit(exit_code)

jnewbery · 2019-09-03T17:02:24Z

test/functional/test_framework/test_framework.py

+        if exit:
+            sys.exit(exit_code)
+
+    def handle_exception(self, e):


my preference would be to not have this as a separate function from main()

Addressed in e163c98.

jnewbery · 2019-09-03T17:07:00Z

test/functional/test_framework/test_framework.py

@@ -440,7 +466,7 @@ def sync_all(self, nodes=None, **kwargs):

    def _start_logging(self):
        # Add logger and logging handlers
-        self.log = logging.getLogger('TestFramework')
+        self.log = logging.getLogger('TestFramework.'+ self.options.tmpdir) # Assign new logger name to prevent temp path reuse.


This isn't quite right. Since logging.shutdown() is called in TestFramework.shutdown(), no more logging should take place (see https://docs.python.org/3/library/logging.html#logging.shutdown).

A better way to do this would be to replace the logging.shutdown() call with:

handlers = self.log.handlers for h in handlers: h.flush() h.flush() self.log.removeHandler(h)

That way, when setup is called again, the old handlers will have been removed and the new handlers will be added to the Logging object.

Thanks for this suggestion. Added to e163c98.

jnewbery · 2019-09-03T17:08:03Z

You should also reword the second commit log to remove reference to TestWrapper if you want to get this merged to Bitcoin Core.

jnewbery · 2019-09-06T18:59:59Z

You should also reword the second commit log to remove reference to TestWrapper if you want to get this merged to Bitcoin Core.

Thinking about this some more, perhaps a good way to get this merged to Bitcoin Core would be to include the TestWrapper and add some documentation about how to run an interactive test. That's quite a compelling rationale for making these changes.

I'd hold off on that while we're still iterating on TestWrapper, but after the workshops let's open a PR against https://github.com/bitcoin/bitcoin.

jachiang · 2019-09-13T18:30:51Z

I am adding another TODO for this PR.

Refactor NetworkThread class in mininode.py, so each BitcoinTestFramework object initialises a different network thread.

jnewbery · 2019-09-14T16:48:50Z

Refactor NetworkThread class in mininode.py, so each BitcoinTestFramework object initialises a different network thread.

I don't think we want more than one instance of BitcoinTestFramework to exist at a time. @jachiang - do you have a reason for wanting this?

jachiang

Refactor NetworkThread class in mininode.py, so each BitcoinTestFramework object initialises a different network thread.

I don't think we want more than one instance of BitcoinTestFramework to exist at a time. @jachiang - do you have a reason for wanting this?

I will omit this from this PR, because it requires limiting one NetworkThread per BitcoinTestFramework, which I am unsure of how to best implement with minimal code changes.

jnewbery · 2019-10-14T14:37:35Z

test/functional/test_framework/test_framework.py

-        sys.exit(exit_code)
+
+        for h in list(self.log.handlers):
+            self.log.removeHandler(h)


Do these need to be flushed before being removed?

jnewbery · 2019-10-14T14:56:31Z

test/functional/test_framework/test_framework.py

+    def shutdown(self, e=None, exit=False):
+        """Call this method to shutdown the test object and optionally handle an exception."""
+
+        if e != None:


why not:

try: self.setup() self.run_test() except BaseException as e: handle_exception(e) finally: exit_code = self.shutdown() sys.exit(exit_code)

jnewbery

A few comments inline. I also think the first commit comment ("Removal of residual NetworkThread state.") could be expanded a bit to explain why you're making that change.

Once the documentation is added, I think this is ready to open as a PR against bitcoin/bitcoin

jnewbery · 2019-10-14T14:57:13Z

test/functional/test_framework/test_framework.py

+
+    def handle_exception(self, e):
+        # Restore traceback from exception object.
+        tb = traceback.format_exception(type(e), e, e.__traceback__)


Is this new traceback functionality necessary? Can it be separated into a new commit so you're not moving and changing code in the same commit?

Not Ideal, I agree. It seems the logger will only capture the traceback if error logging is called directly inside except (see https://stackoverflow.com/a/5191885 and https://docs.python.org/3/library/logging.html#logging.error). Since we are invoking error/exception logging elsewhere, we reconstruct the traceback string from the exception object.

jnewbery · 2019-10-14T14:57:48Z

test/functional/test_framework/test_wrapper.py

+# Copyright (c) 2014-2019 The Bitcoin Core developers
+# Distributed under the MIT software license, see the accompanying
+# file COPYING or http://www.opensource.org/licenses/mit-license.php.
+from test_framework.test_framework import BitcoinTestFramework


pep8 import ordering please.

More generally, can you run flake8 over this new file and resolve any style issues?

jnewbery · 2019-10-14T14:59:35Z

test/functional/test_framework/test_wrapper.py

+
+        def set_test_params(self):
+            # This can be overriden in setup() parameter.
+            self.num_nodes=3


is this required? num_nodes defaults to 3 in the setup() method.

jnewbery · 2019-10-14T14:59:43Z

test/functional/test_framework/test_wrapper.py

+            bitcoind=abspath(join(__file__ ,"../../../..") +  "/src/bitcoind"),
+            bitcoincli=None,
+            setup_clean_chain=True,
+            num_nodes=3,


I suggest you default this to 1

jnewbery · 2019-10-14T15:03:01Z

test/functional/test_framework/test_wrapper.py

+            pdbonfailure=False,
+            usecli = False,
+            perf = False,
+            randomseed = None):


Have you tried changing the method signature to catch *kwargs and then set the defaults below, to avoid this huge function signature?

Yes, it is much cleaner as you suggested. Thank you!

The asyncio.new_event_loop() instance is now removed from the NetworkThread class during shutdown. This enables a NetworkThread instance to be restarted after being closed. The current NetworkThread class guards against an existing new_event_loop during initialization.

Setup and shutdown code now moved into dedicated methods. Test "success" is added as a BitcoinTestFramework member, which can be accessed outside of main.

In order for BitcoinTestFramework to correctly restart after shutdown, the previous logging handlers need to be removed, or else logging will continue in the previous temp directory. "Flush" ensures buffers are emptied, and "close" ensures file handler close logging file.

TestNode objects need to be removed during shutdown, as setup_nodes does not remove previous TestNode objects from previous test runs during setup.

This allows a BitcoinTestFramework child class to set test parameters in an overridden setup() rather than in an overridden set_test_params().

A BitcoinTestFramework child class which can be imported by an external user or project. TestWrapper.setup() initiates an underlying BitcoinTestFramework object with bitcoind subprocesses, rpc interfaces and test logging. TestWrapper.shutdown() safely tears down the BitcoinTestFramework object.

jachiang · 2019-10-26T19:03:37Z

@jnewbery Sorry this took a while, but I've addressed all feedback, reworked commits for easier review and added a documentation file for the TestWrapper class.

I will review again tomorrow and then open a PR against master.
A very big thank you for your time spent reviewing and improving this!

jnewbery · 2019-10-29T19:48:12Z

PR opened against bitcoin/bitcoin here: bitcoin#17288

jachiang requested review from jnewbery and maflcko September 2, 2019 23:58

jnewbery requested changes Sep 3, 2019

View reviewed changes

jachiang mentioned this pull request Sep 13, 2019

Cannot run multiple instances of TestWrapper bitcoinops/taproot-workshop#20

Closed

jachiang force-pushed the master-testframework-main-refactor branch 2 times, most recently from 3f5baf0 to 774ecb1 Compare October 5, 2019 16:57

jachiang commented Oct 5, 2019

View reviewed changes

jachiang requested a review from jnewbery October 6, 2019 02:57

jnewbery reviewed Oct 14, 2019

View reviewed changes

jachiang force-pushed the master-testframework-main-refactor branch 4 times, most recently from 25ebea1 to 5f01275 Compare October 25, 2019 13:15

jachiang added 7 commits October 26, 2019 20:48

Refactored BitcoinTestFramework main method into setup and shutdown.

594364a

Setup and shutdown code now moved into dedicated methods. Test "success" is added as a BitcoinTestFramework member, which can be accessed outside of main.

Clear TestNode objects after shutdown.

2c6fd51

TestNode objects need to be removed during shutdown, as setup_nodes does not remove previous TestNode objects from previous test runs during setup.

Moved assert num_nodes is set into main().

2aa26a6

This allows a BitcoinTestFramework child class to set test parameters in an overridden setup() rather than in an overridden set_test_params().

Added documentation for test_wrapper submodule.

bf4fc41

jachiang force-pushed the master-testframework-main-refactor branch from 5f01275 to bf4fc41 Compare October 26, 2019 18:49

jachiang mentioned this pull request Oct 28, 2019

Added TestShell class for interactive Python environments. bitcoin/bitcoin#17288

Merged

jnewbery closed this Oct 29, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor of BitcoinTestFramework main function. #26

Refactor of BitcoinTestFramework main function. #26

jachiang commented Sep 2, 2019 •

edited

jnewbery Sep 3, 2019

jachiang Oct 5, 2019

jnewbery Sep 3, 2019

jachiang Oct 5, 2019

jnewbery Sep 3, 2019

jachiang Oct 5, 2019 •

edited

jnewbery Oct 14, 2019

jnewbery Sep 3, 2019

jachiang Oct 5, 2019

jnewbery Sep 3, 2019

jachiang Oct 5, 2019

jnewbery commented Sep 3, 2019

jnewbery commented Sep 6, 2019

jachiang commented Sep 13, 2019 •

edited

jnewbery commented Sep 14, 2019

jachiang left a comment

jnewbery Oct 14, 2019

jnewbery Oct 14, 2019

jnewbery left a comment

jnewbery Oct 14, 2019

jachiang Oct 21, 2019

jnewbery Oct 14, 2019

jnewbery Oct 14, 2019

jnewbery Oct 14, 2019

jnewbery Oct 14, 2019

jachiang Oct 21, 2019

jachiang commented Oct 26, 2019

jnewbery commented Oct 29, 2019

Refactor of BitcoinTestFramework main function. #26

Refactor of BitcoinTestFramework main function. #26

Conversation

jachiang commented Sep 2, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jachiang Oct 5, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jnewbery commented Sep 3, 2019

jnewbery commented Sep 6, 2019

jachiang commented Sep 13, 2019 • edited

jnewbery commented Sep 14, 2019

jachiang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jnewbery left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jachiang commented Oct 26, 2019

jnewbery commented Oct 29, 2019

jachiang commented Sep 2, 2019 •

edited

jachiang Oct 5, 2019 •

edited

jachiang commented Sep 13, 2019 •

edited