Fix string concatenation to os.path.join and add exception case #11291

dongsam · 2017-09-08T22:35:11Z

Solved some warnings for Python PEP 8 convention

Method Names and Instance Variables
lowercase with words separated by underscores as necessary to improve readability.
- testDir to test_dir
- inputData to input_data
- ...
Blank Lines
Surround top-level function and class definitions with two blank lines.
Exception Names

and added verification logic about referenced before assignment for output_type

maflcko · 2017-09-09T21:21:41Z

test/util/bitcoin-util-test.py

    """Runs a single test, comparing output and RC to expected output and RC.

    Raises an error if input can't be read, executable fails, or output/RC
    are not as expected. Error is caught by bctester() and reported.
    """
    # Get the exec names and arguments
-    execprog = buildenv["BUILDDIR"] + "/src/" + testObj['exec'] + buildenv["EXEEXT"]
-    execargs = testObj['args']
+    execprog = buildenv["BUILDDIR"] + "/src/" + test_obj['exec'] + buildenv["EXEEXT"]


I'd assume those should probably use os.path.join to make it work on other os as well.

maflcko · 2017-09-09T21:25:50Z

Thank you for putting so much effort in your first contribution. However, according to our developer guidelines we can not accept pull requests that solely change formatting and style.

Nonetheless, you are welcome to address my issue with this file instead. Also note that you can change testObj to test_obj, as formatting fixes are allowed when none-style but refactoring changes are made to the affected lines.

dongsam · 2017-09-10T00:51:41Z

@MarcoFalke Thank you for your advice, I just added commit for use os.path.join instead string concatenation

maflcko · 2017-09-10T01:19:15Z

test/util/bitcoin-util-test.py

    """Runs a single test, comparing output and RC to expected output and RC.

    Raises an error if input can't be read, executable fails, or output/RC
    are not as expected. Error is caught by bctester() and reported.
    """
    # Get the exec names and arguments
-    execprog = buildenv["BUILDDIR"] + "/src/" + testObj['exec'] + buildenv["EXEEXT"]
-    execargs = testObj['args']
+    execprog = os.path.join(buildenv["BUILDDIR"], "src", test_obj["exec"], buildenv["EXEEXT"])


The last concatenation is string concatenation.

You can check if the tests pass locally by make check

Thanks I just fixed it, Sorry for my careless mistake

jnewbery

utACK. A couple of nits inline and commits should be squashed to a single commit before merge.

jnewbery · 2017-09-12T14:15:03Z

test/util/bitcoin-util-test.py


-def bctester(testDir, input_basename, buildenv):
+def bctester(test_dir, input_basename, buildenv):


since you're converting to snake_case, how about build_env

jnewbery · 2017-09-12T14:15:50Z

test/util/bitcoin-util-test.py

    """Runs a single test, comparing output and RC to expected output and RC.

    Raises an error if input can't be read, executable fails, or output/RC
    are not as expected. Error is caught by bctester() and reported.
    """
    # Get the exec names and arguments
-    execprog = buildenv["BUILDDIR"] + "/src/" + testObj['exec'] + buildenv["EXEEXT"]
-    execargs = testObj['args']
+    execprog = os.path.join(buildenv["BUILDDIR"], "src", test_obj["exec"]) + buildenv["EXEEXT"]


exec_prog? (and exec_args and exec_run)

maflcko

Again, I appreciate the effort, but according to the developer notes such refactoring patches can not be accepted.

Do not submit patches solely to modify the style of existing code.

And https://github.com/bitcoin/bitcoin/blob/master/CONTRIBUTING.md#refactoring

This will cause merge conflicts with other open pull requests and consumes precious review time.

Please note that the non-refactoring changes in this pull are much appreciated and I'd like to have them merged cleanly without the formatting changes.

maflcko · 2017-09-12T16:00:28Z

test/util/bitcoin-util-test.py


    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument('-v', '--verbose', action='store_true')
+    parser.add_argument("-v", "--verbose", action="store_true")


I don't it is worth the overhead to change apostrophe to quotation mark.

agree (and for short symbol-like strings as in this line, single quotes seem to be most people's preferred style)

maflcko · 2017-09-12T16:01:29Z

test/util/bitcoin-util-test.py

    # Add the format/level to the logger
    logging.basicConfig(format=formatter, level=level)

-    bctester(config["environment"]["SRCDIR"] + "/test/util/data", "bitcoin-util-test.json", config["environment"])
+    bctester(os.path.join(config["environment"]["SRCDIR"], "test/util/data"), "bitcoin-util-test.json",
+             config["environment"])


Changing this line solely to split it into two causes difficulties grepping in the future.

... and most of our python files don't adhere to the 79 char max line length (which I think is a good thing!)

maflcko · 2017-09-12T16:02:33Z

test/util/bitcoin-util-test.py

+            logging.info("PASSED: " + test_obj["description"])
+        except Exception as e:
+            logging.info("FAILED: " + test_obj["description"])
+            logging.error("Error %s: %s" % (test_obj["description"], e))


Can you explain why it makes sense to first log to info and then the same to the error level?

maflcko · 2017-09-12T16:05:14Z

test/util/bitcoin-util-test.py

+        if not output_data:
+            logging.error("Output data missing for " + output_fn)
+            raise Exception
+        if not output_type:


Generally, I'd prefer if refactoring is separate (at least a separate commit) from changing behavior.

maflcko · 2017-09-12T16:06:02Z

test/util/bitcoin-util-test.py

-    if "error_txt" in testObj:
-        want_error = testObj["error_txt"]
+    if "error_txt" in test_obj:
+        want_error = test_obj["error_txt"]


I don't think it makes sense to replace every variable in the whole file by a lower_case name.

jnewbery · 2017-09-12T17:17:03Z

test/util/bitcoin-util-test.py

@@ -20,40 +20,44 @@
 import subprocess
 import sys

+


For short util modules like this, and even for most of our test scripts, I don't think 2 lines separating each function is necessary. It makes sense for large projects where you want visual separation between top-level classes, but I think it's unnecessary here.

jnewbery · 2017-09-12T17:23:01Z

@dongsam we don't have particularly strict style guidelines for our python code. The only style notes are in https://github.com/bitcoin/bitcoin/blob/master/test/functional/README.md.

Since A Foolish Consistency is the Hobgoblin of Little Minds, we don't usually open PRs to adhere to PEP-8 style, unless there's some good reason to do so (eg making bugs less likely). If I'm doing substantial work on a test script, I often start the PR with a tidy-up commit, but even that is sometimes unpopular and puts off reviewers.

If you're looking for your first commit, I think @MarcoFalke is right that removing the style changes from this PR and just having the os.path changes would make this more palatable for reviewers.

maflcko · 2017-10-05T11:10:45Z

Needs rebase

jnewbery · 2017-10-05T14:06:59Z

test/util/bitcoin-util-test.py

@@ -43,12 +43,11 @@ def main():
    formatter = '%(asctime)s - %(levelname)s - %(message)s'
    # Add the format/level to the logger
    logging.basicConfig(format=formatter, level=level)
-


nit: why remove this line? Blank lines are used within functions to indicate logical sections.

sorry, that was mistake while rebasing, I'll add again now

You've now added trailing whitespace :(

jnewbery · 2017-10-05T14:28:53Z

test/util/bitcoin-util-test.py

@@ -43,12 +43,11 @@ def main():
    formatter = '%(asctime)s - %(levelname)s - %(message)s'
    # Add the format/level to the logger
    logging.basicConfig(format=formatter, level=level)
-


You've now added trailing whitespace :(

jnewbery · 2017-10-05T14:29:54Z

test/util/bitcoin-util-test.py

        inputData = open(filename).read()
        stdinCfg = subprocess.PIPE

    # Read the expected output data (if there is any)
    outputFn = None
    outputData = None
+    outputType = None


Is this change (and the one at line 107) intentional?

Yes, there was potential risk outputType can be None so I added initial outputType and exception line when outputType is None, if you think it is not needed, i can remove again

Yes, this is a good additional check.

jnewbery · 2017-10-09T14:31:23Z

test/util/bitcoin-util-test.py

        except:
            logging.error("Output file " + outputFn + " can not be opened")
            raise
        if not outputData:
            logging.error("Output data missing for " + outputFn)
            raise Exception
+        if not outputType:
+            logging.error("Output type missing for " + outputFn)


suggestion: change the error text slightly here to clarify what the problem is:

logging.error("Output file %s does not have a file extension" % outputFn)

@dongsam Could you address @jnewbery's nit here? Looks like this is pretty much ready for merge.

@fanquake I just change and rebase the error text, sorry for late response

jnewbery · 2017-12-08T19:31:44Z

test/util/bitcoin-util-test.py

@@ -77,32 +77,36 @@ def bctest(testDir, testObj, buildenv):
    are not as expected. Error is caught by bctester() and reported.
    """
    # Get the exec names and arguments
-    execprog = buildenv["BUILDDIR"] + "/src/" + testObj['exec'] + buildenv["EXEEXT"]
+    execprog = os.path.join(buildenv["BUILDDIR"], "src", testObj["exec"]) + buildenv["EXEEXT"]


I think this is better as:

execprog = os.path.join(buildenv["BUILDDIR"], "src", testObj["exec"] + buildenv["EXEEXT"])

(ie buildenv["EXEEXT"] should be appended to testObj["exec"] before joining into a path), but I don't have a windows machine to test this on.

Is there a reason you put buildenv["EXEEXT"] outside the os.path.join?

I thought also but that was mistook, buildenv["EXEEXT"] is not path, so occur error when with os.path.join
that need just string concatenation

right, that's why you should have:

execprog = os.path.join(buildenv["BUILDDIR"], "src", testObj["exec"] + buildenv["EXEEXT"])

(concatenate testObj["exec"] and buildenv["EXEEXT"] as a single argument to os.path.join())

oh, sorry I misunderstood, that's look better, I'll update :)

jnewbery · 2017-12-09T15:55:15Z

utACK a3ac767. Needs to be tested on windows.

maflcko · 2017-12-09T18:41:14Z

test/util/bitcoin-util-test.py

@@ -48,7 +48,7 @@ def main():



^ one line above.

"test/util/data" throws an error on windows:

FileNotFoundError: [Errno 2] No such file or directory: '.../Downloads/workspace/bitcoin\\test/util/data\\bitcoin-util-test.json'

maflcko · 2017-12-09T23:23:01Z

test/util/bitcoin-util-test.py

@@ -48,7 +48,7 @@ def main():

 def bctester(testDir, input_basename, buildenv):
    """ Loads and parses the input file, runs all tests and reports results"""
-    input_filename = testDir + "/" + input_basename
+    input_filename = os.path.join(testDir, input_basename)


Might as well pass in input_filename to bctester and instead do the path.join in line 47, no?

maflcko · 2017-12-30T05:25:55Z

Going to merge since this is a strict improvement.

…ion case a3ac767 Fix string concatenation to os.path.join and add exception case (dongsamb) Pull request description: Solved some warnings for [Python PEP 8 convention](https://www.python.org/dev/peps/pep-0008/) - [Method Names and Instance Variables](https://www.python.org/dev/peps/pep-0008/#method-names-and-instance-variables) lowercase with words separated by underscores as necessary to improve readability. - `testDir` to `test_dir` - `inputData` to `input_data` - ... - [Blank Lines](https://www.python.org/dev/peps/pep-0008/#blank-lines) Surround top-level function and class definitions with two blank lines. - [Exception Names](https://www.python.org/dev/peps/pep-0008/#exception-names) and added verification logic about referenced before assignment for `output_type` Tree-SHA512: 346d08799f03077a2b7257ccdca123b4945b89dbf0677dba452d96b81ce186ec7b5dcdb10b8bb59cfce657a7aedbb7df64921036cbd1bf4ad8bd313d40faa796

… exception case a3ac767 Fix string concatenation to os.path.join and add exception case (dongsamb) Pull request description: Solved some warnings for [Python PEP 8 convention](https://www.python.org/dev/peps/pep-0008/) - [Method Names and Instance Variables](https://www.python.org/dev/peps/pep-0008/#method-names-and-instance-variables) lowercase with words separated by underscores as necessary to improve readability. - `testDir` to `test_dir` - `inputData` to `input_data` - ... - [Blank Lines](https://www.python.org/dev/peps/pep-0008/#blank-lines) Surround top-level function and class definitions with two blank lines. - [Exception Names](https://www.python.org/dev/peps/pep-0008/#exception-names) and added verification logic about referenced before assignment for `output_type` Tree-SHA512: 346d08799f03077a2b7257ccdca123b4945b89dbf0677dba452d96b81ce186ec7b5dcdb10b8bb59cfce657a7aedbb7df64921036cbd1bf4ad8bd313d40faa796

fanquake added the Tests label Sep 8, 2017

maflcko reviewed Sep 9, 2017

View reviewed changes

maflcko reviewed Sep 10, 2017

View reviewed changes

jnewbery reviewed Sep 12, 2017

View reviewed changes

dongsam force-pushed the Fix-PEP8-warnings branch from a334360 to 5256a59 Compare September 12, 2017 15:08

maflcko reviewed Sep 12, 2017

View reviewed changes

jnewbery reviewed Sep 12, 2017

View reviewed changes

dongsam force-pushed the Fix-PEP8-warnings branch 3 times, most recently from c259a68 to 2ce3867 Compare September 13, 2017 19:15

dongsam changed the title ~~Fix PEP8 warnings about formatting~~ Fix string concatenation to os.path.join and add exception case Sep 13, 2017

dongsam force-pushed the Fix-PEP8-warnings branch from 2ce3867 to b7fc18a Compare October 5, 2017 12:22

jnewbery reviewed Oct 5, 2017

View reviewed changes

dongsam force-pushed the Fix-PEP8-warnings branch from b7fc18a to 5bd3600 Compare October 5, 2017 14:15

jnewbery reviewed Oct 5, 2017

View reviewed changes

dongsam force-pushed the Fix-PEP8-warnings branch from 5bd3600 to f1c5277 Compare October 5, 2017 14:50

jnewbery reviewed Oct 9, 2017

View reviewed changes

dongsam force-pushed the Fix-PEP8-warnings branch from f1c5277 to 856b3e2 Compare December 8, 2017 13:53

jnewbery reviewed Dec 8, 2017

View reviewed changes

Fix string concatenation to os.path.join and add exception case

a3ac767

dongsam force-pushed the Fix-PEP8-warnings branch from 856b3e2 to a3ac767 Compare December 9, 2017 06:17

maflcko reviewed Dec 9, 2017

View reviewed changes

maflcko merged commit a3ac767 into bitcoin:master Dec 30, 2017

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021


		def bctester(testDir, input_basename, buildenv):
		def bctester(test_dir, input_basename, buildenv):

Fix string concatenation to os.path.join and add exception case #11291

Fix string concatenation to os.path.join and add exception case #11291

Conversation

dongsam commented Sep 8, 2017 • edited

Choose a reason for hiding this comment

maflcko commented Sep 9, 2017

dongsam commented Sep 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jnewbery left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maflcko left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jnewbery commented Sep 12, 2017

maflcko commented Oct 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jnewbery commented Dec 9, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maflcko commented Dec 30, 2017

dongsam commented Sep 8, 2017 •

edited

jnewbery commented Dec 9, 2017 •

edited