Remove duplicate dictionary keys

[andrewda]

This will automatically remove duplicate dictionary keys with the
--remove-duplicate-keys argument.

Closes #27

Example

--- original/../test.py
+++ fixed/../test.py
@@ -1,15 +1,9 @@
 print({
-  'b': 456,
-  'a': 123,
-  'b': 7834,
   'a': 'wow',
-  'b': 456,
-  'c': 'hello',
   'c': 'hello2',
-  'b': 'hiya',
   'b': 'hiya',
 })
 
-print({ 'a': 123, 'a': 567 })
-print({ 'b': 123, 'b': 567, 'b': 897, 'c': 'hello' })
-print({ 'a': 'first value', 'c': 123, 'd': 'hjfkds', 'c': 567, 'g': 'hi', 'c': 'this value will stay', 'd': 'so will this one' })
+print({ 'a': 567 })
+print({ 'b': 897, 'c': 'hello' })
+print({ 'a': 'first value', 'g': 'hi', 'c': 'this value will stay', 'd': 'so will this one' })

[coveralls]

Coverage decreased (-0.5%) to 97.724% when pulling 3b97ba6 on andrewda:dupe-keys into b482463 on myint:master.

[coveralls]

Coverage decreased (-0.5%) to 97.747% when pulling ac70997 on andrewda:dupe-keys into b482463 on myint:master.

[myint]

@andrewda Excellent work!

There are a few inputs I tried that caused it to crash though:

lambda x: {(0,1): 1, (0,1): 2}

{(0,1): 1, (0,1): 2}

$ ./autoflake.py   --remove-duplicate-keys foo.py
Traceback (most recent call last):
  File "./autoflake.py", line 846, in <module>
    sys.exit(main())
  File "./autoflake.py", line 840, in main
    standard_error=sys.stderr)
  File "./autoflake.py", line 820, in _main
    fix_file(name, args=args, standard_out=standard_out)
  File "./autoflake.py", line 635, in fix_file
    remove_unused_variables=args.remove_unused_variables)
  File "./autoflake.py", line 612, in fix_code
    remove_unused_variables=remove_unused_variables))))
  File "./autoflake.py", line 371, in filter_code
    source)
  File "./autoflake.py", line 448, in filter_duplicate_key
    if is_last_in_object(line, line_number, key, marked_line_numbers, source):
  File "./autoflake.py", line 475, in is_last_in_object
    source
  File "./autoflake.py", line 499, in get_occurences_in_object
    and re.search(r'(?:\'|")%s(?:\'|"): [^,]+' % key, lines[ln - 1]):
TypeError: not all arguments converted during string formatting

I got the examples from the pyflakes test suite for the duplicate key detection feature:

https://github.com/pycqa/pyflakes/blob/master/pyflakes/test/test_dict.py

Thanks

[andrewda]

@myint Thanks for checking that! I had no idea object keys could be tuples in Python. Should be fixed now!

[coveralls]

Coverage decreased (-0.6%) to 97.674% when pulling 3aa1d91 on andrewda:dupe-keys into b482463 on myint:master.

[coveralls]

Coverage decreased (-0.5%) to 97.753% when pulling 3275242 on andrewda:dupe-keys into b482463 on myint:master.

[jayvdb]

Would be good to add a test case for more complex data types being used in the dict keys.

[coveralls]

Coverage decreased (-0.4%) to 97.783% when pulling 60edd2b on andrewda:dupe-keys into b482463 on myint:master.

[jayvdb]

coverage is still decreasing. That means there are new code paths which are not being tested.

[myint]

That could be my fault. Try setting the environment variable AUTOFLAKE_COVERAGE=1 in .travis.yml.

[myint]

Actually, I think you'll need to add it to the Makefile. AUTOFLAKE_COVERAGE=1 should create the coverage output for the end-to-end tests, which you added. See this for an example, where I did it properly:

https://github.com/hhatto/autopep8/blob/4c6a018b7d17b8414155db51abe21adc7e38d496/Makefile#L47

[andrewda]

@jayvdb Any idea why

• https://coveralls.io/builds/14875802/source?filename=autoflake.py#L149
• https://coveralls.io/builds/14875802/source?filename=autoflake.py#L347

aren't covered? Adding prints there shows them to be running fine locally.

Edit: nevermind, what @myint said might fix those

[myint]

I'm noticing that I did things a bit differently here than in autopep8. Here, the coverage rule is used for interactive use only. In the .travis.yml, I'm independently calling the coverage tool without involving make. So you probably need to put AUTOFLAKE_COVERAGE=1 in both Makefile and .travis.yml.

https://github.com/myint/autoflake/blob/b4824635ecdec24a87c32ad6a1c61f6dce09b785/.travis.yml#L26

[jayvdb]

my guess is https://coveralls.io/builds/14875802/source?filename=autoflake.py#L149 is getting covered, but only by some of the jobs.

I dont know coveralls.io very well; I use codecov.io exclusively as it merges reports from multiple CI builds, allowing 100% coverage to be reached even for Windows and Unix specific code.

[coveralls]

Coverage decreased (-0.4%) to 97.783% when pulling 5ded655 on andrewda:dupe-keys into b482463 on myint:master.

[myint]

And if that doesn't work, you could always just test one of the higher-level functions like in:

https://github.com/myint/autoflake/blob/b4824635ecdec24a87c32ad6a1c61f6dce09b785/test_autoflake.py#L479-L486

That way, you don't need to depend on my test coverage subprocess code working. It wouldn't surprise me if there is something wrong with it given that that code was never being exercised previously.

[andrewda]

@myint E2E tests failing with AUTOFLAKE_COVERAGE=1 (but at least they're running) https://travis-ci.org/myint/autoflake/jobs/324440857

[myint]

I think AUTOFLAKE_COVERAGE=1 needs to go on the line that currently says:

- coverage run --include='autoflake.py,test_autoflake.py' test_autoflake.py

So instead of in env:, put it in:

- AUTOFLAKE_COVERAGE=1 coverage run --include='autoflake.py,test_autoflake.py' test_autoflake.py

[coveralls]

Coverage decreased (-0.4%) to 97.783% when pulling 2de59b7 on andrewda:dupe-keys into b482463 on myint:master.

[myint]

I just noticed something else missing in .travis.yml. The coverage results need to be combined. This is because we now have multiple processes generating coverage results. So after:

https://github.com/myint/autoflake/blob/b4824635ecdec24a87c32ad6a1c61f6dce09b785/.travis.yml#L26

you need:

- coverage combine

[coveralls]

Coverage decreased (-0.4%) to 97.783% when pulling 917ff27 on andrewda:dupe-keys into b482463 on myint:master.

[jayvdb]

coverage combine wont do anything, as there is only one coverage run in each travis job, and each travis job is a separate VM. There is a very hacky way to merge the coverage data from multiple travis jobs, but it is typically left to coveralls/codecov to merge reports from multiple CI jobs.

[coveralls]

Coverage decreased (-0.3%) to 97.906% when pulling 51f52c9 on andrewda:dupe-keys into b482463 on myint:master.

[myint]

@jayvdb I tested it locally. It does work using make coverage if combined. We do have multiple processes. That is what the end-to-end tests do.

https://github.com/myint/autoflake/blob/master/test_autoflake.py#L24-L37

[coveralls]

Coverage decreased (-21.08%) to 77.156% when pulling 1f04a5e on andrewda:dupe-keys into b482463 on myint:master.

[myint]

By the way, the massive decrease is expected given that the results aren't being combined. Hence my previous comment. 😄

[myint]

You might also need --branch --parallel-mode in the .travis.yml.

diff --git a/.travis.yml b/.travis.yml
index 8382206..3217848 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -23,7 +23,7 @@ script:

 after_success:
     - pip install --quiet coverage
-    - AUTOFLAKE_COVERAGE=1 coverage run --include='autoflake.py,test_autoflake.py' test_autoflake.py
+    - AUTOFLAKE_COVERAGE=1 coverage run --branch --parallel-mode --include='autoflake.py,test_autoflake.py' test_autoflake.py
     - coverage combine
     - coverage report --show-missing

[jayvdb]

Oh I see coverage run is being invoked within the test suite.

[myint]

Here is the output when I test this locally with --branch --parallel and combine. So it should work on Travis if you add those to .travis.yml. It gives essentially full coverage for autoflake.py, which it originally had:

diff --git a/Makefile b/Makefile
index a469920..5bb3065 100644
--- a/Makefile
+++ b/Makefile
@@ -12,7 +12,8 @@ check:

 coverage:
        @coverage erase
-       @AUTOFLAKE_COVERAGE=1 ${coverage} run test_autoflake.py
+       @AUTOFLAKE_COVERAGE=1 coverage run --branch --parallel --include='autoflake.py,test_autoflake.py' test_autoflake.py
+       @coverage combine
        @coverage report
        @coverage html
        @python -m webbrowser -n "file://${PWD}/htmlcov/index.html"

$ make coverage
..............................................................................................
----------------------------------------------------------------------
Ran 94 tests in 1.572s

OK
Name                Stmts   Miss Branch BrPart  Cover
-----------------------------------------------------
autoflake.py          429      8    198      6    98%
test_autoflake.py     383      0      6      1    99%
-----------------------------------------------------
TOTAL                 812      8    204      7    99%

[coveralls]

Coverage decreased (-21.08%) to 77.156% when pulling 5cd2b3d on andrewda:dupe-keys into b482463 on myint:master.

[myint]

Great work! And thanks for discovering/fixing my coverage bug!

[coveralls]

Coverage increased (+1.3%) to 99.507% when pulling e35b515 on andrewda:dupe-keys into b482463 on myint:master.

[coveralls]

Coverage increased (+1.3%) to 99.507% when pulling 4573c58 on andrewda:dupe-keys into b482463 on myint:master.