diffoscope is broken

[dermetfan]

Steps to reproduce

❯ nix-build -A diffoscope '<nixpkgs>'
these derivations will be built:
  /nix/store/pwm2jh1q49diw89rb1nqay7lwp0ynpq2-python3.6-libarchive-c-2.5.drv
  /nix/store/kfzvsk1p2hvsgr4rvzk2bqn9h2a2m0fd-diffoscope-77.drv
building path(s) ‘/nix/store/2lg2d62j2i6v0hafi478xmhn6lhjj1vp-python3.6-libarchive-c-2.5’
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
unpacking sources
unpacking source archive /nix/store/hg1vfsba6kq1478zxaawdik62r7a9xwi-libarchive-c-2.5.tar.gz
source root is libarchive-c-2.5
setting SOURCE_DATE_EPOCH to timestamp 1469298764 of file libarchive-c-2.5/setup.cfg
patching sources
configuring
building
running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/libarchive
copying libarchive/write.py -> build/lib/libarchive
copying libarchive/read.py -> build/lib/libarchive
copying libarchive/ffi.py -> build/lib/libarchive
copying libarchive/extract.py -> build/lib/libarchive
copying libarchive/exception.py -> build/lib/libarchive
copying libarchive/entry.py -> build/lib/libarchive
copying libarchive/__init__.py -> build/lib/libarchive
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/libarchive
copying build/lib/libarchive/__init__.py -> build/bdist.linux-x86_64/wheel/libarchive
copying build/lib/libarchive/entry.py -> build/bdist.linux-x86_64/wheel/libarchive
copying build/lib/libarchive/exception.py -> build/bdist.linux-x86_64/wheel/libarchive
copying build/lib/libarchive/extract.py -> build/bdist.linux-x86_64/wheel/libarchive
copying build/lib/libarchive/ffi.py -> build/bdist.linux-x86_64/wheel/libarchive
copying build/lib/libarchive/read.py -> build/bdist.linux-x86_64/wheel/libarchive
copying build/lib/libarchive/write.py -> build/bdist.linux-x86_64/wheel/libarchive
running install_egg_info
running egg_info
writing libarchive_c.egg-info/PKG-INFO
writing dependency_links to libarchive_c.egg-info/dependency_links.txt
writing top-level names to libarchive_c.egg-info/top_level.txt
reading manifest file 'libarchive_c.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'libarchive_c.egg-info/SOURCES.txt'
Copying libarchive_c.egg-info to build/bdist.linux-x86_64/wheel/libarchive_c-2.5-py3.6.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/libarchive_c-2.5.dist-info/WHEEL
installing
/tmp/nix-build-python3.6-libarchive-c-2.5.drv-0/libarchive-c-2.5/dist /tmp/nix-build-python3.6-libarchive-c-2.5.drv-0/libarchive-c-2.5
Processing ./libarchive_c-2.5-py2.py3-none-any.whl
Installing collected packages: libarchive-c
Successfully installed libarchive-c-2.5
/tmp/nix-build-python3.6-libarchive-c-2.5.drv-0/libarchive-c-2.5
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/2lg2d62j2i6v0hafi478xmhn6lhjj1vp-python3.6-libarchive-c-2.5
stripping (with flags -S) in /nix/store/2lg2d62j2i6v0hafi478xmhn6lhjj1vp-python3.6-libarchive-c-2.5/lib 
patching script interpreter paths in /nix/store/2lg2d62j2i6v0hafi478xmhn6lhjj1vp-python3.6-libarchive-c-2.5
checking for references to /tmp/nix-build-python3.6-libarchive-c-2.5.drv-0 in /nix/store/2lg2d62j2i6v0hafi478xmhn6lhjj1vp-python3.6-libarchive-c-2.5...
running install tests
============================= test session starts ==============================
platform linux -- Python 3.6.2, pytest-3.2.1, py-1.4.34, pluggy-0.4.0
rootdir: /tmp/nix-build-python3.6-libarchive-c-2.5.drv-0/libarchive-c-2.5, inifile:
collected 16 items

tests/test_convert.py .
tests/test_entry.py ..F...
tests/test_errors.py ....
tests/test_rwx.py ....

=================================== FAILURES ===================================
_________________ test_check_archiveentry_using_python_testtar _________________

    def test_check_archiveentry_using_python_testtar():
>       check_entries(join(data_dir, 'testtar.tar'))

tests/test_entry.py:59: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

test_file = '/tmp/nix-build-python3.6-libarchive-c-2.5.drv-0/libarchive-c-2.5/tests/data/testtar.tar'
regen = False, ignore = []

    def check_entries(test_file, regen=False, ignore=''):
        ignore = ignore.split()
        fixture_file = test_file + '.json'
        if regen:
            entries = list(get_entries(test_file))
            with open(fixture_file, 'w', encoding='UTF-8') as ex:
                json.dump(entries, ex, indent=2)
        with open(fixture_file, encoding='UTF-8') as ex:
            expected = json.load(ex)
        actual = list(get_entries(test_file))
        for e1, e2 in zip(actual, expected):
            for key in ignore:
                e1.pop(key)
                e2.pop(key)
>           assert e1 == e2
E           AssertionError: assert {'isblk': Fal...': False, ...} == {'isblk': Fals...': False, ...}
E             Omitting 12 identical items, use -vv to show
E             Differing items:
E             {'size': 7011} != {'size': 0}
E             Use -v to get the full diff

tests/test_entry.py:92: AssertionError
----------------------------- Captured stderr call -----------------------------
Pathname can't be converted from UTF-8 to current locale.
============================== 1 tests deselected ==============================
============== 1 failed, 14 passed, 1 deselected in 0.19 seconds ===============
builder for ‘/nix/store/pwm2jh1q49diw89rb1nqay7lwp0ynpq2-python3.6-libarchive-c-2.5.drv’ failed with exit code 1
cannot build derivation ‘/nix/store/kfzvsk1p2hvsgr4rvzk2bqn9h2a2m0fd-diffoscope-77.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/kfzvsk1p2hvsgr4rvzk2bqn9h2a2m0fd-diffoscope-77.drv’ failed

Technical details

• System: 18.03pre114421.799435b7ca (Impala)
• Nix version: nix-env (Nix) 1.11.14
• Nixpkgs version: 18.03pre114421.799435b7ca
• Sandboxing enabled: no

[peterhoeg]

We have some WIP on this - #28785 as well as work done by @dtzWill so it will be fixed RealSoonNow(tm).

[edolstra]

The release branch has a workaround for this: 22d4630

Didn't test diffoscope though.

[FRidh]

diffoscope is now fixed on master and release-17.09

[dermetfan]

Sorry, I cannot build on master:

❯ git rev-parse HEAD
3a93d9c516576223029698a3c4aafeb613d3eee5
❯ nix-build -A diffoscope
these derivations will be built:
  /nix/store/zpf60nw8kphffr40i2397wxr5y669lg8-python3.6-mccabe-0.6.1.drv
  /nix/store/zx8n6wkil1bfsvb8ipypl1l0hrh8r718-python3.6-flake8-3.4.1.drv
  /nix/store/2i9fkzwdymg5na6mdjmjs58y2yq4rab3-python3.6-hypothesis-3.11.1.drv
  /nix/store/6rcj9bm8lj9j7habh3y92j8k09a8v6dk-mesa-noglu-17.1.7.drv
  /nix/store/7p37wlp6pygxan3wkh4zcv30ymk47qa8-python3.6-chardet-3.0.4.drv
  /nix/store/bfh1mn206b78mbl9plk4cz0fi9fz62zf-python3.6-python-debian-0.1.23.drv
  /nix/store/q8jfnnm77gzv6hdzch909cfgxzlgsqbx-cairo-1.14.10.drv
  /nix/store/m92289qsrd8bnflgrk1s87m2g29miyci-poppler-utils-0.56.0.drv
  /nix/store/vfjqyr3zvsjrgffffcd75i553c5svd33-diffoscope-85.drv
these paths will be fetched (40.16 MiB download, 259.12 MiB unpacked):
  /nix/store/0615swmkk3y1hpyypw0516mmg124hl9q-llvm-4.0.1-lib
  /nix/store/21v3hpaw4gy2bmrgmngvcs955rpw30fv-glib-2.52.3-dev
  /nix/store/37kc9304nhs2v8xgf7mzvi8pm6m55gzw-glib-2.52.3
  /nix/store/6z161qapcbnn72kp8i4kk0x6q96zgcrq-valgrind-3.13.0-dev
  /nix/store/c651bnp37xgig65zxy2dvfqj3ad9h8h8-llvm-4.0.1
  /nix/store/p6nwfpsl9z0s9isqlmf7blxn9501szkv-valgrind-3.13.0
fetching path ‘/nix/store/37kc9304nhs2v8xgf7mzvi8pm6m55gzw-glib-2.52.3’...
fetching path ‘/nix/store/0615swmkk3y1hpyypw0516mmg124hl9q-llvm-4.0.1-lib’...
fetching path ‘/nix/store/p6nwfpsl9z0s9isqlmf7blxn9501szkv-valgrind-3.13.0’...
building path(s) ‘/nix/store/dgsavzxd9kivwxgzmaibmyw60qyrqqdv-python3.6-mccabe-0.6.1’

*** Downloading ‘https://cache.nixos.org/nar/1hcmfpy1qqsihczsixyr2iz8425g4d7nss0a1nxjma901dm0yw8c.nar.xz’ (signed by ‘cache.nixos.org-1’) to ‘/nix/store/37kc9304nhs2v8xgf7mzvi8pm6m55gzw-glib-2.52.3’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
*** Downloading ‘https://cache.nixos.org/nar/07lkq7n1ksbg0w1r90r2rg73gy2d2lgy6fqvh4fax7f1xsv771bz.nar.xz’ (signed by ‘cache.nixos.org-1’) to ‘/nix/store/0615swmkk3y1hpyypw0516mmg124hl9q-llvm-4.0.1-lib’...

*** Downloading ‘https://cache.nixos.org/nar/1v00w778qw2y5fh1i5vdfa1pxjsnbhfnifzn7mnw9m0vswb95lr7.nar.xz’ (signed by ‘cache.nixos.org-1’) to ‘/nix/store/p6nwfpsl9z0s9isqlmf7blxn9501szkv-valgrind-3.13.0’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0unpacking sources
unpacking source archive /nix/store/fqh1mqbls9k9k6p9dac6wlvr1wrjjadl-mccabe-0.6.1.tar.gz
source root is mccabe-0.6.1
setting SOURCE_DATE_EPOCH to timestamp 1485468785 of file mccabe-0.6.1/setup.cfg
patching sources
configuring
building

  2 4518k    2 97478    0     0  97478      0  0:00:47 --:--:--  0:00:47  135k
  4 1973k    4 97478    0     0  97478      0  0:00:20 --:--:--  0:00:20  129k
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0running bdist_wheel
running build
running build_py
creating build
creating build/lib
copying mccabe.py -> build/lib
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
copying build/lib/mccabe.py -> build/bdist.linux-x86_64/wheel
running install_egg_info
running egg_info
writing mccabe.egg-info/PKG-INFO
writing dependency_links to mccabe.egg-info/dependency_links.txt
writing entry points to mccabe.egg-info/entry_points.txt
writing top-level names to mccabe.egg-info/top_level.txt
reading manifest file 'mccabe.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'mccabe.egg-info/SOURCES.txt'
Copying mccabe.egg-info to build/bdist.linux-x86_64/wheel/mccabe-0.6.1-py3.6.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/mccabe-0.6.1.dist-info/WHEEL
installing
/tmp/nix-build-python3.6-mccabe-0.6.1.drv-0/mccabe-0.6.1/dist /tmp/nix-build-python3.6-mccabe-0.6.1.drv-0/mccabe-0.6.1

  1 12.4M    1  135k    0     0   135k      0  0:01:33  0:00:01  0:01:32 85894
 14 1973k   14  287k    0     0   287k      0  0:00:06  0:00:01  0:00:05  174k
  5 4518k    5  255k    0     0   255k      0  0:00:17  0:00:01  0:00:16  156kProcessing ./mccabe-0.6.1-py2.py3-none-any.whl
Installing collected packages: mccabe
Successfully installed mccabe-0.6.1
/tmp/nix-build-python3.6-mccabe-0.6.1.drv-0/mccabe-0.6.1
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/dgsavzxd9kivwxgzmaibmyw60qyrqqdv-python3.6-mccabe-0.6.1
stripping (with flags -S) in /nix/store/dgsavzxd9kivwxgzmaibmyw60qyrqqdv-python3.6-mccabe-0.6.1/lib 
patching script interpreter paths in /nix/store/dgsavzxd9kivwxgzmaibmyw60qyrqqdv-python3.6-mccabe-0.6.1
checking for references to /tmp/nix-build-python3.6-mccabe-0.6.1.drv-0 in /nix/store/dgsavzxd9kivwxgzmaibmyw60qyrqqdv-python3.6-mccabe-0.6.1...

  9 4518k    9  415k    0     0   207k      0  0:00:21  0:00:02  0:00:19  158k
  2 12.4M    2  356k    0     0   178k      0  0:01:11  0:00:02  0:01:09  135k
 21 1973k   21  431k    0     0   215k      0  0:00:09  0:00:02  0:00:07  162krunning install tests
running pytest
Traceback (most recent call last):
  File "nix_run_setup.py", line 8, in <module>
    exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
  File "setup.py", line 58, in <module>
    'Topic :: Software Development :: Quality Assurance',
  File "/nix/store/mfy3cg23l8h5d7398imx2cwgvb2024fz-python3.6-distutils.cfg/lib/python3.6/site-packages/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/nix/store/mfy3cg23l8h5d7398imx2cwgvb2024fz-python3.6-distutils.cfg/lib/python3.6/site-packages/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/nix/store/mfy3cg23l8h5d7398imx2cwgvb2024fz-python3.6-distutils.cfg/lib/python3.6/site-packages/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/nix/store/dyckx2xxyhrbbb1yagvzij2q9kmnn350-python3.6-pytest-runner-2.6.2/lib/python3.6/site-packages/ptr.py", line 63, in run
    self._build_egg_fetcher()
  File "/nix/store/dyckx2xxyhrbbb1yagvzij2q9kmnn350-python3.6-pytest-runner-2.6.2/lib/python3.6/site-packages/ptr.py", line 97, in _build_egg_fetcher
    for key in opts.keys():
RuntimeError: dictionary changed size during iteration
builder for ‘/nix/store/zpf60nw8kphffr40i2397wxr5y669lg8-python3.6-mccabe-0.6.1.drv’ failed with exit code 1
cannot build derivation ‘/nix/store/zx8n6wkil1bfsvb8ipypl1l0hrh8r718-python3.6-flake8-3.4.1.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/2i9fkzwdymg5na6mdjmjs58y2yq4rab3-python3.6-hypothesis-3.11.1.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/7p37wlp6pygxan3wkh4zcv30ymk47qa8-python3.6-chardet-3.0.4.drv’: 1 dependencies couldn't be built
cannot build derivation ‘/nix/store/bfh1mn206b78mbl9plk4cz0fi9fz62zf-python3.6-python-debian-0.1.23.drv’: 1 dependencies couldn't be built
killing process 31519
killing process 31518
killing process 31517
cannot build derivation ‘/nix/store/vfjqyr3zvsjrgffffcd75i553c5svd33-diffoscope-85.drv’: 1 dependencies couldn't be built
error: build of ‘/nix/store/vfjqyr3zvsjrgffffcd75i553c5svd33-diffoscope-85.drv’ failed

[peterhoeg]

It's pythonPackages.mccabe that's broken. Any thoughts @garbas ?

[FRidh]

This is most likely due to my #29072

[FRidh]

Yep, its due to the setuptools bump in a26ae76.
Its due to to distutils-cfg in b73e3bf
for which I didn't test/rebuild all packages.

[FRidh]

I've reverted b73e3bf for now on master and 17.09. diffoscope now builds again on master.