Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pythonPackages.pycurl: disable flakey memory tests #72039

Closed
wants to merge 1 commit into from

Conversation

@jonringer
Copy link
Contributor

jonringer commented Oct 26, 2019

Motivation for this change

fixes #71975

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Notify maintainers

cc @

@jonringer
Copy link
Contributor Author

jonringer commented Oct 26, 2019

for the port to stable, you may need to cherry-pick a few commits:

$ git log pkgs/development/python-modules/pycurl/default.nix
commit a014874750ab35ceb7dad3adc42bceef3a551029 (HEAD -> fix-pycurl, jonringer/fix-pycurl)
Author: Jonathan Ringer <jonringer117@gmail.com>
Date:   Sat Oct 26 10:44:00 2019 -0700

    pythonPackages.pycurl: disable flakey memory tests

commit 641148260cdde296ea67fea4b6d979bd87d577a8
Author: Jonathan Ringer <jonringer117@gmail.com>
Date:   Thu Oct 17 14:44:05 2019 -0700

    pythonPackages.pycurl: fix tests

commit d5beb9fb5626cd5442ec7549a8960becc48ef192
Author: Mario Rodas <marsam@users.noreply.github.com>
Date:   Thu Aug 22 16:20:00 2019 -0500

    pythonPackages.pycurl: fix build on darwin

@jonringer
Copy link
Contributor Author

jonringer commented Oct 26, 2019

[23 built (4 failed), 59 copied (64.4 MiB), 14.8 MiB DL]
error: build of '/nix/store/6bf90vq6bm5p0m5lnqw7k9v16jda2b5l-env.drv' failed
https://github.com/NixOS/nixpkgs/pull/72039
4 package failed to build:
python27Packages.thumbor python37Packages.patator python38Packages.nvchecker python38Packages.patator

21 package were build:
pyCA python27Packages.blivet python27Packages.koji python27Packages.osc python27Packages.pycurl python27Packages.pykickstart python27Packages.rpkg python27Packages.urlgrabber python37Packages.bugwarrior python37Packages.koji python37Packages.nvchecker python37Packages.osc python37Packages.pycurl python37Packages.urlgrabber python38Packages.bugwarrior python38Packages.koji python38Packages.osc python38Packages.pycurl python38Packages.urlgrabber system-config-printer udocker

the 4 broken, are broken on master:

broken builds logs
[10:52:07] jon@jon-desktop /home/jon/projects/nixpkgs (master)
$ nix build -f . --keep-going python27Packages.thumbor python37Packages.patator python38Packages.nvchecker python38Packages.patator
builder for '/nix/store/58sliyi6fi8dp54d0r5kish0ryl5rzfa-python3.8-pysnmp-4.4.12.drv' failed with exit code 1; last 10 log lines:
  building
  Executing setuptoolsBuildPhase
  Traceback (most recent call last):
    File "nix_run_setup", line 8, in <module>
      exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
    File "setup.py", line 66, in <module>
      observed_version = [int(x) for x in setuptools.__version__.split('.')]
    File "setup.py", line 66, in <listcomp>
      observed_version = [int(x) for x in setuptools.__version__.split('.')]
  ValueError: invalid literal for int() with base 10: 'post20191022'
cannot build derivation '/nix/store/18vd9b1pp7hi7x90v2szpndswm73n688-python3.8-patator-0.7.drv': 1 dependencies couldn't be built
builder for '/nix/store/49sixi010i4f6zk953fk0zg0xyikxc9x-python3.7-pysnmp-4.4.12.drv' failed with exit code 1; last 10 log lines:
  building
  Executing setuptoolsBuildPhase
  Traceback (most recent call last):
    File "nix_run_setup", line 8, in <module>
      exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
    File "setup.py", line 66, in <module>
      observed_version = [int(x) for x in setuptools.__version__.split('.')]
    File "setup.py", line 66, in <listcomp>
      observed_version = [int(x) for x in setuptools.__version__.split('.')]
  ValueError: invalid literal for int() with base 10: 'post20191022'
cannot build derivation '/nix/store/bdd7zm58hrm4zxgf785a7fx4c9z4ai59-python3.7-patator-0.7.drv': 1 dependencies couldn't be built
builder for '/nix/store/ckkriycpas609aah51vv4q17vbza8pz5-python2.7-remotecv-2.2.2.drv' failed with exit code 1; last 10 log lines:
    File "/nix/store/jcs9qj1knjghd07k8xm4mjm0qs12nkgd-python2.7-Pillow-6.2.0/lib/python2.7/site-packages/PIL/Image.py", line 930, in convert
      self.load()
    File "/nix/store/jcs9qj1knjghd07k8xm4mjm0qs12nkgd-python2.7-Pillow-6.2.0/lib/python2.7/site-packages/PIL/ImageFile.py", line 249, in load
      "(%d bytes not processed)" % len(b)
  IOError: image file is truncated (2 bytes not processed)

  ----------------------------------------------------------------------
  Ran 23 tests in 3.295s

  FAILED (errors=3)
cannot build derivation '/nix/store/ndvri99ba6vjvbkpf7cdasv0pkahcmld-python2.7-thumbor-6.7.0.drv': 1 dependencies couldn't be built
builder for '/nix/store/9wpzh2h9kyvwqj450y4h03hcl8y5ki6q-python3.8-python-rapidjson-0.8.0.drv' failed with exit code 1; last 10 log lines:
  E         + [123,123123123123123123123123]

  tests/test_enum.py:22: AssertionError
  =============================== warnings summary ===============================
  /nix/store/sv7c226g55y39ryxnrwxxdg01s184y1p-python3.8-pytest-5.2.1/lib/python3.8/site-packages/_pytest/mark/structures.py:321
    /nix/store/sv7c226g55y39ryxnrwxxdg01s184y1p-python3.8-pytest-5.2.1/lib/python3.8/site-packages/_pytest/mark/structures.py:321: PytestUnknownMarkWarning: Unknown pytest.mark.unit - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/latest/mark.html
      warnings.warn(

  -- Docs: https://docs.pytest.org/en/latest/warnings.html
  ======= 1 failed, 679 passed, 16 skipped, 2 xfailed, 1 warnings in 2.44s =======
cannot build derivation '/nix/store/z1lp4i112cidl030xz0k91kq1l40zrkg-python3.8-structlog-19.1.0.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/yyr7khz8g643y3b889jmmw2rmzldq7ys-python3.8-nvchecker-1.4.4.drv': 1 dependencies couldn't be built
[0 built (4 failed)]
error: build of '/nix/store/18vd9b1pp7hi7x90v2szpndswm73n688-python3.8-patator-0.7.drv', '/nix/store/bdd7zm58hrm4zxgf785a7fx4c9z4ai59-python3.7-patator-0.7.drv', '/nix/store/ndvri99ba6vjvbkpf7cdasv0pkahcmld-python2.7-thumbor-6.7.0.drv', '/nix/store/yyr7khz8g643y3b889jmmw2rmzldq7ys-python3.8-nvchecker-1.4.4.drv' failed
Copy link
Contributor

joepie91 left a comment

This will probably be insufficient, as there are other kinds of failing tests; see eg. #55945 (comment). I don't know if anyone has figured out a common cause yet, but if not, we should probably just disable the tests entirely so as to not break builds?

@jonringer
Copy link
Contributor Author

jonringer commented Oct 27, 2019

Even if it's insufficient, people are still having difficulties on it in a non-deterministic manner

@jonringer
Copy link
Contributor Author

jonringer commented Oct 27, 2019

https://github.com/NixOS/nixpkgs/pull/72039
3 package are marked as broken and were skipped:
fedpkg python27Packages.nixpart python27Packages.nixpart0

4 package failed to build:
python27Packages.thumbor python37Packages.patator python38Packages.nvchecker python38Packages.patator

21 package were build:
pyCA python27Packages.blivet python27Packages.koji python27Packages.osc python27Packages.pycurl python27Packages.pykickstart python27Packages.rpkg python27Packages.urlgrabber python37Packages.bugwarrior python37Packages.koji python37Packages.nvchecker python37Packages.osc python37Packages.pycurl python37Packages.urlgrabber python38Packages.bugwarrior python38Packages.koji python38Packages.osc python38Packages.pycurl python38Packages.urlgrabber system-config-printer udocker
@joepie91
Copy link
Contributor

joepie91 commented Oct 27, 2019

Right, I get that. My point is that those non-deterministic issues aren't solved by the current changes because they occur in other tests as well.

@jonringer
Copy link
Contributor Author

jonringer commented Oct 27, 2019

other than submitting an issue upstream, which some may be nixos specific, I'm not sure what to do

@joepie91
Copy link
Contributor

joepie91 commented Oct 27, 2019

Disabling the install check is the workaround I've applied locally to make the build succeed, and perhaps that'd be the right thing to do here, at least until the root cause gets fixed.

It's perhaps a bit rigorous, but given that the failures occur in various completely unrelated tests, I'm not sure there are any other options - it looks to me like the problem is in how the tests get run (unintentional concurrency?), not in any one specific test, and so any non-concurrency-safe test might fail.

@jonringer
Copy link
Contributor Author

jonringer commented Jan 9, 2020

Closing, no desire currently to tackle this

@jonringer jonringer closed this Jan 9, 2020
@jonringer jonringer deleted the jonringer:fix-pycurl branch Jan 9, 2020
@Infinisil Infinisil mentioned this pull request Jan 19, 2020
1 of 1 task complete
@jonringer jonringer restored the jonringer:fix-pycurl branch Jan 19, 2020
@jonringer jonringer reopened this Jan 19, 2020
@jonringer
Copy link
Contributor Author

jonringer commented Jan 19, 2020

@GrahamcOfBorg build python3Packages.pycurl

@jonringer jonringer closed this Jan 19, 2020
@jonringer
Copy link
Contributor Author

jonringer commented Jan 19, 2020

this was merged, not sure why github just shows it as me pushing to master

@jonringer jonringer deleted the jonringer:fix-pycurl branch Jan 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

2 participants
You can’t perform that action at this time.