Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pythonPackages: fix nix-shell development environments #117157

Closed
wants to merge 1 commit into from

Conversation

cript0nauta
Copy link
Contributor

A change in setuptools v49.0.0 broke the development mode of Python
packages inside nix-shell. This workaround fixes it.

Closes #108525.

Note that this change will require a rebuild of all Python packages. I only tested the change using the example described in the issue (using the sampleproject package). I'm not sure if this change will introduce a regression, so if possible hydra should check that all Python packages work ok before merging this.

Motivation for this change

Fix #108525.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@r-rmcgibbo
Copy link

r-rmcgibbo commented Mar 22, 2021

Result of nixpkgs-review pr 117157 at bb560f83 run on aarch64-linux 1

919 packages marked as broken and skipped:
  • Literate
  • OVMF-CSM
  • acl2
  • adoptopenjdk-hotspot-bin-13
  • adoptopenjdk-hotspot-bin-14
  • adoptopenjdk-jre-hotspot-bin-13
  • adoptopenjdk-jre-hotspot-bin-14
  • agdaPackages.iowa-stdlib
  • aldor
  • alliance
  • ...
1 package failed to build:
21919 packages skipped due to time constraints:
  • AusweisApp2
  • EBTKS
  • EmptyEpsilon
  • Fabric
  • MIDIVisualizer
  • MMA
  • OVMF
  • OVMF-secureBoot
  • R
  • SDL
  • ...
235 packages built successfully:
  • fontconfig
  • fribidi
  • gd
  • glib (gnome2.glib ,gnome3.glib)
  • gtk-doc (gnome2.gtkdoc)
  • gts
  • hidapi
  • libdrm
  • libfido2
  • libglvnd
  • libmicrodns
  • libsysprof-capture
  • libusb1
  • libvdpau
  • libxkbcommon (libxkbcommon_8)
  • meson
  • openssh
  • python38Packages.Babel
  • python38Packages.Mako
  • python38Packages.PyStemmer
  • python38Packages.alabaster
  • python38Packages.apipkg
  • python38Packages.appdirs
  • python38Packages.astor
  • python38Packages.async-timeout
  • python38Packages.async_generator
  • python38Packages.atomicwrites
  • python38Packages.attrs
  • python38Packages.automat
  • python38Packages.backcall
  • python38Packages.blinker
  • python38Packages.certifi
  • python38Packages.click
  • python38Packages.colorama
  • python38Packages.constantly
  • python38Packages.cookies
  • python38Packages.coverage
  • python38Packages.cppy
  • python38Packages.crashtest
  • python38Packages.cryptography_vectors
  • python38Packages.cycler
  • python38Packages.cython
  • python38Packages.decorator
  • python38Packages.distlib
  • python38Packages.docutils
  • python38Packages.ecdsa
  • python38Packages.entrypoints
  • python38Packages.eradicate
  • python38Packages.execnet
  • python38Packages.extras
  • python38Packages.filelock
  • python38Packages.flake8
  • python38Packages.flaky
  • python38Packages.flit-core
  • python38Packages.fontforge
  • python38Packages.freezegun
  • python38Packages.gunicorn
  • python38Packages.gyp
  • python38Packages.hyperlink
  • python38Packages.idna
  • python38Packages.imagesize
  • python38Packages.importlib-metadata
  • python38Packages.incremental
  • python38Packages.iniconfig
  • python38Packages.intreehooks
  • python38Packages.ipython_genutils
  • python38Packages.iso8601
  • python38Packages.itsdangerous
  • python38Packages.jdatetime
  • python38Packages.jinja2
  • python38Packages.jpylyzer
  • python38Packages.jupyterlab-pygments
  • python38Packages.lazy-object-proxy
  • python38Packages.libevdev
  • python38Packages.linecache2
  • python38Packages.lxml
  • python38Packages.m2r
  • python38Packages.markupsafe
  • python38Packages.mccabe
  • python38Packages.mistune
  • python38Packages.mock
  • python38Packages.mongoengine
  • python38Packages.more-itertools
  • python38Packages.nest-asyncio
  • python38Packages.nose
  • python38Packages.nose-exclude
  • python38Packages.nose_warnings_filters
  • python38Packages.olefile
  • python38Packages.outcome
  • python38Packages.packaging
  • python38Packages.pandocfilters
  • python38Packages.pastel
  • python38Packages.pbr
  • python38Packages.pep517
  • python38Packages.pexpect
  • python38Packages.pip
  • python38Packages.pkginfo
  • python38Packages.pluggy
  • python38Packages.pretend
  • python38Packages.prometheus_client
  • python38Packages.psutil
  • python38Packages.ptyprocess
  • python38Packages.py
  • python38Packages.pyasn1
  • python38Packages.pyasn1-modules
  • python38Packages.pycodestyle
  • python38Packages.pycparser
  • python38Packages.pydocstyle
  • python38Packages.pyext
  • python38Packages.pyflakes
  • python38Packages.pygments
  • python38Packages.pyhamcrest
  • python38Packages.pylama
  • python38Packages.pymongo
  • python38Packages.pyparsing
  • python38Packages.pyperf
  • python38Packages.pysocks
  • python38Packages.pytest (python38Packages.pytest_6)
  • python38Packages.pytest-asyncio
  • python38Packages.pytestcov (python38Packages.pytest-cov)
  • python38Packages.pytest-expect
  • python38Packages.pytest-flake8
  • python38Packages.pytest-forked
  • python38Packages.pytest-freezegun
  • python38Packages.pytest-mock
  • python38Packages.pytestrunner (python38Packages.pytest-runner)
  • python38Packages.pytest-subtests
  • python38Packages.pytest-xdist (python38Packages.pytest_xdist)
  • python38Packages.python-dateutil
  • python38Packages.python_mimeparse
  • python38Packages.pytoml
  • python38Packages.pytz
  • python38Packages.pyyaml
  • python38Packages.redis
  • python38Packages.rednose
  • python38Packages.ruamel_base
  • python38Packages.ruamel_yaml
  • python38Packages.ruamel_yaml_clib
  • python38Packages.semantic-version
  • python38Packages.send2trash
  • python38Packages.setuptools
  • python38Packages.setuptools-rust
  • python38Packages.setuptools-scm (python38Packages.setuptools_scm)
  • python38Packages.simplejson
  • python38Packages.six
  • python38Packages.snowballstemmer
  • python38Packages.sortedcontainers
  • python38Packages.soupsieve
  • python38Packages.sphinxcontrib-applehelp
  • python38Packages.sphinxcontrib-devhelp
  • python38Packages.sphinxcontrib-htmlhelp
  • python38Packages.sphinxcontrib-jsmath
  • python38Packages.sphinxcontrib-qthelp
  • python38Packages.sphinxcontrib-serializinghtml
  • python38Packages.sphinxcontrib-websupport
  • python38Packages.sqlparse
  • python38Packages.termstyle
  • python38Packages.testpath
  • python38Packages.text-unidecode
  • python38Packages.toml
  • python38Packages.tomlkit
  • python38Packages.tornado
  • python38Packages.traceback2
  • python38Packages.typing-extensions
  • python38Packages.tzlocal
  • python38Packages.u-msgpack-python
  • python38Packages.ukpostcodeparser
  • python38Packages.umalqurra
  • python38Packages.unittest2
  • python38Packages.vcversioner
  • python38Packages.wcwidth
  • python38Packages.webencodings
  • python38Packages.wheel
  • python38Packages.whoosh
  • python38Packages.wrapt
  • python38Packages.zipp
  • python38Packages.zope_event
  • python38Packages.zope_interface
  • python39Packages.atomicwrites
  • python39Packages.attrs
  • python39Packages.coverage
  • python39Packages.docutils
  • python39Packages.flit-core
  • python39Packages.importlib-metadata
  • python39Packages.iniconfig
  • python39Packages.more-itertools
  • python39Packages.nose
  • python39Packages.packaging
  • python39Packages.pip
  • python39Packages.pluggy
  • python39Packages.py
  • python39Packages.pygments
  • python39Packages.pyparsing
  • python39Packages.setuptools
  • python39Packages.setuptools-scm (python39Packages.setuptools_scm)
  • python39Packages.six
  • python39Packages.sortedcontainers
  • python39Packages.toml
  • python39Packages.wcwidth
  • python39Packages.wheel
  • python39Packages.zipp
  • shared-mime-info
  • systemdMinimal
  • uddup
  • volk
  • xkeyboard_config (xorg.xkeyboardconfig)
  • xlibsWrapper (xorg.xlibsWrapper)
  • xorg.libICE
  • xorg.libSM
  • xorg.libX11
  • xorg.libXau
  • xorg.libXcomposite
  • xorg.libXcursor
  • xorg.libXdamage
  • xorg.libXdmcp
  • xorg.libXext
  • xorg.libXfixes
  • xorg.libXft
  • xorg.libXi
  • xorg.libXinerama
  • xorg.libXrandr
  • xorg.libXrender
  • xorg.libXt
  • xorg.libXtst
  • xorg.libXv
  • xorg.libXvMC
  • xorg.libXxf86vm
  • xorg.libxcb
  • xorg.libxshmfence
  • xorg.xcbutil
  • xorg.xcbutilimage
  • xorg.xcbutilkeysyms
  • xorg.xcbutilrenderutil
  • xorg.xcbutilwm
  • xorg.xorgproto

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement.


Result of nixpkgs-review pr 117157 at bb560f83 run on x86_64-linux 1

825 packages marked as broken and skipped:
  • adobe-reader
  • adoptopenjdk-hotspot-bin-13
  • adoptopenjdk-hotspot-bin-14
  • adoptopenjdk-jre-hotspot-bin-13
  • adoptopenjdk-jre-hotspot-bin-14
  • adoptopenjdk-jre-openj9-bin-13
  • adoptopenjdk-jre-openj9-bin-14
  • adoptopenjdk-openj9-bin-13
  • adoptopenjdk-openj9-bin-14
  • agdaPackages.iowa-stdlib
  • ...
1 package failed to build:
24172 packages skipped due to time constraints:
  • AusweisApp2
  • EBTKS
  • EmptyEpsilon
  • Fabric
  • Literate
  • MIDIVisualizer
  • MMA
  • OSCAR
  • OVMF
  • OVMF-CSM
  • ...
204 packages built successfully:
  • fontconfig
  • fribidi
  • gd
  • glib (gnome2.glib ,gnome3.glib)
  • gtk-doc (gnome2.gtkdoc)
  • gts
  • hidapi
  • libdrm
  • libfido2
  • libglvnd
  • libmicrodns
  • libsysprof-capture
  • libusb1
  • libvdpau
  • libxkbcommon (libxkbcommon_8)
  • meson
  • python38Packages.Babel
  • python38Packages.Mako
  • python38Packages.PyStemmer
  • python38Packages.alabaster
  • python38Packages.apipkg
  • python38Packages.appdirs
  • python38Packages.async-timeout
  • python38Packages.async_generator
  • python38Packages.atomicwrites
  • python38Packages.attrs
  • python38Packages.automat
  • python38Packages.bootstrapped-pip
  • python38Packages.certifi
  • python38Packages.click
  • python38Packages.colorama
  • python38Packages.constantly
  • python38Packages.cookies
  • python38Packages.coverage
  • python38Packages.cppy
  • python38Packages.cryptography_vectors
  • python38Packages.cycler
  • python38Packages.cython
  • python38Packages.dateutil (python38Packages.python-dateutil)
  • python38Packages.decorator
  • python38Packages.distlib
  • python38Packages.docutils
  • python38Packages.ecdsa
  • python38Packages.eggInstallHook
  • python38Packages.eradicate
  • python38Packages.execnet
  • python38Packages.extras
  • python38Packages.filelock
  • python38Packages.flake8
  • python38Packages.flaky
  • python38Packages.flit-core
  • python38Packages.fontforge
  • python38Packages.freezegun
  • python38Packages.gunicorn
  • python38Packages.gyp
  • python38Packages.hyperlink
  • python38Packages.idna
  • python38Packages.imagesize
  • python38Packages.importlib-metadata
  • python38Packages.incremental
  • python38Packages.iniconfig
  • python38Packages.intreehooks
  • python38Packages.iso8601
  • python38Packages.itsdangerous
  • python38Packages.jdatetime
  • python38Packages.jinja2
  • python38Packages.jpylyzer
  • python38Packages.jupyterlab-pygments
  • python38Packages.lazy-object-proxy
  • python38Packages.libevdev
  • python38Packages.linecache2
  • python38Packages.lxml
  • python38Packages.m2r
  • python38Packages.markupsafe
  • python38Packages.mccabe
  • python38Packages.mistune
  • python38Packages.mock
  • python38Packages.mongoengine
  • python38Packages.more-itertools
  • python38Packages.nest-asyncio
  • python38Packages.nose
  • python38Packages.nose-exclude
  • python38Packages.nose_warnings_filters
  • python38Packages.olefile
  • python38Packages.outcome
  • python38Packages.packaging
  • python38Packages.pandocfilters
  • python38Packages.pbr
  • python38Packages.pep517
  • python38Packages.pexpect
  • python38Packages.pip
  • python38Packages.pipBuildHook
  • python38Packages.pipInstallHook
  • python38Packages.pluggy
  • python38Packages.pretend
  • python38Packages.prometheus_client
  • python38Packages.psutil
  • python38Packages.ptyprocess
  • python38Packages.py
  • python38Packages.pyasn1
  • python38Packages.pycodestyle
  • python38Packages.pycparser
  • python38Packages.pydocstyle
  • python38Packages.pyflakes
  • python38Packages.pygments
  • python38Packages.pyhamcrest
  • python38Packages.pylama
  • python38Packages.pymongo
  • python38Packages.pyparsing
  • python38Packages.pyperf
  • python38Packages.pysocks
  • python38Packages.pytest (python38Packages.pytest_6)
  • python38Packages.pytest-asyncio
  • python38Packages.pytestcov (python38Packages.pytest-cov)
  • python38Packages.pytest-expect
  • python38Packages.pytest-forked
  • python38Packages.pytest-freezegun
  • python38Packages.pytest-mock
  • python38Packages.pytestrunner (python38Packages.pytest-runner)
  • python38Packages.pytest-subtests
  • python38Packages.pytestCheckHook
  • python38Packages.pythonCatchConflictsHook
  • python38Packages.python_mimeparse
  • python38Packages.pytoml
  • python38Packages.pytz
  • python38Packages.pyyaml
  • python38Packages.rednose
  • python38Packages.ruamel_base
  • python38Packages.semantic-version
  • python38Packages.send2trash
  • python38Packages.setuptools
  • python38Packages.setuptools-rust
  • python38Packages.setuptools-scm (python38Packages.setuptools_scm)
  • python38Packages.setuptoolsBuildHook
  • python38Packages.setuptoolsCheckHook
  • python38Packages.simplejson
  • python38Packages.six
  • python38Packages.snowballstemmer
  • python38Packages.sortedcontainers
  • python38Packages.soupsieve
  • python38Packages.sphinxcontrib-applehelp
  • python38Packages.sphinxcontrib-devhelp
  • python38Packages.sphinxcontrib-htmlhelp
  • python38Packages.sphinxcontrib-jsmath
  • python38Packages.sphinxcontrib-qthelp
  • python38Packages.sphinxcontrib-serializinghtml
  • python38Packages.sphinxcontrib-websupport
  • python38Packages.sqlparse
  • python38Packages.termstyle
  • python38Packages.testpath
  • python38Packages.text-unidecode
  • python38Packages.toml
  • python38Packages.tomlkit
  • python38Packages.tornado
  • python38Packages.traceback2
  • python38Packages.typing-extensions
  • python38Packages.tzlocal
  • python38Packages.u-msgpack-python
  • python38Packages.ukpostcodeparser
  • python38Packages.umalqurra
  • python38Packages.unittest2
  • python38Packages.vcversioner
  • python38Packages.wcwidth
  • python38Packages.webencodings
  • python38Packages.wheel
  • python38Packages.wheelUnpackHook
  • python38Packages.whoosh
  • python38Packages.wrapt
  • python38Packages.zipp
  • python38Packages.zope_event
  • python38Packages.zope_interface
  • shared-mime-info
  • systemdMinimal
  • xkeyboard_config (xorg.xkeyboardconfig)
  • xlibsWrapper (xorg.xlibsWrapper)
  • xorg.libICE
  • xorg.libSM
  • xorg.libX11
  • xorg.libXScrnSaver
  • xorg.libXau
  • xorg.libXcomposite
  • xorg.libXcursor
  • xorg.libXdamage
  • xorg.libXdmcp
  • xorg.libXext
  • xorg.libXfixes
  • xorg.libXft
  • xorg.libXi
  • xorg.libXinerama
  • xorg.libXrandr
  • xorg.libXrender
  • xorg.libXt
  • xorg.libXtst
  • xorg.libXv
  • xorg.libXvMC
  • xorg.libXxf86vm
  • xorg.libxcb
  • xorg.libxshmfence
  • xorg.xcbutil
  • xorg.xcbutilimage
  • xorg.xcbutilkeysyms
  • xorg.xcbutilrenderutil
  • xorg.xcbutilwm
  • xorg.xorgproto

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement.

@FRidh FRidh self-assigned this Mar 22, 2021
@SuperSandro2000
Copy link
Member

/rebase staging

A change in setuptools v49.0.0 broke the development mode of Python
packages inside nix-shell. This workaround fixes it.

Closes NixOS#108525.
@github-actions github-actions bot changed the base branch from master to staging March 22, 2021 09:11
@github-actions github-actions bot closed this Mar 22, 2021
@github-actions
Copy link
Contributor

Rebased, please reopen the pull request to restart CI

@FRidh FRidh reopened this Mar 22, 2021
# a sitecustomize.py file is a workaround for making nix-shell
# environments work again.
# See https://github.com/pypa/setuptools/issues/2612
cat >"$tmp_path/@pythonSitePackages@/sitecustomize.py" <<EOF
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our interpreter comes with a sitecustomize.py. I suppose this one could take precedence and so it works. This may stop working if we put more logic in the main one though.

I think we could use the existing sitecustomize.py for this by just exporting

NIX_PYTHONPATH="$tmp_path/@pythonSitePackages@"

@FRidh
Copy link
Member

FRidh commented Mar 23, 2021

Thank you for looking into this. I've pushed 7a65bb7 to staging. Please test this.

@FRidh FRidh closed this Mar 23, 2021
@cript0nauta
Copy link
Contributor Author

I tested the staging branch and the problem looks fixed. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Develop local Python package not working since setuptools upgrade
4 participants