Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Another batch of Python 2 removals #75668

Merged
merged 31 commits into from Dec 19, 2019
Merged

Another batch of Python 2 removals #75668

merged 31 commits into from Dec 19, 2019

Conversation

@jtojnar
Copy link
Contributor

@jtojnar jtojnar commented Dec 14, 2019

I was digging through gnome3.gnome-shell’s build-time closure using nix-store -q --graph $(nix-instantiate '<nixpkgs>' -A gnome3.gnome-shell) | dijkstra -da $(nix-instantiate '<nixpkgs>' -A python2) | gvpr -c 'N[dist>1000.0]{delete(NULL, $)}' | xdot - and eliminating Python 2 from derivations referencing it by replacing it with Python 3 whenever possible.

Testing

  • Built individual packages based against unstable:
    • farstream
    • libevdev
    • ninja
    • python3.pkgs.anytree
    • zenity
    • libdbusmenu-gtk3
    • xorg.xcbproto
    • cacert
    • libselinux
    • epoxy
    • anytree + gtk-doc
      • Build glib to test gtk-doc working
    • xorg.libxcb
    • fontforge
    • libglvnd
    • libgit2
    • serd
    • sord
    • sratom
    • lv2
    • lilv
    • fail2ban
    • gamin
    • gst_all_1.gst-plugins-good
    • libgsf
    • doxygen
    • jbig2dec
    • libproxy
    • rustc
    • cargo

Some packages were not handled:

jtojnar added 3 commits Dec 14, 2019
It was never used in the recent history.
@jtojnar jtojnar requested review from FRidh and jonringer as code owners Dec 14, 2019
@jtojnar jtojnar changed the title Another batch of Python 2 removal Another batch of Python 2 removals Dec 14, 2019
jtojnar added 3 commits Dec 14, 2019
* format with nixpkgs-fmt
* drop unneeded dependencies
    * gnome-doc-utils → yelp-tools since 3.6.0
    * intltool → gettext since 3.16.2
* move build-time deps to nativeBuildInputs
* add metadata
@worldofpeace worldofpeace added this to Needs review in Python 2 deprecation Dec 15, 2019
libselinux = callPackage ../os-specific/linux/libselinux { };
libselinux = callPackage ../os-specific/linux/libselinux {
python = python3;
};
Comment on lines -16544 to +16547

This comment has been minimized.

@worldofpeace

worldofpeace Dec 15, 2019
Member

could we do this, but in the expression?
people could override python3 = python?

This comment has been minimized.

@jtojnar

jtojnar Dec 15, 2019
Author Contributor

The reason I did not do that is because it is overridden in python-packages.nix and it felt weird having python3 = python there.

libsemanage = callPackage ../os-specific/linux/libsemanage { };
libsemanage = callPackage ../os-specific/linux/libsemanage {
python = python3;
};
Comment on lines -16548 to +16551

This comment has been minimized.

@worldofpeace

worldofpeace Dec 15, 2019
Member

can we do this in the expression?
python references replaced to python3.

This comment has been minimized.

@jonringer

jonringer Dec 16, 2019
Contributor

for most usages of python outside of python-modules, you generally want to parameterize python like this incase you need to override package versions, or if you want to test with a different version of python

@worldofpeace
Copy link
Member

@worldofpeace worldofpeace commented Dec 15, 2019

I was digging through gnome3.gnome-shell’s build-time closure using nix-store -q --graph $(nix-instantiate '' -A gnome3.gnome-shell) | dijkstra -da $(nix-instantiate '' -A python2) | gvpr -c 'N[dist>1000.0]{delete(NULL, $)}' | xdot - and eliminating Python 2 from derivations referencing it by replacing it with Python 3 whenever possible.

The graph is truly crazy to look at 😄

@jtojnar
Copy link
Contributor Author

@jtojnar jtojnar commented Dec 15, 2019

Good thing you can just python3 into stuff that does not build with it and it will disappear:

diff --git a/pkgs/desktops/gnome-3/core/gnome-shell/default.nix b/pkgs/desktops/gnome-3/core/gnome-shell/default.nix
index e8d5b7dbffd..64ffeadf93e 100644
--- a/pkgs/desktops/gnome-3/core/gnome-shell/default.nix
+++ b/pkgs/desktops/gnome-3/core/gnome-shell/default.nix
@@ -6,7 +6,7 @@
 , accountsservice, gdk-pixbuf, gdm, upower, ibus, networkmanagerapplet, libgnomekbd, gnome-desktop
 , gsettings-desktop-schemas, gnome-keyring, glib, gjs, mutter, evolution-data-server, gtk3
 , sassc, systemd, gst_all_1, adwaita-icon-theme, gnome-bluetooth, gnome-clocks, gnome-settings-daemon
-, gnome-autoar, asciidoc-full }:
+, gnome-autoar, asciidoc }:
 
 # http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/gnome-base/gnome-shell/gnome-shell-3.10.2.1.ebuild?revision=1.3&view=markup
 
@@ -26,7 +26,7 @@ in stdenv.mkDerivation rec {
 
   nativeBuildInputs = [
     meson ninja pkgconfig gettext docbook_xsl docbook_xsl_ns docbook_xml_dtd_42 perl wrapGAppsHook glibcLocales
-    sassc desktop-file-utils libxslt.bin python3 asciidoc-full
+    sassc desktop-file-utils libxslt.bin python3 asciidoc
   ];
   buildInputs = [
     systemd
diff --git a/pkgs/development/compilers/llvm/9/default.nix b/pkgs/development/compilers/llvm/9/default.nix
index 1590d180c3f..cfe88b5d215 100644
--- a/pkgs/development/compilers/llvm/9/default.nix
+++ b/pkgs/development/compilers/llvm/9/default.nix
@@ -42,12 +42,12 @@ let
 
     llvm-manpages = lowPrio (tools.llvm.override {
       enableManpages = true;
-      python = pkgs.python;  # don't use python-boot
+      python = pkgs.python3;  # don't use python-boot
     });
 
     clang-manpages = lowPrio (tools.clang-unwrapped.override {
       enableManpages = true;
-      python = pkgs.python;  # don't use python-boot
+      python = pkgs.python3;  # don't use python-boot
     });
 
     libclang = tools.clang-unwrapped.lib;
diff --git a/pkgs/development/compilers/rust/cargo.nix b/pkgs/development/compilers/rust/cargo.nix
index 91a31be9f8d..f639d330566 100644
--- a/pkgs/development/compilers/rust/cargo.nix
+++ b/pkgs/development/compilers/rust/cargo.nix
@@ -1,4 +1,4 @@
-{ stdenv, file, curl, pkgconfig, python, openssl, cmake, zlib
+{ stdenv, file, curl, pkgconfig, python3, openssl, cmake, zlib
 , makeWrapper, libiconv, cacert, rustPlatform, rustc, libgit2
 , CoreFoundation, Security
 }:
@@ -18,7 +18,7 @@ rustPlatform.buildRustPackage {
   dontUpdateAutotoolsGnuConfigScripts = true;
 
   nativeBuildInputs = [ pkgconfig cmake makeWrapper ];
-  buildInputs = [ cacert file curl python openssl zlib libgit2 ]
+  buildInputs = [ cacert file curl python3 openssl zlib libgit2 ]
     ++ stdenv.lib.optionals stdenv.isDarwin [ CoreFoundation Security libiconv ];
 
   LIBGIT2_SYS_USE_PKG_CONFIG = 1;
diff --git a/pkgs/development/compilers/rust/rustc.nix b/pkgs/development/compilers/rust/rustc.nix
index ca9bf05b7e7..49eaeef0d9c 100644
--- a/pkgs/development/compilers/rust/rustc.nix
+++ b/pkgs/development/compilers/rust/rustc.nix
@@ -1,5 +1,5 @@
 { stdenv, removeReferencesTo, pkgsBuildBuild, pkgsBuildHost, pkgsBuildTarget
-, fetchurl, file, python2
+, fetchurl, file, python3
 , llvm_9, darwin, git, cmake, rust, rustPlatform
 , pkgconfig, openssl
 , which, libffi
@@ -121,7 +121,7 @@ in stdenv.mkDerivation rec {
   dontUseCmakeConfigure = true;
 
   nativeBuildInputs = [
-    file python2 rustPlatform.rust.rustc git cmake
+    file python3 rustPlatform.rust.rustc git cmake
     which libffi removeReferencesTo pkgconfig
   ];
 
diff --git a/pkgs/development/libraries/libplist/default.nix b/pkgs/development/libraries/libplist/default.nix
index 19b700ca5e3..948a8dfefcc 100644
--- a/pkgs/development/libraries/libplist/default.nix
+++ b/pkgs/development/libraries/libplist/default.nix
@@ -1,7 +1,7 @@
-{ stdenv,  autoreconfHook, fetchFromGitHub, pkgconfig, python2Packages, glib }:
+{ stdenv,  autoreconfHook, fetchFromGitHub, pkgconfig, python3Packages, glib }:
 
 let
-  inherit (python2Packages) python cython;
+  inherit (python3Packages) python cython;
 in
 stdenv.mkDerivation rec {
   pname = "libplist";
diff --git a/pkgs/development/tools/build-managers/wafHook/default.nix b/pkgs/development/tools/build-managers/wafHook/default.nix
index 4184bd5fe8c..d33abb88a69 100644
--- a/pkgs/development/tools/build-managers/wafHook/default.nix
+++ b/pkgs/development/tools/build-managers/wafHook/default.nix
@@ -1,7 +1,7 @@
-{ lib, stdenv, pkgs, python, makeSetupHook, waf }:
+{ lib, stdenv, pkgs, python3, makeSetupHook, waf }:
 
 makeSetupHook {
-  deps = [ python ];
+  deps = [ python3 ];
   substitutions = {
     inherit waf;
     crossFlags = lib.optionalString (stdenv.hostPlatform.system != stdenv.targetPlatform.system)
diff --git a/pkgs/tools/typesetting/asciidoc/default.nix b/pkgs/tools/typesetting/asciidoc/default.nix
index 83aa667f58f..98546fd9209 100644
--- a/pkgs/tools/typesetting/asciidoc/default.nix
+++ b/pkgs/tools/typesetting/asciidoc/default.nix
@@ -1,4 +1,4 @@
-{ fetchurl, stdenv, python2
+{ fetchurl, stdenv, python3
 
 , enableStandardFeatures ? false
 , sourceHighlight ? null
@@ -151,7 +151,7 @@ stdenv.mkDerivation rec {
     sha256 = "1w71nk527lq504njmaf0vzr93pgahkgzzxzglrq6bay8cw2rvnvq";
   };
 
-  buildInputs = [ python2 unzip ];
+  buildInputs = [ python3 unzip ];
 
   # install filters early, so their shebangs are patched too
   patchPhase = with stdenv.lib; ''
@@ -251,7 +251,7 @@ stdenv.mkDerivation rec {
         -i a2x.py
   '' + ''
     for n in $(find "$out" . -name \*.py); do
-      sed -i -e "s,^#![[:space:]]*.*/bin/env python,#!${python2}/bin/python,g" "$n"
+      sed -i -e "s,^#![[:space:]]*.*/bin/env python,#!${python3}/bin/python,g" "$n"
       chmod +x "$n"
     done
 
diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index 36f02e50be2..8cbf0b86533 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -8479,6 +8479,7 @@ in
 
   llvmPackages_9 = callPackage ../development/compilers/llvm/9 ({
     inherit (stdenvAdapters) overrideCC;
+    python = python3;
     buildLlvmTools = buildPackages.llvmPackages_9.tools;
     targetLlvmLibraries = targetPackages.llvmPackages_9.libraries;
   } // stdenv.lib.optionalAttrs (buildPackages.stdenv.cc.isGNU && stdenv.hostPlatform.isi686) {
jtojnar added 2 commits Dec 14, 2019
jtojnar added 10 commits Dec 14, 2019
We are dropping Python support from gamin so nothing is holding us any more.
Instead of gamin, we will use systemd on Linux and polling on Darwin;
gamin is the worst backend any way, according to upstream:

fail2ban/fail2ban#1330 (comment)
They are not compatible with Python 3, upstream as dead, and nothing uses the bindings any more.

This is a progress in getting rid of Python 2 from closures of most people (samba, which depends on gamin shared library is common dependency).
Python is only used for a single console test these dayls and it likely requires
non-trivial introspection hacks.
Also move the build-time dependencies to nativeBuildInputs, and drop perl as it no longer seems to be necessary (CMake is not looking for it any more).
Nothing was using them.

Another step in Python 2 removal.
@jtojnar jtojnar force-pushed the jtojnar:py2-cleanup branch from c8598bb to f288a7c Dec 15, 2019
jtojnar added 2 commits Dec 15, 2019
@jtojnar jtojnar requested review from LnL7 and Mic92 as code owners Dec 15, 2019
@ofborg ofborg bot added the 6.topic: rust label Dec 15, 2019
@jtojnar
Copy link
Contributor Author

@jtojnar jtojnar commented Dec 15, 2019

Successfully built rustc and cargo with Python 3 overnight.

@LnL7
Copy link
Member

@LnL7 LnL7 commented Dec 15, 2019

@jtojnar The darwin stdenv (llvm, ninja, etc.) depends on python to bootstrap so switching to python3 for those requires a bit of fiddling with the stdenv stages. Perhaps the next llvm 7 -> 9 upgrade which I've started to look into would be a good opportunity to also tackle python3.

@FRidh FRidh added this to Done in Staging Dec 15, 2019
@FRidh FRidh moved this from Done to Needs review in Staging Dec 15, 2019
@worldofpeace
Copy link
Member

@worldofpeace worldofpeace commented Dec 15, 2019

@GrahamcOfBorg build glib

(darwin gtk-doc sometimes is...)

@jtojnar jtojnar merged commit b171037 into NixOS:staging Dec 19, 2019
16 checks passed
16 checks passed
glib on x86_64-darwin Timed out, unknown build status
Details
Evaluation Performance Report Evaluator Performance Report
Details
glib on aarch64-linux Success
Details
glib on x86_64-linux Success
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A darwin-tested
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release-combined.nix -A tested
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
Staging automation moved this from Needs review to Done Dec 19, 2019
Python 2 deprecation automation moved this from Needs review to Done Dec 19, 2019
@jtojnar jtojnar deleted the jtojnar:py2-cleanup branch Dec 19, 2019
@jtojnar jtojnar mentioned this pull request Dec 25, 2019
17 of 17 tasks complete
@Izorkin

This comment has been minimized.

Copy link
Contributor

@Izorkin Izorkin commented on 7adbfb1 Jan 2, 2020

@jtojnar after this update error start fail2ban:

systemd[1]: Stopped Fail2ban Intrusion Prevention System.
systemd[1]: Starting Fail2ban Intrusion Prevention System...
fail2ban-client[2478]: Traceback (most recent call last):
fail2ban-client[2478]:   File "/nix/store/0hrz4nhxjdhx30573iiz3m8ymri681l3-fail2ban-0.10.4/bin/.fail2ban-client-wrapped", line 35, in <module>
fail2ban-client[2478]:     from fail2ban.client.fail2banclient import exec_command_line, sys
fail2ban-client[2478]:   File "/nix/store/0hrz4nhxjdhx30573iiz3m8ymri681l3-fail2ban-0.10.4/lib/python3.7/site-packages/fail2ban/client/fail2banclient.py", line 39, in <module>
fail2ban-client[2478]:     from ..server.utils import Utils
fail2ban-client[2478]:   File "/nix/store/0hrz4nhxjdhx30573iiz3m8ymri681l3-fail2ban-0.10.4/lib/python3.7/site-packages/fail2ban/server/utils.py", line 54, in <module>
fail2ban-client[2478]:     for name, num in signal.__dict__.iteritems() if name.startswith("SIG"))
fail2ban-client[2478]: AttributeError: 'dict' object has no attribute 'iteritems'

This comment has been minimized.

Copy link
Contributor

@Izorkin Izorkin replied Jan 2, 2020

and missing
propagatedBuildInputs = [ gamin ];

@lopsided98

This comment has been minimized.

Copy link
Contributor

@lopsided98 lopsided98 commented on 56e5572 Jan 10, 2020

This broke the cross build because of #49526

@samueldr samueldr mentioned this pull request Jan 13, 2020
4 of 10 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Staging
  
Done
Linked issues

Successfully merging this pull request may close these issues.

None yet

7 participants
You can’t perform that action at this time.