Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aria2: refactor to avoid python2 dependency #101978

Merged
merged 1 commit into from Oct 29, 2020

Conversation

@braunse
Copy link
Contributor

@braunse braunse commented Oct 28, 2020

Use python3 Sphinx package instead of python2 Sphinx package,
to avoid build-time dependency on python 2.

Motivation for this change

To remove the build dependency on python2, as per #101964

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
Use python3 Sphinx package instead of python2 Sphinx package,
to avoid build-time dependency on python 2, as per
#101964
@SuperSandro2000
Copy link
Member

@SuperSandro2000 SuperSandro2000 commented Oct 28, 2020

Result of nixpkgs-review pr 101978 run on x86_64-linux 1

6 packages failed to build:
  • aria
  • persepolis
  • python37Packages.aria2p
  • python38Packages.aria2p
  • uget
  • uget-integrator

From aria.log:

Can't exec "/bin/sh": No such file or directory at /nix/store/38vxlxrvg3yji3jms44qn94lxdysbj5j-perl-5.32.0/lib/perl5/5.32.0/x86_64-linux-thread-multi/IO/File.pm line 186.
@braunse
Copy link
Contributor Author

@braunse braunse commented Oct 28, 2020

Hmmm.... Here's what happens for me, on Debian if that matters:

❯ nix-shell -p nixpkgs-review --run "nixpkgs-review pr 101978"
$ git -c fetch.prune=false fetch --force https://github.com/NixOS/nixpkgs master:refs/nixpkgs-review/0 pull/101978/head:refs/nixpkgs-review/1
remote: Enumerating objects: 10, done.
remote: Counting objects: 100% (10/10), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 10 (delta 5), reused 7 (delta 5), pack-reused 0
Unpacking objects: 100% (10/10), 3.20 KiB | 252.00 KiB/s, done.
From https://github.com/NixOS/nixpkgs
   0dc7ad6f1fc..4a64cdd39ba  master                -> refs/nixpkgs-review/0
 * [new ref]                 refs/pull/101978/head -> refs/nixpkgs-review/1
$ git worktree add /home/seb/.cache/nixpkgs-review/pr-101978/nixpkgs 4a64cdd39ba5efc692205df8f15bc0924c5e0bf1
Preparing worktree (detached HEAD 4a64cdd39ba)
Updating files: 100% (23067/23067), done.
HEAD is now at 4a64cdd39ba Merge pull request #101952 from mweinelt/p10k
$ nix-env -f /home/seb/.cache/nixpkgs-review/pr-101978/nixpkgs -qaP --xml --out-path --show-trace
$ git merge --no-commit aa8525a8ec73d832b0f7b2f1a0f0fe0d670c2f0d
Automatic merge went well; stopped before committing as requested
$ nix-env -f /home/seb/.cache/nixpkgs-review/pr-101978/nixpkgs -qaP --xml --out-path --show-trace --meta
6 packages updated:
aria2 persepolis python3.7-aria2p python3.8-aria2p uget uget-integrator

$ nix build --no-link --keep-going --option build-use-sandbox relaxed -f /home/seb/.cache/nixpkgs-review/pr-101978/build.nix
[6 built, 482 copied (1316.6 MiB), 293.5 MiB DL]
https://github.com/NixOS/nixpkgs/pull/101978
6 packages built:
aria persepolis python37Packages.aria2p python38Packages.aria2p uget uget-integrator
@Zopieux
Copy link
Contributor

@Zopieux Zopieux commented Oct 28, 2020

Tested working on x86_64 Archlinux starting with an empty /nix directory. Managed to download a magnet torrent with the built aria2c.

Logs
$ sudo mount -t tmpfs -o size=20G tmpfs /nix && curl -L https://nixos.org/nix/install | sh && source ~/.nix-profile/etc/profile.d/nix.sh
$ nix-shell -p nixpkgs-review
$ nixpkgs-review pr 101978
(snip)
[7 built, 490 copied (1341.5 MiB), 297.0 MiB DL]
https://github.com/NixOS/nixpkgs/pull/101978
1 package marked as broken and skipped:
kiwix

6 packages built:
aria persepolis python37Packages.aria2p python38Packages.aria2p uget uget-integrator

$ aria2c 'magnet:?xt=urn:btih:dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c&dn=Big+Buck+Bunny&tr=udp%3A%2F%2Fexplodie.org%3A6969&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.empire-js.us%3A1337&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337&tr=wss%3A%2F%2Ftracker.btorrent.xyz&tr=wss%3A%2F%2Ftracker.fastcast.nz&tr=wss%3A%2F%2Ftracker.openwebtorrent.com&ws=https%3A%2F%2Fwebtorrent.io%2Ftorrents%2F&xs=https%3A%2F%2Fwebtorrent.io%2Ftorrents%2Fbig-buck-bunny.torrent'

10/28 23:54:45 [NOTICE] Downloading 1 item(s)
10/28 23:54:45 [NOTICE] IPv4 DHT: listening on UDP port 6960
10/28 23:54:45 [NOTICE] IPv4 BitTorrent: listening on TCP port 6909
10/28 23:54:45 [NOTICE] IPv6 BitTorrent: listening on TCP port 6909
[#e226df 20KiB/20KiB(100%) CN:20 SD:1]                                                                                                                                                                                                                                                                                      
10/28 23:54:55 [NOTICE] Download complete: [MEMORY][METADATA]Big+Buck+Bunny
[#a6cd3e SEED(0.0) CN:1 SD:0]                                                                                                                                                                                                                                                                                               
10/28 23:55:40 [NOTICE] Shutdown sequence commencing... Press Ctrl-C again for emergency shutdown.
10/28 23:55:40 [NOTICE] Download complete: Big Buck Bunny
10/28 23:55:40 [NOTICE] Your share ratio was 0.0, uploaded/downloaded=0B/263MiB

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
e226df|OK  |       0B/s|[MEMORY][METADATA]Big+Buck+Bunny
a6cd3e|OK  |    11MiB/s|Big Buck Bunny/Big Buck Bunny.en.srt (2more)

Status Legend:
(OK):download completed.
@SuperSandro2000
Copy link
Member

@SuperSandro2000 SuperSandro2000 commented Oct 29, 2020

Could it be that my fail is related to an error in the perl package rather than aria2?

@braunse
Copy link
Contributor Author

@braunse braunse commented Oct 29, 2020

I tested it again on a freshly installed NixOS 20.09 VM, and could not reproduce @SuperSandro2000's problem.

Logs:

[seb@alfons-nixdev:~/dev/nixpkgs]$ nix-shell -p nixpkgs-review --run "nixpkgs-review pr 101978"
$ git -c fetch.prune=false fetch --force https://github.com/NixOS/nixpkgs master:refs/nixpkgs-review/0 pull/101978/head:refs/nixpkgs-review/1
$ git worktree add /home/seb/.cache/nixpkgs-review/pr-101978-1/nixpkgs df6068e934ad2ea8db02609c335b23ac96c3d286
Preparing worktree (detached HEAD df6068e934a)
Updating files: 100% (23069/23069), done.
HEAD is now at df6068e934a Merge pull request #101985 from rnhmjoj/monero
$ git merge --no-commit aa8525a8ec73d832b0f7b2f1a0f0fe0d670c2f0d
Auto-merging pkgs/top-level/all-packages.nix
Automatic merge went well; stopped before committing as requested
$ nix build --no-link --keep-going --option build-use-sandbox relaxed -f /home/seb/.cache/nixpkgs-review/pr-101978-1/build.nix
[7 built, 558 copied (1807.6 MiB), 397.7 MiB DL]
https://github.com/NixOS/nixpkgs/pull/101978
1 package marked as broken and skipped:
kiwix

6 packages built:
aria persepolis python37Packages.aria2p python38Packages.aria2p uget uget-integrator

[0.0 MiB DL]
$ nix-shell /home/seb/.cache/nixpkgs-review/pr-101978-1/shell.nix
these paths will be fetched (0.51 MiB download, 2.47 MiB unpacked):
  /nix/store/82bsy6f0bz7avmy8y62j0z2dnxgi8pxw-bash-interactive-4.4-p23-doc
  /nix/store/b6mp5xw5pjr3bsmk3fphkyxw73vc806j-bash-interactive-4.4-p23-info
  /nix/store/hfvjs6abgzadrv9f387v1ip5mr206jh9-bash-interactive-4.4-p23-dev
  /nix/store/xpmrr7kyzlbny0540kzd1vlqsca69y13-bash-interactive-4.4-p23-man
copying path '/nix/store/82bsy6f0bz7avmy8y62j0z2dnxgi8pxw-bash-interactive-4.4-p23-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/xpmrr7kyzlbny0540kzd1vlqsca69y13-bash-interactive-4.4-p23-man' from 'https://cache.nixos.org'...
copying path '/nix/store/hfvjs6abgzadrv9f387v1ip5mr206jh9-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/b6mp5xw5pjr3bsmk3fphkyxw73vc806j-bash-interactive-4.4-p23-info' from 'https://cache.nixos.org'...

[nix-shell:~/.cache/nixpkgs-review/pr-101978-1]$ aria2
aria2c  aria2p  

[nix-shell:~/.cache/nixpkgs-review/pr-101978-1]$ aria2c http://google.com

10/29 08:45:40 [NOTICE] Downloading 1 item(s)

10/29 08:45:40 [NOTICE] CUID#7 - Redirecting to http://www.google.com/

10/29 08:45:40 [NOTICE] Download complete: /home/seb/.cache/nixpkgs-review/pr-101978-1/index.html

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
18ab42|OK  |    71KiB/s|/home/seb/.cache/nixpkgs-review/pr-101978-1/index.html

Status Legend:
(OK):download completed.

[nix-shell:~/.cache/nixpkgs-review/pr-101978-1]$ exit
$ git worktree prune
@SuperSandro2000
Copy link
Member

@SuperSandro2000 SuperSandro2000 commented Oct 29, 2020

Result of nixpkgs-review pr 101978 run on x86_64-linux 1

1 package marked as broken and skipped:
  • kiwix
2 packages failed to build:
  • uget
  • uget-integrator
4 packages built:
  • aria (aria2)
  • persepolis
  • python37Packages.aria2p
  • python38Packages.aria2p
configuring
configure flags: --disable-dependency-tracking --prefix=/nix/store/pjsfmk5gq3n7vr17znhjrzxmf4d08hra-uget-2.2.3-1
/nix/store/q1zjp9grl4w92qalkdqjs2bj5d0pf8ih-stdenv-linux/setup: ./configure: /bin/sh: bad interpreter: No such file or directory

Uget depends on Aria and would be broken with this PR https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/networking/uget/default.nix#L2. Please fix.

@braunse
Copy link
Contributor Author

@braunse braunse commented Oct 29, 2020

AFAICT, there should be nothing in this PR that relates to whether uget's configure script can find /bin/sh in any way, so I'm unsure how to go about fixing this. Any hints would be appreciated.

FWIW, uget builds fine for me with this PR, both on my Debian machine with Nix installed and on a NixOS 20.09 VM. @SuperSandro2000 can you check that you are able to build uget on the master branch without this PR?

Update: I have a fix for this in https://github.com/braunse/nixpkgs/tree/patch-configure-shebang-in-uget.
Since the fix (patchShebangs . in preConfigure) seems unrelated to aria2, I'm unsure whether I should add it to this PR, or open a fresh PR?

braunse added a commit to braunse/nixpkgs that referenced this pull request Oct 29, 2020
@SuperSandro2000
Copy link
Member

@SuperSandro2000 SuperSandro2000 commented Oct 29, 2020

Result of nixpkgs-review pr 101978 run on x86_64-linux 1

1 package marked as broken and skipped:
  • kiwix
1 package failed to build:
  • persepolis
5 packages built:
  • aria (aria2)
  • python37Packages.aria2p
  • python38Packages.aria2p
  • uget
  • uget-integrator
@SuperSandro2000
Copy link
Member

@SuperSandro2000 SuperSandro2000 commented Oct 29, 2020

Uget gets pulled from cache for me so I can't check if it build or not but I suppose it should.

Edit: Now Uget builds after collecting nix store garbage. The persepolis build failure is unrelated to this PR.

error: can't copy 'man/persepolis.1.gz': doesn't exist or not a regular file

But this is fixed on master.

Copy link
Contributor

@jonringer jonringer left a comment

LGTM

Result of nixpkgs-review pr 101978 1

1 package marked as broken and skipped:
  • kiwix
6 packages built:
  • aria (aria2)
  • persepolis
  • python37Packages.aria2p
  • python38Packages.aria2p
  • uget
  • uget-integrator

@jonringer jonringer merged commit 8bdb520 into NixOS:master Oct 29, 2020
19 checks passed
19 checks passed
@github-actions
tests
Details
@github-actions
action
Details
@ofborg
Evaluation Performance Report Evaluator Performance Report
Details
@github-actions
Wait for ofborg
Details
@ofborg
aria2, aria2.passthru.tests on aarch64-linux Success
Details
@ofborg
aria2, aria2.passthru.tests on x86_64-linux Success
Details
@ofborg
grahamcofborg-eval ^.^!
Details
@ofborg
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
@ofborg
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
@ofborg
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="aa8525a"; rev="aa8525a8ec73d832b0f7b2f1a0f0fe0d670c2f0d"; } ./pkgs/t
Details
@ofborg
grahamcofborg-eval-lib-tests nix-build --arg pkgs import ./. {} ./lib/tests/release.nix
Details
@ofborg
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="aa8525a"; rev="aa8525a8ec73d832b0f7b2f1a0f0fe0d670c2f0d"; } ./nixos/
Details
@ofborg
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="aa8525a"; rev="aa8525a8ec73d832b0f7b2f1a0f0fe0d670c2f0d"; } ./nixos/
Details
@ofborg
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="aa8525a"; rev="aa8525a8ec73d832b0f7b2f1a0f0fe0d670c2f0d"; } ./nixos/
Details
@ofborg
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="aa8525a"; rev="aa8525a8ec73d832b0f7b2f1a0f0fe0d670c2f0d"; } ./pkgs/t
Details
@ofborg
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="aa8525a"; rev="aa8525a8ec73d832b0f7b2f1a0f0fe0d670c2f0d"; } ./pkgs/t
Details
@ofborg
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="aa8525a"; rev="aa8525a8ec73d832b0f7b2f1a0f0fe0d670c2f0d"; } ./pkgs/t
Details
@ofborg
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
@ofborg
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
@braunse braunse deleted the braunse:update-aria2-python2-dep branch Oct 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants