Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update-python-libraries: add github fetcher #99635

Merged
merged 1 commit into from Oct 6, 2020

Conversation

@jonringer
Copy link
Contributor

@jonringer jonringer commented Oct 5, 2020

Motivation for this change

was tired of it only supporting pypi

I use the nix eval -f <NIXPKGS_ROOT>/default.nix python3Packages.<package>.src.meta.homepage to determine the owner and repo. NIXPKGS_ROOT is determined from script location, not an environment variable though.

Some limitations:

  • If the attr name and directory name differ, it will fail the update
  • can't handle "unstable" checkouts due to version logic not being able to handle it ('2018-08-22' vs '1.2.1')?
  • If a package is disabled for python3, it will also fail due to the evaluation will throw the " is not supported for python3.8"
    • Maybe this is a plus? a previously python2-only library will probably need some manual work to get working.
  • If GITHUB_API_TOKEN (taken from vim plugin conventions) is not given, then user will be limited to 100 api requests; instead of 5000

If an update does fail, the entire update job will still succeed, similar to existing behavior.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
@jonringer
Copy link
Contributor Author

@jonringer jonringer commented Oct 5, 2020

cc @FRidh example of running script: #99636

@jonringer
Copy link
Contributor Author

@jonringer jonringer commented Oct 5, 2020

The sha is being incorrectly calculated :(

@jonringer
Copy link
Contributor Author

@jonringer jonringer commented Oct 5, 2020

ah, had to pass --unpack to the prefetcher, as fetchFromGitHub unpacks the tar.gz into a "source" directory.

The other option would be to change the sha into a known "bad" sha, and then see what nix-build complains about. This would be resilient to people passing "name" to fetchFromGitHub as well, but that's not a very common practice at all

@jonringer jonringer force-pushed the add-github-fetcher branch from 88fb619 to 28cd3e6 Oct 5, 2020
@jonringer
Copy link
Contributor Author

@jonringer jonringer commented Oct 5, 2020

built around 8000 python packages, didn't see any sha errors

@jonringer jonringer force-pushed the add-github-fetcher branch from 28cd3e6 to 908300f Oct 5, 2020
Copy link
Member

@FRidh FRidh left a comment

Nice work! If it all works then go ahead and merge it.

@jonringer jonringer force-pushed the add-github-fetcher branch from 908300f to 484748e Oct 6, 2020
@jonringer jonringer merged commit 4eaf246 into NixOS:master Oct 6, 2020
2 of 3 checks passed
2 of 3 checks passed
@github-actions[bot]
tests tests
Details
@github-actions[bot]
action
Details
@github-actions[bot]
Wait for ofborg This failed status will be cleared when ofborg finishes eval.
Details
@jonringer jonringer deleted the add-github-fetcher branch Oct 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants