Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python3.pkgs.scancode-toolkit: init at 21.3.31 #120009

Merged
merged 19 commits into from Apr 23, 2021

Conversation

cole-h
Copy link
Member

@cole-h cole-h commented Apr 20, 2021

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@r-rmcgibbo
Copy link

r-rmcgibbo commented Apr 20, 2021

Result of nixpkgs-review pr 120009 at 09a3b441 run on aarch64-linux 1

13 packages failed to build:
23 packages built successfully:
  • python38Packages.banal
  • python38Packages.debian-inspector
  • python38Packages.extractcode-7z
  • python38Packages.extractcode-libarchive
  • python38Packages.fingerprints
  • python38Packages.gemfileparser
  • python38Packages.normality
  • python38Packages.pymaven-patch
  • python38Packages.saneyaml
  • python38Packages.spdx-tools
  • python38Packages.typecode-libmagic
  • python38Packages.urlpy
  • python39Packages.banal
  • python39Packages.debian-inspector
  • python39Packages.extractcode-7z
  • python39Packages.extractcode-libarchive
  • python39Packages.fingerprints
  • python39Packages.gemfileparser
  • python39Packages.normality
  • python39Packages.pymaven-patch
  • python39Packages.saneyaml
  • python39Packages.typecode-libmagic
  • python39Packages.urlpy
11 suggestions:
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/extractcode/7z.nix:31:0:

       |
    31 |     description = "A ScanCode Toolkit plugin to provide pre-built binary libraries and utilities and their locations";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.019s’
    Near pkgs/development/python-modules/fingerprints/default.nix:29:0:

       |
    29 |     description = "A library to generate entity fingerprints";
       | ^
    
  • warning: python-include-tests

    Consider adding a checkPhase for tests, or if not feasible, pythonImportsCheck.

    Near pkgs/development/python-modules/normality/default.nix:27:0:

       |
    27 |     description = "Micro-library to normalize text strings";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/debian-inspector/default.nix:26:0:

       |
    26 |     description = "Utilities to parse Debian package, copyright and control files";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/typecode/libmagic.nix:32:0:

       |
    32 |     description = "A ScanCode Toolkit plugin to provide pre-built binary libraries and utilities and their locations";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/extractcode/libarchive.nix:37:0:

       |
    37 |     description = "A ScanCode Toolkit plugin to provide pre-built binary libraries and utilities and their locations";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/banal/default.nix:15:0:

       |
    15 |     description = "Commons of banal micro-functions for Python";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/urlpy/default.nix:20:0:

       |
    20 |     description = "Simple URL parsing, canonicalization and equivalence";
       | ^
    
  • warning: unclear-gpl

    lgpl21 is a deprecated license, please check if project uses lgpl21Plus or lgpl21Only and change meta.license accordingly.

    Near pkgs/development/python-modules/extractcode/7z.nix:33:5:

       |
    33 |     license = with licenses; [ asl20 lgpl21 ];
       |     ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/gemfileparser/default.nix:15:0:

       |
    15 |     description = "A library to parse Ruby Gemfile, .gemspec and Cocoapod .podspec file using Python";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/saneyaml/default.nix:26:0:

       |
    26 |     description = "A PyYaml wrapper with sane behaviour to read and write readable YAML safely";
       | ^
    

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement.


Result of nixpkgs-review pr 120009 at 09a3b441 run on x86_64-linux 1

2 packages failed to build:
34 packages built successfully:
  • python38Packages.banal
  • python38Packages.commoncode
  • python38Packages.debian-inspector
  • python38Packages.extractcode
  • python38Packages.extractcode-7z
  • python38Packages.extractcode-libarchive
  • python38Packages.fingerprints
  • python38Packages.gemfileparser
  • python38Packages.intbitset
  • python38Packages.normality
  • python38Packages.plugincode
  • python38Packages.pymaven-patch
  • python38Packages.saneyaml
  • python38Packages.scancode-toolkit
  • python38Packages.spdx-tools
  • python38Packages.typecode
  • python38Packages.typecode-libmagic
  • python38Packages.urlpy
  • python39Packages.banal
  • python39Packages.commoncode
  • python39Packages.debian-inspector
  • python39Packages.extractcode
  • python39Packages.extractcode-7z
  • python39Packages.extractcode-libarchive
  • python39Packages.fingerprints
  • python39Packages.gemfileparser
  • python39Packages.intbitset
  • python39Packages.normality
  • python39Packages.plugincode
  • python39Packages.pymaven-patch
  • python39Packages.saneyaml
  • python39Packages.typecode
  • python39Packages.typecode-libmagic
  • python39Packages.urlpy
16 suggestions:
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/debian-inspector/default.nix:26:0:

       |
    26 |     description = "Utilities to parse Debian package, copyright and control files";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/urlpy/default.nix:20:0:

       |
    20 |     description = "Simple URL parsing, canonicalization and equivalence";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/banal/default.nix:15:0:

       |
    15 |     description = "Commons of banal micro-functions for Python";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/typecode/libmagic.nix:32:0:

       |
    32 |     description = "A ScanCode Toolkit plugin to provide pre-built binary libraries and utilities and their locations";
       | ^
    
  • warning: unclear-gpl

    lgpl21 is a deprecated license, please check if project uses lgpl21Plus or lgpl21Only and change meta.license accordingly.

    Near pkgs/development/python-modules/extractcode/7z.nix:33:5:

       |
    33 |     license = with licenses; [ asl20 lgpl21 ];
       |     ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/gemfileparser/default.nix:15:0:

       |
    15 |     description = "A library to parse Ruby Gemfile, .gemspec and Cocoapod .podspec file using Python";
       | ^
    
  • warning: python-include-tests

    Consider adding a checkPhase for tests, or if not feasible, pythonImportsCheck.

    Near pkgs/development/python-modules/normality/default.nix:27:0:

       |
    27 |     description = "Micro-library to normalize text strings";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/typecode/default.nix:34:0:

       |
    34 |     description = "Comprehensive filetype and mimetype detection using libmagic and Pygments";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/plugincode/default.nix:30:0:

       |
    30 |     description = "A library that provides plugin functionality for ScanCode toolkit";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.028s’
    Near pkgs/development/python-modules/fingerprints/default.nix:29:0:

       |
    29 |     description = "A library to generate entity fingerprints";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/extractcode/7z.nix:31:0:

       |
    31 |     description = "A ScanCode Toolkit plugin to provide pre-built binary libraries and utilities and their locations";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/saneyaml/default.nix:26:0:

       |
    26 |     description = "A PyYaml wrapper with sane behaviour to read and write readable YAML safely";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/commoncode/default.nix:38:0:

       |
    38 |     description = "A set of common utilities, originally split from ScanCode";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.026s’
    Near pkgs/development/python-modules/fingerprints/default.nix:29:0:

       |
    29 |     description = "A library to generate entity fingerprints";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/extractcode/libarchive.nix:37:0:

       |
    37 |     description = "A ScanCode Toolkit plugin to provide pre-built binary libraries and utilities and their locations";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.000s’
    Near pkgs/development/python-modules/extractcode/default.nix:28:0:

       |
    28 |     description = "A mostly universal archive extractor using z7zip, libarchve, other libraries and the Python standard library for reliable archive extraction";
       | ^
    

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement.


Result of nixpkgs-review pr 120009 at ea558328 run on aarch64-linux 1

2 packages failed to build:
38 packages built successfully:
  • python38Packages.banal
  • python38Packages.commoncode
  • python38Packages.debian-inspector
  • python38Packages.dparse
  • python38Packages.extractcode
  • python38Packages.extractcode-7z
  • python38Packages.extractcode-libarchive
  • python38Packages.fingerprints
  • python38Packages.gemfileparser
  • python38Packages.intbitset
  • python38Packages.normality
  • python38Packages.plugincode
  • python38Packages.pymaven-patch
  • safety-cli (python38Packages.safety)
  • python38Packages.saneyaml
  • python38Packages.spdx-tools
  • python38Packages.typecode
  • python38Packages.typecode-libmagic
  • python38Packages.urlpy
  • python39Packages.banal
  • python39Packages.commoncode
  • python39Packages.debian-inspector
  • python39Packages.dparse
  • python39Packages.extractcode
  • python39Packages.extractcode-7z
  • python39Packages.extractcode-libarchive
  • python39Packages.fingerprints
  • python39Packages.gemfileparser
  • python39Packages.intbitset
  • python39Packages.normality
  • python39Packages.plugincode
  • python39Packages.pymaven-patch
  • python39Packages.safety
  • python39Packages.saneyaml
  • python39Packages.spdx-tools
  • python39Packages.typecode
  • python39Packages.typecode-libmagic
  • python39Packages.urlpy
5 suggestions:
  • warning: unused-argument

    Unused argument: pytest.
    Near pkgs/development/python-modules/typecode/libmagic.nix:7:3:

      |
    7 | , pytest
      |   ^
    
  • warning: unused-argument

    Unused argument: pytestCheckHook.
    Near pkgs/development/python-modules/extractcode/libarchive.nix:13:3:

       |
    13 | , pytestCheckHook
       |   ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.031s’
    Near pkgs/development/python-modules/fingerprints/default.nix:37:0:

       |
    37 |     description = "A library to generate entity fingerprints";
       | ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.033s’
    Near pkgs/development/python-modules/fingerprints/default.nix:37:0:

       |
    37 |     description = "A library to generate entity fingerprints";
       | ^
    
  • warning: unclear-gpl

    lgpl21 is a deprecated license, please check if project uses lgpl21Plus or lgpl21Only and change meta.license accordingly.

    Near pkgs/development/python-modules/extractcode/7z.nix:44:5:

       |
    44 |     license = with licenses; [ asl20 lgpl21 ];
       |     ^
    

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement.


Result of nixpkgs-review pr 120009 at 6f993e72 run on aarch64-linux 1

2 packages failed to build:
38 packages built successfully:
  • python38Packages.banal
  • python38Packages.commoncode
  • python38Packages.debian-inspector
  • python38Packages.dparse
  • python38Packages.extractcode
  • python38Packages.extractcode-7z
  • python38Packages.extractcode-libarchive
  • python38Packages.fingerprints
  • python38Packages.gemfileparser
  • python38Packages.intbitset
  • python38Packages.normality
  • python38Packages.plugincode
  • python38Packages.pymaven-patch
  • safety-cli (python38Packages.safety)
  • python38Packages.saneyaml
  • python38Packages.spdx-tools
  • python38Packages.typecode
  • python38Packages.typecode-libmagic
  • python38Packages.urlpy
  • python39Packages.banal
  • python39Packages.commoncode
  • python39Packages.debian-inspector
  • python39Packages.dparse
  • python39Packages.extractcode
  • python39Packages.extractcode-7z
  • python39Packages.extractcode-libarchive
  • python39Packages.fingerprints
  • python39Packages.gemfileparser
  • python39Packages.intbitset
  • python39Packages.normality
  • python39Packages.plugincode
  • python39Packages.pymaven-patch
  • python39Packages.safety
  • python39Packages.saneyaml
  • python39Packages.spdx-tools
  • python39Packages.typecode
  • python39Packages.typecode-libmagic
  • python39Packages.urlpy
5 suggestions:
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.031s’
    Near pkgs/development/python-modules/fingerprints/default.nix:37:0:

       |
    37 |     description = "A library to generate entity fingerprints";
       | ^
    
  • warning: unused-argument

    Unused argument: pytest.
    Near pkgs/development/python-modules/typecode/libmagic.nix:7:3:

      |
    7 | , pytest
      |   ^
    
  • warning: unclear-gpl

    lgpl21 is a deprecated license, please check if project uses lgpl21Plus or lgpl21Only and change meta.license accordingly.

    Near pkgs/development/python-modules/extractcode/7z.nix:44:5:

       |
    44 |     license = with licenses; [ asl20 lgpl21 ];
       |     ^
    
  • warning: no-python-tests

    Test runner could not discover any test cases: ‘Ran 0 tests in 0.030s’
    Near pkgs/development/python-modules/fingerprints/default.nix:37:0:

       |
    37 |     description = "A library to generate entity fingerprints";
       | ^
    
  • warning: unused-argument

    Unused argument: pytestCheckHook.
    Near pkgs/development/python-modules/extractcode/libarchive.nix:13:3:

       |
    13 | , pytestCheckHook
       |   ^
    

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement.

Copy link
Member

@SuperSandro2000 SuperSandro2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry about the amount of changes.

collected:

  • setuptools-scm goes into nativeBuildInputs
  • https should be used for GitHub
  • pythonImportsCheck are really nice to have even when checkPhase is run
  • pytest should be executed explicitly through pytestCheckHook to avoid silent failures in the future
  • if nose is used please executed that directly instead of through setuptools check or pytest
  • the new line after }: is just stylistic

pkgs/development/python-modules/urlpy/default.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/plugincode/default.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/pymaven-patch/default.nix Outdated Show resolved Hide resolved
@cole-h
Copy link
Member Author

cole-h commented Apr 21, 2021

I think I got everything, but this PR is a doozy... feel free to re-open any resolved conversations that I missed.

EDIT: I don't think having bitarray_0_9 in the top-level is necessary, but I'll happily do it (later).

pkgs/development/python-modules/normality/default.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/pymaven-patch/default.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/saneyaml/default.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/saneyaml/default.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/commoncode/default.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/commoncode/default.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/commoncode/default.nix Outdated Show resolved Hide resolved
pytestCheckHook
];

checkPhase = ''
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was unable to get tests to work if I set preCheck = "export HOME=$(mktemp -d)"; (the tests would complain about an unwritable home), so this is necessary (unless there's something I overlooked).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That happens in the pythonImportsCheckPhase, so we can set $HOME in another phase like postInstall and add a comment as to why.
To solve this problem completely, we should introduce prePythonImportsCheck or something. Alternatively, pythonImportsCheck should be executed after pytestCheckPhase.
cc @jonringer @FRidh

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be able to do as part of preCheck. As I believe pythonImportsCheck is part of the preDist phase

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The real problem is actually addressed later: #120009 (comment)

Importing needs a writable home, and preCheck happens in between pythonImportsCheckPhase and pytestCheckPhase.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about we just run preCheck before pythonImportsCheck as well? #120300

@cole-h cole-h force-pushed the scancode branch 3 times, most recently from f0669d4 to b0b724a Compare April 22, 2021 01:02
@cole-h
Copy link
Member Author

cole-h commented Apr 22, 2021

aarch64 should be fixed by bdbdfe9.

I also fixed python39.pkgs in whatever packages it was broken for (by disabling the tests that don't work on python 3.9).

@dotlambda
Copy link
Member

Can we also correct dparse's homepage?

@cole-h
Copy link
Member Author

cole-h commented Apr 22, 2021

I noticed the same thing -- I'll get to that in the morning.

@cole-h
Copy link
Member Author

cole-h commented Apr 22, 2021

dparse's homepage has been corrected.

This fixes builds on aarch64.
@cole-h
Copy link
Member Author

cole-h commented Apr 23, 2021

@ofborg build python3.pkgs.scancode-toolkit

@grahamc
Copy link
Member

grahamc commented Apr 23, 2021

It looks like ofborg is done, but the "Wait for ofborg" check is confused about the state of things.

@grahamc grahamc merged commit 8d0340a into NixOS:master Apr 23, 2021
@grahamc grahamc deleted the scancode branch April 23, 2021 16:41
@jonringer
Copy link
Contributor

It looks like ofborg is done, but the "Wait for ofborg" check is confused about the state of things.

@grahamc AFAIK, the darwin check never completed. Also the aarch64 build failed

@cole-h
Copy link
Member Author

cole-h commented Apr 23, 2021

aarch64 failed because of bitarray's tests failing: #120358.

@SuperSandro2000
Copy link
Member

the darwin check never completed

That usually takes forever if it completes at all. I think it can be safely ignored.

@pombredanne
Copy link

@cole-h

aarch64 failed because of bitarray's tests failing: #120358.

I am not sure bitarray was ever built on ARM :|

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants