-
-
Notifications
You must be signed in to change notification settings - Fork 538
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for external licenses in scans #2979
Add support for external licenses in scans #2979
Commits on Sep 4, 2022
-
Add support for external licenses in scans aboutcode-org#480
This adds `-dir` or `--additional-directories` as a command line option in license detection. This allows users to specify paths to directories of licenses and rules they'd like to use during license detection, but would not like to add to the ScanCode database of licenses. This involves adding a new option in `licensedcode/plugin_license.py`, and this option is used as a parameter in `scancode/api.py`. In this approach, the licenses and rules contained in these additional directories are combined with the existing licenses and rules in the ScanCode database to produce a single index. The code for this is found in `licensedcode/cache.py` and the helper methods for loading these licenses and rules are found in `licensedcode/models.py`. This commit also includes a unit test to verify that license detection succeeds with an additional directory found in `tests/licensedcode/test_plugin_license.py`. Part of the setup for the unit test and future tests involves creating a new directory in `tests/licensedcode/data` that contains sample external licenses used in the unit tests. Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 05d9197 - Browse repository at this point
Copy the full SHA 05d9197View commit details -
Add documentation for new
--dir
CLI optionSigned-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 4989205 - Browse repository at this point
Copy the full SHA 4989205View commit details -
Enable using installed licenses in scans aboutcode-org#2994
This allows users to use licenses that they install via wheels in license scans. It also adds a sample package containing an external license that can be used to manually verify expected behavior. Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 0cbfff1 - Browse repository at this point
Copy the full SHA 0cbfff1View commit details -
Add CI job to test detecting installed license
This adds a new CI job in azure-pipelines.yml that installs an external license and tests that we can detect it. Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 3376b9c - Browse repository at this point
Copy the full SHA 3376b9cView commit details -
Add documentation for installed license plugins
This adds documentation for how to set up directories of installed licenses and how to install them for license detection. Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 681436d - Browse repository at this point
Copy the full SHA 681436dView commit details -
Enable installed rules to be used in detection
This adds files in the example installed license that can be used in license detection. It also adds a test to be run as part of a CI job. Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for e741ffd - Browse repository at this point
Copy the full SHA e741ffdView commit details -
Move
licensedcode_test_utils
into main wheelThis moves `licensedcode_test_utils` from tests into src so that users can write their own tests for installed licenses. It also adds documentation for how to write and run these tests. Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for afae692 - Browse repository at this point
Copy the full SHA afae692View commit details -
Add Windows and MacOS images to Azure pipelines
This adds Windows and MacOS images as part of the CI job to test the installed external license detection. Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for f576dfa - Browse repository at this point
Copy the full SHA f576dfaView commit details -
Add rule and license validation when index is made
This performs the checks `check_ignorable_clues()` in test_detection_validate.py and `test_validate_license_library_data()` in test_models.py when the index is created. This will allow us to validate additional rules and licenses that are added into the index. Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for a68c4cc - Browse repository at this point
Copy the full SHA a68c4ccView commit details -
Add SPDX license key to example licenses
Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 789cf76 - Browse repository at this point
Copy the full SHA 789cf76View commit details -
Refactor CLI option for external licenses
This removes ``-dir`` as an option and renames ``--additional_directories`` to ``--external-license-directory``. Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for e7809ee - Browse repository at this point
Copy the full SHA e7809eeView commit details -
revise documentation for --additional-license-directory
Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for cdf627f - Browse repository at this point
Copy the full SHA cdf627fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7e66c9a - Browse repository at this point
Copy the full SHA 7e66c9aView commit details -
refactor API to not use additional_directories except when reindexing
Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 9679feb - Browse repository at this point
Copy the full SHA 9679febView commit details -
Always consider multiple directories when generating index
Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 6aae7e2 - Browse repository at this point
Copy the full SHA 6aae7e2View commit details -
Ensure licenses are unique when loading licenses from multiple direct…
…ories Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for f13ed1d - Browse repository at this point
Copy the full SHA f13ed1dView commit details -
add callback for --additional-license-directory and include additiona…
…l directories in scancode output Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 9b03eea - Browse repository at this point
Copy the full SHA 9b03eeaView commit details -
fix help.txt to include --additional-license-directory
Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 40f3be9 - Browse repository at this point
Copy the full SHA 40f3be9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 983024e - Browse repository at this point
Copy the full SHA 983024eView commit details -
Configuration menu - View commit details
-
Copy full SHA for c8391d0 - Browse repository at this point
Copy the full SHA c8391d0View commit details -
add check in cli.py to see if cached directories file actually exists
Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 1bc43af - Browse repository at this point
Copy the full SHA 1bc43afView commit details -
fix expected test results directory path
Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for fc7b967 - Browse repository at this point
Copy the full SHA fc7b967View commit details -
Configuration menu - View commit details
-
Copy full SHA for 597c616 - Browse repository at this point
Copy the full SHA 597c616View commit details -
fix expected results for external and installed license tests
Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 85001c1 - Browse repository at this point
Copy the full SHA 85001c1View commit details -
put license installation into posix azure pipeline
Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 7497009 - Browse repository at this point
Copy the full SHA 7497009View commit details -
remove setuptools and wheel from setup.py
Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for d6068c6 - Browse repository at this point
Copy the full SHA d6068c6View commit details -
change from scan to reindex licenses in license library validation test
Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 16513ff - Browse repository at this point
Copy the full SHA 16513ffView commit details -
Add is_builtin field to Licenses and Rules and modify url output
Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 17df9d0 - Browse repository at this point
Copy the full SHA 17df9d0View commit details
Commits on Sep 5, 2022
-
fix methods based on previous changes
Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 3762ca5 - Browse repository at this point
Copy the full SHA 3762ca5View commit details -
add new license provider plugin for additional licenses
Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for ba9740b - Browse repository at this point
Copy the full SHA ba9740bView commit details
Commits on Sep 30, 2022
-
Test that additional license plugin works
This test can run only in the CI and use a new pytest marker. This requires isolation as it installs new license from a new location plugin. Co-authored-by: Kevin Ji <kyji1011@gmail.com> Co-authored-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com> Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Configuration menu - View commit details
-
Copy full SHA for ba11f05 - Browse repository at this point
Copy the full SHA ba11f05View commit details -
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Configuration menu - View commit details
-
Copy full SHA for 61c3283 - Browse repository at this point
Copy the full SHA 61c3283View commit details -
Use new "scanplugins" pytest marker
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Configuration menu - View commit details
-
Copy full SHA for a4ebbe0 - Browse repository at this point
Copy the full SHA a4ebbe0View commit details -
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Configuration menu - View commit details
-
Copy full SHA for 1db9437 - Browse repository at this point
Copy the full SHA 1db9437View commit details
Commits on Oct 2, 2022
-
fix expected scan results after installed license CI change
Signed-off-by: Kevin Ji <kyji1011@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 8df0e27 - Browse repository at this point
Copy the full SHA 8df0e27View commit details
Commits on Oct 12, 2022
-
Reorganize additional license tests
* Organize tests data under one single folder. * Organize test functions under one single file. * Keep three tests, one for additional directory, one for additional plugin, and one for combined testing. * Edit the CI file accordingly. Co-authored-by: Philippe Ombredanne <pombredanne@nexb.com> Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for f53886f - Browse repository at this point
Copy the full SHA f53886fView commit details -
Move reindex licenses to a seperate script
Delete the reindex licenses options from scancode CLI and add a new script which does the reindexing, also with additional directories optinally. Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 5361052 - Browse repository at this point
Copy the full SHA 5361052View commit details -
Configuration menu - View commit details
-
Copy full SHA for a477e54 - Browse repository at this point
Copy the full SHA a477e54View commit details -
Add external licenses info in header
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 6412039 - Browse repository at this point
Copy the full SHA 6412039View commit details
Commits on Oct 20, 2022
-
Add is_builtin flag to matched_rule data
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 6e14d8a - Browse repository at this point
Copy the full SHA 6e14d8aView commit details -
Do not return empty strings in license data
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 044f60d - Browse repository at this point
Copy the full SHA 044f60dView commit details
Commits on Oct 21, 2022
-
Add --only-builtin falg for scancode-reindex-licenses
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for f201faa - Browse repository at this point
Copy the full SHA f201faaView commit details
Commits on Oct 28, 2022
-
Update docs for external licenses
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 095c8ed - Browse repository at this point
Copy the full SHA 095c8edView commit details -
Refactor external licenses code
Refactor according to the feedback. Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 54fb102 - Browse repository at this point
Copy the full SHA 54fb102View commit details -
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Configuration menu - View commit details
-
Copy full SHA for f2b1e13 - Browse repository at this point
Copy the full SHA f2b1e13View commit details