Detect unknown licenses #1675 #2592
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: akugarg <akanksha.garg2k@gmail.com>
akugarg
force-pushed
the
improve_license_detection
branch
from
190554b
to
aecab91
Compare
AyanSinhaMahapatra
added this to In progress
in GSoC 2021: Improve Unknown License Detection @akugarg
pombredanne
requested changes
Aug 6, 2021
pombredanne
requested changes
Aug 14, 2021
Signed-off-by: akugarg <akanksha.garg2k@gmail.com>
akugarg
force-pushed
the
improve_license_detection
branch
from
962d1f5
to
de2e0d0
Compare
|
This should add some tests on licenses that ideally scancode would never add (some proprietary/sdk licenses), to check the accuracy of unknown license detection. Linking some of them here: |
Rather than using a function argument Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
This is a batch of misc. license detection rules and a new license. These have been mostly found thanks to the upcoming unknown license detection. Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
This makes it available in the CLI Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
This is cleaner and more composable than old-style interpolation Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Add split_weak_matches() function to pre-filter weak unknown matches. Make unknown matches eligible for filter_spurious_matches() and lower minimum density to 0.6=5 for longer matches. Move the call for filter_spurious_matches() earlier in the refine pipeline. Add new filter_invalid_contained_unknown_matches() function to discard unknown matches found inside the matched queryregion of larger regular matches Extract get_full_qspan_matched_text() function from get_full_matched_text() for improve reusability. This is designed to be called when crafting new rules absed on a match (which is what is done with unknown matches). Use format for matched license text highlight This is cleaner and more composable than old-style interpolation. Improve debug tracing of matched texts. Apply other minor refactoring and doc impropvements Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Create unique rule id based on a checksum of the rule content
Also improve key phrases parsing for dnagling {{ {{ braces.
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Create proper synthethic Rule and LicenseMatch on match and return a match or None. Include unknown licenses matching as an option to Index.match Add tests Use shorter ngrams of length 6 rather than 7 for better sensitivity This is balanced by the addition of filters: - Filter weak unknown matches at match time - Filter out several weak unknown ngrams at index time Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
|
@akugarg @AyanSinhaMahapatra your review is welcomed! |
|
Note that you can ignore most of the rules additions, they should have been in another PR, but I could not resist adding them... I forced an unknown licenses detection on our whole test suite and this helped spots several issues that I fixed on the fly. :P |
pombredanne
approved these changes
Jan 8, 2022
pombredanne
changed the title
Note that this is NOT YET returned in the API and outputs Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
|
I am merging now. Your feedback is still mucho welcomed on the refinements, |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: akugarg akanksha.garg2k@gmail.com
This PR introduces a new and effective way for detection of unknown licenses by making use of n-grams.
Refer #1675 for more details.
Tasks
Run tests locally to check for errors.