Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

License detection improvements and review #3346

Merged
merged 95 commits into from
Jun 11, 2023
Merged

License detection improvements and review #3346

merged 95 commits into from
Jun 11, 2023

Conversation

pombredanne
Copy link
Member

@pombredanne pombredanne commented Apr 24, 2023 •
edited by AyanSinhaMahapatra

This PR adds misc. license and copyright detection improvements and the --todo scancode CLI option for review items summary.

Tasks

  • Reviewed contribution guidelines
  • PR is descriptively titled 📑 and links the original issue above 🔗
  • Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
    Run tests locally to check for errors.
  • Commits are in uniquely-named feature branch and has no merge conflicts 📁
  • Looked for possible updates in documentation and added updates if applicable
  • Updated CHANGELOG.rst

DennisClark and others added 30 commits April 24, 2023 17:55
@DennisClark @pombredanne
Ensure MIT License is not mistakenly detected
ae9b5c0 
Signed-off-by: Dennis Clark <dmclark@nexb.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Do not test license data as binary
5ae2284 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Bump the relevance of the Apache license URL
17ef223 
This is a 100% relevant reference

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add new and improved license detection rules
8ccc6e9 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add new license detection rules
5736b27 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Refine and rename license detection rules
534a1bf 
This helps to get a more accurate and correct detection

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add new license detection rules #3256
3653b26 
Add new rules to improve detection accuracy in phpwiki.

Reference: #3256
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add new copyleft license detection rules
dfa8a62 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add new permissive license detection rules
7c7288c 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add new misc. license detection rules
5b4c61b 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Improve rule building and error reporting
dd63c23 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add new license detection rules
8fbf43b 
These are mostly seen in some Debian copyright files

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Collect (c)opyright copyright notices
789248e 
These are weird but seen frequently enough

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Do not detect JUNK copyright
d3c2e57 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Do not treat @link as relevant to authorship
83e72d3 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Do not return weird dates in holder
f295155 
Also treat some capitalized words as no proper nouns

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Improve copyright detections
ab3caed 
Dot not treat solo period as NN
Allow trailing "authors" as part of copyright
Handle more case of truncated copyrights

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Rename rules to avoid conflicts
b84a8cc 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Merge remote-tracking branch 'master5/add-new-and-improved-license-ru…
a00774e 
…les' into license-detection-improvements-and-review
@pombredanne
Improve copyright and authors detection
bbf3c56 
* Remove duplicated tests
* Do not strip leading @ sign in words
* Streamline some lexer regex (they were incorrectly handling dots)
* Add new copyright statement variants

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@AyanSinhaMahapatra
Ambiguous Detections ToDo items #3122
c9dc26b 
* Adds a new --todo post scan option to review potential
  ambiguous license/package detections
* Adds a `todo` top level attribute with these review items
  which is a `ToDo` list for the reviewer

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Smoke test --from-json for --todo
7cbc78e 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Fix --from-json in --todo option
f885196 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Add support for license clues in --todo
d4124b2 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Add low relevance case in --todo
faf24c3 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Fix test expectations and tests
92299a1 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Add new license detection rules
ccdc40b 
These solve the following license detection bugs:

* #3361
* #3360
* #3358
* #3355

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Add unknown license reference rules
bfb7087 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Regenerate test expectations for maven
9d42b81 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Add CHANGELOG entry for --todo option
259eb01 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@pombredanne
Improve morecopyright detection corner cases
31c5f52 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Merge remote-tracking branch 'upstream/license-detection-improvements…
8f89254 
…-and-review' into license-detection-improvements-and-review
@pombredanne
Add new license detection rules
341c9c1 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Improve copyright detection in markup
9159a27 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Add new and improved license detection rules
93097a0 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the license-detection-improvements-and-review branch from b36bb73 to 86151a8 Compare May 30, 2023 10:51
@AyanSinhaMahapatra
Fix bug in nuget license detection
886bc39 
Reference: #3409
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the license-detection-improvements-and-review branch from 86151a8 to 886bc39 Compare May 30, 2023 11:39
AyanSinhaMahapatra and others added 9 commits May 31, 2023 14:29
@AyanSinhaMahapatra
Add new apache-1.1 RULE
2b51b4d 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Merge branch 'develop' into license-detection-improvements-and-review
9cc72ce
@AyanSinhaMahapatra
Fix test failures with regen
2629008 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra
Fix unknown-spdx detections for dangling markup
713a605 
Fixes unknown-spdx detections for nuget nuspecs file license
detection where a dangling </licenseurl> markup would result
in the SPDX license detection not working.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@pombredanne
Improve copyright detection
7417bc6 
For the form of "Contributors to the project..."

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Remove unused method
eecd896 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Clarify doc and format code
1c1acdc 
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Fail correctly on timeout
6252270 
Somehow we need to re-raise an exception fo this to work correctly

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne
Keep trucking on timeout
ca92534 
Do not fail to scan on timeout, instead keep on trucking and log errors.

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>
@pombredanne pombredanne force-pushed the license-detection-improvements-and-review branch from b439760 to ca92534 Compare June 7, 2023 10:48
@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the license-detection-improvements-and-review branch 2 times, most recently from 05dd2ba to 6d439eb Compare June 9, 2023 14:27
@AyanSinhaMahapatra
Fix top level license detection bugs
1bad407 
Fixes a bug where top level license detections had license expressions
and IDs which were not present anywhere in the scan. This was because
we were dereferencing unknown license references after the unique
license detection collections and so the modified identifiers and
license expressions were not present in the top-level license
detections.

Fixes a bug where top level unique license detections
were not properly collected from package license detections.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the license-detection-improvements-and-review branch from 6d439eb to 1bad407 Compare June 9, 2023 15:02
@AyanSinhaMahapatra
Merge branch 'develop' into license-detection-improvements-and-review
a87a4d5 
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
@AyanSinhaMahapatra AyanSinhaMahapatra force-pushed the license-detection-improvements-and-review branch from 48fe068 to a87a4d5 Compare June 9, 2023 17:21
pombredanne
pombredanne commented Jun 11, 2023
View reviewed changes
Copy link
Member Author

@pombredanne pombredanne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM ... time to merge thos!

@pombredanne pombredanne merged commit b52f7df into develop Jun 11, 2023
34 checks passed
@pombredanne pombredanne deleted the license-detection-improvements-and-review branch June 11, 2023 17:53
@AyanSinhaMahapatra AyanSinhaMahapatra restored the license-detection-improvements-and-review branch June 12, 2023 10:38
@AyanSinhaMahapatra AyanSinhaMahapatra mentioned this pull request Jun 29, 2023
Merged
@pombredanne pombredanne deleted the license-detection-improvements-and-review branch June 29, 2023 16:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@AyanSinhaMahapatra AyanSinhaMahapatra

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@pombredanne @AyanSinhaMahapatra @DennisClark @johnmhoran