monk - full text license scanner #2

wuan · 2014-07-04T17:52:15Z

This pull request adds another license scanner called monk.

Monk is scanning files and compares their content to the full text of all licenses known to fossology or other text sources.

Monk is based on the following workflow:

First both texts are tokenized word by word.
Then both texts are compared token by token.
Monk differentiates between full matches and matches with additions/removal and provides a full diff like view of its result

Currently the performance of monks algorithm is good enough to check every source file agains every known license. Differing from the first proposal in #7065 (see sourceforge) monk can be used as a standalone license scanner and its operation does not depend on e. g. nomos results.

In addition to the monk agent we started to generalize the license browser and viewer, which was extended to show licenses found by different license scanning agents (currently nomos and monk) while browsing the file tree. We would like to gather more feedback from the community as this general license browser might replace agent specific license browsers later.

…ory), thanks for your testing.

…nd List Files for License)

bucket syntax error due to adding specific uploadtree tables in sql

… for licenses matche with a _LT_FREE_x signature)

…7149, #6831, #7263.

…igemo, Logica-OSL-1.0, Nvidia , Unidex, Giftware, GPL-3.0+-with-classpath-exception and GPL-2.0+-with-classpath-exception.

…recursively scan a directory), thanks for your testing.

…pdel'.

…rn error msg when the licesne shortname is empty.

…ces are trying to delete same files.

wuan · 2014-08-08T20:27:17Z

Merged on sourceforge master

commit e5cedc6bc2cac6f3a73d7a8279c080d03ae00551 Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Thu Jan 31 09:42:37 2019 +0100 un-comment stage 'docker-release-job' commit 940fddb41c4353bbcf847befdabdab982f8314be Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Thu Jan 31 09:08:25 2019 +0100 really configure stage 'docker-audit-clair-job' commit 78d002fcb19982a30fc53a57c3e7e5bb9c4e868f Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Thu Jan 31 08:58:32 2019 +0100 un-comment security stage 'docker-audit-clair-job' commit 2e176bf4c1fcbf0acf4e15d8152744a693953881 Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Tue Jan 29 22:23:29 2019 +0100 Revert "install directly VIM in docker image" This reverts commit 42f9bde724aa72b6f7bba68373295988ee841cf2. commit 42f9bde724aa72b6f7bba68373295988ee841cf2 Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Tue Jan 29 17:54:36 2019 +0100 install directly VIM in docker image commit 0b3bf5aabd294358c6a8ead6a557b8eec0886923 Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Tue Jan 29 17:24:07 2019 +0100 add EXPOSE stanza to Dockerfile commit 11c8174cda584b9f523c7bf26bc2dd004982ee40 Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Tue Jan 29 16:22:55 2019 +0100 use appropriate command for the Alipne docker image (apk command) commit dcfd0a8e487af9c05003c1d673a0e13e839e4d7a Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Tue Jan 29 15:52:34 2019 +0100 Install git commit a525e49ffe97777c574cd3d7ca408186c0fbac4b Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Tue Jan 29 15:47:24 2019 +0100 Activate Octopus Merge commit dfed1b1072e90c0467e69d349fb197126a444b70 Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Tue Jan 29 15:23:29 2019 +0100 re-disable test stage for now commit f0aaacdba8b2b3138473f6e516d35d64f92a01df Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Tue Jan 29 15:10:36 2019 +0100 un-remove previously un-removede --cache-from option + remove new lines commit ad8becb7e40dd2ad0b046a7f03bd1c8d9329d181 Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Tue Jan 29 14:39:25 2019 +0100 remove un-removed --cache-from option commit 13c73b2c482e4f0d694f1d733cdb9230bd400400 Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Tue Jan 29 14:33:40 2019 +0100 linty: use --permissive option rather than previous hack commit f66363612fa832a50251396db636248151ad0ad2 Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Tue Jan 29 14:30:29 2019 +0100 un-remove --cache-from option commit 2a264f3aa49a6d6ae6ee1dc193615c5681452441 Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Tue Jan 29 12:04:09 2019 +0100 workaround Lint test errors commit dfbf11b33e3104d0284f8d2dea1b08134834ec53 Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Tue Jan 29 11:34:37 2019 +0100 Remove double Maintener labes commit 566578911568208865b61c05e2ac9267d1b7f820 Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Tue Jan 29 11:13:15 2019 +0100 Try to comply with Dockerfile checking rules commit f5f7f7c5507dfdbf2345ef41ffdf25199d6d65d1 Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Tue Jan 29 11:03:12 2019 +0100 uncomment all Tests sections, fix the stages commit 4d03b7f64870fc44416badc3699858b2ca4c85ee Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Tue Jan 29 11:00:04 2019 +0100 uncomment the Test section commit f1a33e405bbdc9ed8f391022933423a691f34333 Author: bdrozd01 <bartlomiej.drozdz@orange.com> Date: Tue Jan 29 09:44:11 2019 +0100 fix no_proxy syntax commit 5566b93a2077524fa9d760eb82778b5118fa13f5 Author: bdrozd01 <bartlomiej.drozdz@orange.com> Date: Tue Jan 29 09:38:47 2019 +0100 docker build fix attempt commit d12044d619059b761e0ba217739379eb4cdc6596 Author: bdrozd01 <bartlomiej.drozdz@orange.com> Date: Tue Jan 29 09:32:21 2019 +0100 remove --cache-fom in docker build commit 52d6e51aee27031e95db5a59f5519f0743432462 Author: bdrozd01 <bartlomiej.drozdz@orange.com> Date: Tue Jan 29 09:23:54 2019 +0100 pass proxy context to container build commit 13a950b474e8391b7447d4116ee4273675fa7c8a Merge: 5777195c4 fafb96fb2 Author: bdrozd01 <bartlomiej.drozdz@orange.com> Date: Tue Jan 29 09:15:37 2019 +0100 resolve local conflicts commit fafb96fb2009111c34342536da8bc02c6b096f96 Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Tue Jan 29 09:09:14 2019 +0100 no proxy for registry.forge.orange-labs.fr commit 684cd7575c6755e9f1ba771ffa369de93c5c7536 Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Mon Jan 28 18:12:38 2019 +0100 Add proxy setting commit ca9d83a1270367fd5565b5f38f96a18e66e524f4 Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Mon Jan 28 17:04:10 2019 +0100 Adapt gitlab-ci file commit ae4056a67f9c4b5f91c26277baf00c7bd6a719fd Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Mon Jan 28 16:57:32 2019 +0100 remove lint and test stages commit cd34a4341b284a6483b7bc5ff4b94a1edb0b7be4 Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Mon Jan 28 16:45:29 2019 +0100 reuse gitLab-CI from gitlab-ci-docker-image-hello-world branch commit 49b0729c6b078bf2ec31e82652338a77b47b7b3f Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Mon Jan 28 16:27:12 2019 +0100 tags in build_step_one stage #2 commit 5777195c487255b64bb4fe1edf6d863aee0ab4ca Author: bdrozd01 <bartlomiej.drozdz@orange.com> Date: Mon Jan 28 16:23:25 2019 +0100 other changes commit 0f55061f8f09a66b8f4298f4ba22c814f810819c Author: bdrozd01 <bartlomiej.drozdz@orange.com> Date: Mon Jan 28 16:18:14 2019 +0100 tags in build_step_one stage commit 04dffc38e7dafbc05bf6a7d4d493ada50e7d8c09 Author: Toussaint Nicolas <nicolas1.toussaint@orange.com> Date: Mon Jan 28 16:10:22 2019 +0100 build docker image, try #1

yaobinshi and others added 29 commits July 15, 2014 16:43

fixed for ununpack unit test failures because of patch from kevin lo.

7d91513

fixed for #6015(Upgrade cp2foss to allow upload from a source reposit…

21129c8

…ory), thanks for your testing.

fixed for #6057(Implement new path filters on List Files for Bucket a…

fa5d110

…nd List Files for License)

Added performance/stree test script.

d547888

Merge branch 'master' of ssh://git.code.sf.net/p/fossology/fossology

bdb970a

Fix for http://www.fossology.org/issues/5829

bb46c56

bucket syntax error due to adding specific uploadtree tables in sql

Implements enhancement #7227 (Report "Freeware" instead of "BSD-lite"…

bdbd2ec

… for licenses matche with a _LT_FREE_x signature)

fixed and added tests for #7197, #7209, #7203, #7191, #7173, #7167, #…

6f31369

…7149, #6831, #7263.

delete Nvidia-EULA directory.

ca238a9

Merge branch 'master' of ssh://git.code.sf.net/p/fossology/fossology

1bc2417

added new licenses and related license text, Freeware, JISP, Qmail, M…

059c8f4

…igemo, Logica-OSL-1.0, Nvidia , Unidex, Giftware, GPL-3.0+-with-classpath-exception and GPL-2.0+-with-classpath-exception.

fixed for #7323(JISP license has an white space in license admin page)

eec6818

Fix issue #7329: fossinit.php fails if Nomos doesn't return any version

e5f4ef5

Merge branch 'master' of ssh://git.code.sf.net/p/fossology/fossology

d570716

fixed for http://www.fossology.org/issues/7137 (nomos new feature to …

4064891

…recursively scan a directory), thanks for your testing.

fixed for nomos cli tests failures.

557d911

get rid of a debug msg.

5554051

Fixed bug6921, on Rhel/Fedora system commands are 'userdel' and 'grou…

db9cf89

…pdel'.

fixed for #7347(nomossa compile error and caused package build failed)

3c2821e

added warning msg and changed the default value of process_count.

21d7ed2

fixed one defect in list_license.php (report all license as default)

b35f76d

fixed for #7323, trim the license short name and full name, also retu…

10b238d

…rn error msg when the licesne shortname is empty.

2 scripts to get which licenses nomos can return.

4c687fe

fixed for nomos running on directory. this fix is for multiple instan…

5c853a9

…ces are trying to delete same files.

add more helpful error msg.

323d223

change fopen mode from 'a' to 'w'.

2dd6c6b

Added test cases for migration testing.

6f4d5a4

Remove the hard code home directory.

f170db4

monk full text license scanner

45d1ecf

yaobinshi mentioned this pull request Jan 9, 2015

parts of URLs reported as e-mail addresses even though they are not #229

Open

bobgob mentioned this pull request Jan 28, 2015

Dashboard failure in 2.6.2 #384

Closed

This was referenced Mar 23, 2015

Fatal error when load the license browse page #404

Closed

[Admin -> Dashboard] crashes with both browser and log error text displayed. #409

Closed

rlintu mentioned this pull request Apr 9, 2015

Database migration from 2.6.2 to master branch fails #418

Closed

pguttmann mentioned this pull request May 27, 2015

fo-postinstall script is failing in current code base b80b858465 #446

Closed

ag4ums mentioned this pull request Feb 20, 2019

rest api endpoints for jobs, uploads, report and folder are failing #1287

Closed

NicolasToussaint mentioned this pull request Jun 20, 2019

Associate a link to an internal ticket system to an upload #1378

Open

abirw-zz mentioned this pull request Jul 16, 2019

A fresh install requires decision for alter DB (clearing_decision.clearing_pk) #415

Closed

helloWorldHitech mentioned this pull request Aug 1, 2019

Reuser fail on rev 3.5 #1411

Closed

jl431 mentioned this pull request Dec 4, 2019

Install of Fossology 3.6.0 fails when configured with a database that isn't called "fossology" #1543

Open

NicolasToussaint referenced this pull request in Orange-OpenSource/fossology May 29, 2020

Also merge docker-build branch #2

b883deb

NicolasToussaint referenced this pull request in Orange-OpenSource/fossology May 29, 2020

Also merge docker-build branch #2

e988f6a

lucian-balea mentioned this pull request Nov 19, 2020

Option to highlight and count files with no license information #1775

Open

Pirx69 mentioned this pull request Nov 30, 2020

ReDeployment of Fossology Docker fails due to Exception in Fossology StartUp Script #1841

Open

mcjaeger mentioned this pull request Apr 6, 2021

feat(copyright): save deleted copyrights in copyright_event table #1914

Merged

dl9pf mentioned this pull request Aug 25, 2021

ununpack lockup on libzip source tree #2090

Open

lakshmibhavani1 mentioned this pull request Mar 31, 2022

Reuser fails after upgrading to 4.0 version #2200

Closed

scanoss-qg mentioned this pull request Jul 1, 2022

feat(scanoss-agent): Initial version of scanoss agent for Fossology #2167

Closed

wang-fut mentioned this pull request Aug 2, 2023

scheduler keeping restarting but health check looks just fine #2558

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

monk - full text license scanner #2

monk - full text license scanner #2

wuan commented Jul 4, 2014

wuan commented Aug 8, 2014

monk - full text license scanner #2

monk - full text license scanner #2

Conversation

wuan commented Jul 4, 2014

wuan commented Aug 8, 2014