New tool - redistribution check suggestion

[bobgob]

Given that an audited codebase is different from what is redistributed, part of my job is to scan the redistribution. I run it through extractcode/scancode (only-findings) and produce a json-pp file which I manually go through. These are usually over 100MB. For example, I look for “GPL 3”, “Commercial”, “Restricted”, “Proprietary”, “Patent” and other company important licenses. I also look for jars without source jars. This is rudimentary but catches a surprising number of errors. It’s also time consuming.

It would be more efficient to be able to remove items from the output that cause uninteresting hits. For example,
Binutils
Gdb
.txt, COPYING, Whence, html, … files
LC_MESSAGES/
Testsuite/
Fonts
Etc.
Files that have multiple licenses where one of them is gplv2 or a very permissive license.

Perhaps I've included too many details but I hope I've conveyed the need for some sort of tool help.

[pombredanne]

That's a great idea that could likely be implemented though the new plugin architecture by @yashdsaraf

There could be a couple ways that come to mind:

• a pre scan plugin: before scanning (like the --ignore flag) a file or path, apply some custom filtering to avoid scanning some parts of a codebase entirely. This could be based on the various type and name information, some file classification, etc and could skip scanning entirely some parts of the codebase
• a post scan plugin: once the scans are completed and before the output is generated we could have a plugin that does some custom filtering or transformation of the results. Since the scans are available then, this can be based on more data such as the detected licenses, copyrights, etc. and could perform some summarization as needed
• less appealing, an output plugin that would write to database (e.g. sqlite) and that you could filter externally

These could also be filtering that could be handled in AboutCode Manager, but they might be best at scan time.

[yashdsaraf]

@pombredanne The post-scan plugin approach seems best to me for handling

Files that have multiple licenses where one of them is gplv2 or a very permissive license

Perhaps even a pre-scan and post-scan plugin working in conjunction? so some obvious files can be ignored before scan and ones which require the scan to complete, like determining the permissive license of the two, might be handled by a post-scan plugin.

[pombredanne]

@yashdsaraf good ideas. @bobgob could you elaborate a bit on which kind of filtering you are looking after?

[bobgob]

It seems like the easiest would be a pre-scan like an rsync exclude file. Logic based exclusions would need to be done as a post-scan.
I've not been able to use AboutCode Manager with my large json files but it would be great to use ACM and filter on the fly.
It would also be great if there were file groups (that I could edit). For example, if there were a "permissive" group (a text file with all the licenses you consider permissive) and I could do filtering by group and not just individual license. The scancode project would supply default groups so people can get started using them quickly. Filtering only by individual license would be very tedious.

[pombredanne]

@bobgob you wrote:

It seems like the easiest would be a pre-scan like an rsync exclude file. Logic based exclusions would need to be done as a post-scan.

Could the current --ignore option do most of what you need?

It would also be great if there were file groups (that I could edit). For example, if there were a "permissive" group (a text file with all the licenses you consider permissive) and I could do filtering by group and not just individual license. The scancode project would supply default groups so people can get started using them quickly. Filtering only by individual license would be very tedious.

Filtering based on licensing would have to be a post-scan plugin. Note that while there is no such grouping done, the license "Category" attribute is returned for all license matches and would likely the grouping you might want to use as a base. It is available right now.

[bobgob]

Yes, --ignore would work. Just more cumbersome than an exclude file.
Category is what I use today (manually). It is just not fine grained as much as I would like. For example, I don't want GPLv3 and GPLv2 in the came group. With user defined categories I could duplicate Copyleft, remove GPLv3 and have two categories, one with and one without GPLv3. That implies that a file can be in multiple categories. If file and category need to be 1:1 then I would remove GPLv3 from Copyleft and create a new GPLv3 category (for my usage).
All this said, the best solution for me would be an interactive tool because I don't know what I'm getting into until after the scan. For example, I may find a large directory that needs to be ignored. Or I find a whole directory structure that I can categorize. For example, say I find a proprietary file. The next thing I'd do is check the other files in that directory. If they are also proprietary I would look at the parent directory and repeat. This bottom up classification can be very time consuming but is needed so that I can identify the root directory of those proprietary files. I believe ScanCode Toolkit and AboutCode can do this but they are not able to handle the number of files in my scans.

[pombredanne]

@bobgob you wrote

Yes, --ignore would work. Just more cumbersome than an exclude file.

This has been designed as the front of an ignore file which should be next as an enhancement

Category is what I use today (manually). It is just not fine grained as much as I would like. For example, I don't want GPLv3 and GPLv2 in the came group. With user defined categories I could duplicate Copyleft, remove GPLv3 and have two categories, one with and one without GPLv3. That implies that a file can be in multiple categories. If file and category need to be 1:1 then I would remove GPLv3 from Copyleft and create a new GPLv3 category (for my usage).

I think the best approach here would be to update the scans with your own categories. This could be a post scan plugin alright.

All this said, the best solution for me would be an interactive tool because I don't know what I'm getting into until after the scan. For example, I may find a large directory that needs to be ignored. Or I find a whole directory structure that I can categorize. For example, say I find a proprietary file. The next thing I'd do is check the other files in that directory. If they are also proprietary I would look at the parent directory and repeat. This bottom up classification can be very time consuming but is needed so that I can identify the root directory of those proprietary files. I believe ScanCode Toolkit and AboutCode can do this but they are not able to handle the number of files in my scans.

Visual navigation and "tagging" would definitely be a job for AboutCode Manager... Yet if you are dealing with very large scans this may be an issue at least for now.

A couple of avenues to improve this are:

• divide and conquer, e.g. do not run one large scan but several smaller ones?
• improve AboutCode Manager to deal with large codebase. One things I chatted with @jdaguil there could be an output plugin that would spit a sqlite DB directly in the format expected by ABCMger. This is not a panacea but this means that no JSON transformation would be needed to open a scan in ABCMger with this and this may help dealing with very large scans.