This adds `-dir` or `--additional-directories` as a command line
option in license detection. This allows users to specify paths to
directories of licenses and rules they'd like to use during license
detection, but would not like to add to the ScanCode database of
licenses.

This involves adding a new option in `licensedcode/plugin_license.py`,
and this option is used as a parameter in `scancode/api.py`. In this
approach, the licenses and rules contained in these additional
directories are combined with the existing licenses and rules in the
ScanCode database to produce a single index. The code for this is found
in `licensedcode/cache.py` and the helper methods for loading these
licenses and rules are found in `licensedcode/models.py`. This commit
also includes a unit test to verify that license detection succeeds
with an additional directory found in
`tests/licensedcode/test_plugin_license.py`. Part of the setup for the
unit test and future tests involves creating a new directory in
`tests/licensedcode/data` that contains sample external licenses used
in the unit tests.

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

This allows users to use licenses that they install via wheels in
license scans. It also adds a sample package containing an external
license that can be used to manually verify expected behavior.

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

This adds a new CI job in azure-pipelines.yml that installs an external
license and tests that we can detect it.

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

This adds documentation for how to set up directories of installed
licenses and how to install them for license detection.

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

This adds files in the example installed license that can be used in
license detection. It also adds a test to be run as part of a CI job.

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

This moves `licensedcode_test_utils` from tests into src so that
users can write their own tests for installed licenses.
It also adds documentation for how to write and run these tests.

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

This adds Windows and MacOS images as part of the CI job to test
the installed external license detection.

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

This performs the checks `check_ignorable_clues()` in
test_detection_validate.py and `test_validate_license_library_data()`
in test_models.py when the index is created.
This will allow us to validate additional rules and licenses that are
added into the index.

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

This removes ``-dir`` as an option and renames
``--additional_directories`` to ``--external-license-directory``.

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

…ories

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

…l directories in scancode output

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

This test can run only in the CI and use a new pytest marker.
This requires isolation as it installs new license from a new location
plugin.

Co-authored-by: Kevin Ji <kyji1011@gmail.com>
Co-authored-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>
Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>

Signed-off-by: Kevin Ji <kyji1011@gmail.com>

* Organize tests data under one single folder.
* Organize test functions under one single file.
* Keep three tests, one for additional directory, one for
  additional plugin, and one for combined testing.
* Edit the CI file accordingly.

Co-authored-by: Philippe Ombredanne <pombredanne@nexb.com>
Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Delete the reindex licenses options from scancode
CLI and add a new script which does the reindexing,
also with additional directories optinally.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Refactor according to the feedback.

Signed-off-by: Ayan Sinha Mahapatra <ayansmahapatra@gmail.com>

Signed-off-by: Philippe Ombredanne <pombredanne@nexb.com>