Adds GitHub wiki of Scancode-Toolkit and Scancode-Workbench

docs/source/conf.py

@@ -14,6 +14,14 @@
 # import sys
 # sys.path.insert(0, os.path.abspath('.'))
 
+# Adding Support for GIFs in Sphinx
+from sphinx.builders.html import StandaloneHTMLBuilder
+StandaloneHTMLBuilder.supported_image_types = [
+    'image/svg+xml',
+    'image/gif',
+    'image/png',
+    'image/jpeg'
+]
 
 # -- Project information -----------------------------------------------------
 

docs/source/doc_maintenance.rst

@@ -47,6 +47,10 @@ Assuming that your Sphinx installation was successful, Sphinx should build a loc
 
     open build/html/index.html
 
+In case this command did not work, for example on Ubuntu 18.04 you may get a message like “Couldn’t get a file descriptor referring to the console”, try: ::
+
+    see build/html/index.html
+
 You now have a local build of the AboutCode documents.
 
 Improve AboutCode Documents

docs/source/index.rst

@@ -5,8 +5,10 @@ Guide
 *****
 
 .. toctree::
-   :maxdepth: 2
+   :maxdepth: 3
 
+   scancode-toolkit/index
+   scancode-workbench/index
    license
    help
    doc_maintenance

docs/source/scancode-toolkit/comprehensive_installation.rst

@@ -0,0 +1,107 @@
+Comprehensive Installation
+==========================
+
+ScanCode requires Python 2.7.x and is tested on Linux, Mac, and Windows. Make sure Python 2.7 is installed first.
+
+System Requirements
+-------------------
+
+- Hardware : ScanCode will run best with a modern X86 processor and at least 2GB of RAM and 250MB of disk.
+
+- Supported operating systems : ScanCode should run on these OSes:
+
+	#. Linux: on most recent 64-bit Linux distributions (32-bit distros are only partially supported),
+	#. Mac: on recent Mac OSX (10.6.8 and up),
+	#. Windows: on Windows 7 and up (32- or 64-bit) using a 32-bit Python.
+
+Prerequisites
+-------------
+ScanCode needs a Python 2.7 interpreter.
+
+- On Linux: Use your package manager to install python2.7. If Python 2.7 is not available from your package manager, you must compile it from sources. For instance, visit https://github.com/dejacode/about-code-tool/wiki/BuildingPython27OnCentos6 for instructions to compile Python from sources on Centos.
+
+- On Ubuntu 12.04, 14.04 and 16.04, you will need to install these packages first: ``python-dev bzip2 xz-utils zlib1g libxml2-dev libxslt1-dev``
+
+- On Debian and Debian-based distros you will need to install these packages first: ``python-dev libbz2-1.0 xz-utils zlib1g libxml2-dev libxslt1-dev``
+
+- On RPM-based distros, you will need to install these packages first: ``python-devel zlib bzip2-libs xz-libs libxml2-devel libxslt-devel``
+
+- **On Windows**:
+
+	Use the Python 2.7 32-bit (e.g. the Windows x86 MSI installer) for X86 regardless of whether you run Windows on 32-bit or 64-bit. DO NOT USE Python X86_64 installer even if you run 64 bit Windows. Download Python from this url: https://www.python.org/ftp/python/2.7.13/python-2.7.13.msi
+
+	Install Python on the c: drive and use all default installer options(scancode will try to find python just in c:\python27\python.exe). See the Windows installation section for more installation details.
+
+- On Mac: Download and install Python from this url: https://www.python.org/ftp/python/2.7.13/python-2.7.13-macosx10.6.pkg
+
+Do not use Unicode, non-ASCII in your installation Path
+-------------------------------------------------------
+There is a bug in underlying libraries that prevent this.
+
+Installation on Linux and Mac
+-----------------------------
+
+Download and extract the latest ScanCode release from:
+https://github.com/nexB/scancode-toolkit/releases/latest
+
+Open a terminal in the extracted directory and run::
+
+	./scancode --help       
+
+This will configure ScanCode and display the command line help.
+
+Installation on Windows
+-----------------------
+
+- Download the latest ScanCode release zip file from https://github.com/nexB/scancode-toolkit/releases/latest
+
+- In Windows Explorer (called File Explorer on Windows 10), select the downloaded ScanCode zip and right-click.
+
+- In the pop-up menu select 'Extract All...'
+
+- In the pop-up window 'Extract zip folders' ('Extract Compressed (Zipped) Folders' on Windows 10) use the default options to extract.
+
+- Once the extraction is complete, a new Windows Explorer/File Explorer window will pop up.
+
+- In this Explorer window, select the new folder that was created and right-click.
+
+	* On Windows 10, double-click the new folder, select one of the files inside the folder (e.g., 'setup.py'), and right-click.
+- In the pop-up menu select 'Properties'.
+
+- In the pop-up window 'Properties', select the Location value. Copy this to the clipboard and close the 'Properties' window.
+
+- Press the start menu button. (On Windows 10, click the search box or search icon in the taskbar.)
+
+- In the search box type::
+
+	cmd
+
+- Select 'cmd.exe' listed in the search results. (On Windows 10, you may see 'Command Prompt' instead -- select that.)
+
+- A new 'cmd.exe' window ('Command Prompt' on Windows 10) pops up.
+
+- In this window (aka a 'command prompt'), type the following (i.e., 'cd' followed by a space)::
+
+	cd
+
+- Right-click in this window and select Paste. This will paste the path where you extracted ScanCode.
+
+- Press Enter.
+
+- This will change the current location of your command prompt to the root directory where scancode is installed.
+
+- Then type::
+
+	scancode -h                                                                       
+
+- Press enter. This will configure your ScanCode installation.
+
+- Several messages are displayed followed by the scancode command help.
+
+- The installation is complete.
+
+Un-installation
+---------------
+
+- Delete the directory in which you extracted ScanCode.
+- Delete any temporary files created in your system temp directory under a scancode directory.

docs/source/scancode-toolkit/developement.rst

@@ -0,0 +1,106 @@
+Development
+===========
+
+See CONTRIBUTING.rst for details: https://github.com/nexB/scancode-toolkit/blob/master/CONTRIBUTING.rst
+
+Code layout and conventions
+---------------------------
+
+Source code is in ``src/`` Tests are in ``tests/``.
+
+There is one Python package for each major feature under ``src/`` and a corresponding directory with the same name under ``tests`` (but this is not a package by design).
+
+Each test script is named ``test_XXXX`` and while we love to use ``py.test`` as a test runner, most tests have no dependencies on ``py.test``, only on the ``unittest`` module (with the exception of some command line tests that depend on pytest monkeypatching capabilities.
+
+When source or tests need data files, we store these in a ``data`` subdirectory.
+
+We use PEP8 conventions with a relaxed line length that can be up to 90'ish characters long when needed to keep the code clear and readable.
+
+We store pre-built bundled native binaries in ``bin/`` sub-directories of each ``src/`` packages. These binaries are organized by OS and architecture. This ensure that ScanCode works out of the box either using a checkout or a download, without needing a compiler and toolchain to be installed. The corresponding source code for the pre-built binaries are store in a separate repository at https://github.com/nexB/scancode-thirdparty-src
+
+We store bundled thirdparty components and libraries in the ``thirdparty`` directory. Python libraries are stored as wheels, eventually pre-built if the corresponding wheel is not available in the Pypi repository. Some of these components may be advanced builds with bug fixes or advanced patches.
+
+We write tests, a lot of tests, thousands of tests. Several tests are data-driven and use data files as test input and sometimes data files as test expectation (in this case using either JSON or YAML files). The tests should pass on Linux 64 bits, Windows 32 and 64 bits and on MacOSX 10.6.8 and up. We maintain two CI loops with Travis (Linux) at https://travis-ci.org/nexB/scancode-toolkit and Appveyor (Windows) at https://ci.appveyor.com/project/nexB/scancode-toolkit
+
+When finding bugs or adding new features, we add tests. See existing test code for examples.
+
+Running tests
+-------------
+
+ScanCode comes with over 13,000 unit tests to ensure detection accuracy and stability across Linux, Windows and macOS OSes: we kinda love tests, do we?
+
+We use pytest to run the tests: call the ``py.test`` script to run the whole test suite. This is installed by ``pytest`` which is bundled with a ScanCode checkout and installed when you run ``./configure``).
+
+If you are running from a fresh git clone and you run ``./configure`` and then ``source bin/activate`` the ``py.test`` command will be available in your path.
+
+Alternatively if you have already configured but are not in an activated "virtualenv" the ``py.test`` command is available under ``<root of your checkout>/bin/py.test``
+
+(Note: paths here are for POSIX, but mostly the same applies to Windows)
+
+If you have a multiprocessor machine you might want to run the tests in parallel (and faster) For instance: ``py.test -n4`` runs the tests on 4 CPUs. We typically run the tests in verbose mode with ``py.test -vvs -n4``
+
+You can also run a subset of the test suite as shown in the CI configs https://github.com/nexB/scancode-toolkit/blob/develop/appveyor.yml#L6 e,g, ``py.test -n 2 -vvs tests/scancode`` runs only the tests scripts present in the ``tests/scancode`` directory. (You can pass a path to a specific test script file there too).
+
+See also https://docs.pytest.org for details or use the ``py.test -h`` command to show the many other options available.
+
+One useful option is to run a select subset of the test functions matching a pattern with the ``-k`` option for instance: ``py.test -vvs -k tcpdump`` would only run test functions that contain the string "tcpdump" in their name or their class name or module name .
+
+Another useful option after a test run with some failures is to re-run only the failed tests with the ``--lf`` option for instance: ``py.test -vvs --lf`` would only run only test functions that failed in the previous run.
+
+pip requirements and the configure script
+-----------------------------------------
+
+ScanCode use the ``configure`` and ``configure.bat`` (and ``etc/configure.py`` behind the scenes) scripts to install a `virtualenv <https://virtualenv.pypa.io/en/stable/>`_ , install required packaged dependencies as `pip <https://github.com/pypa/pip>`_ requirements and more configure tasks such that ScanCode can be installed in a self-contained way with no network connectivity required.
+
+Earlier unreleased versions of ScanCode where using ``buildout`` to install and configure eventually complex dependencies. We had some improvements that were merged in the upstream ``buildout`` to support bootstrapping and installing without a network connection and When we migrated to use ``pip`` and ``wheels`` as new, improved and faster way to install and configure dependencies we missed some of the features of ``buildout`` like the ``recipes``, being able to invoke arbitrary Python or shell scripts after installing packages and have scripts or requirements that are operating system-specific.
+
+ScanCode requirements and third-party Python libraries
+------------------------------------------------------
+
+In a somewhat unconventional way, all the required libraries are bundled aka. copied in the repo itself in the thirdparty/ directory. If ScanCode were only a library it would not make sense. But its is first an application and having a well defined frozen set of dependent packages is important for an app. The benefit of this approach (combined with the ``configure`` script) means that a mere checkout of the repository contains everything needed to run ScanCode except for a Python interpreter.
+
+Using ScanCode as a Python library
+----------------------------------
+
+ScanCode can be used alright as a Python library and is available as as a Python wheel in Pypi and installed with ``pip install scancode-toolkit``.
+
+Steps to cut a new release:
+---------------------------
+
+- run bumpversion with major, minor or patch to bump the version in:
+
+	- ``src/scancode/__init__.py``
+	- ``setup.py``
+	- Update the CHANGELOG.rst
+
+- commit changes and push changes to develop:
+
+	- ``git commit -m "commit message"``
+	- ``git push --set-upstream origin develop``
+
+- merge develop branch in master and tag the release.
+
+	- ``git checkout master``
+	- ``git merge develop``
+	- ``git tag -a v1.6.1 -m "Release v1.6.1"``
+	- ``git push --set-upstream origin master``
+	- ``git push --set-upstream origin v1.6.1``
+
+- draft a new release in GitHub, using the previous release blurb as a base. Highlight new and noteworthy changes from the CHANGELOG.rst.
+
+- run ``etc/release/release.sh`` locally.
+
+- upload the release archives created in the ``dist/`` directory to the GitHub release page.
+
+- save the release as a draft. Use the previous release notes to create notes in the same style. Ensure that the link to thirdparty source code is present.
+
+- test the downloads.
+
+- publish the release on GitHub
+
+- then build and publish the released wheel on Pypi. For this you need your own Pypi credentials (and get authorized to publish Pypi release: ask @pombredanne) and you need to have the ``twine`` package installed and configured.
+
+	- Build a ``.whl`` with ``python setup.py bdist_wheel``
+	- Run twine with ``twine upload dist/<path to the built wheel>``
+	- Once uploaded check the published release at https://pypi.python.org/pypi/scancode-toolkit/
+	- Then create a new fresh local virtualenv and test the wheel installation with: ``pip install scancode-toolkit``

docs/source/scancode-toolkit/documentation.rst

@@ -0,0 +1,29 @@
+Documentation
+=============
+
+This page provides an index of current ScanCode user documentation.
+
+Download
+--------
+
+Download the latest release of ScanCode from our `release page <https://github.com/nexB/scancode-toolkit/releases.>`_ 
+
+Installation
+------------
+
+See https://github.com/nexB/scancode-toolkit/blob/master/README.rst for more.
+
+User Guide
+----------
+
+The goal of ScanCode is to help you detect accurately provenance information in a codebase.
+The output of the scan is either a JSON file, an HTML app or a plain HTML file. You can visualize the HTML format in a tree view format.
+This view contains the following elements:
+
+- Code tree view - On the left side, you are able to navigate the code tree to understand what ScanCode has detected in each file.
+- Path - The directory path of the analyzed file.
+- Start/End Line - The line number where the Copyright or License has been detected.
+- What - The type of detection, either Copyright or a License.
+- Info - The name of the detected output.
+
+You can sort any column by clicking on its title. Search is also available in the top right corner for faster access to a specified resource or a type of detected license or copyright.