Enhance and create detailed documentation for scancode-config.yml

[pombredanne]

We need to have detailed documentation for scancode-config.yml fields and syntax with concrete examples such we have a well-rounded and guided experience for users.

We have no proper documentation for these fields:

• "Ignored patterns": today: Provide one or more path patterns to be ignored, one per line.
• "License score": today: Do not return license matches with a score lower than this score. A number between 0 and 100.
• "Attribution template": Custom attribution template.

We need to also track some important data attributes to guide scanning and integration with other AboutCode tools:

• product name: to use for integration with DejaCode
• product version: to use when linking back and integrating with DejaCode
• ignored_dependency_scopes. see Add support for ignored_dependency_scopes field for configuration #1235

As a nice to have we should also:

• provide a downloadable YAML template/sample scancode-config.yml file in the documentation

future (for reference only -- needs further evaluation):
specific list of package type/ecosystems in use including paths to the product package manifests and lockfiles as an alternative to nexB/dependency-inspector#2 (comment)

[pombredanne]

We may want to also exclude certain package dependency scopes

[DennisClark]

We may also want to include one or more fields to identify where SCIO should send notifications about the status of a project, including link(s) to the result files. working title of the new field could be notification_address .

[tdruez]

1. The help for existing settings fields is being enhanced at Enhance help text documentation for Project settings form #1197  #1229

2. The product name and version fields were added in the settings.

3. Now about:

specific list of package type/ecosystems in use including paths to the product package manifests and lockfiles as an alternative to "Create a Lockfile checker and generator dependency-inspector#2 (comment)"

@pombredanne Could you enter a detailed issue about the implementation you have in mind, including concrete data examples? I have no clues about what is required here.

[pombredanne]

Some notes:

• for now, the product version is static, and the config file will be stored in the git repo (or provided additionally). Updating the product version will implying editing the config file.

[pombredanne]

For dependency scopes, I would like to be able to flag as ignored certain a dependency scopes of a given package type. For instance the config file could contain this:

ignored_dependency_scopes:
   - package_type: npm
     scope: devDependencies
   - package_type: pypi
     scope: tests

These are actual scope names. There are no wildcards. Only exact matched strings

The question is then to determine what we do with this:

1. We collect (and in the future resolve) all dependencies even if for ignored scope, and we pass along these ignores in the scan results for filtering in DejaCode
2. We do not collect (and in the future do not resolve) ignored dependencies from the type and scope (meaning no record is created in SCIO)
3. We collect (and in the future resolve) all dependencies, but we filter out ignored scoped from reporting

2.) is solution that looks the cleanest and most obvious approach and is consistent with paths, also the fastest.

[DennisClark]

see related issue:

• Support storing scancode-config.yml files in the scanned codebase #1236

[tdruez]

@pombredanne provide a downloadable YAML template/sample scancode-config.yml file in the documentation.

Which style/content should we adopt for such template? Here's a few ideas:

• No content:

product_name:
product_version:
ignored_patterns:
ignored_dependency_scopes:

• No content with docs (no examples):

# The product name of this project, as specified within the DejaCode application.
product_name:

# The product version of this project, as specified within the DejaCode application.
product_version:

# Provide one or more path patterns to be ignored.
# Each pattern should follow the syntax of Unix shell-style wildcards:
# - Use ``*`` to match multiple characters.
# - Use ``?`` to match a single character.
ignored_patterns:


# Specify certain dependency scopes to be ignored for a given package type.
ignored_dependency_scopes:

• Commented with concrete examples:

#product_name: My Product Name
#product_version: '1.0'

#ignored_patterns:
#  - '*.tmp'
#  - 'tests/*'

#ignored_dependency_scopes:
# - package_type: npm
#   scope: devDependencies
# - package_type: pypi
#   scope: tests

• Commented with concrete examples and inline docs:

# The product name of this project, as specified within the DejaCode application.
#product_name: My Product Name

# The product version of this project, as specified within the DejaCode application.
#product_version: '1.0'

# Provide one or more path patterns to be ignored.
# Each pattern should follow the syntax of Unix shell-style wildcards:
# - Use ``*`` to match multiple characters.
# - Use ``?`` to match a single character.
#ignored_patterns:
#  - '*.tmp'
#  - 'tests/*'

# Specify certain dependency scopes to be ignored for a given package type.
#ignored_dependency_scopes:
# - package_type: npm
#   scope: devDependencies
# - package_type: pypi
#   scope: tests

[pombredanne]

Commented with concrete examples and inline docs:

this is the best IMHO