Skip to content

Commit

Permalink
🎉 Add --custom-plugins feature
Browse files Browse the repository at this point in the history 
- 🐍 Add add_shared_arguments() function in usage.py for DRYness
- 🐛 Fix issue #242 via passing `should_verify_secrets=not args.no_verify` to `from_parser_builder` call
- 🐛 Fix sorting issue in format_baseline_for_output() where --update and regular scan had different secret order
- 💯 All non-separated out files again :D
- 🎓 Mention `--custom-plugins` in README
- 🎓 Standardize NOTE -> Note
- 🐛 Fix test pollution due to `all_plugins` cls attribute
- 🐍 Change all relative imports to absolute, to avoid broken imports if someone copies an existing plugin to make a custom plugin

🙈 Hacks located in `def parse_args` of usage.py
  • Loading branch information
@KevinHock
KevinHock committed Apr 19, 2020
1 parent 332d9ef commit 2d4b90e
Show file tree
Hide file tree
Showing 42 changed files with 738 additions and 279 deletions.
14 changes: 14 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ For a look at recent changes, please see [CHANGELOG.md](https://github.com/Yelp/

If you are looking to contribute, please see [CONTRIBUTING.md](https://github.com/Yelp/detect-secrets/blob/master/CONTRIBUTING.md).


## Example Usage

### Setting Up a Baseline
Expand All @@ -44,6 +45,7 @@ If you are looking to contribute, please see [CONTRIBUTING.md](https://github.co
$ detect-secrets scan > .secrets.baseline
```


### pre-commit Hook

```
Expand All @@ -56,12 +58,14 @@ $ cat .pre-commit-config.yaml
exclude: .*/tests/.*
```


### Auditing a Baseline

```
$ detect-secrets audit .secrets.baseline
```


### Upgrading Baselines

This is only applicable for upgrading baselines that have been created after version 0.9.
Expand All @@ -71,6 +75,7 @@ For upgrading baselines lower than that version, just recreate it.
$ detect-secrets scan --update .secrets.baseline
```


### Command Line

`detect-secrets` is designed to be used as a git pre-commit hook, but you can also invoke `detect-secrets scan [path]` directly being `path` the file(s) and/or directory(ies) to scan (`path` defaults to `.` if not specified).
Expand All @@ -93,6 +98,7 @@ either the client-side pre-commit hook, or the server-side secret scanner.
3. **Secrets Baseline**, to allowlist pre-existing secrets in the repository,
so that they won't be continuously caught through scan iterations.


### Client-side `pre-commit` Hook

See [pre-commit](https://github.com/pre-commit/pre-commit) for instructions
Expand All @@ -104,11 +110,13 @@ Hooks available:
- `detect-secrets`: This hook detects and prevents high entropy strings from
entering the codebase.


### Server-side Secret Scanning

Please see the [detect-secrets-server](https://github.com/Yelp/detect-secrets-server)
repository for installation instructions.


### Secrets Baseline

```
Expand Down Expand Up @@ -144,6 +152,7 @@ This may be a convenient way for you to allowlist secrets, without having to
regenerate the entire baseline again. Furthermore, this makes the allowlisted
secrets easily searchable, auditable, and maintainable.


## Currently Supported Plugins

The current heuristic searches we implement out of the box include:
Expand Down Expand Up @@ -175,18 +184,23 @@ See [detect_secrets/
plugins](https://github.com/Yelp/detect-secrets/tree/master/detect_secrets/plugins)
for more details.

There is also a `--custom-plugins` option in which you can bring your own plugins, e.g. `detect-secrets scan --custom-plugins testing/custom_plugins_dir/ --custom-plugins testing/hippo_plugin.py`.


## Caveats

This is not meant to be a sure-fire solution to prevent secrets from entering
the codebase. Only proper developer education can truly do that. This pre-commit
hook merely implements several heuristics to try and prevent obvious cases of
committing secrets.


### Things that won't be prevented

* Multi-line secrets
* Default passwords that don't trigger the `KeywordDetector` (e.g. `login = "hunter2"`)


### Plugin Configuration

One method that this package uses to find secrets is by searching for high
Expand Down
146 changes: 93 additions & 53 deletions detect_secrets/core/audit.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,16 @@
from copy import deepcopy
from functools import lru_cache

from ..plugins.common import initialize
from ..plugins.common.util import get_mapping_from_secret_type_to_class_name
from ..util import get_git_remotes
from ..util import get_git_sha
from .baseline import merge_results
from .bidirectional_iterator import BidirectionalIterator
from .code_snippet import CodeSnippetHighlighter
from .color import AnsiColor
from .color import colorize
from .common import write_baseline_to_file
from detect_secrets.core.baseline import merge_results
from detect_secrets.core.bidirectional_iterator import BidirectionalIterator
from detect_secrets.core.code_snippet import CodeSnippetHighlighter
from detect_secrets.core.color import AnsiColor
from detect_secrets.core.color import colorize
from detect_secrets.core.common import write_baseline_to_file
from detect_secrets.plugins.common import initialize
from detect_secrets.plugins.common.util import get_mapping_from_secret_type_to_class_name
from detect_secrets.util import get_git_remotes
from detect_secrets.util import get_git_sha


class SecretNotFoundOnSpecifiedLineError(Exception):
Expand Down Expand Up @@ -48,7 +48,7 @@ class RedundantComparisonError(Exception):
'config': {},
}
EMPTY_STATS_RESULT = {
'signal': 0,
'signal': '0.00%',
'true-positives': {
'count': 0,
'files': defaultdict(int),
Expand Down Expand Up @@ -87,11 +87,12 @@ def audit_baseline(baseline_filename):

try:
_print_context(
filename,
secret,
current_secret_index,
total_choices,
original_baseline['plugins_used'],
filename=filename,
secret=secret,
count=current_secret_index,
total=total_choices,
plugins_used=original_baseline['plugins_used'],
custom_plugin_paths=original_baseline['custom_plugin_paths'],
)
decision = _get_user_decision(can_step_back=secret_iterator.can_step_back())
except SecretNotFoundOnSpecifiedLineError:
Expand Down Expand Up @@ -144,7 +145,7 @@ def compare_baselines(old_baseline_filename, new_baseline_filename):
This means that we won't have cases where secrets are moved around;
only added or removed.
NOTE: We don't want to do a version check, because we want to be able to
Note: We don't want to do a version check, because we want to be able to
use this functionality across versions (to see how the new version fares
compared to the old one).
"""
Expand All @@ -170,6 +171,7 @@ def compare_baselines(old_baseline_filename, new_baseline_filename):
header = '{} {}'
if is_removed:
plugins_used = old_baseline['plugins_used']
custom_plugin_paths = old_baseline['custom_plugin_paths']
header = header.format(
colorize('Status:', AnsiColor.BOLD),
'>> {} <<'.format(
Expand All @@ -178,6 +180,7 @@ def compare_baselines(old_baseline_filename, new_baseline_filename):
)
else:
plugins_used = new_baseline['plugins_used']
custom_plugin_paths = new_baseline['custom_plugin_paths']
header = header.format(
colorize('Status:', AnsiColor.BOLD),
'>> {} <<'.format(
Expand All @@ -187,13 +190,14 @@ def compare_baselines(old_baseline_filename, new_baseline_filename):

try:
_print_context(
filename,
secret,
current_index,
total_reviews,
plugins_used,
filename=filename,
secret=secret,
count=current_index,
total=total_reviews,
plugins_used=plugins_used,
custom_plugin_paths=custom_plugin_paths,
additional_header_lines=header,
force=is_removed,
force_line_printing=is_removed,
)
decision = _get_user_decision(
can_step_back=secret_iterator.can_step_back(),
Expand Down Expand Up @@ -258,7 +262,9 @@ def determine_audit_results(baseline, baseline_path):
'stats': deepcopy(EMPTY_STATS_RESULT),
}

secret_type_to_plugin_name = get_mapping_from_secret_type_to_class_name()
secret_type_to_plugin_name = get_mapping_from_secret_type_to_class_name(
custom_plugin_paths=baseline['custom_plugin_paths'],
)

total = 0
for filename, secret in all_secrets:
Expand All @@ -269,7 +275,8 @@ def determine_audit_results(baseline, baseline_path):
try:
secret_info['plaintext'] = get_raw_secret_value(
secret=secret,
plugin_settings=baseline['plugins_used'],
plugins_used=baseline['plugins_used'],
custom_plugin_paths=baseline['custom_plugin_paths'],
file_handle=io.StringIO(file_contents),
filename=filename,
)
Expand All @@ -283,13 +290,13 @@ def determine_audit_results(baseline, baseline_path):
audit_results['stats'][audit_result]['count'] += 1
audit_results['stats'][audit_result]['files'][filename] += 1
total += 1
audit_results['stats']['signal'] = str(
audit_results['stats']['signal'] = '{:.2f}'.format(
(
float(audit_results['stats']['true-positives']['count'])
/
total
) * 100,
)[:4] + '%'
) + '%'

for plugin_config in baseline['plugins_used']:
plugin_name = plugin_config['name']
Expand Down Expand Up @@ -473,9 +480,10 @@ def _print_context( # pragma: no cover
secret,
count,
total,
plugin_settings,
plugins_used,
custom_plugin_paths,
additional_header_lines=None,
force=False,
force_line_printing=False,
):
"""
:type filename: str
Expand All @@ -490,15 +498,24 @@ def _print_context( # pragma: no cover
:type total: int
:param total: total number of secrets in baseline
:type plugin_settings: list
:param plugin_settings: plugins used to create baseline.
:type plugins_used: list
:param plugins_used: output of "plugins_used" in baseline. e.g.
>>> [
... {
... 'name': 'Base64HighEntropyString',
... 'base64_limit': 4.5,
... },
... ]
:type custom_plugin_paths: List[str]
:param custom_plugin_paths: possibly empty list of paths that have custom plugins.
:type additional_header_lines: str
:param additional_header_lines: any additional lines to add to the
header of the interactive audit display.
:type force: bool
:param force: if True, will print the lines of code even if it doesn't
:type force_line_printing: bool
:param force_line_printing: if True, will print the lines of code even if it doesn't
find the secret expected
:raises: SecretNotFoundOnSpecifiedLineError
Expand All @@ -523,10 +540,11 @@ def _print_context( # pragma: no cover
error_obj = None
try:
secret_with_context = _get_secret_with_context(
filename,
secret,
plugin_settings,
force=force,
filename=filename,
secret=secret,
plugins_used=plugins_used,
custom_plugin_paths=custom_plugin_paths,
force_line_printing=force_line_printing,
)
print(secret_with_context)
except SecretNotFoundOnSpecifiedLineError as e:
Expand Down Expand Up @@ -608,9 +626,10 @@ def _get_file_line(filename, line_number):
def _get_secret_with_context(
filename,
secret,
plugin_settings,
plugins_used,
custom_plugin_paths,
lines_of_context=5,
force=False,
force_line_printing=False,
):
"""
Displays the secret, with surrounding lines of code for better context.
Expand All @@ -621,15 +640,24 @@ def _get_secret_with_context(
:type secret: dict, PotentialSecret.json() format
:param secret: the secret listed in baseline
:type plugin_settings: list
:param plugin_settings: plugins used to create baseline.
:type plugins_used: list
:param plugins_used: output of "plugins_used" in baseline. e.g.
>>> [
... {
... 'name': 'Base64HighEntropyString',
... 'base64_limit': 4.5,
... },
... ]
:type custom_plugin_paths: List[str]
:param custom_plugin_paths: possibly empty list of paths that have custom plugins.
:type lines_of_context: int
:param lines_of_context: number of lines displayed before and after
secret.
:type force: bool
:param force: if True, will print the lines of code even if it doesn't
:type force_line_printing: bool
:param force_line_printing: if True, will print the lines of code even if it doesn't
find the secret expected
:raises: SecretNotFoundOnSpecifiedLineError
Expand All @@ -649,18 +677,19 @@ def _get_secret_with_context(
)

raw_secret_value = get_raw_secret_value(
secret,
plugin_settings,
io.StringIO(file_content),
filename,
secret=secret,
plugins_used=plugins_used,
custom_plugin_paths=custom_plugin_paths,
file_handle=io.StringIO(file_content),
filename=filename,
)

try:
snippet.highlight_line(raw_secret_value)
except ValueError:
raise SecretNotFoundOnSpecifiedLineError(secret['line_number'])
except SecretNotFoundOnSpecifiedLineError:
if not force:
if not force_line_printing:
raise

snippet.target_line = colorize(
Expand All @@ -673,16 +702,26 @@ def _get_secret_with_context(

def get_raw_secret_value(
secret,
plugin_settings,
plugins_used,
custom_plugin_paths,
file_handle,
filename,
):
"""
:type secret: dict
:param secret: see caller's docstring
:type plugin_settings: list
:param plugin_settings: see caller's docstring
:type plugins_used: list
:param plugins_used: output of "plugins_used" in baseline. e.g.
>>> [
... {
... 'name': 'Base64HighEntropyString',
... 'base64_limit': 4.5,
... },
... ]
:type custom_plugin_paths: List[str]
:param custom_plugin_paths: possibly empty list of paths that have custom plugins.
:type file_handle: file object
:param file_handle: Open handle to file where the secret is
Expand All @@ -692,8 +731,9 @@ def get_raw_secret_value(
as a means of comparing whether two secrets are equal.
"""
plugin = initialize.from_secret_type(
secret['type'],
plugin_settings,
secret_type=secret['type'],
plugins_used=plugins_used,
custom_plugin_paths=custom_plugin_paths,
)

plugin_secrets = plugin.analyze(file_handle, filename)
Expand Down
Loading

0 comments on commit 2d4b90e

Please sign in to comment.