Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce sequential string false positives #64

Merged
merged 2 commits into from
Jul 31, 2018
Merged

Reduce sequential string false positives #64

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
451cd46
Added 'defaults to n.n' to the --base64-limit and --hex-limit help me…
KevinHock Jul 30, 2018
b5135fc
Added IGNORED_SEQUENTIAL_STRINGS to high_entropy_strings.py and adjus…
KevinHock Jul 31, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions detect_secrets/core/usage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -282,20 +282,20 @@ def consolidate_args(args):
def _add_custom_limits(self):
high_entropy_help_text = (
'Sets the entropy limit for high entropy strings. '
'Value must be between 0.0 and 8.0.'
'Value must be between 0.0 and 8.0, '
)

self.parser.add_argument(
'--base64-limit',
type=self._argparse_minmax_type,
nargs='?',
help=high_entropy_help_text,
help=high_entropy_help_text + 'defaults to 4.5.',
)
self.parser.add_argument(
'--hex-limit',
type=self._argparse_minmax_type,
nargs='?',
help=high_entropy_help_text,
help=high_entropy_help_text + 'defaults to 3.0.',
)
return self

Expand Down
18 changes: 17 additions & 1 deletion detect_secrets/plugins/high_entropy_strings.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,18 @@
from detect_secrets.plugins.core.yaml_file_parser import YamlFileParser


IGNORED_SEQUENTIAL_STRINGS = (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use the string library for this.

e.g.

import string

IGNORED_SEQUENTIAL_STRINGS = (
    string.ascii_uppercase + string.digits + string.ascii_uppercase,
    string.hexdigits + string.hexdigits,
)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++
Changed it to

 IGNORED_SEQUENTIAL_STRINGS = (
-    'ABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789+/',  # upper/lower, numbers +/
-    '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHIJKLMNOPQRSTUVWXYZ+/',  # numbers, upper/lower +/
-    'ABCDEFABCDEF0123456789ABCDEFABCDEF',  # Hex
-    'ABCDEFGHIJKLMNOPQRSTUVWXYZ=/',
+    (
+        string.ascii_uppercase +
+        string.ascii_uppercase +
+        string.digits +
+        string.ascii_uppercase +
+        string.ascii_uppercase +
+        '+/'
+    ),
+    string.hexdigits.upper() + string.hexdigits.upper(),
+    string.ascii_uppercase + '=/',
 )

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying to use rebasing to make my commit history super clean, like someone I know. (@bcaller)

(
string.ascii_uppercase +
string.ascii_uppercase +
string.digits +
string.ascii_uppercase +
string.ascii_uppercase +
'+/'
),
string.hexdigits.upper() + string.hexdigits.upper(),
string.ascii_uppercase + '=/',
)
YAML_EXTENSIONS = (
'.yaml',
'.yml',
Expand Down Expand Up @@ -75,12 +87,16 @@ def analyze_string(self, string, line_num, filename):
"""Searches string for custom pattern, and captures all high entropy strings that
match self.regex, with a limit defined as self.entropy_limit.
"""

output = {}

if WHITELIST_REGEX.search(string):
return output

uppercased_string = string.upper()
for sequential_string in IGNORED_SEQUENTIAL_STRINGS:
if uppercased_string in sequential_string:
return output

for result in self.secret_generator(string):
secret = PotentialSecret(self.secret_type, filename, line_num, result)
output[secret] = secret
Expand Down
1 change: 1 addition & 0 deletions test_data/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ credentials:
some_value_here: not_a_secret
other_value_here: 1234567890a
nested:
value: AKIAabcdefghijklmnop
value: abcdefghijklmnop
list_of_keys:
- 123
Expand Down
2 changes: 1 addition & 1 deletion test_data/short_files/last_line.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
secrets_for_no_one_to_find =
hunter2
password123
0123456789a
BEEF0123456789a
2 changes: 1 addition & 1 deletion tests/main_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ def test_old_baseline_ignored_with_update_flag(
2:secrets_for_no_one_to_find =
3: hunter2
4: password123
5: 0123456789a
5: BEEF0123456789a
""")[1:-1],
),
],
Expand Down
3 changes: 1 addition & 2 deletions tests/plugins/high_entropy_strings_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,12 +148,11 @@ def test_yaml_file(self):
with open('test_data/config.yaml') as f:
secrets = plugin.analyze(f, 'test_data/config.yaml')

assert len(secrets.values()) == 2
assert len(secrets.values()) == 1
for secret in secrets.values():
location = str(secret).splitlines()[1]
assert location in (
'Location: test_data/config.yaml:3',
'Location: test_data/config.yaml:5',
)

def test_entropy_lower_limit(self):
Expand Down