Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: support unicode in markdown #128

Merged
merged 1 commit into from
Feb 12, 2019

Conversation

killuazhu
Copy link
Contributor

What's in the PR

  • Fixes an UnicodeEncodeError when scanning non-ini file (such as markdown) containing unicode.
  • Add one more ini test case
  • Parametrize ini test cases

Offending File content

# Sample markdown file

[guides](http://localhost/guilds)

Test Unicode in non ini file would not fail on python 2.7.

╭─ diagnose
╰» ssh to server x:22324241234423414

key="ToCynx5Se4e2PtoZxEhW7lUJcOX15c54"

Current error message (fixed after using the PR)

Traceback (most recent call last):
  File "detect_secrets/main.py", line 200, in <module>
    sys.exit(main())
  File "detect_secrets/main.py", line 47, in main
    baseline_dict = _perform_scan(args, plugins)
  File "detect_secrets/main.py", line 146, in _perform_scan
    scan_all_files=args.all_files,
  File "/Users/killua/workspace/ww/whitewater-detect-secrets/detect_secrets/core/baseline.py", line 58, in initialize
    output.scan_file(file)
  File "/Users/killua/workspace/ww/whitewater-detect-secrets/detect_secrets/core/secrets_collection.py", line 206, in scan_file
    self._extract_secrets_from_file(f, filename_key)
  File "/Users/killua/workspace/ww/whitewater-detect-secrets/detect_secrets/core/secrets_collection.py", line 306, in _extract_secrets_from_file
    results.update(plugin.analyze(f, filename))
  File "/Users/killua/workspace/ww/whitewater-detect-secrets/detect_secrets/plugins/high_entropy_strings.py", line 68, in analyze
    output = analyze_function(file, filename)
  File "/Users/killua/workspace/ww/whitewater-detect-secrets/detect_secrets/plugins/high_entropy_strings.py", line 184, in wrapped
    exclude_lines_regex=self.exclude_lines_regex,
  File "/Users/killua/workspace/ww/whitewater-detect-secrets/detect_secrets/plugins/common/ini_file_parser.py", line 25, in __init__
    self.parser.read_file(file)
  File "/Users/killua/.pyenv/versions/2.7.15/lib/python2.7/site-packages/backports/configparser/__init__.py", line 715, in read_file
    self._read(f, source)
  File "/Users/killua/.pyenv/versions/2.7.15/lib/python2.7/site-packages/backports/configparser/__init__.py", line 1100, in _read
    optname = self.optionxform(optname.rstrip())
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-1: ordinal not in range(128)

Copy link
Collaborator

@KevinHock KevinHock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks great to me 🚢

@killuazhu
Copy link
Contributor Author

Hey @KevinHock are we going to create a new release anytime soon?

@KevinHock
Copy link
Collaborator

Hi @killuazhu, @domanchi was going to test the basic auth changes internally a little bit before we bumped to 0.12.0. But as soon as he does we will make a new release 👍 :)

@killuazhu
Copy link
Contributor Author

@KevinHock Once this PR is merged, I have another one to address Unicode issue during the audit process. It would be great if both of these PRs can be merged before we cut a new release.

@KevinHock KevinHock merged commit cbb130d into Yelp:master Feb 12, 2019
KevinHock added a commit that referenced this pull request Feb 12, 2019
@KevinHock
Copy link
Collaborator

Sounds good to me @killuazhu, looking forward to it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants