Identify strings made of repeated characters as having low entropy

[mjambon]

This should work but I want to add tests to make sure it does. The issue is that the semgrep matcher returns quoted strings e.g. "xxxxxxxxxxxx" and the entropy analysis is supposed to be smart and eliminate the quotes. This is necessary for the string to be recognized as a repeated character.

Edit: added tests and fixed implementation accordingly.

PR checklist:

• Tests
• Documentation is up-to-date
• Changelog is up-to-date
• Change has no security implications (otherwise, ping security team)

[mjambon]

@kurt-r2c if you find a link to the original issue about entropy analysis, we should add it to CHANGES.md, otherwise I think it's no big deal. This is what I wrote:

- Entropy analysis: strings made of repeated characters such as
  `'xxxxxxxxxxxxxx'` are no longer reported has having high entropy (#4833)

[mjambon]

mmm, a test fails with generic mode when binding a metavariable to something that looks like an int, which ends up being represented as an Int kind of value:

(Metavariable.E
   { e = (L (Int ((Some 123456789012), ()))); e_id = 0; e_range = None })

Trying to figure out how this works.

[mjambon]

The new behavior is to unquote the captured string value if possible. If it's not a string, use the original source code fragment.

[github-actions]

🔥 Potential speedup in benchmark semgrep.bench.coinbase.std: -23.6% (-3.137 s)

14 benchmarks, 3.8% faster on average.

Individual deviations greater than 20% from the baseline are reported. An individual performance degradation of over 30% or a global degradation of over 7% is an error and will block the pull request. See run output for full results ('Show all checks' > 'Tests / semgrep benchmark tests' 'Details').

[mjambon]

Getting what looks like a network error (twice in a row). The URL being reported (https://semgrep.dev/c/auto) works for me from home. I'm going to re-run the job again.

https://github.com/returntocorp/semgrep/runs/5610188578?check_suite_focus=true

--- stdout from semgrep process ---
{"errors": [{"code": 2, "level": "error", "message": "Failed to download config from https://semgrep.dev/c/auto: ('Connection aborted.', RemoteDisconnected('Remote end closed\nconnection without response'))", "type": "SemgrepError"}, {"code": 7, "level": "error", "message": "invalid configuration file found (1 configs were invalid)", "type": "SemgrepError"}], "paths": {"_comment": "<add --verbose for a list of skipped paths>", "scanned": []}, "results": []}

--- end semgrep stdout ---

[kurt-r2c]

@mjambon we were tracking this internally via Linear: RULES-446. There's no associated GitHub issue. The statement as-is should be fine.