Python: Can't match unicode snowman literals in strings

[craigds]

Describe the bug

I'm trying to match a string, in python, which contains a literal snowman character (☃). I can't find a way to do so.

To Reproduce

https://semgrep.dev/s/craigds:unicode-snowman

Expected behavior
I expected to just be able to use the string itself

semgrep --lang=python --pattern '"Test ☃"'

It didn't match. Neither did any of the other things I tried, e.g.:

• "Test \x{FE0F}"
• "Test \u2603"

What is the priority of the bug to you?

• P0: blocking your adoption of Semgrep or workflow
• P1: important to fix or quite annoying
• P2: regular bug that should get fixed

Environment

• semgrep 0.70.0 via homebrew on MacOS 11.6
• Doesn't work on semgrep.dev either though, see link above

[aryx]

might be a unicode issue.

[aryx]

@mjambon can you have a look at it? You're the unicode expert ...

[mjambon]

@pad here's what I got:

$ semgrep-core -lang python -e '"snowman ☃"' <(echo '"snowman ☃"') -fast
/tmp/semgrep-core-cf24d0-63:1
 "snowman ☃"
$ semgrep-core -lang python -co <(echo '"snowman ☃"')
/tmp/semgrep-core-da6d22-63:1 with rule -
 "snowman ☃"
$ semgrep-core -lang python -config snowman.yml <(echo '"snowman ☃"') -fast

with the following rule file snowman.yml:

rules:
- id: '-'
  pattern: '"snowman ☃"'
  message: '"snowman ☃"'
  languages:
  - python
  severity: ERROR

The problem is when using a rule file, -fast, and some non-ascii input.

[mjambon]

For background, see #2111. The hack that was implemented to work around bad locations replaces each non-ascii byte by a Z, resulting in false positives like the following:

$ semgrep-core -lang python -e '"😀"' <(echo '"🚀"')
/tmp/semgrep-core-124947-63:1
 "🚀"

This match happens because both 😀 and 🚀 and parsed as ZZZZ due to our unicode hack. However it breaks our -fast ("filter irrelevant rules") optimization which compares the parsed pattern (containing ZZZZ or ZZZ) against the raw source file still containing the original utf8 🚀 or ☃. It sees that the string ZZZZ doesn't exist in the source file and decides to skip it.

[mjambon]

A proper fix would involve eliminating the hack that replaces non-ascii bytes by Zs, and then should do whatever is necessary to report proper locations. This would be a bit of work and needs to be done right.

Alternatively, we could extend the Unicode hack to work with the -fast optimization. This could make the optimization slower, in addition to making the code more complicated.

[emjin]

imo we should hack a solution for -fast. Can we just exclude Z* from string filtering?

[mjambon]

@emjin great idea. I was worried about the cost of editing each target but editing the pattern should be fine.

[mjambon]

@craigds we just merged a "fix" for this, which is a hack on top of another hack. It will be in the next semgrep release. At some point, we'll need to handle Unicode correctly. Meanwhile, expect some false positives. See #4415.