Regex does not match on unicode input

[smbecker]

Description

I have a sample here that demonstrates a valid regex pattern that successfully matches against a string but fails to match against the same input when it is converted to unicode. This does not seem like correct behavior as I would expect it to match regardless.

Configuration

• Net 5.0
• Ubuntu 20.10

Tagging subscribers to this area: @eerhardt, @pgovind
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

I have a sample here that demonstrates a valid regex pattern that successfully matches against a string but fails to match against the same input when it is converted to unicode. This does not seem like correct behavior as I would expect it to match regardless.

Configuration

• Net 5.0
• Ubuntu 20.10

Author:	smbecker
Assignees:	-
Labels:	area-System.Text.RegularExpressions, untriaged
Milestone:	-

[stephentoub]

In your sample, this:

var unicodeContent = new StreamReader(new MemoryStream(Encoding.Unicode.GetBytes(content))).ReadToEnd();

produces the string:

"E\0m\0a\0i\0l\0s\0 \0a\0d\0d\0r\0e\0s\0s\0e\0s\0\r\0\n\0*\0 \0t\0e\0s\0t\01\0@\0d\0o\0m\0a\0i\0n\0.\0c\0o\0m\0\r\0\n\0*\0 \0t\0e\0s\0t\02\0@\0d\0o\0m\0a\0i\0n\0.\0c\0o\0m\0"

That legitimately doesn't match the supplied regex pattern:

"\b[\w\.=-]+@[\w\.-]+\.[\w]{2,3}\b"

For example, when it gets to the first address, 't' will match \w, but then '\0' is not a word char, '.', '=', '-', or '@', and thus will fail to match.

[pgovind]

Was just going to say the same thing @stephentoub did. (Tested locally on my machine)

[smbecker]

I knew that I must be missing something obvious but for the life of me, couldn't see it since the debugger wasn't showing the \0s. In this simple case, the fix was to just pass in Encoding.Unicode to the StreamReader. Sorry guys, and thanks for jumping on this so quickly.