Regex Pattern Fails to Hit When It Shouldn't

[i-nino]

I am experiencing an issue when scanning a variety of binary files for a regex pattern that I know exists in all of them. For one of those binary files (all are PE executables), the regex fails to find the full pattern (even though it's there), but does find the pattern when I shorten it; specifically when I remove the '.' at the end of the query. The '.' in the middle of the query work as expected.

Full query looks like this:
r"\x1E\x1F\x1E\x1F\x1E\x1E**..**\x1E\x1C\x1E\x1F\x1E\x1C..\x1E"

r"\x1E\x1F\x1E\x1F\x1E\x1E**..**\x1E\x1C\x1E\x1F\x1E\x1C" <== removed the (. .)

I want to believe I am doing something wrong in the structure of my very simple regex, since if I am not, that means I cannot trust the results of any scan. The full regex hits successfully and on a couple of files that contain the full pattern, it misses. Once I remove the "any 2 characters", it hits on the one that it misses - so it can definitely find them. Somehow these special characters appear to be the issue, but only for certain files, because on other files, everything works as expected.

[BurntSushi]

Please provide a Rust program that I compile, along with input, expected output and actual output.

[i-nino]

@BurntSushi Thanks for the quick response. It is code for an internal tool that I am porting from C++, give me a sec to remove all of the noise and will provide what you asked for shortly.

[i-nino]

use regex::bytes::Regex;
use std::fs;

fn main() {
    let args: Vec<String> = std::env::args().collect();
    if args.len() != 2 {
        return;
    }

    let ref path = &args[1];
    let rslt = fs::read( path );
    match rslt {
        Ok( img_buffer ) => {
            let re = Regex::new( r"\x2E\x2F\x2E\x2F\x2E\x2C..\x2E\x2C\x2E\x2F\x2E\x2C..\x2E" ).unwrap();
            ///let re = Regex::new( r"\x2E\x2F\x2E\x2F\x2E\x2C..\x2E\x2C\x2E\x2F\x2E\x2C" ).unwrap(); // this one hits
            if re.is_match( &img_buffer ) {
                println!( "Regex hit!" );
            }
            else {
                println!( "Regex missed!" );
            }
        }
        Err( e ) => {
            println!( "Error processing input file: {}", e );
        }
    }
}

Please let me know how I can send you the two files to verify the output is not as expected for the miss.dmp.

[i-nino]

@BurntSushi https://tmpfiles.org/download/117096/regex_test.7z

Link to download the files. They will be up for the next two hours.

[BurntSushi]

The behavior looks correct to me. The reproduction is a lot simpler if you try to minimize it, which can be done by taking the thing you're trying to match in the file directly:

use regex::bytes::Regex;

fn main() {
    let haystack =
        &b"\x2E\x2F\x2E\x2F\x2E\x2C\x2E\x26\x2E\x2C\x2E\x2F\x2E\x2C\x2F\x95\x2E"[..];

    let re = Regex::new(
        r"\x2E\x2F\x2E\x2F\x2E\x2C..\x2E\x2C\x2E\x2F\x2E\x2C..\x2E"
    ).unwrap();
    assert!(re.is_match(haystack));
}

This assert fails and I would expect it to. Notice that the .. sub-expression in the regex is trying to match the bytes \x2F\x95. The first . will match \x2F no problem, since it's an ASCII byte. But \x95 is not an ASCII byte, and moreover, is not a valid UTF-8 starting byte, so it can never follow an ASCII byte in valid UTF-8 contents. When Unicode mode is enabled (which is the default), . will only match valid UTF-8 encodings of Unicode scalar values. Thus, no match is produced.

Presumably, you want . to match any arbitrary byte rather than valid UTF-8. Since bytes::Regex is permitted to match invalid UTF-8, all you need to do is disable Unicode mode. You can do that via (?-u:..) or just by putting a (?-u) at the beginning of the regex or by setting bytes::RegexBuilder::unicode to false.

This program passes the assert:

use regex::bytes::Regex;

fn main() {
    let haystack =
        &b"\x2E\x2F\x2E\x2F\x2E\x2C\x2E\x26\x2E\x2C\x2E\x2F\x2E\x2C\x2F\x95\x2E"[..];

    let re = Regex::new(
        r"(?-u)\x2E\x2F\x2E\x2F\x2E\x2C..\x2E\x2C\x2E\x2F\x2E\x2C..\x2E"
    ).unwrap();
    assert!(re.is_match(haystack));
}