issues getting pcre to use jit

[alubbe]

I've been trying to get pcre to use jit, but to no avail. My goal is to reproduce these benchmarks: http://sljit.sourceforge.net/regex_perf.html
But no matter what regex I try (Tom|Sawyer|Huckleberry|Finn, [a-z]shing, ...), it doesn't JIt:

2015/10/03 21:55:57 [debug] 58278#0: *1 lua compiling gsub regex "Tom|Sawyer|Huckleberry|Finn" with options "oj" (compile once: 1) (dfa mode: 0) (jit mode: 1)
2015/10/03 21:55:57 [debug] 58278#0: *1 pcre JIT compiling result: 0
...
2015/10/03 22:53:21 [debug] 58278#0: *1 lua compiling gsub regex "[a-z]shing" with options "oj" (compile once: 1) (dfa mode: 0) (jit mode: 1)
2015/10/03 22:53:21 [debug] 58278#0: *1 pcre JIT compiling result: 0

I used the --with-pcre-jit flag and it seems to be installed:

checking for PCRE library ... found
checking for PCRE JIT support ... found
checking for OpenSSL library ... found
checking for zlib library ... found
creating objs/Makefile

Configuration summary
  + using system PCRE library

Am I doing something wrong? These regular expressions should be JITable as shown by the benchmark, right?

[agentzh]

@alubbe Yes, those regexes should be JIT compiled. My hunch is that maybe your system PCRE library is broken or something.

You can try downloading the latest source code tarball of PCRE from its official site and build it into nginx (or openresty) with the ./configure options --with-pcre=PATH and --with-pcre-jit? Below are the steps verified on my side:

mkdir -p ~/work
cd ~/work
wget ftp://ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-8.37.tar.gz
tar -xvf pcre-8.37.tar.gz
cd /path/to/ngx_openresty-1.9.3.1/
./configure --with-debug --with-pcre=$HOME/work/pcre-8.37 --with-pcre-jit --prefix=/opt/or -j9
make -j9
sudo make install
cd /opt/or/nginx
sudo vim conf/nginx.conf
# add the location = /t using content_by_lua with ngx.re.gsub() and etc.
sudo ./sbin/nginx
curl localhost/t
grep -iE 'regex|pcre' logs/error.log

The output of the last command on my side is

2015/10/05 10:54:31 [debug] 97783#0: *3 lua regex cache miss for gsub regex "[a-z]shing" with options "jo" and replace ""
2015/10/05 10:54:31 [debug] 97783#0: *3 lua compiling gsub regex "[a-z]shing" with options "jo" (compile once: 1) (dfa mode: 0) (jit mode: 1)
2015/10/05 10:54:31 [debug] 97783#0: *3 pcre JIT compiling result: 1
2015/10/05 10:54:31 [debug] 97783#0: *3 lua saving compiled sub regex (0 captures) into the cache (entries 0)

Please note the pcre JIT compiling result: 1 line above.

Good luck!

[alubbe]

Thanks, that solved it!

[alubbe]

By the way, are there any plans to support pcre2?

[agentzh]

@alubbe I'd rather it be a separate Lua library, like lua-resty-pcre2 :)

[agentzh]

@alubbe FWIW, for quite some time I've been working on my own regex engine, sregex, which supports streaming processing (i.e., strictly nonbuffered processing) as well as multi-regex parallel matching:

https://github.com/openresty/sregex

And the benchmark for the quick prototype of its new DFA engine looks promising as compared to PCRE2 JIT and RE2:

http://openresty.org/misc/re/bench/

Just thinking that you might be interested :)

[alubbe]

Definitely interested :)
What's your timeline on the new DFA engine?
In the meantime, your benchmarks also show that PCRE2 is a lot faster than PCRE, so is there any way I can motivate you to update the adapter? 😄

[agentzh]

Hello!

On Tue, Oct 6, 2015 at 6:38 PM, Andreas Lubbe notifications@github.com wrote:

What's your timeline on the new DFA engine?

The sregex DFA engine is still under early (active) development. No
timeline for its first production release yet, nor is its Lua binding
library.

In the meantime, your benchmarks also show that PCRE2 is a lot faster than PCRE, so is there any way I can motivate you to update the adapter?

ngx_lua's ngx.re API will not migrate to PCRE2 unless the nginx core
does that as well.

As I mentioned, if you really want to use PCRE2 NOW, then you'd better
rely on a separate Lua binding library for PCRE2, say,
lua-resty-pcre2. Well, you can consider getting your employer to
sponsor the work around lua-resty-pcre2 (if you really want to, please
contact me privately).

Regards,
-agentzh