Tcl core 8.5 with PCRE (and DFA) regular expressions engine

[sebres]

Artificial PR for core 8.5 with PCRE (and DFA) regular expressions engine.

I've rebased original not completed patch to newer 8.5 (many conflicts resolved) relative my branch sebres-8-5-timerate (in order to test performance also).

Compared to original variant provided von Jeffrey with patch pcre-20080121, it is complete:

• many bugs fixed, regsub working as expected now, more robust and faster;
• has better backwards compatibility to the classic NFA-regexp of tcl;
• e. g. UCP (Unicode properties for \d, \w, etc.) and many others.

Additionally:

• better UTF8 and UCP support;
• types are compiled now;
• I've fixed all known bugs (of the classic regexp) for this new engine;
• added DFA type (still draft, but it works well), thus for example
  `regexp -type dfa -inline {a|ab|abc} -abc-` returns dfa-alternatives `{abc ab a}` ;)
• new test-cases added in order to explain differences (advantages and disadvantages) of all 3 variants.
• I've also a build for windows resp. auto-scripts and makefile for windows, thus if somebody needs, just ask :)

Todo's:

• I should review and rewrite a few of code pieces for better understanding and to avoid code duplication;
• regexp -binary for real binary capability through PCRE;
• regexp -dict for capturing named groups of PCRE;
• provide real DFA-workspace (ATM fixed in stack) with reallocate on demand if to small;
• docu.

As regards the performance, the PCRE as well as DFA are very faster as classic NFA of tcl (up to 10 times and on large regexp still faster).
Here an excerpt as a foretaste:

% timerate -calibrate {}

% foreach t {c p d} { 
    proc test_$t {} \
    [string map [list _REENG_ $t] \
      {puts _REENG_:[timerate {regsub -type _REENG_ -all -line {^((\d{2})-(\d{2})-(\d{2,4})|NULL)$} "10-10-2017\nNULL\n20-10-2017" {**\1**\2**}}]} 
]; puts "% [info body test_$t]"; test_$t }

% puts c:[timerate {regsub -type c -all -line {^((\d{2})-(\d{2})-(\d{2,4})|NULL)$} "10-10-2017\nNULL\n20-10-2017" {**\1**\2**}}]
-c:32.4819 µs/# 30763 # 30786.4 #/sec 999.241 net-ms
% puts p:[timerate {regsub -type p -all -line {^((\d{2})-(\d{2})-(\d{2,4})|NULL)$} "10-10-2017\nNULL\n20-10-2017" {**\1**\2**}}]
+p:1.266349 µs/# 774477 # 789671 #/sec 980.758 net-ms
% puts d:[timerate {regsub -type d -all -line {^((\d{2})-(\d{2})-(\d{2,4})|NULL)$} "10-10-2017\nNULL\n20-10-2017" {**\1**\2**}}]
+d:1.204760 µs/# 813269 # 830040 #/sec 979.794 net-ms

Note, the DFA has not realy a back-references here (but match alternatives), just the regsub used in order to minimize overhead of some tcl internal by measure (round about setting of variables or creating the lists by -indices or -inline).

Tested with PCRE 8.40 up to 8.45.

If interested by TCT I'll rebase it to fossil as soon as possible and provide my 8.6th and 8.7th branches for this.
I'll just spare this work (rebase) if nobody needs it.

Ah, yes, don't forget: Thanks to Jeffrey for the original work!

[sebres]

implemented real DFA-workspace with reallocate on demand if to small; small code review.