Merge branch 'merge-pcre' into 10.0

pcre/AUTHORS

@@ -8,7 +8,7 @@ Email domain:     cam.ac.uk
 University of Cambridge Computing Service,
 Cambridge, England.
 
-Copyright (c) 1997-2015 University of Cambridge
+Copyright (c) 1997-2016 University of Cambridge
 All rights reserved
 
 
@@ -19,7 +19,7 @@ Written by:       Zoltan Herczeg
 Email local part: hzmester
 Emain domain:     freemail.hu
 
-Copyright(c) 2010-2015 Zoltan Herczeg
+Copyright(c) 2010-2016 Zoltan Herczeg
 All rights reserved.
 
 
@@ -30,7 +30,7 @@ Written by:       Zoltan Herczeg
 Email local part: hzmester
 Emain domain:     freemail.hu
 
-Copyright(c) 2009-2015 Zoltan Herczeg
+Copyright(c) 2009-2016 Zoltan Herczeg
 All rights reserved.
 
 

pcre/CMakeLists.txt

@@ -65,6 +65,7 @@
 #            so it has been removed.
 # 2013-10-08 PH got rid of the "source" command, which is a bash-ism (use ".")
 # 2013-11-05 PH added support for PARENS_NEST_LIMIT
+# 2016-03-01 PH applied Chris Wilson's patch for MSVC static build
 
 PROJECT(PCRE C CXX)
 

pcre/ChangeLog

@@ -4,12 +4,104 @@ ChangeLog for PCRE
 Note that the PCRE 8.xx series (PCRE1) is now in a bugfix-only state. All
 development is happening in the PCRE2 10.xx series.
 
+Version 8.39 14-June-2016
+-------------------------
+
+1.  If PCRE_AUTO_CALLOUT was set on a pattern that had a (?# comment between
+    an item and its qualifier (for example, A(?#comment)?B) pcre_compile()
+    misbehaved. This bug was found by the LLVM fuzzer.
+
+2.  Similar to the above, if an isolated \E was present between an item and its
+    qualifier when PCRE_AUTO_CALLOUT was set, pcre_compile() misbehaved. This
+    bug was found by the LLVM fuzzer.
+
+3.  Further to 8.38/46, negated classes such as [^[:^ascii:]\d] were also not
+    working correctly in UCP mode.
+
+4.  The POSIX wrapper function regexec() crashed if the option REG_STARTEND
+    was set when the pmatch argument was NULL. It now returns REG_INVARG.
+
+5.  Allow for up to 32-bit numbers in the ordin() function in pcregrep.
+
+6.  An empty \Q\E sequence between an item and its qualifier caused
+    pcre_compile() to misbehave when auto callouts were enabled. This bug was
+    found by the LLVM fuzzer.
+
+7.  If a pattern that was compiled with PCRE_EXTENDED started with white
+    space or a #-type comment that was followed by (?-x), which turns off
+    PCRE_EXTENDED, and there was no subsequent (?x) to turn it on again,
+    pcre_compile() assumed that (?-x) applied to the whole pattern and
+    consequently mis-compiled it. This bug was found by the LLVM fuzzer.
+
+8.  A call of pcre_copy_named_substring() for a named substring whose number
+    was greater than the space in the ovector could cause a crash.
+
+9.  Yet another buffer overflow bug involved duplicate named groups with a
+    group that reset capture numbers (compare 8.38/7 below). Once again, I have
+    just allowed for more memory, even if not needed. (A proper fix is
+    implemented in PCRE2, but it involves a lot of refactoring.)
+
+10. pcre_get_substring_list() crashed if the use of \K in a match caused the
+    start of the match to be earlier than the end.
+
+11. Migrating appropriate PCRE2 JIT improvements to PCRE.
+
+12. A pattern such as /(?<=((?C)0))/, which has a callout inside a lookbehind
+    assertion, caused pcretest to generate incorrect output, and also to read
+    uninitialized memory (detected by ASAN or valgrind).
+
+13. A pattern that included (*ACCEPT) in the middle of a sufficiently deeply
+    nested set of parentheses of sufficient size caused an overflow of the
+    compiling workspace (which was diagnosed, but of course is not desirable).
+
+14. And yet another buffer overflow bug involving duplicate named groups, this
+    time nested, with a nested back reference. Yet again, I have just allowed
+    for more memory, because anything more needs all the refactoring that has
+    been done for PCRE2. An example pattern that provoked this bug is:
+    /((?J)(?'R'(?'R'(?'R'(?'R'(?'R'(?|(\k'R'))))))))/ and the bug was
+    registered as CVE-2016-1283.
+
+15. pcretest went into a loop if global matching was requested with an ovector
+    size less than 2. It now gives an error message. This bug was found by
+    afl-fuzz.
+
+16. An invalid pattern fragment such as (?(?C)0 was not diagnosing an error
+    ("assertion expected") when (?(?C) was not followed by an opening
+    parenthesis.
+
+17. Fixed typo ("&&" for "&") in pcre_study(). Fortunately, this could not
+    actually affect anything, by sheer luck.
+
+18. Applied Chris Wilson's patch (Bugzilla #1681) to CMakeLists.txt for MSVC
+    static compilation.
+
+19. Modified the RunTest script to incorporate a valgrind suppressions file so
+    that certain errors, provoked by the SSE2 instruction set when JIT is used,
+    are ignored.
+
+20. A racing condition is fixed in JIT reported by Mozilla.
+
+21. Minor code refactor to avoid "array subscript is below array bounds"
+    compiler warning.
+
+22. Minor code refactor to avoid "left shift of negative number" warning.
+
+23. Fix typo causing compile error when 16- or 32-bit JIT is compiled without
+    UCP support.
+
+24. Refactor to avoid compiler warnings in pcrecpp.cc.
+
+25. Refactor to fix a typo in pcre_jit_test.c
+
+26. Patch to support compiling pcrecpp.cc with Intel compiler.
+
+
 Version 8.38 23-November-2015
 -----------------------------
 
 1.  If a group that contained a recursive back reference also contained a
     forward reference subroutine call followed by a non-forward-reference
-    subroutine call, for example /.((?2)(?R)\1)()/, pcre2_compile() failed to
+    subroutine call, for example /.((?2)(?R)\1)()/, pcre_compile() failed to
     compile correct code, leading to undefined behaviour or an internally
     detected error. This bug was discovered by the LLVM fuzzer.
 

pcre/LICENCE

@@ -25,7 +25,7 @@ Email domain:     cam.ac.uk
 University of Cambridge Computing Service,
 Cambridge, England.
 
-Copyright (c) 1997-2015 University of Cambridge
+Copyright (c) 1997-2016 University of Cambridge
 All rights reserved.
 
 
@@ -36,7 +36,7 @@ Written by:       Zoltan Herczeg
 Email local part: hzmester
 Emain domain:     freemail.hu
 
-Copyright(c) 2010-2015 Zoltan Herczeg
+Copyright(c) 2010-2016 Zoltan Herczeg
 All rights reserved.
 
 
@@ -47,7 +47,7 @@ Written by:       Zoltan Herczeg
 Email local part: hzmester
 Emain domain:     freemail.hu
 
-Copyright(c) 2009-2015 Zoltan Herczeg
+Copyright(c) 2009-2016 Zoltan Herczeg
 All rights reserved.
 
 

pcre/NEWS

@@ -1,6 +1,15 @@
 News about PCRE releases
 ------------------------
 
+Release 8.39 14-June-2016
+-------------------------
+
+Some appropriate PCRE2 JIT improvements have been retro-fitted to PCRE1. Apart
+from that, this is another bug-fix release. Note that this library (now called
+PCRE1) is now being maintained for bug fixes only. New projects are advised to
+use the new PCRE2 libraries.
+
+
 Release 8.38 23-November-2015
 -----------------------------
 

pcre/RunGrepTest

@@ -67,6 +67,15 @@ fi
 ./pcretest -C utf >/dev/null
 utf8=$?
 
+# We need valgrind suppressions when JIT is in use. (This isn't perfect because
+# some tests are run with -no-jit, but as PCRE1 is in maintenance only, I have
+# not bothered about that.)
+
+./pcretest -C jit >/dev/null
+if [ $? -eq 1 -a "$valgrind" != "" ] ; then
+  valgrind="$valgrind --suppressions=./testdata/valgrind-jit.supp"
+fi
+
 echo "Testing pcregrep main features"
 
 echo "---------------------------- Test 1 ------------------------------" >testtrygrep

pcre/RunTest

@@ -178,6 +178,7 @@ nojit=
 sim=
 skip=
 valgrind=
+vjs=
 
 # This is in case the caller has set aliases (as I do - PH)
 unset cp ls mv rm
@@ -357,6 +358,9 @@ $sim ./pcretest -C jit >/dev/null
 jit=$?
 if [ $jit -ne 0 -a "$nojit" != "yes" ] ; then
   jitopt=-s+
+  if [ "$valgrind" != "" ] ; then
+    vjs="--suppressions=$testdata/valgrind-jit.supp"
+  fi
 fi
 
 # If no specific tests were requested, select all. Those that are not
@@ -423,7 +427,7 @@ for bmode in "$test8" "$test16" "$test32"; do
 if [ $do1 = yes ] ; then
   echo $title1
   for opt in "" "-s" $jitopt; do
-    $sim $valgrind ./pcretest -q $bmode $opt $testdata/testinput1 testtry
+    $sim $valgrind ${opt:+$vjs} ./pcretest -q $bmode $opt $testdata/testinput1 testtry
     if [ $? = 0 ] ; then
       $cf $testdata/testoutput1 testtry
       if [ $? != 0 ] ; then exit 1; fi
@@ -441,7 +445,7 @@ fi
 if [ $do2 = yes ] ; then
   echo $title2 "(not UTF-$bits)"
   for opt in "" "-s" $jitopt; do
-    $sim $valgrind ./pcretest -q $bmode $opt $testdata/testinput2 testtry
+    $sim $valgrind ${opt:+$vjs} ./pcretest -q $bmode $opt $testdata/testinput2 testtry
     if [ $? = 0 ] ; then
       $cf $testdata/testoutput2 testtry
       if [ $? != 0 ] ; then exit 1; fi
@@ -504,7 +508,7 @@ if [ $do3 = yes ] ; then
   if [ "$locale" != "" ] ; then
     echo $title3 "(using '$locale' locale)"
     for opt in "" "-s" $jitopt; do
-      $sim $valgrind ./pcretest -q $bmode $opt $infile testtry
+      $sim $valgrind ${opt:+$vjs} ./pcretest -q $bmode $opt $infile testtry
       if [ $? = 0 ] ; then
         if $cf $outfile testtry >teststdout || \
            $cf $outfile2 testtry >teststdout || \
@@ -540,7 +544,7 @@ if [ $do4 = yes ] ; then
     echo "  Skipped because UTF-$bits support is not available"
   else
     for opt in "" "-s" $jitopt; do
-      $sim $valgrind ./pcretest -q $bmode $opt $testdata/testinput4 testtry
+      $sim $valgrind ${opt:+$vjs} ./pcretest -q $bmode $opt $testdata/testinput4 testtry
       if [ $? = 0 ] ; then
         $cf $testdata/testoutput4 testtry
         if [ $? != 0 ] ; then exit 1; fi
@@ -560,7 +564,7 @@ if [ $do5 = yes ] ; then
     echo "  Skipped because UTF-$bits support is not available"
   else
     for opt in "" "-s" $jitopt; do
-      $sim $valgrind ./pcretest -q $bmode $opt $testdata/testinput5 testtry
+      $sim $valgrind ${opt:+$vjs} ./pcretest -q $bmode $opt $testdata/testinput5 testtry
       if [ $? = 0 ] ; then
         $cf $testdata/testoutput5 testtry
         if [ $? != 0 ] ; then exit 1; fi
@@ -580,7 +584,7 @@ if [ $do6 = yes ] ; then
     echo "  Skipped because Unicode property support is not available"
   else
     for opt in "" "-s" $jitopt; do
-      $sim $valgrind ./pcretest -q $bmode $opt $testdata/testinput6 testtry
+      $sim $valgrind ${opt:+$vjs} ./pcretest -q $bmode $opt $testdata/testinput6 testtry
       if [ $? = 0 ] ; then
         $cf $testdata/testoutput6 testtry
         if [ $? != 0 ] ; then exit 1; fi
@@ -602,7 +606,7 @@ if [ $do7 = yes ] ; then
     echo "  Skipped because Unicode property support is not available"
   else
     for opt in "" "-s" $jitopt; do
-      $sim $valgrind ./pcretest -q $bmode $opt $testdata/testinput7 testtry
+      $sim $valgrind ${opt:+$vjs} ./pcretest -q $bmode $opt $testdata/testinput7 testtry
       if [ $? = 0 ] ; then
         $cf $testdata/testoutput7 testtry
         if [ $? != 0 ] ; then exit 1; fi
@@ -698,7 +702,7 @@ if [ $do12 = yes ] ; then
   if [ $jit -eq 0 -o "$nojit" = "yes" ] ; then
     echo "  Skipped because JIT is not available or not usable"
   else
-    $sim $valgrind ./pcretest -q $bmode $testdata/testinput12 testtry
+    $sim $valgrind $vjs ./pcretest -q $bmode $testdata/testinput12 testtry
     if [ $? = 0 ] ; then
       $cf $testdata/testoutput12 testtry
       if [ $? != 0 ] ; then exit 1; fi
@@ -735,7 +739,7 @@ if [ "$do14" = yes ] ; then
     cp -f $testdata/saved16 testsaved16
     cp -f $testdata/saved32 testsaved32
     for opt in "" "-s" $jitopt; do
-      $sim $valgrind ./pcretest -q $bmode $opt $testdata/testinput14 testtry
+      $sim $valgrind ${opt:+$vjs} ./pcretest -q $bmode $opt $testdata/testinput14 testtry
       if [ $? = 0 ] ; then
         $cf $testdata/testoutput14 testtry
         if [ $? != 0 ] ; then exit 1; fi
@@ -759,7 +763,7 @@ if [ "$do15" = yes ] ; then
     echo "  Skipped because UTF-$bits support is not available"
   else
     for opt in "" "-s" $jitopt; do
-      $sim $valgrind ./pcretest -q $bmode $opt $testdata/testinput15 testtry
+      $sim $valgrind ${opt:+$vjs} ./pcretest -q $bmode $opt $testdata/testinput15 testtry
       if [ $? = 0 ] ; then
         $cf $testdata/testoutput15 testtry
         if [ $? != 0 ] ; then exit 1; fi
@@ -783,7 +787,7 @@ if [ $do16 = yes ] ; then
     echo "  Skipped because Unicode property support is not available"
   else
     for opt in "" "-s" $jitopt; do
-      $sim $valgrind ./pcretest -q $bmode $opt $testdata/testinput16 testtry
+      $sim $valgrind ${opt:+$vjs} ./pcretest -q $bmode $opt $testdata/testinput16 testtry
       if [ $? = 0 ] ; then
         $cf $testdata/testoutput16 testtry
         if [ $? != 0 ] ; then exit 1; fi
@@ -805,7 +809,7 @@ if [ $do17 = yes ] ; then
     echo "  Skipped when running 8-bit tests"
   else
     for opt in "" "-s" $jitopt; do
-      $sim $valgrind ./pcretest -q $bmode $opt $testdata/testinput17 testtry
+      $sim $valgrind ${opt:+$vjs} ./pcretest -q $bmode $opt $testdata/testinput17 testtry
       if [ $? = 0 ] ; then
         $cf $testdata/testoutput17 testtry
         if [ $? != 0 ] ; then exit 1; fi
@@ -829,7 +833,7 @@ if [ $do18 = yes ] ; then
     echo "  Skipped because UTF-$bits support is not available"
   else
     for opt in "" "-s" $jitopt; do
-      $sim $valgrind ./pcretest -q $bmode $opt $testdata/testinput18 testtry
+      $sim $valgrind ${opt:+$vjs} ./pcretest -q $bmode $opt $testdata/testinput18 testtry
       if [ $? = 0 ] ; then
         $cf $testdata/testoutput18-$bits testtry
         if [ $? != 0 ] ; then exit 1; fi
@@ -853,7 +857,7 @@ if [ $do19 = yes ] ; then
     echo "  Skipped because Unicode property support is not available"
   else
     for opt in "" "-s" $jitopt; do
-      $sim $valgrind ./pcretest -q $bmode $opt $testdata/testinput19 testtry
+      $sim $valgrind ${opt:+$vjs} ./pcretest -q $bmode $opt $testdata/testinput19 testtry
       if [ $? = 0 ] ; then
         $cf $testdata/testoutput19 testtry
         if [ $? != 0 ] ; then exit 1; fi

pcre/configure.ac

@@ -9,18 +9,18 @@ dnl The PCRE_PRERELEASE feature is for identifying release candidates. It might
 dnl be defined as -RC2, for example. For real releases, it should be empty.
 
 m4_define(pcre_major, [8])
-m4_define(pcre_minor, [38])
+m4_define(pcre_minor, [39])
 m4_define(pcre_prerelease, [])
-m4_define(pcre_date, [2015-11-23])
+m4_define(pcre_date, [2016-06-14])
 
 # NOTE: The CMakeLists.txt file searches for the above variables in the first
 # 50 lines of this file. Please update that if the variables above are moved.
 
 # Libtool shared library interface versions (current:revision:age)
-m4_define(libpcre_version, [3:6:2])
-m4_define(libpcre16_version, [2:6:2])
-m4_define(libpcre32_version, [0:6:0])
-m4_define(libpcreposix_version, [0:3:0])
+m4_define(libpcre_version, [3:7:2])
+m4_define(libpcre16_version, [2:7:2])
+m4_define(libpcre32_version, [0:7:0])
+m4_define(libpcreposix_version, [0:4:0])
 m4_define(libpcrecpp_version, [0:1:0])
 
 AC_PREREQ(2.57)

pcre/doc/html/pcreapi.html

@@ -315,9 +315,8 @@ <h1>pcreapi man page</h1>
 building PCRE, for use in environments that have limited stacks. Because of the
 greater use of memory management, it runs more slowly. Separate functions are
 provided so that special-purpose external code can be used for this case. When
-used, these functions are always called in a stack-like manner (last obtained,
-first freed), and always for memory blocks of the same size. There is a
-discussion about PCRE's stack usage in the
+used, these functions always allocate memory blocks of the same size. There is
+a discussion about PCRE's stack usage in the
 <a href="pcrestack.html"><b>pcrestack</b></a>
 documentation.
 </P>
@@ -2913,9 +2912,9 @@ <h1>pcreapi man page</h1>
 </P>
 <br><a name="SEC26" href="#TOC1">REVISION</a><br>
 <P>
-Last updated: 09 February 2014
+Last updated: 18 December 2015
 <br>
-Copyright &copy; 1997-2014 University of Cambridge.
+Copyright &copy; 1997-2015 University of Cambridge.
 <br>
 <p>
 Return to the <a href="index.html">PCRE index page</a>.