Move to versioned symbols requires a soname bump

[MatthewVernon]

Hi,

The pcre2 10.47 release enables versioned symbols (by default). Unfortunately, this has the effect of changing almost all the symbols in the resulting library; and this requires a soname increment. Software built against 10.46 won't work with 10.47

I see this in Debian's CI if I run the glib2.0 tests: where the package was built against 10.46, but is running with 10.47 libraries installed. Signal 6 is SIGABRT

511s # FAIL: glib/regex.test (Child process killed by signal 6)

Equally, if I take pcre2test from 10.46, and run it with 10.47 libraries, it's obviously broken:

matthew@tsk:~/bin$ pcre2test
PCRE2 version 10.47 2025-10-21 (8-bit)
  re> /[abc]*/
data> abcadbce
Start of matched string is beyond its end - displaying from end to start.
 0: ERROR: bad value(s) for offset(s): 0x1 0xdeadbe00

whereas, with the 10.46 libraries, it behaves as expected:

PCRE2 version 10.46 2025-08-27 (8-bit)
  re> /[abc]*/
data> abcadbce
 0: abca

This was reported as Debian bug 1119027 which contains a diff of the symbols file (showing essentially every symbol replaced - e.g. pcre2_code_copy_8@Base is replaced by pcre2_code_copy_8@PCRE2_10.47).

I'd like not to get Debian and PCRE2 upstream out of sync regarding soname. So we could bump the soname now, and if there is another soname bump needed before 10.48, bump it again. Or I could disable versioned symbols in 10.47 in Debian (or you could decide that it's worth putting out 10.48 now with a soname bump since presumably Debian won't be the only people with this issue).

What do you think?

[NWilson]

Oh no, I had no idea. I was following some documentation which seemed to say that adding versions was OK.

The quickfix is for Debian to simply compiled without the symbol versions for now, to un-break things.

I was hoping to avoid a soname bump, on the grounds that is the worst of API breakages, and I didn't want to do it unnecessarily.

I'm so sorry - I'll have to think about what to do.

[NWilson]

I'm scared that I've done something even worse.

If I build 10.47 without symbols, it seems to be able to link and run just fine against 10.47 with symbols.

The only problem I can reproduce so far is building 10.46 and running against 10.47 - even 10.47 without versions!

What have I done? This is pretty terrible. It looks like some sort of an ABI break, totally independent of the versioning issue.

[NWilson]

The diff of pcre2.h from 10.46 to 10.47:

(It all looks completely tame, and no signatures changed in a bad way at all.)

--- ../interface-46/pcre2.h     2025-10-27 10:45:26.033201749 +0000
+++ ../interface-47/pcre2.h     2025-10-27 10:42:42.686207252 +0000
@@ -40,44 +40,41 @@
 #define PCRE2_H_IDEMPOTENT_GUARD
 
 /* The current PCRE version information. */
 
 #define PCRE2_MAJOR           10
-#define PCRE2_MINOR           46
-#define PCRE2_PRERELEASE      
-#define PCRE2_DATE            2025-08-27
+#define PCRE2_MINOR           48
+#define PCRE2_PRERELEASE      -DEV
+#define PCRE2_DATE            2025-10-21
 
-/* When an application links to a PCRE DLL in Windows, the symbols that are
+/* When an application links to a PCRE2 DLL in Windows, the symbols that are
 imported have to be identified as such. When building PCRE2, the appropriate
-export setting is defined in pcre2_internal.h, which includes this file. So we
-don't change existing definitions of PCRE2_EXP_DECL. */
+export setting is defined in pcre2_internal.h, which includes this file. So, we
+don't change existing definitions of PCRE2_EXP_DECL.
 
-#if defined(_WIN32) && !defined(PCRE2_STATIC)
-#  ifndef PCRE2_EXP_DECL
-#    define PCRE2_EXP_DECL  extern __declspec(dllimport)
-#  endif
-#endif
-
-/* By default, we use the standard "extern" declarations. */
+By default, we use the standard "extern" declarations. */
 
 #ifndef PCRE2_EXP_DECL
-#  ifdef __cplusplus
+#  if defined(_WIN32) && !defined(PCRE2_STATIC)
+#    define PCRE2_EXP_DECL  extern __declspec(dllimport)
+#  elif defined __cplusplus
 #    define PCRE2_EXP_DECL  extern "C"
 #  else
 #    define PCRE2_EXP_DECL  extern
 #  endif
 #endif
 
 /* When compiling with the MSVC compiler, it is sometimes necessary to include
-a "calling convention" before exported function names. (This is secondhand
-information; I know nothing about MSVC myself). For example, something like
+a "calling convention" before exported function names. For example:
 
   void __cdecl function(....)
 
-might be needed. In order so make this easy, all the exported functions have
-PCRE2_CALL_CONVENTION just before their names. It is rarely needed; if not
-set, we ensure here that it has no effect. */
+might be needed. In order to make this easy, all the exported functions have
+PCRE2_CALL_CONVENTION just before their names.
+
+PCRE2 normally uses the platform's standard calling convention, so this should
+not be set unless you know you need it. */
 
 #ifndef PCRE2_CALL_CONVENTION
 #define PCRE2_CALL_CONVENTION
 #endif
 
@@ -341,10 +338,14 @@
 #define PCRE2_ERROR_ECLASS_HINT_SQUARE_BRACKET     212
 #define PCRE2_ERROR_PERL_ECLASS_UNEXPECTED_EXPR    213
 #define PCRE2_ERROR_PERL_ECLASS_EMPTY_EXPR         214
 #define PCRE2_ERROR_PERL_ECLASS_MISSING_CLOSE      215
 #define PCRE2_ERROR_PERL_ECLASS_UNEXPECTED_CHAR    216
+#define PCRE2_ERROR_EXPECTED_CAPTURE_GROUP         217
+#define PCRE2_ERROR_MISSING_OPENING_PARENTHESIS    218
+#define PCRE2_ERROR_MISSING_NUMBER_TERMINATOR      219
+#define PCRE2_ERROR_NULL_ERROROFFSET               220
 
 /* "Expected" matching error codes: no match and partial match. */
 
 #define PCRE2_ERROR_NOMATCH          (-1)
 #define PCRE2_ERROR_PARTIAL          (-2)
@@ -430,10 +431,15 @@
 #define PCRE2_ERROR_DFA_UINVALID_UTF  (-66)
 #define PCRE2_ERROR_INVALIDOFFSET     (-67)
 #define PCRE2_ERROR_JIT_UNSUPPORTED   (-68)
 #define PCRE2_ERROR_REPLACECASE       (-69)
 #define PCRE2_ERROR_TOOLARGEREPLACE   (-70)
+#define PCRE2_ERROR_DIFFSUBSPATTERN   (-71)
+#define PCRE2_ERROR_DIFFSUBSSUBJECT   (-72)
+#define PCRE2_ERROR_DIFFSUBSOFFSET    (-73)
+#define PCRE2_ERROR_DIFFSUBSOPTIONS   (-74)
+#define PCRE2_ERROR_BAD_BACKSLASH_K   (-75)
 
 
 /* Request types for pcre2_pattern_info() */
 
 #define PCRE2_INFO_ALLOPTIONS            0
@@ -482,10 +488,11 @@
 #define PCRE2_CONFIG_VERSION                11
 #define PCRE2_CONFIG_HEAPLIMIT              12
 #define PCRE2_CONFIG_NEVER_BACKSLASH_C      13
 #define PCRE2_CONFIG_COMPILED_WIDTHS        14
 #define PCRE2_CONFIG_TABLES_LENGTH          15
+#define PCRE2_CONFIG_EFFECTIVE_LINKSIZE     16
 
 /* Optimization directives for pcre2_set_optimize().
 For binary compatibility, only add to this list; do not renumber. */
 
 #define PCRE2_OPTIMIZATION_NONE    0
@@ -741,18 +748,18 @@
 PCRE2_EXP_DECL pcre2_match_data *PCRE2_CALL_CONVENTION \
   pcre2_match_data_create(uint32_t, pcre2_general_context *); \
 PCRE2_EXP_DECL pcre2_match_data *PCRE2_CALL_CONVENTION \
   pcre2_match_data_create_from_pattern(const pcre2_code *, \
     pcre2_general_context *); \
+PCRE2_EXP_DECL void PCRE2_CALL_CONVENTION \
+  pcre2_match_data_free(pcre2_match_data *); \
 PCRE2_EXP_DECL int PCRE2_CALL_CONVENTION \
   pcre2_dfa_match(const pcre2_code *, PCRE2_SPTR, PCRE2_SIZE, PCRE2_SIZE, \
     uint32_t, pcre2_match_data *, pcre2_match_context *, int *, PCRE2_SIZE); \
 PCRE2_EXP_DECL int PCRE2_CALL_CONVENTION \
   pcre2_match(const pcre2_code *, PCRE2_SPTR, PCRE2_SIZE, PCRE2_SIZE, \
     uint32_t, pcre2_match_data *, pcre2_match_context *); \
-PCRE2_EXP_DECL void PCRE2_CALL_CONVENTION \
-  pcre2_match_data_free(pcre2_match_data *); \
 PCRE2_EXP_DECL PCRE2_SPTR PCRE2_CALL_CONVENTION \
   pcre2_get_mark(pcre2_match_data *); \
 PCRE2_EXP_DECL PCRE2_SIZE PCRE2_CALL_CONVENTION \
   pcre2_get_match_data_size(pcre2_match_data *); \
 PCRE2_EXP_DECL PCRE2_SIZE PCRE2_CALL_CONVENTION \
@@ -760,11 +767,13 @@
 PCRE2_EXP_DECL uint32_t PCRE2_CALL_CONVENTION \
   pcre2_get_ovector_count(pcre2_match_data *); \
 PCRE2_EXP_DECL PCRE2_SIZE *PCRE2_CALL_CONVENTION \
   pcre2_get_ovector_pointer(pcre2_match_data *); \
 PCRE2_EXP_DECL PCRE2_SIZE PCRE2_CALL_CONVENTION \
-  pcre2_get_startchar(pcre2_match_data *);
+  pcre2_get_startchar(pcre2_match_data *); \
+PCRE2_EXP_DECL int PCRE2_CALL_CONVENTION \
+  pcre2_next_match(pcre2_match_data *, PCRE2_SIZE *, uint32_t *);
 
 
 /* Convenience functions for handling matched substrings. */
 
 #define PCRE2_SUBSTRING_FUNCTIONS \
@@ -940,10 +949,11 @@
 #define pcre2_match_context_create            PCRE2_SUFFIX(pcre2_match_context_create_)
 #define pcre2_match_context_free              PCRE2_SUFFIX(pcre2_match_context_free_)
 #define pcre2_match_data_create               PCRE2_SUFFIX(pcre2_match_data_create_)
 #define pcre2_match_data_create_from_pattern  PCRE2_SUFFIX(pcre2_match_data_create_from_pattern_)
 #define pcre2_match_data_free                 PCRE2_SUFFIX(pcre2_match_data_free_)
+#define pcre2_next_match                      PCRE2_SUFFIX(pcre2_next_match_)
 #define pcre2_pattern_convert                 PCRE2_SUFFIX(pcre2_pattern_convert_)
 #define pcre2_pattern_info                    PCRE2_SUFFIX(pcre2_pattern_info_)
 #define pcre2_serialize_decode                PCRE2_SUFFIX(pcre2_serialize_decode_)
 #define pcre2_serialize_encode                PCRE2_SUFFIX(pcre2_serialize_encode_)
 #define pcre2_serialize_free                  PCRE2_SUFFIX(pcre2_serialize_free_)

[MatthewVernon]

I should confess I've not tried running pcre2test compiled against one version and running with another before. And, in fact, if I try and take one built against 10.42 (which is what's installed on my Debian oldstable system) and try it with 10.46 (rebuilt on bookworm) I see a similar failure:

PCRE2 version 10.46 2025-08-27
  re> /[abc]*/
data> abcadbce
Start of matched string is beyond its end - displaying from end to start.
 0: ERROR: bad value(s) for offset(s): 0x0 0x1deadbe00

So it may be that pcre2test is unusual.

The only symbols change I see in the library is the addition of pcre2_next_match_16@Base (and similar 8,32), which I agree doesn't seem a big change.

What I'll do for now is upload 10.47-2 (with symbol versioning disabled) to Debian's experimental distribution and once that's been built re-run the tests of glib2.0 (built against 10.46) against 10.47-2.

[NWilson]

Thank you. I have done a git bisect:

#! /bin/sh

# Run with:
#   git bisect start `git merge-base pcre2-10.47 master` `git merge-base pcre2-10.46 master`
#   git bisect run ./bisect-script.sh
#
# The output is...
#    broken in eb3bd3cf1418cb1a0eabf984b0b1e80b6bdd9314

set -e

git submodule update
rm -rf build
mkdir build
cd build
CC='clang -fsanitize=undefined,address,integer -fno-sanitize-recover=undefined,integer -fno-sanitize=unsigned-integer-overflow,unsigned-shift-base,function' cmake -G Ninja .. -DBUILD_SHARED_LIBS=ON -DBUILD_STATIC_LIBS=OFF -DCMAKE_BUILD_TYPE=Debug -DPCRE2_SYMVERS=OFF
ninja

cd ../build-46
printf "/[abc]*/\n  abcadbce\n\n/zz/\n  z\n" | ./pcre2test

And the guilty commit is eb3bd3c.

And indeed, now I see it, it's clear what the issue is. I changed the layout of an internal structure.

pcre2test is special - it reaches inside the library and touches the fields of the (hidden, not exported) structures created by the PCRE2 API. So I don't think this is a fair test. With hindsight it should have been immediately obvious to me as maintainer - I just haven't ever tried running pcre2test built against 10.46 and running against 10.47. It won't work, and it's not a problem.

---

But back to the versioning issue... If I build pcre2test 10.47 without versioning, and then run it against a build of 10.47 library with versioning, I get no errors or failures. I can run the entire test suite without problems.

So I'm not going completely mad.

I did test this scenario! And it appears to work. So as for what the glib tests are doing... I have no idea (yet).

[carenas]

I did test this scenario! And it appears to work. So as for what the glib tests are doing... I have no idea (yet).

ironically, the pcre2 library versioning was meant to specially help libraries like glib that depend on pcre2, so there are no conflicts when they are used by a program that ALSO links with pcre2, and might had been built against a different version.

the documentation about soname changes is a little misleading, because in this case the MAJOR number was meant to be increased.

in retrospective (specially considering that the version numbers is only enabled in a few systems) it should had been disabled by default to avoid forcing this ABI break.

[NWilson]

I've dug into the GLib failure.

SIGABRT (signal 6) sounds serious. But actually, it's just what their test harness returns when a test fails.

Here's the upstream test (in 2.86.1, which is the version you were testing in the log you linked): https://gitlab.gnome.org/GNOME/glib/-/blob/2.86.1/glib/tests/regex.c?ref_type=tags#L2645

They are testing that the regex ^(a)\g{3 fails with a particular syntax error. But, I changed/improved the granularity of the syntax errors, so this is now diagnosed as "unterminated backreference" rather than "missing backreference".

Hence, the test fails.

---

Unless there's further evidence, I don't think (yet) that adding symbols is actually a major version bump, or an ABI break.

[NWilson]

PCRE2 version 10.46 2025-08-27 (8-bit)
  re> /^(a)\g{3/
Failed: error 157 at offset 6: \g is not followed by a braced, angle-bracketed, or quoted name/number or by a plain number

PCRE2 version 10.47 2025-10-21 (8-bit)
  re> /^(a)\g{3/
Failed: error 219 at offset 8: syntax error in subpattern number (missing terminator?)
        here: ^(a)\g{3 |<--|

The vague PCRE2_ERROR_BACKSLASH_G_SYNTAX has been changed to the more granular PCRE2_ERROR_MISSING_NUMBER_TERMINATOR in this particular case.

[MatthewVernon]

The vague PCRE2_ERROR_BACKSLASH_G_SYNTAX has been changed to the more granular PCRE2_ERROR_MISSING_NUMBER_TERMINATOR in this particular case.

Yes, which was already causing a build failure in glib2.0 (cf Debian bug 1118572); it's also caused similar problems in mariadb and python-pcre2 at least which I still need to dig into. I think that's likely unrelated to the soname bump question.

[NWilson]

I made the following change in GLib:

diff --git a/glib/tests/regex.c b/glib/tests/regex.c
index d7a698ec6..a94acbd68 100644
--- a/glib/tests/regex.c
+++ b/glib/tests/regex.c
@@ -2642,7 +2642,7 @@ G_GNUC_END_IGNORE_DEPRECATIONS
   TEST_NEW_FAIL ("\\666", G_REGEX_RAW, G_REGEX_ERROR_INVALID_OCTAL_VALUE);
   TEST_NEW_FAIL ("^(?(DEFINE) abc | xyz ) ", 0, G_REGEX_ERROR_TOO_MANY_BRANCHES_IN_DEFINE);
   TEST_NEW_FAIL ("a", G_REGEX_NEWLINE_CRLF | G_REGEX_NEWLINE_ANYCRLF, G_REGEX_ERROR_INCONSISTENT_NEWLINE_OPTIONS);
-  TEST_NEW_FAIL ("^(a)\\g{3", 0, G_REGEX_ERROR_MISSING_BACK_REFERENCE);
+  TEST_NEW_FAIL ("^(a)\\g\"3", 0, G_REGEX_ERROR_MISSING_BACK_REFERENCE);
   TEST_NEW_FAIL ("^(a)\\g{0}", 0, G_REGEX_ERROR_INEXISTENT_SUBPATTERN_REFERENCE);
   TEST_NEW ("abc(*FAIL:123)xyz", 0, 0);
   TEST_NEW_FAIL ("a(*FOOBAR)b", 0, G_REGEX_ERROR_UNKNOWN_BACKTRACKING_CONTROL_VERB);

I then built PCRE2 without symbol versions, installed it in /usr/local/lib. I build & linked Glib 2.0.

Then I rebuilt PCRE2 with symbol versions, installed it in the same place, and did meson test -C _build/ --no-rebuild regex to re-run the tests, without rebuilding.

$ ldd ./_build/glib/tests/regex
        linux-vdso.so.1 (0x00007ffe224dd000)
        libglib-2.0.so.0 => /workspaces/glib/_build/glib/tests/../libglib-2.0.so.0 (0x00007ac6b9914000)
        libpcre2-8.so.0 => /usr/local/lib/libpcre2-8.so.0 (0x00007ac6b9880000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ac6b9669000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ac6b9a96000)

All the tests pass. So, I believe I have evidence here that the addition of symbol versions is not causing any problems with GLib. It's solely the issue with the new error code.

[NWilson]

The upstream GLib issue is: https://gitlab.gnome.org/GNOME/glib/-/issues/3809

It has been assigned to milestone 2.86.2, so I guess they'll release a version that works with PCRE2 10.47 fairly soon.

[NWilson]

The mariadb issue is very similar to the Glib one:

331s CURRENT_TEST: main.func_regexp_pcre
331s --- /usr/share/mariadb/mariadb-test/main/func_regexp_pcre.result	2025-07-28 19:35:03.000000000 +0000
331s +++ /tmp/tmp.57yAEs5aSK/var/7/log/func_regexp_pcre.reject	2025-10-26 22:46:11.505754917 +0000
331s @@ -791,7 +791,7 @@
331s  1
331s  SET default_regex_flags=DEFAULT;
331s  SELECT 'Aq' RLIKE 'A\\q';
331s -ERROR 42000: Regex error 'unrecognized character follows \ at offset 2'
331s +ERROR 42000: Regex error 'unrecognized character follows \ at offset 3'
331s  SET default_regex_flags='EXTRA';
331s  SELECT 'A' RLIKE 'B';
331s  'A' RLIKE 'B'
331s 
331s Result content mismatch
331s 
331s  - skipping '/tmp/tmp.57yAEs5aSK/var/7/log/main.func_regexp_pcre/'

They just have a rather brittle test which asserts that we aren't going to update our error messages (basically), and we did. The error offsets are purely for human consumption... but I guess I should have expected someone to hardcode them in their tests.

I do apologise.

---

The python-pcre2 one is more interesting.

 73s =================================== FAILURES ===================================
 73s _______________ test_match_expand[[abc]*--dabacbaccbacccb-0-1-d] _______________
 73s 
 73s pattern = '[abc]*', replacement = '', subject = 'dabacbaccbacccb', options = 0
 73s offset = 1, result = 'd'
 73s 
 73s     @pytest.mark.parametrize(
 73s         "pattern,replacement,subject,options,offset,result", test_data_match_expand
 73s     )
 73s     def test_match_expand(pattern, replacement, subject, options, offset, result):
 73s         p = pcre2.compile(pattern, options=options)
 73s         m = p.match(subject, offset=offset)
 73s >       assert m.expand(replacement) == result
 73s                ^^^^^^^^^^^^^^^^^^^^^
 73s 
 73s tests/test_match.py:40: 
 73s _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 73s match.pyx:243: in pcre2.match.Match.expand
 73s     ???
 73s _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 73s 
 73s >   ???
 73s E   pcre2.exceptions.LibraryError: Substitute start offset differs from prior match call

This is caused by a change which I made on the request of the Rust language bindings. It looks like the Python bindings are not happy though. The match object is keeping a reference to the subject string, for use in expand() but it looks like it somehow passes in a different subject string. I'd have to look into what the Python bindings are doing, but it's clearly caused by #807.

[NWilson]

I sent an update to the python-pcre2 project: grtetrault/pcre2.py#18

The MariaDB problem has some chatter on the OpenSUSE mailing list, and a bug for it there. It looks like they just disabled/patched-out the test that's failing. I can't find any record of them reporting it upstream to MariaDB.

GLib2.0 is in progress, as linked above.

Matthew, do let me know if any additional packages are failing.

[MatthewVernon]

OK, so an update - I can reproduce the same failure modes with glib2.0 from unstable (built against 10.46) running against 10.47-2 from experimental (i.e. without the versioned symbols) - arm64 and amd64. And there's been an update to the Debian bug where someone is pretty clear that symbol versioning shouldn't require a soname bump (but also that packages built against pcre-with-versioned-symbols aren't usable when linked to pcre-without-versioned-symbols). So there is obviously disagreement on the point!

So I think the answer is to file (and hopefully fix) fails-to-build-from-source bugs against the packages that no longer build with 10.47, then get 10.47-2 into unstable and testing, and then try restoring symbol versioning and do a test rebuild of all the things that depend directly on pcre2 in Debian; then we can retry the glib2.0 runtime tests in CI and if that's all good push 10.47-3 (with versioned symbols again).

[MatthewVernon]

Looking at all the packages that failed to build with 10.47-1 at debusine, they nearly all have existing FTBFS errors (i.e. are not regressions). The exceptions are the ones already discussed here (python3-pcre2, mariadb, glib2.0), and remctl output which is failing in its server/acl tests which the source suggests is doing some pcre2 based things. I'll report that to the remctl maintainers.

For my future reference, since debusine logs won't remain forever, the false-positive-failures are:

• 389-ds-base
• android-platform-external-libselinux
• crystal
• ganglia
• libsynthesis
• libt3widget
• mp4h
• mvdsv
• octave
• prelude-lml
• qxw
• squeak-vm