PCRE2 with Thread Local Storage #289
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rpluem
reviewed
Feb 1, 2022
|
close/reopen to restart ci |
PCRE2 wants an opaque context by providing the API to allocate and free it, so to minimize these calls we maintain one opaque context per thread (in Thread Local Storage, TLS) grown as needed, and while at it we do the same for PCRE1 ints vectors. Note that this requires a fast TLS mechanism to be worth it, which is the case of apr_thread_data_get/set() from/to apr_thread_current() when APR_HAS_THREAD_LOCAL; otherwise we'll do the allocation and freeing for each ap_regexec(). The small stack vector is used for PCRE1 && !APR_HAS_THREAD_LOCAL only now. git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897240 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897241 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897242 13f79535-47bb-0310-9956-ffa450edef68
No more (useless), no less (or PCRE will allocate a new buffer by itself to satisfy the needs), so we should base our buffer size solely on the number of captures in the regex (determined at compile time from the pattern). The nmatch provided by the user is used to fill in pmatch only (up to that), but "our" buffers are sized exactly as needed to avoid oversized allocations or PCRE allocating by itself. git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897244 13f79535-47bb-0310-9956-ffa450edef68
…ue at limits. Don't write to pmatch[nlimit:] when ncaps > nlimit, rc should not exceed nmatch either as before r1897244. git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897248 13f79535-47bb-0310-9956-ffa450edef68
…s!). #include "apr_thread_proc.h" is enough/needed by util_pcre.c and main.c. Fix compilation (vector => ovector) for !HAVE_PCRE2 && APR_HAS_THREAD_LOCAL. Check pcre2_match_data_create() return value for HAVE_PCRE2 && !APR_HAS_THREAD_LOCAL. git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897250 13f79535-47bb-0310-9956-ffa450edef68
Even though APR_HAS_THREAD_LOCAL is compiled in, ap_regexec() might still be called by non a apr_thread_t thread, let's fall back to alloc/free in this case too. git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897260 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897261 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897263 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897386 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897459 13f79535-47bb-0310-9956-ffa450edef68
their apr-1.8+ equivalent if available, or implement them using the compiler's
thread_local mechanism if available, or finally provide stubs otherwise.
AP_THREAD_LOCAL is defined to the compiler's keyword iff AP_HAS_THREAD_LOCAL.
Replace all apr_thread_create() calls with ap_thread_create() so that httpd
threads can use ap_thread_current()'s pool data as Thread Local Storage.
Bump MMN minor.
* include/httpd.h():
Define AP_HAS_THREAD_LOCAL, AP_THREAD_LOCAL (eventually), ap_thread_create(),
ap_thread_current_create() and ap_thread_current().
* server/util.c:
Implement ap_thread_create(), ap_thread_current_create() and
ap_thread_current() when APR < 1.8.
* modules/core/mod_watchdog.c, modules/http2/h2_workers.c,
modules/ssl/mod_ssl_ct.c:
Use ap_thread_create() instead of apr_thread_create.
* server/main.c:
Use AP_HAS_THREAD_LOCAL and ap_thread_current_create instead of APR's.
* server/util_pcre.c:
Use AP_HAS_THREAD_LOCAL and ap_thread_current instead of APR's.
* server/mpm/event/event.c, server/mpm/worker/worker.c,
server/mpm/prefork/prefork.c:
Use ap_thread_create() instead of apr_thread_create.
Create an apr_thread_t/ap_thread_current() for the main chaild thread usable
at child_init().
* server/mpm/winnt/child.c:
Use ap_thread_create() instead of CreateThread().
Create an apr_thread_t/ap_thread_current() for the main chaild thread usable
git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897460 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897461 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897462 13f79535-47bb-0310-9956-ffa450edef68
…r_fork().
thread_local variables are not (always?) reset on fork(), so we need a way
to set the current_thread to NULL in the child process.
Implement and use ap_thread_current_after_fork() for that.
* include/httpd.h:
Define ap_thread_current_after_fork().
* server/util.c:
Implement ap_thread_current_after_fork().
* server/mpm/event/event.c, server/mpm/prefork/prefork.c,
server/mpm/worker/worker.c:
Use ap_thread_current_after_fork().
* server/mpm/winnt/child.c:
Windows processes are not fork()ed and each child runs the main(), so
ap_thread_current_create() was already called there.
git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897472 13f79535-47bb-0310-9956-ffa450edef68
Replace ap_thread_current_create() by ap_thread_main_create() which is how it's used by httpd. The former is now a local helper only to implement the latter. This allows to consolidate/factorize common code in the main() of httpd and the unix MPMs. git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897543 13f79535-47bb-0310-9956-ffa450edef68
to "httpd -V" output and to mod_info page. git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1612934 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1612940 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1613189 13f79535-47bb-0310-9956-ffa450edef68
Possibly(?) pcre2_match() can modifiy the given pcre2_match_data and invalidate the old ovector, be safe and fetch it after. git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897651 13f79535-47bb-0310-9956-ffa450edef68
Don't stderr printf the "stat" and "failed" results from the previous apr_app_initialize() call for an error in ap_thread_main_create(). git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897680 13f79535-47bb-0310-9956-ffa450edef68
…cre's usage. If the compiler's thread_local is not efficient enough on some platforms, or not desired, have a way to disable its usage in httpd (at compile time). Handle -DAP_NO_THREAD_LOCAL and/or -DAPREG_NO_THREAD_LOCAL as build opt-out for thread_local usage in httpd gobally and/or in ap_regex only (respectively). git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897689 13f79535-47bb-0310-9956-ffa450edef68
For completness, and possibly to ease backport to 2.4.x for MPM winnt. git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897691 13f79535-47bb-0310-9956-ffa450edef68
git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1897692 13f79535-47bb-0310-9956-ffa450edef68
asfgit
pushed a commit
that referenced
this pull request
Feb 28, 2022
PCRE2 wants an opaque context by providing the API to allocate and free it, so
to minimize these calls we maintain one opaque context per thread (in Thread
Local Storage, TLS) grown as needed, and while at it we do the same for PCRE1
ints vectors. Note that this requires a fast TLS mechanism to be worth it,
which is the case of apr_thread_data_get/set() from/to apr_thread_current()
when APR_HAS_THREAD_LOCAL; otherwise we'll do the allocation and freeing for
each ap_regexec().
The small stack vector is used for PCRE1 && !APR_HAS_THREAD_LOCAL only now.
Follow up to r1897240: APR_HAS_THREAD_LOCAL wants #ifdef instead of #if.
Follow up to r1897240: CHANGES entry.
ap_regex: PCRE needs buffers sized against the number of captures only.
No more (useless), no less (or PCRE will allocate a new buffer by itself to
satisfy the needs), so we should base our buffer size solely on the number
of captures in the regex (determined at compile time from the pattern).
The nmatch provided by the user is used to fill in pmatch only (up to that),
but "our" buffers are sized exactly as needed to avoid oversized allocations
or PCRE allocating by itself.
ap_regex: Follow up to r1897244: Fix pmatch overflow and returned value at limits.
Don't write to pmatch[nlimit:] when ncaps > nmatch, rc should not exceed nmatch
either as before r1897244.
ap_regex: Follow up to r1897240: Fix issues spotted by Rüdiger (thanks!).
#include "apr_thread_proc.h" is enough/needed by util_pcre.c and main.c.
Fix compilation (vector => ovector) for !HAVE_PCRE2 && APR_HAS_THREAD_LOCAL.
Check pcre2_match_data_create() return value for HAVE_PCRE2 && !APR_HAS_THREAD_LOCAL.
ap_regex: Follow up to r1897240: runtime fallback to alloc/free.
Even though APR_HAS_THREAD_LOCAL is compiled in, ap_regexec() might still be
called by non a apr_thread_t thread, let's fall back to alloc/free in this
case too.
ap_regex: Follow up to r1897240: no ap_thread_current() yet.
ap_regex: Follow up to r1897240: cleanups.
ap_regex: Follow up to r1897240: cleanup PCRE2 match data on exit.
ap_regex: Follow up to r1897240: #if APR_HAS_THREAD_LOCAL, not #ifdef.
core: Efficient ap_thread_current() with APR < 1.8.
#define ap_thread_create, ap_thread_current_create and ap_thread_current to
their apr-1.8+ equivalent if available, or implement them using the compiler's
thread_local mechanism if available, or finally provide stubs otherwise.
#define AP_HAS_THREAD_LOCAL to 1 in the two former case or 0 otherwise, while
AP_THREAD_LOCAL is defined to the compiler's keyword iff AP_HAS_THREAD_LOCAL.
Replace all apr_thread_create() calls with ap_thread_create() so that httpd
threads can use ap_thread_current()'s pool data as Thread Local Storage.
Bump MMN minor.
* include/httpd.h():
Define AP_HAS_THREAD_LOCAL, AP_THREAD_LOCAL (eventually), ap_thread_create(),
ap_thread_current_create() and ap_thread_current().
* server/util.c:
Implement ap_thread_create(), ap_thread_current_create() and
ap_thread_current() when APR < 1.8.
* modules/core/mod_watchdog.c, modules/http2/h2_workers.c,
modules/ssl/mod_ssl_ct.c:
Use ap_thread_create() instead of apr_thread_create.
* server/main.c:
Use AP_HAS_THREAD_LOCAL and ap_thread_current_create instead of APR's.
* server/util_pcre.c:
Use AP_HAS_THREAD_LOCAL and ap_thread_current instead of APR's.
* server/mpm/event/event.c, server/mpm/worker/worker.c,
server/mpm/prefork/prefork.c:
Use ap_thread_create() instead of apr_thread_create.
Create an apr_thread_t/ap_thread_current() for the main chaild thread usable
at child_init().
* server/mpm/winnt/child.c:
Use ap_thread_create() instead of CreateThread().
Create an apr_thread_t/ap_thread_current() for the main chaild thread usable
Follow up to r1897460: APLOGNOs.
Follow up to r1897460: !APR_HAS_THREAD implies no ap_thread_* either.
core: Follow up to r1897460: Implement and use ap_thread_current_after_fork().
thread_local variables are not (always?) reset on fork(), so we need a way
to set the current_thread to NULL in the child process.
Implement and use ap_thread_current_after_fork() for that.
* include/httpd.h:
Define ap_thread_current_after_fork().
* server/util.c:
Implement ap_thread_current_after_fork().
* server/mpm/event/event.c, server/mpm/prefork/prefork.c,
server/mpm/worker/worker.c:
Use ap_thread_current_after_fork().
* server/mpm/winnt/child.c:
Windows processes are not fork()ed and each child runs the main(), so
ap_thread_current_create() was already called there.
core: Follow up to r1897460: Provide ap_thread_main_create().
Replace ap_thread_current_create() by ap_thread_main_create() which is how
it's used by httpd. The former is now a local helper only to implement the
latter.
This allows to consolidate/factorize common code in the main() of httpd and
the unix MPMs.
ap_regex: Follow up to r1897240: Fetch the ovector _after_ the match.
Possibly(?) pcre2_match() can modifiy the given pcre2_match_data and invalidate
the old ovector, be safe and fetch it after.
main: Follow up to r1897240: Fix bad log copypasta.
Don't stderr printf the "stat" and "failed" results from the previous
apr_app_initialize() call for an error in ap_thread_main_create().
core: Follow up to r1897240: Opt-out for AP_HAS_THREAD_LOCAL and/or pcre's usage.
If the compiler's thread_local is not efficient enough on some platforms, or
not desired, have a way to disable its usage in httpd (at compile time).
Handle -DAP_NO_THREAD_LOCAL and/or -DAPREG_NO_THREAD_LOCAL as build opt-out for
thread_local usage in httpd gobally and/or in ap_regex only (respectively).
core: Follow up to r1897240: Provide/export ap_thread_current_create()
For completness, and possibly to ease backport to 2.4.x for MPM winnt.
core: Follow up to r1897240 & r1897691: Syntax.
Add compiled and loaded PCRE version numbers
to "httpd -V" output and to mod_info page.
Forgotten file needed for r1612934.
Minor mmn bump due to r1612940.
Backports: r1897240, r1897241, r1897242, r1897244, r1897248, r1897250,
r1897260, r1897261, r1897263, r1897386, r1897459, r1897460,
r1897461, r1897462, r1897472, r1897543, r1897651, r1897680,
r1897689, r1897691, r1897692, r1612934, r1612940, r1613189
Submitted by: ylavic, rjung, rjung, rjung
Reviewed by: ylavic, rpluem, covener, steffenal, wrowe
GH: closes #289 (#289)
git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/branches/2.4.x@1898467 13f79535-47bb-0310-9956-ffa450edef68
|
Merged in r1898467 / 285b628 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
First commit is the current backport proposal.(now in 2.4.x as r1898399 / c602ba1)Next is TLS work:
r1897240
r1897241
r1897242
r1897244
r1897248
r1897250
r1897260
r1897261
r1897263
r1897386
r1897459
r1897460
r1897461
r1897462
r1897472
r1897543
r1897651
r1897680
r1897689
r1897691
r1897692
Finally (but not chronologically), PCRE version printing:
r1612934
r1612940
r1613189