Merge branch 'master' into quic-latest

* master:
  Doc: Fix typo in negative_revalidating_lifetime (apache#7427)
  Change comment handling for long lines in url_sig plugin (apache#7421)
  Add unit tests for PROXY Protocol v1 parser (apache#7332)
  LGTM: Remove superfluous const qualifier in return type (apache#7412)
  Fix issue with unavailable server retry codes (apache#7410)
  Remove the warning statement (apache#7414)
  default to throttling and subsequently simplify the transfer code (apache#7257)
  Improvement to lua plugin (apache#7413)
  Make places to bind/unbind SSL object with/from NetVC (apache#7399)
  traffic_ctl - plugin msg  now require only the tag as mandatory field data field is now optional. (apache#7364)
  API - Add new api function TSHttpTxnServerSsnTransactionCount() to retrieve the number of transactions between TS proxy and the origin server from a single session. (apache#7387)
  Fix clang compiler complaint about an unused parameter in SNIAction. (apache#7409)
  Add compression support to stats_over_http (apache#7393)
  Doc: Fix INPUT tag of Doxyfile (apache#7404)
  Remove unneeded variables in UnixNetVConnection (apache#7403)
  Correctly pass back errno to HttpSM (apache#7402)
  Reverting to old negative_caching conditional behavior (apache#7401)
  Remove unused MAYBE_ABORT state (apache#7400)
  traffic_manager should not retry on disk failure (apache#7397)
  Eliminate dangling pointer into stack space. (apache#7392)
  This PR aims to address some of the lock contention found and (apache#7377)
  Remove a special treatment for SSLNetVC in migrateToCurrentThread() (apache#7384)
  Replace ::exit() with _exit() to avoid secondary cleanup cores (apache#7395)
  [Doc] Fix build warnings (apache#7391)
  Clear call_sm on tunnel reset (apache#7352)
  Unused code: HostDBContinuation::removeEvent (apache#7383)
  Traffic Dump: Fix stream-id printing after first transaction. (apache#7311)
  Add comments to ink_queue.h. (apache#7376)
  Cleanup incoming PROXY Protocol v1 (apache#7331)
  In CI, only run autopep8 on branches that enforce autopep8 (apache#7270)
  Fix FreeBSD 12 link issue in test_libhttp2. (apache#7367)
  Adjust flags to ensure tunnel producer is cleaned up (apache#7336)
  Cleanup: Remove SSL Wire Trace releated code in UnixNetVConnection (apache#7368)
  Use EVP MAC API if available (apache#7363)
  Use EVP API instead of MD5_Init/Update/Final (secure_link plugin) (apache#7355)
  Use ERR_get_error_all if available (apache#7354)
  Use OpeSSL EVP API instead of SHA256_Init/Update/Final (apache#7342)
  Cleanup: Get rid of NetVConnection::outstanding() (apache#7366)
  Cleanup: Remove unused functions (apache#7365)
  Add a post case to the conn_timeout test (apache#7334)
  Fix sni ip_allow and host_sni_policy (apache#7349)
  AuTest for Split DNS (apache#7325)
  Make reloading client certificate configuration more reliable (apache#7313)
  Add negative caching tests and fixes. (apache#7361)
  ESI: Ensure gzip header is always initialized (apache#7360)
  Allow for regex_remap of pristine URL. (apache#7347)
  Set thread mutex to the DNSHandler mutex of SplitDNS (apache#7321)
  Fix lookup split dns rule with fast path (apache#7320)
  Add note to background fetch about include/exclude (apache#7343)
  AuTest for incoming PROXY Protocol v1 (apache#7326)
  Fix vc close migration race condition (apache#7337)
  TLS Session Reuse: Downgrade add_session messages to debug (apache#7345)
  TLS Session Reuse: Downgrade noisy log to debug (apache#7344)
  Remove the last remnants of the enable_url_expandomatic (apache#7276)
  Remove unnecessary cast from ReverseProxy. (apache#7329)
  Updates the Dockerfile with more packages (apache#7323)
  fixup in HttpSM to only set [TS_MILESTONE_SERVER_CLOSE if TS_MILESTONE_SERVER_CONNECT has been set (apache#7259)
  Add option for hybrid global and thread session pools (apache#6978)
  Get appropriate locks on SSN_START hook delays (apache#7295)
  s3_auth: demote noisy errors around configuration that doesn't affect plugin usability (apache#7306)
  Follow the comments in I_Thread.h, add an independent ink_thread_key for EThread. (apache#6288)
  Reduce the number of write operation on H2 (apache#7282)

.gitignore

@@ -94,6 +94,7 @@ lib/perl/lib/Apache/TS.pm
 
 iocore/net/test_certlookup
 iocore/net/test_UDPNet
+iocore/net/test_libinknet
 iocore/net/quic/test_QUIC*
 iocore/aio/test_AIO
 iocore/eventsystem/test_IOBuffer

build/crypto.m4

@@ -318,3 +318,50 @@ AC_DEFUN([TS_CHECK_EARLY_DATA], [
 
   AC_SUBST(has_tls_early_data)
 ])
+
+dnl
+dnl Since OpenSSL 1.1.1
+dnl
+dnl SSL_CTX_set_tlsext_ticket_key_evp_cb function is for OpenSSL 3.0
+dnl SSL_CTX_set_tlsext_ticket_key_cb macro is for OpenSSL 1.1.1
+dnl SSL_CTX_set_tlsext_ticket_key_cb function is for BoringSSL
+AC_DEFUN([TS_CHECK_SESSION_TICKET], [
+  _set_ssl_ctx_set_tlsext_ticket_key_evp_cb_saved_LIBS=$LIBS
+
+  TS_ADDTO(LIBS, [$OPENSSL_LIBS])
+  AC_CHECK_HEADERS(openssl/ssl.h)
+  session_ticket_check=no
+  has_tls_session_ticket=0
+  AC_MSG_CHECKING([for SSL_CTX_set_tlsext_ticket_key_cb macro])
+  AC_COMPILE_IFELSE(
+    [AC_LANG_PROGRAM([[#include <openssl/ssl.h>]],
+                     [[
+                     #ifndef SSL_CTX_set_tlsext_ticket_key_cb
+                     #error
+                     #endif
+                     ]])
+    ],
+    [
+      AC_DEFINE(HAVE_SSL_CTX_SET_TLSEXT_TICKET_KEY_CB, 1, [Whether SSL_CTX_set_tlsext_ticket_key_cb is available])
+      session_ticket_check=yes
+      has_tls_session_ticket=1
+    ],
+    []
+  )
+  AC_MSG_RESULT([$session_ticket_check])
+  AC_CHECK_FUNCS(
+    SSL_CTX_set_tlsext_ticket_key_evp_cb SSL_CTX_set_tlsext_ticket_key_cb,
+    [
+      session_ticket_check=yes
+      has_tls_session_ticket=1
+    ],
+    []
+  )
+
+  LIBS=$_set_ssl_ctx_set_tlsext_ticket_key_evp_cb_saved_LIBS
+
+  AC_MSG_CHECKING([for session ticket support])
+  AC_MSG_RESULT([$session_ticket_check])
+
+  AC_SUBST(has_tls_session_ticket)
+])

ci/docker/deb/Dockerfile

@@ -53,7 +53,7 @@ RUN apt-get update; apt-get -y dist-upgrade; \
     git distcc file wget openssl hwloc intltool-debian; \
     # Devel packages that ATS needs
     apt-get -y install libssl-dev libexpat1-dev libpcre3-dev libcap-dev \
-    libhwloc-dev libhwloc5 libunwind8 libunwind8-dev \
+    libhwloc-dev libhwloc5 libunwind8 libunwind8-dev zlib1g-dev \
     tcl-dev tcl8.6-dev libjemalloc-dev libluajit-5.1-dev liblzma-dev \
     libhiredis-dev libbrotli-dev libncurses-dev libgeoip-dev libmagick++-dev; \
     # Optional: This is for the OpenSSH server, and Jenkins account + access (comment out if not needed)

ci/docker/yum/Dockerfile

@@ -52,7 +52,7 @@ RUN yum -y update; \
     # Devel packages that ATS needs
     yum -y install openssl-devel expat-devel pcre-devel libcap-devel hwloc-devel libunwind-devel \
     xz-devel libcurl-devel ncurses-devel jemalloc-devel GeoIP-devel luajit-devel brotli-devel \
-    ImageMagick-devel ImageMagick-c++-devel hiredis-devel \
+    ImageMagick-devel ImageMagick-c++-devel hiredis-devel zlib-devel \
     perl-ExtUtils-MakeMaker perl-Digest-SHA perl-URI; \
     # This is for autest stuff
     yum -y install python3 httpd-tools procps-ng nmap-ncat pipenv \

ci/jenkins/bin/clang-format.sh

@@ -44,8 +44,12 @@ autoreconf -if && ./configure
 ${ATS_MAKE} clang-format
 [ "0" != "$?" ] && exit 1
 
-${ATS_MAKE} autopep8
-[ "0" != "$?" ] && exit 1
+# Only enforce autopep8 on branches where the pre-commit hook was updated to
+# check it. Otherwise, none of the PRs for older branches will pass this check.
+if grep -q autopep8 tools/git/pre-commit; then
+    ${ATS_MAKE} autopep8
+    [ "0" != "$?" ] && exit 1
+fi
 
 git diff --exit-code
 [ "0" != "$?" ] && exit 1

configure.ac

@@ -1298,6 +1298,9 @@ TS_CHECK_CRYPTO_SET_CIPHERSUITES
 # Check for openssl early data support
 TS_CHECK_EARLY_DATA
 
+# Check for openssl session ticket support
+TS_CHECK_SESSION_TICKET
+
 saved_LIBS="$LIBS"
 TS_ADDTO([LIBS], ["$OPENSSL_LIBS"])
 
@@ -1307,6 +1310,7 @@ AC_CHECK_FUNCS([ \
   CRYPTO_set_mem_functions \
   HMAC_CTX_new \
   X509_get0_signature \
+  ERR_get_error_all \
 ])
 
 AC_CHECK_FUNC([ASN1_STRING_get0_data], [],

doc/Doxyfile

@@ -746,11 +746,12 @@ WARN_LOGFILE           =
 # Note: If this tag is empty the current directory is searched.
 
 INPUT                  = mainpage.doc \
+                         ../include \
                          ../iocore \
-                         ../lib/cppapi/include/atscppapi \
                          ../lib/records \
-                         ../lib/ts \
-                         ../proxy
+                         ../mgmt \
+                         ../proxy \
+                         ../src
 
 # This tag can be used to specify the character encoding of the source files
 # that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses

doc/admin-guide/files/records.config.en.rst

@@ -994,8 +994,20 @@ mptcp
    ========== =================================================================
    ``global`` Re-use sessions from a global pool of all server sessions.
    ``thread`` Re-use sessions from a per-thread pool.
+   ``hybrid`` Try to work as a global pool, but release server sessions to the
+              per-thread pool if there is lock contention on the global pool.
    ========== =================================================================
 
+
+   Setting :ts:cv:`proxy.config.http.server_session_sharing.pool` to global can reduce
+   the number of connections to origin for some traffic loads.  However, if many
+   execute threads are active, the thread contention on the global pool can reduce the
+   lifetime of connections to origin and reduce effective origin connection reuse.
+
+   For a hybrid pool, the operation starts as the global pool, but sessons are returned
+   to the local thread pool if the global pool lock is not acquired rather than just
+   closing the origin connection as is the case in standard global mode.
+
 .. ts:cv:: CONFIG proxy.config.http.attach_server_session_to_client INT 0
    :overridable:
 
@@ -1652,11 +1664,9 @@ Negative Response Caching
    ====================== =====================================================
    ``204``                No Content
    ``305``                Use Proxy
-   ``400``                Bad Request
    ``403``                Forbidden
    ``404``                Not Found
    ``414``                URI Too Long
-   ``405``                Method Not Allowed
    ``500``                Internal Server Error
    ``501``                Not Implemented
    ``502``                Bad Gateway
@@ -1674,7 +1684,7 @@ Negative Response Caching
    How long (in seconds) |TS| keeps the negative responses  valid in cache. This value only affects negative
    responses that do NOT have explicit ``Expires:`` or ``Cache-Control:`` lifetimes set by the server.
 
-.. ts:cv:: CONFIG proxy.config.http.negative_caching_list STRING 204 305 403 404 405 414 500 501 502 503 504
+.. ts:cv:: CONFIG proxy.config.http.negative_caching_list STRING 204 305 403 404 414 500 501 502 503 504
    :reloadable:
 
    The HTTP status code for negative caching. Default values are mentioned above. The unwanted status codes can be
@@ -1693,7 +1703,7 @@ Negative Response Caching
 
 .. ts:cv:: CONFIG proxy.config.http.negative_revalidating_lifetime INT 1800
 
-   How long, in seconds, to consider a stale cached document valid if If
+   How long, in seconds, to consider a stale cached document valid if
    :ts:cv:`proxy.config.http.negative_revalidating_enabled` is enabled and |TS| receives a negative
    (``5xx`` only) response from the origin server during revalidation.
 
@@ -1884,7 +1894,7 @@ Security
 
    You can override this global setting on a per domain basis in the :file:`sni.yaml` file using the :ref:`host_sni_policy attribute<override-host-sni-policy>` action.
 
-   Currently, only the verify_client policy is checked for host name and SNI matching.
+   Currently, only the verify_client and ip_allow policies are checked for host name and SNI matching.
 
 Cache Control
 =============
@@ -3901,6 +3911,27 @@ HTTP/2 Configuration
    Clients that send smaller window increments lower than this limit will be immediately disconnected with an error
    code of ENHANCE_YOUR_CALM.
 
+.. ts:cv:: CONFIG proxy.config.http2.write_buffer_block_size INT 262144
+   :reloadable:
+
+   Specifies the size of a buffer block that is used for buffering outgoing
+   HTTP/2 frames. The size will be rounded up based on power of 2.
+
+.. ts:cv:: CONFIG proxy.config.http2.write_size_threshold FLOAT 0.5
+   :reloadable:
+
+   Specifies the size threshold for triggering write operation for sending HTTP/2
+   frames. The default value is 0.5 and it measn write operation is going to be
+   triggered when half or more of the buffer is occupied.
+
+.. ts:cv:: CONFIG proxy.config.http2.write_time_threshold INT 100
+   :reloadable:
+   :units: milliseconds
+
+   Specifies the time threshold for triggering write operation for sending HTTP/2
+   frames. Write operation will be triggered at least once every this configured
+   number of millisecond regardless of pending data size.
+
 HTTP/3 Configuration
 ====================
 

doc/admin-guide/performance/index.en.rst

@@ -495,7 +495,7 @@ Error responses from origins are consistent and costly
 If error responses are costly for your origin server to generate, you may elect
 to have |TS| cache these responses for a period of time. The default behavior is
 to consider all of these responses to be uncacheable, which will lead to every
-client request to result in an origin request.
+client request resulting in an origin request.
 
 This behavior is controlled by both enabling the feature via
 :ts:cv:`proxy.config.http.negative_caching_enabled` and setting the cache time
@@ -504,7 +504,7 @@ status code for negative caching can be set with :ts:cv:`proxy.config.http.negat
 
     CONFIG proxy.config.http.negative_caching_enabled INT 1
     CONFIG proxy.config.http.negative_caching_lifetime INT 10
-    CONFIG proxy.config.http.negative_caching_list STRING 204 305 403 404 405 414 500 501 502 503 504
+    CONFIG proxy.config.http.negative_caching_list STRING 204 305 403 404 414 500 501 502 503 504
 
 SSL-Specific Options
 ~~~~~~~~~~~~~~~~~~~~

doc/admin-guide/plugins/background_fetch.en.rst

@@ -74,6 +74,16 @@ The contents of the config-file could be as below::
    exclude X-Foo-Bar text
    exclude Content-Length <1000
 
+.. important::
+
+   The ``include`` configuration directive is only used when there is a corresponding ``exclude`` to exempt.
+   For example, a single line directive, ``include Host example.com`` would not make the plugin
+   *only* act on example.com. To acheive classic allow (only) lists, one would need to have a broad
+   exclude line, such as::
+
+      exclude Host *
+      include Host example.com
+
 The plugin also now supports per remap activation. To activate the plugin for a given remap, add the
 below on the remap line::
 

doc/admin-guide/plugins/esi.en.rst

@@ -80,11 +80,11 @@ Enabling ESI
 
 2. There are four options you can add to the above.
 
-- "--private-response" will add private cache control and expires header to the processed ESI document.
-- "--packed-node-support" will enable the support for using packed node, which will improve the performance of parsing
+- ``--private-response`` will add private cache control and expires header to the processed ESI document.
+- ``--packed-node-support`` will enable the support for using packed node, which will improve the performance of parsing
   cached ESI document.
-- "--disable-gzip-output" will disable gzipped output, which will NOT gzip the output anyway.
-- "--first-byte-flush" will enable the first byte flush feature, which will flush content to users as soon as the entire
+- ``--disable-gzip-output`` will disable gzipped output, which will NOT gzip the output anyway.
+- ``--first-byte-flush`` will enable the first byte flush feature, which will flush content to users as soon as the entire
   ESI document is received and parsed without all ESI includes fetched (the flushing will stop at the ESI include markup
   till that include is fetched).
 

doc/admin-guide/plugins/header_rewrite.en.rst

@@ -482,6 +482,15 @@ Refer to `Requests vs. Responses`_ for more information on determining the
 context in which the transaction's URL is evaluated.  The ``<part>`` may be
 specified according to the options documented in `URL Parts`_.
 
+SSN-TXN-COUNT
+~~~~~~~~~~~~~
+::
+
+    cond %{SSN-TXN-COUNT} <operand>
+
+Returns the number of transactions between the Traffic Server proxy and the origin server from a single session.
+Any value greater than zero indicates connection reuse.
+
 Condition Operands
 ------------------
 

doc/admin-guide/plugins/lua.en.rst

@@ -2977,7 +2977,7 @@ ts.http.transaction_count
 
 **context:** do_remap/do_os_response or do_global_* or later
 
-**description:** This function returns the number of transaction in this connection
+**description:** This function returns the number of transaction in this client connection
 
 Here is an example
 
@@ -2991,6 +2991,16 @@ Here is an example
 
 :ref:`TOP <admin-plugins-ts-lua>`
 
+ts.http.server_transaction_count
+--------------------------------
+**syntax:** *ts.http.server_transaction_count()*
+
+**context:** do_remap/do_os_response or do_global_* or later
+
+**description:** This function returns the number of transaction in this server connection
+
+:ref:`TOP <admin-plugins-ts-lua>`
+
 ts.http.redirect_url_set
 ------------------------
 **syntax:** *ts.http.redirect_url_set()*
@@ -4235,6 +4245,21 @@ ts.mgmt.get_string
 
 :ref:`TOP <admin-plugins-ts-lua>`
 
+ts.mgmt.add_config_file
+-----------------------
+**syntax:** *ts.mgmt.add_config_file(parent, filename)*
+
+**context:** do_remap/do_os_response or do_global_* or later.
+
+**description:** This function invokes ``TSMgmtConfigFileAdd`` API.
+
+::
+
+    remap = ts.mgmt.get_string('proxy.config.url_remap.filename')
+    ts.mgmt.add_config_file(remap, '/etc/my.config')
+
+:ref:`TOP <admin-plugins-ts-lua>`
+
 ts.stat_create
 --------------
 **syntax:** *val = ts.stat_create(STAT_NAME, RECORDDATA_TYPE, PERSISTENT, SYNC)*

doc/admin-guide/plugins/regex_remap.en.rst

@@ -54,6 +54,16 @@ profile dump, you can do ::
     $ sudo touch remap.config
     $ sudo traffic_ctl config reload
 
+By default, this plugin operates on the post-remap URL (including any
+remappings done by preceding plugins in the remap rule).  This behavior
+can be modified with the optional parameter ::
+
+    @pparam=[no-]pristine            [default: off]
+
+With ``@pparam=pristine``, the plugin will operate on the pre-remap, or
+pristine, URL.  (But, if no regular expression in the config file is
+matched, the resulting URL will still be the post-remap URL.)
+
 By default, only the path and query string of the URL are provided for
 the regular expressions to match. The following optional parameters can
 be used to modify the plugin instance behavior ::

doc/appendices/command-line/traffic_ctl.en.rst

@@ -235,7 +235,9 @@ traffic_ctl plugin
    :cpp:enumerator:`TSLifecycleHookID::TS_LIFECYCLE_MSG_HOOK` will receive a callback for that hook.
    The :arg:`TAG` and :arg:`DATA` will be available to the plugin hook processing. It is expected
    that plugins will use :arg:`TAG` to select relevant messages and determine the format of the
-   :arg:`DATA`.
+   :arg:`DATA`. The :arg:`DATA` is optional and may not be available to consume, if not available then size will be 0
+   and the data will be NULL. Any extra passed value beside the tag and the optional data will be ignored.
+   Check :c:type:`TSPluginMsg` for more info.
 
 traffic_ctl host
 ----------------

doc/conf.py

@@ -179,6 +179,9 @@
                   ('cpp:identifier', 'F'),  # template arg
                   ('cpp:identifier', 'Args'),  # variadic template arg
                   ('cpp:identifier', 'Rest'),  # variadic template arg
+                  ('c:type', 'uint64_t'),
+                  ('c:type', 'uint8_t'),
+                  ('c:type', 'int32_t')
                   ]
 
 # Autolink issue references.