Merge branch 'master' into quic-latest

* master:
  Fix ALPN support on QUIC connections (#7593)
  fix mem leak in session cache (#7707)
  Parent Select Plugin (#7467)
  Add new TS API function TSUrlRawPortGet. (#7568)
  Add NixOS support (#7697)
  Remove support for --enable-remote-cov-commit (#7700)
  Remove configure-time loopback interface detection (#7702)
  Add sqpv log field for server protocol (#7680)
  Call do_io_close instead of HTTP2_SESSION_EVENT_FINI handler (#7594)
  Fix a bug in tspush that pushes corrupted content to cache (#7696)
  Automatically marks PRs and issues stale (#7675)
  New rate_limit plugin for simple resource limitations (#7623)
  Remove undefined method HttpSM::perform_nca_cache_action (#7692)
  Remove undefined method HttpSM::setup_client_header_nca (#7691)
  Scalar; Move "tag" struct to be inside the "ts" namespace to avoid collisions. (#7690)
  Rollback LAZY_BUF_ALLOC remove in HttpTunnel (#7583)
  Add class to normalize handling of pending action (#7667)
  Make HTTP/2 Curl AuTest gold files case insensitive (#7683)
  Add STL compliant field iteration to MIMEHdr. - rebase. (#7476)
  Fix use of -mcx16 flag - only use if it compiles cleanly. (#7684)
  Refine connection failure logging and messages and eliminate suprious connection errors (#7580)
  Add close header normalize openclose test (#7679)
  Fix has_consumer_besides_client to deal with no clients (#7685)
  create a new cache status RWW_HIT (#7670)
  Updating to AuTest 1.10.0 (#7682)
  sslheaders AuTest: Skip if plugin does not exist (#7678)
  Add AuTest for Background Fill (#7613)
  Do NOT kill tunnel if it has any consumer besides HT_HTTP_CLIENT (#7641)
  AuTest: address various permissions issues (#7668)
  Adding TCP Info header support to header rewrite (#7516)
  Refine Inline.cc carveout for arm64 darwin builds (#7662)
  Comment why log eviction isn't implemented via a log field. (#7648)
  Fixing Throttler.h for older clang and gcc compilers (#7651)
  Update -with-profile and add some profiling documentation (#7601)
  Use correct default value for verify.server.policy (#7636)
  Update server_response_body_bytes when background fill worked (#7621)
  Remove erroneous manager.log mesg with remap include file reload (#7646)
  Change ROUNDUP from function-like macro to function template. (#7614)
  Document http.default_buffer_water_mark (#7612)
  Add proxy.config.cache.log.alternate.eviction (#7629)
  Fix HttpSessionManager::acquireSession from previous rebase error (#7631)
  Fix tls_client_versions and tls_hooks18 tests (#7645)
  Updating documentation for negative_revalidating_lifetime (#7633)
  Remove reference to client.verify.server from tests and other bits (#7639)
  Add pooled_server_connections metric (#7627)
  Expose URL element methods through HTTPHdr (#7628)
  Add default implementation for allow_half_open (#7630)
  Add thread yeield to avoid busy waiting in LogObject::_checkout_write(). (#7576)
  Add proxy.process.http.background_fill_total_count (#7625)
  statichit: misc. fixes (#7634)
  Remove unused variables (#7626)
  Adding negative revalidating AuTests. (#7620)
  Add failed state to hostdb to better track failing origins (#7291)
  Use standard isdigit library function (#7619)
  Typo in output when forcing kqueue for configure (#7617)
  Implement log throttling (#7279)
  Increase Proxy Verifier caching delay. (#7616)
  Set pcre_malloc/free function pointers in core main() only. (#7608)

.github/workflows/stale.yml

@@ -0,0 +1,27 @@
+name: Mark stale issues and pull requests
+
+on:
+  push:
+    paths:
+      - .github/workflows/stale.yml
+  schedule:
+  - cron: "30 1 * * *"
+
+jobs:
+  stale:
+
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/stale@v3
+      with:
+        repo-token: ${{ secrets.GITHUB_TOKEN }}
+        stale-issue-message: 'This issue has been automatically marked as stale because it has not had recent activity. Marking it stale to flag it for further consideration by the community.'
+        stale-pr-message: 'This pull request has been automatically marked as stale because it has not had recent activity. Marking it stale to flag it for further consideration by the community.'
+        stale-issue-label: 'Stale'
+        stale-pr-label: 'Stale'
+        days-before-pr-stale: 90
+        days-before-issue-stale: 365
+        days-before-pr-close: -1
+        days-before-issue-close: -1
+        debug-only: true

build/network.m4

@@ -18,24 +18,6 @@ dnl -----------------------------------------------------------------
 dnl network.m4: Trafficserver's autoconf macros for testing network support
 dnl
 
-dnl
-dnl TS_CHECK_LOOPBACK_IFACE: try to figure out default loopback interface
-dnl
-AC_DEFUN([TS_CHECK_LOOPBACK_IFACE], [
-default_loopback_iface=""
-AC_MSG_CHECKING([for loopback network interface])
-case $host_os in
-  linux*)
-    default_loopback_iface=lo
-  ;;
-darwin* | freebsd* | solaris*)
-  default_loopback_iface=lo0
-  ;;
-esac
-AC_MSG_RESULT([$default_loopback_iface])
-AC_SUBST([default_loopback_iface])
-])
-
 dnl
 dnl Check on IN6_IS_ADDR_UNSPECIFIED. We can't just check if it is defined
 dnl because some releases of FreeBSD and Solaris define it incorrectly.

config.layout

@@ -277,3 +277,23 @@
     logdir:        /var/log+
     cachedir:      /var/lib+
 </Layout>
+
+<Layout NixOS>
+    prefix:        /opt/trafficserver
+    exec_prefix:   ${prefix}
+    bindir:        ${exec_prefix}/bin
+    sbindir:       ${exec_prefix}/sbin
+    libdir:        ${exec_prefix}/lib
+    libexecdir:    ${exec_prefix}/libexec
+    infodir:       ${prefix}/share/info
+    mandir:        ${prefix}/share/man
+    sysconfdir:    /etc+
+    datadir:       ${prefix}/share+
+    docdir:        ${prefix}/share/doc+
+    installbuilddir: ${datadir}/build
+    includedir:    ${prefix}/include
+    localstatedir: /var/lib+
+    runtimedir:    /run+
+    logdir:        /var/log+
+    cachedir:      /var/cache+
+</Layout>

configs/records.config.default.in

@@ -129,6 +129,8 @@ CONFIG proxy.config.cache.ram_cache.size INT -1
 CONFIG proxy.config.cache.ram_cache_cutoff INT 4194304
     # https://docs.trafficserver.apache.org/records.config#proxy-config-cache-limits-http-max-alts
 CONFIG proxy.config.cache.limits.http.max_alts INT 5
+    # https://docs.trafficserver.apache.org/records.config#proxy-config-cache-log-alternate-eviction
+CONFIG proxy.config.cache.log.alternate.eviction INT 0
     # https://docs.trafficserver.apache.org/records.config#proxy-config-cache-max-doc-size
 CONFIG proxy.config.cache.max_doc_size INT 0
 CONFIG proxy.config.cache.min_average_object_size INT 8000

configure.ac

@@ -359,18 +359,6 @@ AC_MSG_RESULT([Will build documentation: $enable_doc_build])
 AM_CONDITIONAL([BUILD_DOCS], [test "xyes" = "x$enable_doc_build"])
 AM_CONDITIONAL([BUILD_MANPAGES], [test "xtrue" = "x$build_manpages"])
 
-#
-# Remote Coverity Prevent commit
-#
-AC_MSG_CHECKING([whether to commit cov defects to remote host])
-AC_ARG_ENABLE([remote-cov-commit],
-  [AS_HELP_STRING([--enable-remote-cov-commit[=HOST]], [commit cov defects to remote host [HOST=localhost]])],
-  [],
-  [enable_remote_cov_commit=localhost]
-)
-AC_MSG_RESULT([$enable_remote_cov_commit])
-AC_SUBST([enable_remote_cov_commit])
-
 #
 # WCCP
 #
@@ -1534,7 +1522,7 @@ AS_IF([test "x$event_interface" = "xauto"], [
       ;;
     xkqueue)
       use_kqueue=1
-      AC_MSG_RESULT([forced to port])
+      AC_MSG_RESULT([forced to kqueue])
       ;;
     *)
       AC_MSG_RESULT([failed])
@@ -1569,6 +1557,8 @@ __saved_CXXFLAGS="${CXXFLAGS}"
 __saved_CFLAGS="${CFLAGS}"
 
 has_128bit_cas=0
+# Don't add the -mcx16 flag unless needed and it compiles cleanly.
+needs_mcx16_for_cas=0
 
 TS_TRY_COMPILE_NO_WARNING([],[
     __int128_t x = 0;
@@ -1587,6 +1577,7 @@ TS_TRY_COMPILE_NO_WARNING([],[
       ], [
         AC_MSG_RESULT(yes)
         has_128bit_cas=1
+        needs_mcx16_for_cas=1
       ], [
         AC_MSG_RESULT(no)
     ])
@@ -1597,11 +1588,9 @@ CFLAGS="${__saved_CFLAGS}"
 AC_LANG_POP
 AC_SUBST(has_128bit_cas)
 
-AS_IF([test "x$has_128bit_cas" = "x1"], [
-  AS_IF([test "x$ax_cv_c_compiler_vendor" != "xintel"], [
+AS_IF([test "x$needs_mcx16_for_cas" = "x1"], [
     TS_ADDTO(AM_CFLAGS, [-mcx16])
     TS_ADDTO(AM_CXXFLAGS, [-mcx16])
-  ])
 ])
 
 # Check for POSIX capabilities library.
@@ -1993,7 +1982,7 @@ if test "x${ac_cv_member_struct_tcp_info_tcpi_data_segs_out}" = "xyes"; then
 fi
 
 if test "x${with_profiler}" = "xyes"; then
-AC_CHECK_HEADERS([google/profiler.h \
+AC_CHECK_HEADERS([gperftools/profiler.h \
                   ], [], [])
 fi
 
@@ -2101,7 +2090,6 @@ AC_SUBST(has_so_mark)
 AC_SUBST(has_ip_tos)
 AC_SUBST(has_so_peercred)
 
-TS_CHECK_LOOPBACK_IFACE
 TS_CHECK_MACRO_IN6_IS_ADDR_UNSPECIFIED
 
 AC_CHECK_TYPE([struct tcp_info],

doc/admin-guide/configuration/cache-basics.en.rst

@@ -583,6 +583,9 @@ To force document caching:
 
 #. Run the command :option:`traffic_ctl config reload` to apply the configuration changes.
 
+
+.. _CachingHttpAlternates:
+
 Caching HTTP Alternates
 =======================
 

doc/admin-guide/files/records.config.en.rst

@@ -1130,6 +1130,17 @@ mptcp
    request bodies which lack a ``Content-length`` header.
 
 .. ts:cv:: CONFIG proxy.config.http.default_buffer_water_mark INT 32768
+   :reloadable:
+   :overridable:
+
+   Number of bytes |TS| is allowed to read ahead of the client from the origin. Note that when
+   :ref:`Read While Write <admin-configuration-reducing-origin-requests>` settings are in place,
+   this setting will apply to the first client to request the object, regardless if subsequent,
+   simultaneous clients of that object can read faster. The buffered bytes will consume memory
+   while waiting for the client to consume them.
+
+   While this setting is reloadable, dramatic changes can cause bigger memory usage than expected
+   and is thus not recommended.
 
 .. ts:cv:: CONFIG proxy.config.http.request_buffer_enabled INT 0
    :overridable:
@@ -1660,14 +1671,35 @@ Negative Response Caching
    to network or HTTP errors. If it is enabled, rather than caching the negative response, the
    current stale content is preserved and served. Note this is considered only on a revalidation of
    already cached content. A revalidation failure means a connection failure or a 50x response code.
+   When considering replying with a stale response in these negative revalidating circumstances,
+   |TS| will respect the :ts:cv:`proxy.config.http.cache.max_stale_age` configuration and will not
+   use a cached response older than ``max_stale_age`` seconds.
 
    A value of ``0`` disables serving stale content and a value of ``1`` enables keeping and serving stale content if revalidation fails.
 
 .. ts:cv:: CONFIG proxy.config.http.negative_revalidating_lifetime INT 1800
 
-   How long, in seconds, to consider a stale cached document valid if
-   :ts:cv:`proxy.config.http.negative_revalidating_enabled` is enabled and |TS| receives a negative
-   (``5xx`` only) response from the origin server during revalidation.
+   When replying with a stale cached response in negative revalidating circumstances (see
+   :ts:cv:`proxy.config.http.negative_revalidating_enabled`), |TS| includes an ``Expires:`` HTTP
+   header field in the cached response with a future time so that upstream caches will not try to
+   revalidate their respective stale objects. This configuration specifies how many seconds in the
+   future |TS| will calculate the value of this inserted ``Expires:`` header field.
+
+   There is a limitation to this method to be aware of: per specification (see IETF RFC 7234,
+   section 4.2.1), ``Cache-Control:`` response directives take precedence over the ``Expires:``
+   header field when determining object freshness. Thus if the cached response contains either a
+   ``max-age`` or an ``s-maxage`` ``Cache-Control:`` response directive, then these directives would
+   take precedence for the upstream caches over the inserted ``Expires:`` field, rendering the
+   ``Expires:`` header ineffective in specifying the configured freshness lifetime.
+
+   Finally, be aware that the only way this configuration is used is as input into calculating the
+   value of these inserted ``Expires:`` header fields. This configuration does not direct |TS|
+   behavior with regard to whether it considers a stale object to be fresh enough to serve out of
+   cache when revalidation fails. As mentioned above in
+   :ts:cv:`proxy.config.http.negative_revalidating_enabled`,
+   :ts:cv:`proxy.config.http.cache.max_stale_age` is used for that determination.
+
+   This configuration defaults to 1,800 seconds (30 minutes).
 
 Proxy User Variables
 ====================
@@ -2076,7 +2108,7 @@ Cache Control
    :reloadable:
    :overridable:
 
-   The maximum age allowed for a stale response before it cannot be cached.
+   The maximum age in seconds allowed for a stale response before it cannot be cached.
 
 .. ts:cv:: CONFIG proxy.config.http.cache.guaranteed_min_lifetime INT 0
    :reloadable:
@@ -2221,6 +2253,22 @@ Cache Control
    The maximum number of alternates that are allowed for any given URL.
    Disable by setting to 0.
 
+.. ts:cv:: CONFIG proxy.config.cache.log.alternate.eviction INT 0
+
+   When enabled (``1``), |TS| will emit a Status level log entry every time an
+   alternate for an object is evicted due to the number of its alternates
+   exceeding the value of :ts:cv:`proxy.config.cache.limits.http.max_alts`. The
+   URI for the evicted alternate is included in the log. This logging may be
+   useful to determine whether :ts:cv:`proxy.config.cache.limits.http.max_alts`
+   is tuned correctly for a given environment. It also provides visibility into
+   alternate eviction for individual objects, which can be helpful for
+   diagnosing unexpected `Vary:` header behavior from particular origins.
+
+   For further details concerning the caching of alternates, see :ref:`Caching
+   HTTP Alternates <CachingHttpAlternates>`.
+
+   By default, alternate eviction logging is disabled (set to ``0``).
+
 .. ts:cv:: CONFIG proxy.config.cache.target_fragment_size INT 1048576
 
    Sets the target size of a contiguous fragment of a file in the disk cache.
@@ -2969,6 +3017,29 @@ Logging Configuration
 
    How often |TS| executes log related periodic tasks, in seconds
 
+.. ts:cv:: CONFIG proxy.config.log.proxy.config.log.throttling_interval_msec INT 60000
+   :reloadable:
+   :units: milliseconds
+
+   The minimum amount of milliseconds between repeated throttled |TS| log
+   events. A value of 0 implies no throttling. Note that for performance
+   reasons only certain logs are compiled with throttling applied to them.
+
+   Throttling is applied to all log events for a particular message which is
+   emitted within its throttling interval. That is, once a throttled log is
+   emitted, none will be emitted until the next log event for that message
+   which occurs outside of this configured interval. As mentioned above, this
+   message is applied not broadly but rather to potentially noisy log messages,
+   such as ones that might occur thousands of times a second under certain
+   error conditions. Once the next log event occurs outside of its interval, a
+   summary message is printed conveying how many messages of that type were
+   throttled since the last time it was emitted.
+
+   It is possible that a log is emitted, followed by more of its type in an
+   interval, then none are emitted after that. Be aware this would result in no
+   summary log message for that interval until the message is emitted again
+   outside of the throttled interval.
+
 .. ts:cv:: CONFIG proxy.config.http.slow.log.threshold INT 0
    :reloadable:
    :units: milliseconds
@@ -3112,6 +3183,17 @@ Diagnostic Logging Configuration
    |TS| plugins will typically log debug messages using the :c:func:`TSDebug`
    API, passing the plugin name as the debug tag.
 
+.. ts:cv:: CONFIG proxy.config.diags.debug.throttling_interval_msec INT 0
+   :reloadable:
+   :units: milliseconds
+
+   The minimum amount of milliseconds between repeated |TS| `diag` and `debug`
+   log events. A value of 0 implies no throttling. All diags and debug logs
+   are compiled with throttling applied to them.
+
+   For details about how log throttling works, see
+   :ts:cv:`log.throttling_interval_msec
+   <proxy.config.log.proxy.config.log.throttling_interval_msec>`.
 
 .. ts:cv:: CONFIG proxy.config.diags.logfile_perm STRING rw-r--r--
 
@@ -3554,7 +3636,7 @@ SSL Termination
 Client-Related Configuration
 ----------------------------
 
-.. ts:cv:: CONFIG proxy.config.ssl.client.verify.server.policy STRING STRICT
+.. ts:cv:: CONFIG proxy.config.ssl.client.verify.server.policy STRING ENFORCED
    :reloadable:
    :overridable:
 

doc/admin-guide/logging/cache-results.en.rst

@@ -102,6 +102,14 @@ TCP_HIT
 A valid copy of the requested object was in the cache and Traffic Server sent
 the object to the client.
 
+.. _crc-tcp-cf-hit:
+
+TCP_CF_HIT
+-------
+
+A valid copy of the requested object is being updated in the cache and Traffic Server sent
+the object to the client.
+
 .. _crc-tcp-miss:
 
 TCP_MISS

doc/admin-guide/logging/formatting.en.rst

@@ -170,7 +170,8 @@ crsc  Proxy Cache    Cache Result Sub-Code. More specific code to complement the
                      Cache Result Code.
 chm   Proxy Cache    Cache Hit-Miss status. Specifies the level of cache from
                      which this request was served by |TS|. Currently supports
-                     only RAM (``HIT_RAM``) vs disk (``HIT_DISK``).
+                     only RAM (``HIT_RAM``), disk (``HIT_DISK``) and Read While Write (``HIT_RWW``).
+                     HIT_RWW could imply either HIT_RAM or HIT_DISK.
 cwr   Proxy Cache    Cache Write Result. Specifies the result of attempting to
                      write to cache: not relevant (``-``), no cache write
                      (``WL_MISS``), write interrupted (``INTR``), error while
@@ -567,6 +568,7 @@ Field Source                Description
 cqhv  Client Request        Client request HTTP version. Deprecated since 9.0.
                             Use ``cqpv`` instead.
 cqpv  Client Request        Client request protocol and version.
+sqpv  Proxy Request         Origin negotiated protocol and version
 csshv Cached Proxy Response Origin server's HTTP version from cached version of
                             the document in |TS| proxy cache.
 sshv  Origin Response       Origin server's response HTTP version.

doc/admin-guide/monitoring/statistics/core/cache.en.rst

@@ -168,12 +168,22 @@ Cache
    :ungathered:
 
 .. ts:stat:: global proxy.process.http.background_fill_current_count integer
+   :type: gauge
    :ungathered:
 
+   Represents the current number of background fill
+
+.. ts:stat:: global proxy.process.http.background_fill_total_count integer
+   :type: counter
+   :ungathered:
+
+   Represents the total number of background fill
+
 .. ts:stat:: global proxy.process.http.cache_deletes integer
 .. ts:stat:: global proxy.process.http.cache_hit_fresh integer
 .. ts:stat:: global proxy.process.http.cache_hit_ims integer
 .. ts:stat:: global proxy.process.http.cache_hit_mem_fresh integer
+.. ts:stat:: global proxy.process.http.cache_hit_rww integer
 .. ts:stat:: global proxy.process.http.cache_hit_revalidated integer
 .. ts:stat:: global proxy.process.http.cache_hit_stale_served integer
 .. ts:stat:: global proxy.process.http.cache_lookups integer
@@ -223,4 +233,3 @@ Cache
 .. ts:stat:: global proxy.process.http.tcp_refresh_miss_count_stat integer
 .. ts:stat:: global proxy.process.http.tcp_refresh_miss_origin_server_bytes_stat integer
 .. ts:stat:: global proxy.process.http.tcp_refresh_miss_user_agent_bytes_stat integer
-

doc/admin-guide/monitoring/statistics/core/http-connection.en.rst

@@ -138,6 +138,12 @@ HTTP Connection
 
    This tracks the number of origin connections denied due to being over the :ts:cv:`proxy.config.http.per_server.connection.max` limit.
 
+.. ts:stat:: global proxy.process.http.pooled_server_connections integer
+   :type: counter
+
+   This metric tracks the number of server connections currently in the server session sharing pools. The server session sharing is
+   controlled by settings :ts:cv:`proxy.config.http.server_session_sharing.pool` and :ts:cv:`proxy.config.http.server_session_sharing.match`.
+
 
 HTTP/2
 ------