Merge ASF 9.2.x into our Edge branch (apache#721)

* add a metric to track how often the range seek bug is detected (apache#8970)

Co-authored-by: Chris McFarlen <cmcfarlen@apple.com>
(cherry picked from commit b23e8a0)

* Fix reverting PR#7302 (apache#8975)

PR#7302 was reverted by PR#8316 as an incompatible change for 9.2.0.
It looks like the revert commit has a mistake that made a crash by
calling `HttpSM::send_origin_throttled_response()` twice.

(cherry picked from commit 3cccd2d)

* Fixes issue with file size calculation for existing logs (apache#8971)

* Issue arises with existing log files at startup

* Because the existing bytes are not accounted for, log rolling does not occur at the correct time

* Existing code can lead to logging being suspended indefinitely without manual intervention if thresholds are exceeded and no rolled log files can be deleted

* Corner case more evident when other data not rolled by ATS is present in the logging directory

(cherry picked from commit 6225b12)

* Proxy Verifier: Update to version 2.4.1 (apache#8965)

This updates the Proxy Verifier version used by our AuTests to version
v2.4.1. This Proxy Verifier version prints the ALPN used by the proxy
(ATS in our case) to the server. This will allow us to verify ALPN
behavior for an upcoming PR for HTTP/2 to origin.

(cherry picked from commit fb02ef8)

* Destroy ssl context after use. (apache#8531)

As per the docs this needs to be released after use, this was missing from the cert_reporting_tool plugin.
This also fixes the example in the docs.

(cherry picked from commit 57015b7)

* Extend milestone api time tracking to remap. (apache#8520)

(cherry picked from commit a9405ac)

* Add 5xx's to be allowed to be used for simple retries (apache#8518)

* Add 5xx's to be allowed to be used for simple retries

Remove unnecessary functions in transact for finding ranges

Change PS response checking to not use internal state. Now pass in retry type and code

(cherry picked from commit 30096b4)

* Updated ChangeLog

* Pin flask to version 2.1.3 (apache#9008)

This resolves an AuTest Pipenv package dependency conflict for Werkzeug,
which is used by httpbin. Latest versions of flask require newer
versions of flask which conflicts with our pin to keep httpbin working.

(cherry picked from commit 46c1a0a)

* Add back validatation that the scheme matches the wire protocol (apache#9005)

This adds back in the scheme and wire protocol check (see apache#8465) along
with a configuration to be able to disable the check if the verification
is not desired.

(cherry picked from commit 7ec147e)

* Ignore POST request case from a check for background fill (apache#9013)

(cherry picked from commit 1f3e111)

* Add content length mismatch check on handling HEADERS frame and CONTINUATION frame (apache#9012)

* Add content length mismatch check on handling HEADERS frame and CONTINUATION frame

* Correct error class of HTTP/2 malformed requests

(cherry picked from commit e921228)

* Restrict unknown scheme of HTTP/2 request (apache#9010)

Strictly following RFC 3986 Section 3.1

```
scheme      = ALPHA *( ALPHA / DIGIT / "+" / "-" / "." )
```

(cherry picked from commit c56f872)

* Fail fast on HTTP/2 header validation (apache#9009)

Co-authored-by: Masakazu Kitajo <maskit@apache.org>
(cherry picked from commit eaef5e8)

* Add stack guard pages (apache#8996)

Use r/w protected pages on top of stacks to guard against stack
overflow.  The number of VM pages to use for guarding can be set
via the config option proxy.config.thread.default.stackguard_pages.

(cherry picked from commit 1abf6c0)

* Fix compile on M1 Mac (apache#8999)

Add arm64 to the list of known stack growth directions.

(cherry picked from commit 697da39)

* Add RangeTransform::m_write_vio state checks (apache#8980)

(cherry picked from commit e912ece)

* Update slice to only prefetch when first block is miss/hit-stale (apache#8890)

* Update slice to only prefetch when first block is miss/hit-stale

* Remove extra line spaces & generalize autest output

* Verify cont is valid

* Use xdebug to only prefetch when first block is cacheable with miss/hit-stale status

* Precompile via regex pattern in config

* Remove plugin dependency, add cache status header between slice and crr

* Only enable prefetching from CRR on 206 partial case from origin

* Update header type, fix 206 case for 304 in CRR, allow header to be used for debugging

* update header val type

Co-authored-by: Serris Lew <lserris@apple.com>
(cherry picked from commit f14cce4)

 Conflicts:
	doc/admin-guide/plugins/slice.en.rst

* Setup UA consumer only if ua_entry is not nullptr (apache#8949)

(cherry picked from commit cbe0bea)

* Update roadmap doc with latest releases (apache#8977)

(cherry picked from commit 80a0ff9)

* Report an error if configure can't find zlib (apache#8446)

(cherry picked from commit a100761)

* Add thread safety to PendingAction operations. (apache#8443)

(cherry picked from commit 29a5092)

* Doc: Add proxy.config.cacvhe.mutex_retry_delay (apache#8376)

(cherry picked from commit 3ad1587)

* test_MMH: fix memory leak in unit test (apache#8357)

(cherry picked from commit 0eccef0)

* crash fix (apache#8268)

(cherry picked from commit cd1139b)

* Fix length bug in validate_unmapped_url_path (apache#8080)

(cherry picked from commit ac16a3b)

* Updated ChangeLog

Co-authored-by: Chris McFarlen <chris@mcfarlen.us>
Co-authored-by: Masaori Koshiba <masaori@apache.org>
Co-authored-by: Jeff Elsloo <elsloo@users.noreply.github.com>
Co-authored-by: Brian Neradt <brian.neradt@gmail.com>
Co-authored-by: Damian Meden <damian.meden@gmail.com>
Co-authored-by: Alan M. Carroll <amc@apache.org>
Co-authored-by: Evan Zelkowitz <eze@apache.org>
Co-authored-by: Leif Hedstrom <zwoop@apache.org>
Co-authored-by: Mo Chen <uncorrupt@gmail.com>
Co-authored-by: Serris Lew <serrisnlew@gmail.com>
Co-authored-by: Matt Williams <gh@mattyw.net>
Co-authored-by: Bryan Call <bcall@apache.org>
Co-authored-by: Brian Olsen <bnolsen@gmail.com>
Co-authored-by: Fei Deng <duke8253@gmail.com>
Co-authored-by: bneradt <bneradt@yahooinc.com>

CHANGELOG-9.2.0

@@ -199,6 +199,7 @@ Changes with Apache Traffic Server 9.2.0
   #8075 - Add plugin parent_select reloading
   #8076 - Fix plugin parent_select missing null checks.
   #8078 - Clean up constructors and initialization in LogAccess class.
+  #8080 - Fix length bug in validate_unmapped_url_path
   #8083 - Cancel active timeout when releasing session
   #8084 - Defer canceling UA inactivity timeout for chunked bodies too
   #8085 - Add cache condition in header_rewrite plugin for lookup results
@@ -226,7 +227,6 @@ Changes with Apache Traffic Server 9.2.0
   #8187 - Ran clang-tidy over the master branch
   #8188 - lua: add ts.is_debug_tag_set function
   #8190 - Add TSMgmtConfigFileAdd api doc
-  #8191 - Fix error connection logging crash
   #8194 - Remove global g_rec_config_contents_llq
   #8201 - Fix H2 logic when setting EOS flag on DATA frame
   #8204 - transaction data sink: only stream body bytes
@@ -260,6 +260,7 @@ Changes with Apache Traffic Server 9.2.0
   #8264 - Prefetch plugin: add  --fetch-query option
   #8266 - Do not turn off cache for internal requests
   #8267 - Align TS API Au test with Yahoo internal version.
+  #8268 - Avoid chunking in the post redirect case
   #8269 - Fix leaks in ConfigManager::configName
   #8271 - Add warning message about why enhance your calm is being sent
   #8274 - Use better done file locations for some AuTests
@@ -303,6 +304,7 @@ Changes with Apache Traffic Server 9.2.0
   #8348 - remove unused RecConfigFileEntry from RecConfigParse
   #8354 - Adds support for TCP_NOTSENT_LOWAT sockopt
   #8356 - free sessions when timeout
+  #8357 - test_MMH: fix memory leak in unit test
   #8358 - Locking around SSLSecret::secret_map access
   #8359 - Fix regex_remap plugin redirection.
   #8360 - Stabilize regex_revalidate Au test.
@@ -313,6 +315,7 @@ Changes with Apache Traffic Server 9.2.0
   #8372 - Add Au test for pqsi and pqsp log fields.
   #8374 - 9.2.x autest fixes
   #8375 - Add a set-http-cntl operator for header_rewrite
+  #8376 - Doc: Add proxy.config.cache.mutex_retry_delay
   #8381 - Fix a potential H2 stall
   #8392 - Update the feature lists available for autest
   #8395 - Added metrics to the rate limit plugin and document the new options
@@ -334,7 +337,9 @@ Changes with Apache Traffic Server 9.2.0
   #8435 - Fix unused-but-set-variable warnings
   #8440 - Update the Multiplexer Docs for Multplexed HTTPS Connections
   #8441 - bigobj: use automake to build test utilities
+  #8443 - Add thread safety to PendingAction operations.
   #8445 - Include CPPFLAGS when building CompileParseRules
+  #8446 - Report an error if configure can't find zlib
   #8449 - Add AuTest for HTTP CONNECT
   #8450 - Document proxy.config.memory.max_usage
   #8453 - Ignore ECONNABORTED on blocking accept
@@ -360,9 +365,11 @@ Changes with Apache Traffic Server 9.2.0
   #8508 - Makes proxy_serve_stale version independent
   #8509 - Marks tls_engine test as OpenSSL-only
   #8517 - Remove commented out EventProcessor::eventthread
+  #8518 - Add 5xx's to be allowed to be used for simple retries
+  #8520 - Extend milestone api time tracking to remap.
   #8523 - url_sig: change cleanup msg from TSError to TSDebug
-  #8524 - Set an appropriate callback function for OpenSSL 3
   #8525 - Remove unused function "ats_mallopt"
+  #8531 - Destroy ssl context after use.
   #8532 - Adds autest to test error cases loading ssl_multicert.config
   #8537 - docs: fix fedora install notes and spelling issues
   #8540 - Enable conf_remap_float test and remove special case for local config file
@@ -482,6 +489,7 @@ Changes with Apache Traffic Server 9.2.0
   #8882 - Make ats_scoped_obj a synonym for std::unique_ptr .
   #8884 - Upgrade to Proxy Verifier 2.4.0.
   #8886 - Add escape json for logging
+  #8890 - Update slice to only prefetch when first block is miss/hit-stale
   #8892 - Cleanup SNIConfig
   #8897 - Removed references to the throttle option from the slice plugin. (#8373)
   #8898 - slice and cache_range_requests: allow header override (#8666)
@@ -490,7 +498,6 @@ Changes with Apache Traffic Server 9.2.0
   #8909 - Move HKDF to OpenSSL 3 interfaces
   #8924 - Update location for core rule set in modsecurity example
   #8925 - Cleanup. Remove unnecessary use of a memory arena when logging.
-  #8926 - Allows errors from plugin initialization to bubble up
   #8927 - Add docs for remap_stats plugin
   #8928 - proxy_serve_stale: Test updates
   #8931 - Fix clang-format installation with multiple threads
@@ -499,4 +506,20 @@ Changes with Apache Traffic Server 9.2.0
   #8942 - Fix doc formatting for plugin remap_stats
   #8943 - Fix doc formatting for rate_limit plugin
   #8944 - Make the autopep8 clang-format targets quieter
+  #8949 - Setup UA consumer only if ua_entry is not nullptr
   #8950 - .fit/fmt/.clang-format-installed prerequisite
+  #8965 - Proxy Verifier: Update to version 2.4.1
+  #8970 - add a metric to track how often the range seek bug is detected
+  #8971 - Fixes issue with file size calculation for existing logs
+  #8975 - Fix reverting PR#7302
+  #8977 - Update roadmap doc with latest releases
+  #8980 - Add RangeTransform::m_write_vio state checks
+  #8996 - Add stack guard pages
+  #8999 - Fix compile on M1 Mac
+  #9005 - Add back validatation that the scheme matches the wire protocol
+  #9008 - Pin flask to version 2.1.3
+  #9009 - Fail fast on HTTP/2 header validation
+  #9010 - Restrict unknown scheme of HTTP/2 request
+  #9011 - Add control char check in MIME Parser
+  #9012 - Add content length mismatch check on handling HEADERS frame and CONTINUATION frame
+  #9013 - Ignore POST request case from a check for background fill

configure.ac

@@ -1389,6 +1389,9 @@ AM_CONDITIONAL([BUILD_JA3_PLUGIN], [test "x${enable_ja3_plugin}" = "xyes"])
 #
 # Check for zlib presence and usability
 TS_CHECK_ZLIB
+if test "x${enable_zlib}" != "xyes"; then
+  AC_MSG_ERROR([Cannot find zlib library. Configure --with-zlib=DIR])
+fi
 
 #
 # Check for lzma presence and usability

doc/admin-guide/files/parent.config.en.rst

@@ -214,7 +214,7 @@ The following list shows the possible actions and their allowed values.
 
 ``parent_retry``
     - ``simple_retry`` - If the parent returns a 404 response or if the response matches
-      a list of http 4xx responses defined in ``simple_server_retry_responses`` on a request
+      a list of http 4xx and/or 5xx responses defined in ``simple_server_retry_responses`` on a request
       a new parent is selected and the request is retried.  The number of retries is controlled
       by ``max_simple_retries`` which is set to 1 by default.
     - ``unavailable_server_retry`` - If the parent returns a 503 response or if the response matches
@@ -224,11 +224,17 @@ The following list shows the possible actions and their allowed values.
     - ``both`` - This enables both ``simple_retry`` and ``unavailable_server_retry`` as described above.
     - If not set, by default all response codes will be considered a success, and parents will not be retried based on any HTTP response code.
 
+    .. Note::
+
+        If a response code exists in both the simple and unavailable lists and both
+        is the retry type then simple_retry will take precedence and unavailable_server_retry
+        will not be used for that code.
+
 .. _parent-config-format-simple-server-retry-responses:
 
 ``simple_server_retry_responses``
    If ``parent_retry`` is set to either ``simple_retry`` or ``both``, this parameter is a comma separated list of
-   http 4xx response codes that will invoke the ``simple_retry`` described in the ``parent_retry`` section. By
+   http 4xx and/or 5xx response codes that will invoke the ``simple_retry`` described in the ``parent_retry`` section. By
    default, ``simple_server_retry_responses`` is set to 404.
 
 .. _parent-config-format-unavailable-server-retry-responses:

doc/admin-guide/files/records.config.en.rst

@@ -2401,6 +2401,17 @@ Cache Control
    write vector. For further details on cache write vectors, refer to the
    developer documentation for :cpp:class:`CacheVC`.
 
+.. ts::cv:: CONFIG proxy.config.cache.mutex_retry_delay INT 2
+   :reloadable:
+   :units: milliseconds
+
+   The retry delay for missing a lock on a mutex in the cache component. This is used generically
+   for most locks, except those that have an explicit configuration for the retry delay. For
+   instance, if the cache component is notifying another continuation of a cache event and fails to
+   get the lock for that continuation, it will use this as the delay for the retry. This is also
+   used from the asynchronous IO threads when IO finishes and the ``CacheVC`` lock or stripe lock is
+   required.
+
 RAM Cache
 =========
 

doc/admin-guide/plugins/slice.en.rst

@@ -113,13 +113,6 @@ The slice plugin supports the following options::
         Requires setting up an intermediate loopback remap rule.
         -r for short
 
-    --prefetch-count=<int> (optional)
-        Default is 0
-        Prefetches successive 'n' slice block requests in the background
-        and cached. Especially for large objects, prefetching can improve
-        cache miss latency.
-        -f for short
-
     --skip-header=<header name> (default: X-Slicer-Info)
         Header name used by the slice plugin after the loopback
         to indicate that the slice plugin should be skipped.
@@ -133,6 +126,14 @@ The slice plugin supports the following options::
         `cache_range_requests` plugin.
         -i for short
 
+    --prefetch-count=<int> (optional)
+        Default is 0
+        Prefetches successive 'n' slice block requests in the background
+        and caches (with `cache_range_requests` plugin). Prefetching is only
+        enabled when first block is a cacheable object with miss or hit-stale status.
+        Especially for large objects, prefetching can improve cache miss latency.
+        -f for short
+
 Examples::
 
     @plugin=slice.so @pparam=--blockbytes=1000000 @plugin=cache_range_requests.so

doc/developer-guide/api/functions/TSSslClientContext.en.rst

@@ -59,4 +59,9 @@ Server source distribution. It demonstrates how to use :func:`TSSslClientContext
 
 .. literalinclude:: ../../../../example/plugins/c-api/client_context_dump/client_context_dump.cc
   :language: c
-  :lines: 137-145
+  :lines: 154-166
+
+
+.. literalinclude:: ../../../../example/plugins/c-api/client_context_dump/client_context_dump.cc
+  :language: c
+  :lines: 51-55

doc/release-notes/roadmap.en.rst

@@ -26,8 +26,10 @@ ATS Release Roadmap
    :maxdepth: 1
 
 Apache Traffic Server is currently on a two year major release cycle. At any
-given time, we guarantee to have two major versions supported. Currently,
-this means v7.1.x and v8.0.x, with a possible v8.1.x coming.
+given time, we guarantee to have two major versions supported. Please refer to
+our `download page <https://trafficserver.apache.org/downloads>`_ to see the
+current supported versions.
+
 
 For details on the actual developer release process, See
 

include/tscore/PendingAction.h

@@ -73,13 +73,15 @@ class PendingAction
    *
    * @param action @c Action to check.
    *
+   * @return @c true if the action was cleared, @c false if not.
+   *
    * This clears the internal pointer without any side effect. it is used when the @c Action
    * is handled and therefore should no longer be canceled.
    */
-  void clear_if_action_is(Action *action);
+  bool clear_if_action_is(Action *action);
 
 private:
-  Action *pending_action = nullptr;
+  std::atomic<Action *> pending_action = nullptr;
 };
 
 inline bool
@@ -91,22 +93,32 @@ PendingAction::empty() const
 inline PendingAction &
 PendingAction::operator=(Action *action)
 {
-  // Apparently HttpSM depends on not canceling the previous action if anew
+  // Apparently @c HttpSM depends on not canceling the previous action if a new
   // one completes immediately. Canceling the contained action in that case
-  // cause the HttpSm to permanently stall.
+  // cause the @c HttpSM to permanently stall.
   if (ACTION_RESULT_DONE != action) {
-    if (action != pending_action && pending_action != nullptr) {
-      pending_action->cancel();
+    Action *expected; // Need for exchange, and to load @a pending_action only once.
+    // Avoid race conditions - for each assigned action, ensure exactly one thread
+    // cancels it. Assigning @a expected in the @c while expression avoids potential
+    // races if two calls to this method have the same @a action.
+    while ((expected = pending_action) != action) {
+      if (pending_action.compare_exchange_strong(expected, action)) {
+        // This thread did the swap and now no other thread can get @a expected which
+        // must be canceled if it's not @c nullptr and then this thread is done.
+        if (expected != nullptr) {
+          expected->cancel();
+        }
+        break;
+      }
     }
-    pending_action = action;
   }
   return *this;
 }
 
 inline Continuation *
 PendingAction::get_continuation() const
 {
-  return pending_action ? pending_action->continuation : nullptr;
+  return pending_action ? pending_action.load()->continuation : nullptr;
 }
 
 inline Action *
@@ -118,14 +130,20 @@ PendingAction::get() const
 inline PendingAction::~PendingAction()
 {
   if (pending_action) {
-    pending_action->cancel();
+    pending_action.load()->cancel();
   }
 }
 
-inline void
+inline bool
 PendingAction::clear_if_action_is(Action *action)
 {
-  if (action == pending_action) {
-    pending_action = nullptr;
+  if (action != nullptr) {
+    while (action == pending_action) {
+      if (pending_action.compare_exchange_strong(action, nullptr)) {
+        // do NOT cancel - this is called when the event is handled.
+        return true;
+      }
+    }
   }
+  return false;
 }

iocore/cache/Cache.cc

@@ -3086,6 +3086,7 @@ register_cache_stats(RecRawStatBlock *rsb, const char *prefix)
   REG_INT("read.active", cache_read_active_stat);
   REG_INT("read.success", cache_read_success_stat);
   REG_INT("read.failure", cache_read_failure_stat);
+  REG_INT("read.seek.failure", cache_read_seek_fail_stat);
   REG_INT("write.active", cache_write_active_stat);
   REG_INT("write.success", cache_write_success_stat);
   REG_INT("write.failure", cache_write_failure_stat);

iocore/cache/CacheRead.cc

@@ -749,6 +749,8 @@ CacheVC::openReadMain(int /* event ATS_UNUSED */, Event * /* e ATS_UNUSED */)
 
       doc->magic = DOC_CORRUPT;
 
+      CACHE_INCREMENT_DYN_STAT(cache_read_seek_fail_stat);
+
       CACHE_TRY_LOCK(lock, vol->mutex, mutex->thread_holding);
       if (!lock.is_locked()) {
         SET_HANDLER(&CacheVC::openReadDirDelete);

iocore/cache/P_CacheInternal.h

@@ -113,6 +113,7 @@ enum {
   cache_read_active_stat,
   cache_read_success_stat,
   cache_read_failure_stat,
+  cache_read_seek_fail_stat,
   cache_write_active_stat,
   cache_write_success_stat,
   cache_write_failure_stat,