Merge branch 'master' into quic-latest

* master: Get rid of code for OpenSSL that has old QUIC API (apache#7599) Fixed warning in gcc 11 about array not being initalized (apache#7840) Don't call next next dup on destroyed mime field mloc. (apache#7833) build_h3_tools: use OpenSSL_1_1_1k+quic (apache#7836) Address assert on captive_action (apache#7807) Fix so EOS are delivered to sessions in the pool (apache#7828) Fix a format specifier for size_t (apache#7830) Fix stall on sending response for request with trailer header (apache#7831) Simplification dir_init_done (apache#7817) Remove unused member from HttpSM (apache#7835) AuTest: use exteneded help output to determin curl feature support (apache#7834) Apply fmt compile time argument checking to log functions (apache#7829) Adds new X-Cache-Info header to the xdebug plugin (apache#7784) Cleanup: Remove unused members of Http2Stream (apache#7813) Cleanup: unused functions of Http2ClientSession (apache#7812) Cancel cross_thread_event on clear_io_events (apache#7815) Cleanup: Remove a meaningless Http2Stream::do_io_close() call (apache#7814) Eliminate next dup call using stale mime field mloc is s3_auth plugin. (apache#7825) NetEvent cleanup - replace #define with constexpr (apache#7804) fix origin session related crashes (apache#7808) Update HTTP version info in HostDB on new outbound connection (apache#7816) Remove a redundant argument (apache#7811) SSL Cert lookup using PP dest ip when ProxyProtocol is enabled (apache#7802) Fix MLoc assert caused by s3auth (apache#7790) Fix cpu utilization problem in session cache (apache#7719) Fix to cookie_remap.cc tp avoid Intel compiler warning. (apache#7792) TSHttpTxnCacheDiskPathGet - tighten up the code a bit. (apache#7806) Doc: tcpinfo plugin table formatting (apache#7805) fix DNS spike issue for TCP_RETRY mode (apache#7307) Adds new TS API TSHttpTxnCacheDiskPathGet (apache#7783) tests: Fixes spelling (apache#7789) Traffic Dump: Add an HTTP/3 AuTest (apache#7758) use sendmsg and recvmsg (apache#7793) HTTP: clean up the http_hdr_describe format error (apache#7797) Fixes an issue where next hop unit tests crash when run on macOS. (apache#7787) Apply log throttling to HTTP/2 session error rate messages (apache#7772) Cleans up uninitialized warning in LogMessage.cc (apache#7788) Short circuit remap reload when a valid remap file is not specified (apache#7782) DNS: Clean up argument passing to DNS queries. (apache#7778) Remove extra verify-callback (apache#7540) Augment test cases for tls_verify_override test (apache#7736) Make when_to_revalidate setting available on HTTPS (apache#7753) Add traffic_server command line option for debugging in Au test. (apache#7762) Test: Update tls_partial_blind_tunnel to have a nameserver. (apache#7773) Test: update tls_forward_nonhttp to have a nameserver. (apache#7774) Test: add nameserver to log-filter test. (apache#7776) BWF: Add support for std::error_code. (apache#7777) Test: add nameserver to log-field test. (apache#7779) Test: add nameserver to regex_remap test. (apache#7775) Elevate privileges for traffic_manager during SSL cert reload (apache#7770) Clean up HTTP version processing (apache#7766) Remove proxy.config.http.down_server.abort_threshold (apache#7748) Remove undocumented keepalive_internal_vc setting (apache#7693) doc: header_rewrite random function not inclusive (apache#7760) Experimental Cache fill plugin (apache#7470) Remove references to removed options (apache#7756) Propagate TLS errors (apache#7714) AuTest extension: check for unrecognized configurations (apache#7752) Fixes errors in the strategies.yaml documentation. (apache#7745) Updates to Nexthop strategies to limit the number of simultaneous (apache#7744) Fixes Issue apache#7739 - Next hop strategy with bad 'to' URL causes TS crash. (apache#7749) header_rewrite: Various fixes for MaxMind support (apache#7746) Remove unused variable is_revalidation_necessary (apache#7747) Fix simple remapping in regex_remap plugin. (apache#7718) Adding DNS TTL AuTests. (apache#7742) Add a chunked disabled test. (apache#7743) Fix monitor threads in lib records to exit on system shutdown. (apache#7731) Add overload for memcpy to take a destination buffer and source string_view / TextView (apache#7732) Test: Add nameserver to TLS tunnel forward test. (apache#7733) AIO_NOT_IN_PROGRESS should not be 0 (apache#7734) if transaction status non-success, bypass intercept plugin (apache#7724) ink_utf8_to_latin1 is not defined, removing declaration (apache#7737) Fix build on FreeBSD 13 (apache#7730) Update VSCode CPP Standard (apache#7723) Updating to use Proxy Verifier 2.2.0 (apache#7729) header_rewrite: Allow for relative path to geo database files (apache#7727) Override proxy.config.ssl.client.sni_policy from sni.yaml (apache#7703) compress.test.py: Reference config file from Test.RunDirectory (apache#7725) Ran clang-tidy over the code (apache#7708) Deny unknown transfer encoding values (apache#7694) Fix doc for http2.no_activity_timeout_in (apache#7721) Add DynamicStats (apache#7704) header_rewrite: allow for use of maxminddb as source of geo truth (apache#7695) Include in parentselectdefs.h in install target (apache#7713) uri_signing: fix warning which affects ubuntu:20.04 builds (apache#7717) Increase the maximum slice block size from 32MB to 128MB (apache#7709)
maskit · May 17, 2021 · 312cf39 · 312cf39
2 parents f90e8dd + 5cdc145
commit 312cf39
Show file tree

Hide file tree

Showing 251 changed files with 4,257 additions and 1,951 deletions.
diff --git a/.gitignore b/.gitignore
@@ -90,6 +90,7 @@ src/tscore/test_X509HostnameValidator
 src/tscore/test_tscore
 src/tscpp/util/test_tscpputil
 lib/records/test_librecords
+lib/records/test_librecords_on_eventsystem
 lib/perl/lib/Apache/TS.pm
 
 iocore/net/test_certlookup

diff --git a/.vscode/c_cpp_properties.json b/.vscode/c_cpp_properties.json
@@ -92,7 +92,9 @@
                 ],
                 "limitSymbolsToIncludedHeaders": true,
                 "databaseFilename": ""
-            }
+            },
+            "cStandard": "c11",
+            "cppStandard": "c++17"
         },
         {
             "name": "Win32",

diff --git a/configs/records.config.default.in b/configs/records.config.default.in
@@ -35,7 +35,6 @@ CONFIG proxy.config.http.insert_response_via_str INT 0
 #    https://docs.trafficserver.apache.org/en/latest/admin-guide/files/parent.config.en.html
 ##############################################################################
 CONFIG proxy.config.http.parent_proxy.retry_time INT 300
-CONFIG proxy.config.http.parent_proxy.connect_attempts_timeout INT 30
 CONFIG proxy.config.http.forward.proxy_auth_to_parent INT 0
 CONFIG proxy.config.http.uncacheable_requests_bypass_parent INT 1
 
@@ -60,7 +59,6 @@ CONFIG proxy.config.http.connect_attempts_max_retries INT 3
 CONFIG proxy.config.http.connect_attempts_max_retries_dead_server INT 1
 CONFIG proxy.config.http.connect_attempts_rr_retries INT 3
 CONFIG proxy.config.http.connect_attempts_timeout INT 30
-CONFIG proxy.config.http.post_connect_attempts_timeout INT 1800
 CONFIG proxy.config.http.down_server.cache_time INT 60
 CONFIG proxy.config.http.down_server.abort_threshold INT 10
 

diff --git a/configure.ac b/configure.ac
@@ -1260,20 +1260,11 @@ AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <openssl/ssl.h>]],
                     AC_CHECK_FUNCS(SSL_set_quic_early_data_enabled)
                     LIBS=$_quic_saved_LIBS
                   ],
-                  [AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <openssl/ssl.h>]],
-                                   [[
-                                     #ifdef SSL_MODE_QUIC_HACK
-                                     #else
-                                     # error no hack for quic
-                                     #endif
-                                   ]])
-                  ],
-                  [AC_MSG_RESULT([yes]); enable_quic=yes; enable_quic_old_api=yes],
-                  [AC_MSG_RESULT([no])])
+                  [
+                    AC_MSG_RESULT([no])
                   ])
 
 AM_CONDITIONAL([ENABLE_QUIC], [test "x$enable_quic" = "xyes"])
-AM_CONDITIONAL([ENABLE_QUIC_OLD_API], [test "x$enable_quic_old_api" = "xyes"])
 TS_ARG_ENABLE_VAR([use], [quic])
 AC_SUBST(use_quic)
 
@@ -1685,27 +1676,66 @@ AC_SUBST(use_hwloc)
 #
 AC_CHECK_HEADERS([GeoIP.h], [
   AC_CHECK_LIB([GeoIP], [GeoIP_new], [
-    AC_SUBST([GEO_LIBS], ["-lGeoIP"])
+    AC_SUBST([GEOIP_LIBS], ["-lGeoIP"])
+    AC_SUBST(has_geoip, 1)
   ], [
-    AC_SUBST([GEO_LIBS], [""])
+    AC_SUBST([GEOIP_LIBS], [""])
+    AC_SUBST(has_geoip, 0)
   ])
 ])
 
+AM_CONDITIONAL([HAS_GEOIP], [test "x${has_geoip}" = "x1" ])
+
 #
 # Check for libmaxmind.  This is the maxmind v2 API where GeoIP is the legacy
 # v1 dat file based API
 #
 AC_CHECK_HEADERS([maxminddb.h], [
   AC_CHECK_LIB([maxminddb], [MMDB_open], [
     AC_SUBST([MAXMINDDB_LIBS], ["-lmaxminddb"])
-    AC_SUBST(has_maxmind, 1)
+    AC_SUBST(has_maxminddb, 1)
   ], [
     AC_SUBST([MAXMINDDB_LIBS], [""])
-    AC_SUBST(has_maxmind, 0)
+    AC_SUBST(has_maxminddb, 0)
   ])
 ])
 
-AM_CONDITIONAL([BUILD_MAXMIND_ACL_PLUGIN], [test "x${has_maxmind}" = "x1" ])
+AM_CONDITIONAL([HAS_MAXMINDDB], [test "x${has_maxminddb}" = "x1" ])
+
+AC_ARG_WITH([hrw-geo-provider],
+  [AS_HELP_STRING([--with-hrw-geo-provider=geoip|maxminddb],[geo provider to use with header_rewrite [default=auto] ])],
+  [geo_provider=$withval],
+  [geo_provider="auto"]
+)
+use_hrw_geoip=0
+use_hrw_maxminddb=0
+
+AS_IF([test "x$geo_provider" = "xauto"], [
+  if test "x$has_geoip" = "x1"; then
+    use_hrw_geoip=1
+    AC_MSG_NOTICE([Using GeoIP interface for header_rewrite])
+  elif test "x$has_maxminddb" = "x1"; then
+    use_hrw_maxminddb=1
+    AC_MSG_NOTICE([Using MaxMindDB interface for header_rewrite])
+  fi
+],[
+  case "x$geo_provider" in
+    xgeoip)
+      use_hrw_geoip=1
+      AC_MSG_RESULT([forced to GeoIP])
+      ;;
+    xmaxminddb)
+      use_hrw_maxminddb=1
+      AC_MSG_RESULT([forced to MaxMindDB])
+      ;;
+    *)
+      AC_MSG_RESULT([failed])
+      AC_MSG_FAILURE([unknown geo interface $geo_provider])
+  esac
+])
+
+AC_SUBST(use_hrw_geoip)
+AC_SUBST(use_hrw_maxminddb)
 
 # Right now, the healthcheck plugins requires inotify_init (and friends)
 AM_CONDITIONAL([BUILD_HEALTHCHECK_PLUGIN], [ test "$ac_cv_func_inotify_init" = "yes" ])

diff --git a/doc/admin-guide/files/records.config.en.rst b/doc/admin-guide/files/records.config.en.rst
@@ -1574,13 +1574,6 @@ Origin Server Connect Attempts
 
    Specifies how long (in seconds) |TS| remembers that an origin server was unreachable.
 
-.. ts:cv:: CONFIG proxy.config.http.down_server.abort_threshold INT 10
-   :reloadable:
-   :overridable:
-
-   The number of seconds before |TS| marks an origin server as unavailable after a client abandons a request
-   because the origin server was too slow in sending the response header.
-
 .. ts:cv:: CONFIG proxy.config.http.uncacheable_requests_bypass_parent INT 1
    :reloadable:
    :overridable:
@@ -2609,6 +2602,13 @@ DNS
    ``2`` TCP_ONLY:  |TS| always talks to nameservers over TCP.
    ===== ======================================================================
 
+.. ts:cv:: CONFIG proxy.config.dns.max_tcp_continuous_failures INT 10
+
+   If DNS connection mode is TCP_RETRY, set the threshold of the continuous TCP
+   query failures count for the TCP connection, reset the TCP connection immediately
+   if the continuous TCP query failures conut over the threshold. If the threshold
+   is 0 (or less than 0) we close this feature.
+
 .. ts:cv:: CONFIG proxy.config.dns.max_dns_in_flight INT 2048
 
    Maximum inflight DNS queries made by |TS| at any given instant
@@ -3914,7 +3914,6 @@ HTTP/2 Configuration
 
 .. ts:cv:: CONFIG proxy.config.http2.accept_no_activity_timeout INT 120
    :reloadable:
-   :overridable:
 
    Specifies how long |TS| keeps connections to clients open if no
    activity is received on the connection. Lowering this timeout can ease
@@ -3923,7 +3922,6 @@ HTTP/2 Configuration
 
 .. ts:cv:: CONFIG proxy.config.http2.no_activity_timeout_in INT 120
    :reloadable:
-   :overridable:
 
    Specifies how long |TS| keeps connections to clients open if a
    transaction stalls. Lowering this timeout can ease pressure on the proxy if

diff --git a/doc/admin-guide/files/sni.yaml.en.rst b/doc/admin-guide/files/sni.yaml.en.rst
@@ -114,6 +114,10 @@ client_key                The file containing the client private key that corres
                           |TS| tries to use a private key in client_cert.  Otherwise,
                           :ts:cv:`proxy.config.ssl.client.private_key.filename` is used.
 
+client_sni_policy         Policy of SNI on outbound connection.
+
+                          If not specified, the value of :ts:cv:`proxy.config.ssl.client.sni_policy` is used.
+
 http2                     Indicates whether the H2 protocol should be added to or removed from the
                           protocol negotiation list.  The valid values are :code:`on` or :code:`off`.
 

diff --git a/doc/admin-guide/files/strategies.yaml.en.rst b/doc/admin-guide/files/strategies.yaml.en.rst
@@ -76,7 +76,7 @@ Example::
         - scheme: https
           port: 443
           health_check_url: https://192.168.1.1:443
-  	- &p2
+    - &p2
       host: p2.foo.com
       protocol:
         - scheme: http
@@ -98,23 +98,22 @@ The field definitions in the examples below are defined in the **hosts** section
 Example using **YAML** anchors and references::
 
   groups:
-  	- &g1
+    - &g1
       - <<: *p1
-      	weight: 1.5
+        weight: 1.5
       - <<: *p2
-      	weight: 0.5
-  	- &g2
+        weight: 0.5
+    - &g2
       - <<: *p3
-      	weight: 0.5
+        weight: 0.5
       - <<: *p4
         weight: 1.5
 
 Explicitly defined Example, no **YAML** references::
 
   groups:
     - &g1
-      - p1
-        host: p1.foo.com
+      - host: p1.foo.com
         protocol:
           - scheme: http
             port: 80
@@ -123,8 +122,7 @@ Explicitly defined Example, no **YAML** references::
             port: 443
             health_check_url: https://192.168.1.1:443
         weight: 0.5
-      - p2
-        host: p2.foo.com
+      - host: p2.foo.com
         protocol:
           - scheme: http
             port: 80
@@ -134,8 +132,7 @@ Explicitly defined Example, no **YAML** references::
             health_check_url: https://192.168.1.2:443
         weight: 0.5
     - &g2
-      - p3
-        host: p3.foo.com
+      - host: p3.foo.com
         protocol:
           - scheme: http
             port: 80
@@ -144,8 +141,7 @@ Explicitly defined Example, no **YAML** references::
             port: 443
             health_check_url: https://192.168.1.3:443
         weight: 0.5
-      - p4
-        host: p4.foo.com
+      - host: p4.foo.com
         protocol:
           - scheme: http
             port: 80

diff --git a/doc/admin-guide/monitoring/statistics/core/dns.en.rst b/doc/admin-guide/monitoring/statistics/core/dns.en.rst
@@ -35,6 +35,18 @@ DNS
 
    The number of DNS lookups currently in progress.
 
+.. ts:stat:: global proxy.process.dns.tcp_retries integer
+   :type: gauge
+   :ungathered:
+
+   The number of DNS query over TCP in TCP_RETRY connection mode.
+
+.. ts:stat:: global proxy.process.dns.tcp_reset integer
+   :type: gauge
+   :ungathered:
+
+   The number of resetting TCP connection in TCP_RETRY connection mode.
+
 .. ts:stat:: global proxy.process.dns.lookup_avg_time integer
    :type: derivative
    :units: milliseconds

diff --git a/doc/admin-guide/monitoring/statistics/core/ssl.en.rst b/doc/admin-guide/monitoring/statistics/core/ssl.en.rst
@@ -91,12 +91,18 @@ SSL/TLS
 .. ts:stat:: global proxy.process.ssl.ssl_session_cache_hit integer
    :type: counter
 
+.. ts:stat:: global proxy.process.ssl.ssl_origin_session_cache_hit integer
+   :type: counter
+
 .. ts:stat:: global proxy.process.ssl.ssl_session_cache_lock_contention integer
    :type: counter
 
 .. ts:stat:: global proxy.process.ssl.ssl_session_cache_miss integer
    :type: counter
 
+.. ts:stat:: global proxy.process.ssl.ssl_origin_session_cache_miss integer
+   :type: counter
+
 .. ts:stat:: global proxy.process.ssl.ssl_session_cache_new_session integer
    :type: counter
 

diff --git a/doc/admin-guide/plugins/cache_fill.en.rst b/doc/admin-guide/plugins/cache_fill.en.rst
@@ -0,0 +1,46 @@
+.. Licensed to the Apache Software Foundation (ASF) under one or more contributor license
+   agreements.  See the NOTICE file distributed with this work for additional information regarding
+   copyright ownership.  The ASF licenses this file to you under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with the License.  You may obtain
+   a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software distributed under the License
+   is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
+   or implied.  See the License for the specific language governing permissions and limitations
+   under the License.
+
+.. _admin-plugins-cache-fill.so:
+.. include:: /common.defs
+
+Cache Fill Plugin
+***********************
+
+The speed of the response served from the cache depends on the cache speed and the client filling the object.
+This dependency could significantly impact all the clients requesting the object.
+This plugin tries to eliminate the dependence by making the original request spawn a background request to fill the cache.
+The initial version of this plugin relays the initial request to the origin server instead of waiting for the background request to start filling the cache as there is no easier way to find the wait time.
+This plugin doesn't provide any improvement for smaller objects but could also degrade the performance as two outgoing requests for every cache update.
+
+
+Using the plugin
+----------------
+
+This plugin functions as a per remap plugin.
+
+To activate the plugin, in :file:`remap.config`, simply append the
+below to the specific remap line::
+
+   @plugin=cache_fill.so @pparam=<config-file>
+
+Functionality
+-------------
+
+Plugin decides to trigger a background fetch of the original (Client) request if the request/response is cacheable and cache status is TS_CACHE_LOOKUP_MISS/TS_CACHE_LOOKUP_HIT_STALE.
+
+Future additions
+----------------
+
+*  Fetching the original request from the cache.
+
diff --git a/doc/admin-guide/plugins/header_rewrite.en.rst b/doc/admin-guide/plugins/header_rewrite.en.rst
@@ -82,11 +82,16 @@ This plugin may be enabled globally, so that the conditions and header
 rewriting rules are evaluated for every request made to your |TS| instance.
 This is done by adding the following line to your :file:`plugin.config`::
 
-  header_rewrite.so config_file_1.conf config_file_2.conf ...
+  header_rewrite.so [--geo-db-path=path/to/geoip.db] config_file_1.conf config_file_2.conf ...
 
 You may specify multiple configuration files. Their rules will be evaluated in
 the order the files are listed.
 
+The plugin takes an optional switch ``--geo-db-path``. If MaxMindDB support has
+been compiled in, use this switch to point at your .mmdb file. This also applies to
+the remap context.
+
+
 Enabling Per-Mapping
 --------------------
 
@@ -219,7 +224,7 @@ GEO
     cond %{GEO:<part>} <operand>
 
 Perform a GeoIP lookup of the client-IP, using a 3rd party library and
-DB. Currently only the MaxMind GeoIP API is supported. The default is to
+DB. Currently the MaxMind GeoIP and MaxMindDB APIs are supported. The default is to
 do a Country lookup, but the following qualifiers are supported::
 
     %{GEO:COUNTRY}      The country code (e.g. "US")
@@ -424,7 +429,7 @@ RANDOM
 
     cond %{RANDOM:<n>} <operand>
 
-Generates a random integer between ``0`` and ``<n>``, inclusive.
+Generates a random integer from ``0`` up to (but not including) ``<n>``. Mathmatically, ``[0,n)`` or ``0 <= r < n``.
 
 STATUS
 ~~~~~~
@@ -495,7 +500,7 @@ TCP-INFO
 ~~~~~~~~
 ::
 
-	cond %{<name>}
+    cond %{<name>}
         add-header @PropertyName "%{TCP-INFO}"
 
 This operation records TCP Info struct field values as an Internal remap as well as global header at the event hook specified by the condition. Supported hook conditions include TXN_START_HOOK, SEND_RESPONSE_HEADER_HOOK and TXN_CLOSE_HOOK in the Global plugin and REMAP_PSEUDO_HOOK, SEND_RESPONSE_HEADER_HOOK and TXN_CLOSE_HOOK in the Remap plugin. Conditions supported as request headers include TXN_START_HOOK and REMAP_PSEUDO_HOOK. The other conditions are supported as response headers. TCP Info fields currently recorded include rtt, rto, snd_cwnd and all_retrans. This operation is not supported on transactions originated within Traffic Server (for e.g using the |TS| :c:func:`TSHttpTxnIsInternal`)

diff --git a/doc/admin-guide/plugins/index.en.rst b/doc/admin-guide/plugins/index.en.rst
@@ -147,6 +147,7 @@ directory of the |TS| source tree. Experimental plugins can be compiled by passi
    :hidden:
 
    Access Control <access_control.en>
+   Cache Fill <cache_fill.en>
    Certifier <certifier.en>
    Cert Reporting Tool <cert_reporting_tool.en>
    Collapsed-Forwarding <collapsed_forwarding.en>