Merge branch 'master' into quic-latest

* master:
  Select lua context per thread (#7465)
  Fix out of bounds access error in jtest (#7526)
  Disable compiling Inline.cc on macOS (#7389)
  Makes sure the types are correct, avoiding compiler warnings (#7523)
  Move has_request_body to ProxyTransaction (#7499)
  Make the H3 build script work properly on Debian platforms (#7522)
  slice/handleFirstServerHeader: return sooner on requested range errors (#7486)
  Add new log field for negotiated ALPN Protocol ID with the client (#7491)
  Add Outbound PROXY Protocol (v1/v2) Support (#7446)
  Updates the Dockerfile for debian (#7518)
  Disable client inactivity timeout while server is processing POST request (#7309)
  Upgrade Catch.hpp to v2.13.4 (#7464)
  Move reopen_moved_log_files to log flushing thread (#7450)
  replace psutil.pid() with psutil.process_iter() for safer execution (#7515)
  Fix spacing in clang-analyzer.sh script (#7480)
  Fix out of bounds access error in ats_base64_decode (#7490)
  Updated to build lastest versions of Fedora and CentOS docker images (#7505)
  Fix QUIC unit tests build issue on GNU ld (#7496)
  Fix QUIC unit test failures (#7497)
  Fixed build issues with Fedora 34 (#7506)
  Fixing DNS local_ipv* config option (#7507)
  traffic_dump: AuTests to use Proxy Verifier. (#7502)
  Disable ja3 plugin when building with boringssl (#7500)
  Avoid -Warray-bounds on PROXY Protocol Builder (#7488)
  AuTest: Upgrade to Proxy Verifier 2.0.2 (#7493)
  fix certs (#7494)
  Add zlib1g-dev to Debian dependencies in README (#7495)
  Unit Test -  Increase openssl's key size. Place test certs into a common test folder. (#7451)
  Add basic type aliases for std::chrono types to ink_time.h for future use. (#7482)
  traffic_ctl - Fix lookup key for run-root option (#7484)
  update thread config tests (#7370)
  Perf: Replace casecmp with memcmp in HPACK static table lookup (#6521)
  Add PROXY Protocol Builder (#7445)
  Adjust so transfer-encoding header can be treated hop-by-hop (#7473)
  Convert auxkey form 2 uint32_t to 1 uint64_t. (#7350)
  Remove the queuing option from proxy.config.http.per_server.connection (#7302)
  Remove unused function ink_microseconds. (#7481)
  use std::unordered_map to store sessions (#7405)
  drop use of BIO_f_base64 and EVP_PKEY_new_mac_key (#7106)
  Do not write to the cache if the plugin decides not to write to the cache (#7461)
  API to retrieve NoStore set by plugins (#7439)
  Update AuTest version update directions for pipenv (#7469)
  Add command line utility to help convert remap plugin usage to ATS9. (#7426)
  Cleanup: Get rid of MIMEFieldWrapper from HPACK encoding (#6520)
  Proxy Verifier: Making use of delay directives for caching tests. (#7468)
  Cleanup: Add SNIRoutingType (#7453)
  Updating to Proxy Verifier v2.0.0 (#7454)
  Adjust to actually try a server address more than once (#7288)
  Change atoi to atol, causing obvious issues on what needs to be int64's (#7466)
  Cleans up duplicated TSOutboundConnectionMatchType definition (#7090)
  Fixing compress expectation for new microserver (#7463)
  Update to the new MicroServer 1.0.6 release (#7460)
  CacheRead: clear dir entry if doc is found to be truncated (#7064)
  Do not provide a stale negative cache (#7422)
  Generalize SNI support (#6870)
  Add synchronization between UDPNetProcessor::UDPBind in main Thread and initialize_thread_for_udp_net in ET_UDP Thread (#7407)
  Fix heap use after free in DNSProcessor::getby() (#3871)
  Fix comment in include/tscore/Filenames.h. (#7457)
  Fix Makefile target for creating changelogs (#7455)
  Change squid log code for self looping (#7443)
  Enhancements for compress plugin (#7416)
  Add incoming PROXY Protocol v2 support (#7340)
  Cleanup: Remove unused members of NextHopProperty (#7436)
  Small fix to regex_remap PR # 7347. (#7437)
  PoolableSession (#6828)
  option to disable compression for range request's response (#7287)
  Make TSUrlSchemeGet() return scheme implied by URL type when there is no explicit scheme. (#7262)

Makefile.am

@@ -58,7 +58,7 @@ doxygen:
 	@cd doc && $(MAKE) $(AM_MAKEFLAGS) $@
 
 changelog:
-	./tools/changelog.pl apache trafficserver $(VERSION) $(AUTHTOKEN) > CHANGELOG-$(VERSION)
+	./tools/git/changelog.pl -o apache -r trafficserver -m $(VERSION) -a $(AUTHTOKEN) > CHANGELOG-$(VERSION)
 
 asf-dist: asf-distdir
 	tardir=$(distdir) && $(am__tar) --mtime=./configure.ac | bzip2 -9 -c >$(distdir).tar.bz2

README

@@ -101,6 +101,7 @@ plugins to build large scale web applications.
     pkg-config
     libmodule-install-perl
     gcc/g++ or clang/clang++
+    zlib1g-dev
     libssl-dev
     libpcre3-dev
     libcap-dev (optional, highly recommended)

ci/docker/Makefile

@@ -19,40 +19,36 @@
 .DEFAULT_GOAL := help
 
 help:
-	@echo 'fedora26         create ATS docker image for Fedora 26'
-	@echo 'fedora27         create ATS docker image for Fedora 27'
-	@echo 'fedora28         create ATS docker image for Fedora 28'
-	@echo 'fedora29         create ATS docker image for Fedora 29'
-	@echo 'fedora30         create ATS docker image for Fedora 30'
-	@echo 'centos6          create ATS docker image for Centos 6'
+	@echo 'fedora31         create ATS docker image for Fedora 31'
+	@echo 'fedora32         create ATS docker image for Fedora 32'
+	@echo 'fedora33         create ATS docker image for Fedora 33'
+	@echo 'fedora34         create ATS docker image for Fedora 34'
 	@echo 'centos7          create ATS docker image for Centos 7'
+	@echo 'centos8          create ATS docker image for Centos 8'
 	@echo 'all              build all images'
 
 all: fedora centos
 
 # Fedora Docker images
-fedora: fedora26 fedora27 fedora28
+fedora: fedora31 fedora32 fedora33 fedora34
 
-fedora26:
-	docker build -t ats_$@ --build-arg OS_VERSION=26 --build-arg OS_TYPE=fedora yum/
+fedora31:
+	docker build -t ats_$@ --build-arg OS_VERSION=31 --build-arg OS_TYPE=fedora yum/
 
-fedora27:
-	docker build -t ats_$@ --build-arg OS_VERSION=27 --build-arg OS_TYPE=fedora yum/
+fedora32:
+	docker build -t ats_$@ --build-arg OS_VERSION=32 --build-arg OS_TYPE=fedora yum/
 
-fedora28:
-	docker build -t ats_$@ --build-arg OS_VERSION=28 --build-arg OS_TYPE=fedora yum/
+fedora33:
+	docker build -t ats_$@ --build-arg OS_VERSION=33 --build-arg OS_TYPE=fedora yum/
 
-fedora29:
-	docker build -t ats_$@ --build-arg OS_VERSION=29 --build-arg OS_TYPE=fedora yum/
-
-fedora30:
-	docker build -t ats_$@ --build-arg OS_VERSION=30 --build-arg OS_TYPE=fedora yum/
+fedora34:
+	docker build -t ats_$@ --build-arg OS_VERSION=34 --build-arg OS_TYPE=fedora yum/
 
 # Centos Docker images
-centos: centos6 centos7
-
-centos6:
-	docker build -t ats_$@ --build-arg OS_VERSION=6 --build-arg OS_TYPE=centos yum/
+centos: centos7 centos8
 
 centos7:
 	docker build -t ats_$@ --build-arg OS_VERSION=7 --build-arg OS_TYPE=centos yum/
+
+centos8:
+	docker build -t ats_$@ --build-arg OS_VERSION=8 --build-arg OS_TYPE=centos yum/

ci/docker/deb/Dockerfile

@@ -53,7 +53,7 @@ RUN apt-get update; apt-get -y dist-upgrade; \
     git distcc file wget openssl hwloc intltool-debian; \
     # Devel packages that ATS needs
     apt-get -y install libssl-dev libexpat1-dev libpcre3-dev libcap-dev \
-    libhwloc-dev libhwloc5 libunwind8 libunwind8-dev zlib1g-dev \
+    libhwloc-dev libunwind8 libunwind-dev zlib1g-dev \
     tcl-dev tcl8.6-dev libjemalloc-dev libluajit-5.1-dev liblzma-dev \
     libhiredis-dev libbrotli-dev libncurses-dev libgeoip-dev libmagick++-dev; \
     # Optional: This is for the OpenSSH server, and Jenkins account + access (comment out if not needed)

ci/jenkins/bin/autest.sh

@@ -46,6 +46,9 @@ URL="https://ci.trafficserver.apache.org/autest"
 JOB_ID=${ghprbPullId:-${ATS_BRANCH:-master}}
 AUSB="ausb-${JOB_ID}.${BUILD_NUMBER}"
 SANDBOX="/var/tmp/${AUSB}"
+PROXY_VERIFIER_VERSIONS="/home/jenkins/proxy-verifier"
+PROXY_VERIFIER_VERSION_FILE="tests/proxy-verifier-version.txt"
+PROXY_VERIFIER_PREPARE="tests/prepare_proxy_verifier.sh"
 
 # Optional settings
 CCACHE=""
@@ -57,6 +60,7 @@ QUIC=""
 CURL=""
 AUTEST_DEBUG=""
 AUTEST_VERBOSE=""
+PROXY_VERIFIER_ARGUMENT=""
 
 [ "1" == "$enable_ccache" ] && CCACHE="--enable-ccache"
 [ "1" == "$enable_werror" ] && WERROR="--enable-werror"
@@ -86,7 +90,8 @@ set -x
 
 # Configure
 autoreconf -if
-./configure --prefix="${INSTALL}" \
+./configure \
+    --prefix="${INSTALL}" \
     --with-user=jenkins \
     --enable-experimental-plugins \
     --enable-example-plugins \
@@ -111,7 +116,24 @@ AUTEST="/usr/bin/autest"
 [ ! -x ${AUTEST} ] && AUTEST="/usr/local/bin/autest"
 set -x
 
-${AUTEST} -D ./tests/gold_tests --sandbox "$SANDBOX" --ats-bin "${INSTALL}/bin" $AUTEST_DEBUG $AUTEST_VERBOSE
+pv_version=""
+if [ -f "${PROXY_VERIFIER_VERSION_FILE}" ]; then
+  pv_version=`cat "${PROXY_VERIFIER_VERSION_FILE}"`
+elif [ -f "${PROXY_VERIFIER_PREPARE}" ]; then
+  pv_version=`awk -F'"' '/^pv_version/ {print $2}' "${PROXY_VERIFIER_PREPARE}"`
+fi
+if [ "x${pv_version}" != "x" ]; then
+  PROXY_VERIFIER_BIN="${PROXY_VERIFIER_VERSIONS}/${pv_version}/bin"
+  PROXY_VERIFIER_ARGUMENT="--proxy-verifier-bin ${PROXY_VERIFIER_BIN}"
+fi
+
+${AUTEST} \
+    -D ./tests/gold_tests \
+    --sandbox "$SANDBOX" \
+    --ats-bin "${INSTALL}/bin" \
+    $PROXY_VERIFIER_ARGUMENT \
+    $AUTEST_DEBUG \
+    $AUTEST_VERBOSE
 status=$?
 
 set +x

ci/jenkins/bin/clang-analyzer.sh

@@ -47,7 +47,7 @@ if [ "${JOB_NAME#*-github}" != "${JOB_NAME}" ]; then
     ATS_BRANCH="github"
     if [ -w "${OUTPUT_BASE}/${ATS_BRANCH}" ]; then
         output="${OUTPUT_BASE}/${ATS_BRANCH}/${ghprbPullId}"
-        [ ! -d "${output}"] && mkdir "${output}"
+        [ ! -d "${output}" ] && mkdir "${output}"
     fi
     github_pr=" PR #${ghprbPullId}"
     results_url="https://ci.trafficserver.apache.org/clang-analyzer/${ATS_BRANCH}/${ghprbPullId}/"

configure.ac

@@ -1363,7 +1363,7 @@ AC_EGREP_CPP(yes, [
   #endif
   ], [
     AC_MSG_RESULT(yes)
-    AS_IF([test "x${enable_experimental_plugins}" = "xyes"], [
+    AS_IF([test "x${enable_experimental_plugins}" = "xyes" && -z "$openssl_is_boringssl"], [
       enable_ja3_plugin=yes
     ])
   ], [AC_MSG_RESULT(no)])

doc/admin-guide/configuration/proxy-protocol.en.rst

@@ -31,17 +31,12 @@ TLS connections.
 
 .. note::
 
-    The current version only supports transforming client IP from PROXY Version 1
-    header to the Forwarded: header.
-
-In the current implementation, the client IP address in the PROXY protocol header
-is passed to the origin server via an HTTP `Forwarded:
-<https://tools.ietf.org/html/rfc7239>`_ header.
+    The current implementation doesn't support TLV fields of Version 2.
 
 The Proxy Protocol must be enabled on each port.  See
 :ts:cv:`proxy.config.http.server_ports` for information on how to enable the
 Proxy Protocol on a port.  Once enabled, all incoming requests must be prefaced
-with the PROXY v1 header.  Any request not preface by this header will be
+with the PROXY v1/v2 header.  Any request not preface by this header will be
 dropped.
 
 As a security measure, an optional list of trusted IP addresses may be
@@ -50,13 +45,20 @@ configured with :ts:cv:`proxy.config.http.proxy_protocol_allowlist`.
    .. important::
 
        If the allowlist is configured, requests will only be accepted from these
-       IP addresses and must be prefaced with the PROXY v1 header.
+       IP addresses and must be prefaced with the PROXY v1/v2 header.
+
+1. HTTP Forwarded Header
 
-See :ts:cv:`proxy.config.http.insert_forwarded` for configuration information.
+The client IP address in the PROXY protocol header is passed to the origin server via an HTTP `Forwarded:
+<https://tools.ietf.org/html/rfc7239>`_ header. See :ts:cv:`proxy.config.http.insert_forwarded` for configuration information.
 Detection of the PROXY protocol header is automatic.  If the PROXY header
 precludes the request, it will automatically be parse and made available to the
 Forwarded: request header sent to the origin server.
 
+2. Outbound PROXY protocol
+
+See :ts:cv:`proxy.config.http.proxy_protocol_out` for configuration information.
+
 Example
 -------
 

doc/admin-guide/files/records.config.en.rst

@@ -1487,15 +1487,9 @@ Origin Server Connect Attempts
 
    Set a limit for the number of concurrent connections to an upstream server group. A value of
    ``0`` disables checking. If a transaction attempts to connect to a group which already has the
-   maximum number of concurrent connections the transaction either rechecks after a delay or a 503
+   maximum number of concurrent connections a 503
    (``HTTP_STATUS_SERVICE_UNAVAILABLE``) error response is sent to the user agent. To configure
 
-   Number of transactions that can be delayed concurrently
-      See :ts:cv:`proxy.config.http.per_server.connection.queue_size`.
-
-   How long to delay before rechecking
-      See :ts:cv:`proxy.config.http.per_server.connection.queue_delay`.
-
    Upstream server group definition
       See :ts:cv:`proxy.config.http.per_server.connection.match`.
 
@@ -1529,26 +1523,6 @@ Origin Server Connect Attempts
    This setting is independent of the :ts:cv:`setting for upstream session sharing matching
    <proxy.config.http.server_session_sharing.match>`.
 
-.. ts:cv:: CONFIG proxy.config.http.per_server.connection.queue_size INT 0
-   :reloadable:
-
-   Controls the number of transactions that can be waiting on an upstream server group.
-
-   ``-1``
-      Unlimited.
-
-   ``0``
-      Never wait. If the connection maximum has been reached immediately respond with an error.
-
-   A positive number
-      If there are less than this many waiting transactions, delay this transaction and try again. Otherwise respond immediately with an error.
-
-.. ts:cv:: CONFIG proxy.config.http.per_server.connection.queue_delay INT 100
-   :reloadable:
-   :units: milliseconds
-
-   If a transaction is delayed due to too many connections in an upstream server group, delay this amount of time before checking again.
-
 .. ts:cv:: CONFIG proxy.config.http.per_server.connection.alert_delay INT 60
    :reloadable:
    :units: seconds
@@ -1818,6 +1792,21 @@ Proxy User Variables
 
    See :ref:`proxy-protocol` for more discussion on how |TS| transforms the `Forwarded: header`.
 
+.. ts:cv:: CONFIG proxy.config.http.proxy_protocol_out INT ``-1``
+   :reloadable:
+   :overridable:
+
+   Set the behavior of outbound PROXY Protocol.
+
+   =========== ======================================================================
+   Value       Description
+   =========== ======================================================================
+   ``-1``      Disable (default)
+   ``0``       Forward received PROXY protocol to the next hop
+   ``1``       Send client information in PROXY protocol version 1
+   ``2``       Send client information in PROXY protocol version 2
+   =========== ======================================================================
+
 .. ts:cv:: CONFIG proxy.config.http.normalize_ae INT 1
    :reloadable:
    :overridable:
@@ -2596,6 +2585,15 @@ DNS
 
    Maximum number of retries made by |TS| on a given DNS query
 
+.. ts:cv:: CONFIG proxy.config.dns.local_ipv4 STRING NULL
+
+   Local IPV4 address to bind to in order to make DNS requests
+
+.. ts:cv:: CONFIG proxy.config.dns.local_ipv6 STRING NULL
+
+   Local IPV6 address to bind to in order to make DNS requests
+
+
 HostDB
 ======
 

doc/admin-guide/logging/formatting.en.rst

@@ -604,6 +604,7 @@ SSL / Encryption
 .. _cqssv:
 .. _cqssc:
 .. _cqssu:
+.. _cqssa:
 .. _pqssl:
 .. _pscert:
 
@@ -628,6 +629,7 @@ cqssv  Client Request SSL version used to communicate with the client.
 cqssc  Client Request SSL Cipher used by |TS| to communicate with the client.
 cqssu  Client Request SSL Elliptic Curve used by |TS| to communicate with the
                       client when using an ECDHE cipher.
+cqssa  Client Request ALPN Protocol ID negotiated with the client.
 pqssl  Proxy Request  Indicates whether the connection from |TS| to the origin
                       was over SSL or not.
 pscert Proxy Request  1 if origin requested certificate from |TS| during TLS

doc/admin-guide/plugins/compress.en.rst

@@ -103,6 +103,12 @@ versions of the content as :term:`alternates <alternate>`. When set to
 ``false``, |TS| will cache only the compressed or decompressed variant returned
 by the origin. Enabled by default.
 
+range-request
+-------------
+
+When set to ``true``, causes |TS| to compress responses to Range Requests.
+Disabled by default. Setting this to true while setting cache to false leads to delivering corrupted content.
+
 compressible-content-type
 -------------------------
 

doc/appendices/command-line/cvtremappi.en.rst

@@ -0,0 +1,69 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+   or more contributor license agreements.  See the NOTICE file
+   distributed with this work for additional information
+   regarding copyright ownership.  The ASF licenses this file
+   to you under the Apache License, Version 2.0 (the
+   "License"); you may not use this file except in compliance
+   with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing,
+   software distributed under the License is distributed on an
+   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied.  See the License for the
+   specific language governing permissions and limitations
+   under the License.
+
+.. include:: ../../common.defs
+
+.. _cvtremappi:
+
+cvtremappi
+**********
+
+Description
+===========
+
+To help convert your remapping configuration from pre-ATS9 to ATS9 and later.  It may be useful if you use any
+of the core plugins regex_remap.so, header_rewrite.so or gzip.so.  (For this script to work, the python3
+command has to be in your path.)  You can specify where your remap configuration file is with the option:
+
+--filepath FILEPATH
+
+If this parameter is omitted, it defaults to ``./remap.config`` .  The script will make necessary modifications
+to this file, and any files it includes with ``.include`` .  It will change `@plugin=gzip.so` to its new name,
+`@plugin=compress.so` .  When regex_remap.so is invoked as the first remap plugin, it will add the parameter
+@pparam=pristine .  (This makes it work the same as in pre-9 ATS, where the request URL is the pre-remapping
+URL for the first plugin for a remap rule.)  When `header_rewrite.so` is used as a remap plugin, no changes
+are needed in the remap configuration line invoking it.  However, changes may be necessary to the
+configuration files passed to it as parameters.  If a header rewrite configuration file is used for both the
+invocation of header rewrite as the first plugin for remap rules, and for other invocations, it may be
+necessary to generate two new versions of it.  In these cases, the prefix `1st-` is added to file's name,
+for the version used with header rewrite as the first plugin.  If you prefer that a different prefix be added,
+you can specify it with this option:
+
+--prefix PREFIX
+
+If you are also using header rewrite as a global plugin, you should also provide the filepath of the global
+plugin configuration file with this option:
+
+--plugin PLUGIN
+
+(Note that, if the PLUGIN filepath is relative, it should be relative to the directory containing the remap
+configuration file, not relative to the directory the script is run from.  Note also that, if relative paths
+for include files for header rewrite config files appear in the configuration files, they are assumed to be
+relative to the directory containing the remap configuration file.)
+
+Header rewrite previously had some logic that has been eliminated in ATS9.  If a line in a header rewrite
+configuration file relies on this deprecated logic, an error message will be output to standard error.  The
+text `ERROR:` will be prepended to the line in the configuration file causing the error.
+
+The script writes, one per line, a list of the files it is changing or creating to the standard output.  But
+both new and changed files will be written into entirely new files with the suffix `.new` added to the filepath.
+For example, if `remap.config` is changed by the script, it will put the changed version of the file in
+`remap.config.new` .  This gives you a chance to review the changes the script has made.  You can then put the
+changed files into effect with the tool script `insnew`.  This script reads a list of filepaths, one per line,
+from the standard input.  For each filepath `FP`, if it specifies an existing file, it will rename it to
+`FP.old`.  It will then rename the file `FP.new` to `FP`.  This second script should be run from the same
+current directory as the first script was run from.