bug: Verification of upstream TLS certificates for L4 (stream) protocols is not performed

[deekshith-hadil]

Current Behavior

When APISIX is configured as a TCP (stream) proxy and the upstream is configured with tls.verify = true and a client certificate, APISIX successfully establishes a connection to the upstream even if the configured CA for verifying the upstream certificate is invalid. In other words, APISIX is not verifying the upstream server's TLS certificate for L4 (stream) routes and allows the connection to proceed regardless of verification failure.

Expected Behavior

When upstream.tls.verify is true, APISIX should validate the upstream server certificate against the configured CA (client.ca referenced by client_cert_id) and abort/close the connection if verification fails (e.g., if the CA does not match).

Actual behavior

APISIX does not reject the upstream TLS connection when an invalid CA is provided; data is successfully proxied to the upstream despite the CA mismatch.

Error Logs

No Error logs

Steps to Reproduce

1. Install APISIX using the Helm chart:

helm repo add apisix https://apache.github.io/apisix-helm-chart
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
kubectl create ns apisix
helm install apisix apisix/apisix \
  --set etcd.enabled=true \
  --set apisix.enableIPv6=false \
  --set ingress-controller.enabled=false \
  --set apisix.admin.allow.ipList="" \
  --set service.type=LoadBalancer \
  --set apisix.admin.type=LoadBalancer \
  --set service.stream.enabled=true \
  --set 'service.stream.tcp[0]=5443' \
  -n apisix

2. Generate certificates:

# Root CA1 (used to sign upstream server)
openssl genrsa -out ca.key 2048
openssl req -new -sha256 -key ca.key -out ca.csr -subj "/CN=ROOTCA"
openssl x509 -req -days 36500 -sha256 -extensions v3_ca -signkey ca.key -in ca.csr -out ca.cer

# Client certificate (self signed)
openssl req -x509 -newkey rsa:2048 -sha256 -days 36500 -nodes -keyout mtls-client.key -out mtls-client.cer -subj "/CN=MTLSCLIENT" -addext "extendedKeyUsage = clientAuth"

3. Create a client SSL object in APISIX referencing CA1 (initially correct CA):

curl -X PUT 'http://127.0.0.1:9180/apisix/admin/ssls/5443' \
  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' \
  -H 'Content-Type: application/json' \
  -d '{
    "sni": "test.com",
    "cert": "'"$(cat mtls-client.cer)"'",
    "key": "'"$(cat mtls-client.key)"'",
    "type": "client",
    "client": {
        "ca": "'"$(cat ca.cer)"'"
    }
  }'

4. Create a stream_route with upstream.tls.verify = true and client_cert_id pointing to the SSL object:

curl -X PUT http://127.0.0.1:9180/apisix/admin/stream_routes/5443 \
  -H "X-API-KEY: edd1c9f034335f136f87ad84b625c8f1" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "mtlstest",
    "server_port": 5443,
    "upstream": {
      "name": "mtlsupstream",
      "desc": "Upstream for the example service",
      "scheme": "tls",
      "nodes": [
        {
          "host": "10.12.130.244",
          "port": 5000,
          "weight": 100,
          "priority": 0
        }
      ],
      "tls": {
        "verify": true,
        "client_cert_id": 5443
      }
    }
  }'

5. Start an upstream TLS server (no client-cert verification on upstream for this test because of known issue bug: mTLS stream data to external server, APISIX not sending client certificate #12472):

# Create server certificates
# Server certificate (CN=test.com), signed by CA1
openssl genrsa -out server.key 2048
openssl req -new -sha256 -key server.key -out server.csr -subj "/CN=test.com"
openssl x509 -req -days 36500 -sha256 -extensions v3_req -CA ca.cer -CAkey ca.key -CAserial ca.srl -CAcreateserial -in server.csr -out server.cer

# Run server
openssl s_server -accept 5000 -cert server.cer -key server.key

6. Send raw TCP data to APISIX (APISIX should proxy via TLS to upstream):

echo 'test' | nc 127.0.0.1 5443

• At this point, data is received by the upstream server (expected).

7. Now patch the client SSL in APISIX to use the wrong CA (CA2) that does not match the upstream certificate:

# Generate new CA certificate
# Root CA2 (different CA, NOT used to sign client/server certs)
openssl genrsa -out ca2.key 2048
openssl req -new -sha256 -key ca2.key -out ca2.csr -subj "/CN=ROOTCA"
openssl x509 -req -days 36500 -sha256 -extensions v3_ca -signkey ca2.key -in ca2.csr -out ca2.cer


curl -X PUT 'http://127.0.0.1:9180/apisix/admin/ssls/5443' \
  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' \
  -H 'Content-Type: application/json' \
  -d '{
    "sni": "test.com",
    "cert": "'"$(cat mtls-client.cer)"'",
    "key": "'"$(cat mtls-client.key)"'",
    "type": "client",
    "client": {
        "ca": "'"$(cat ca2.cer)"'"
    }
  }'

8. Repeat sending TCP data:

echo 'test' | nc 127.0.0.1 5443

• Actual result: Data is still successfully received by the upstream. APISIX did not terminate the connection even though the CA used to verify the upstream certificate is incorrect.

Environment

• APISIX version (run apisix version): 3.14.1
• APISIX helm chart version: 2.12.4
• Kubernetes server version: v1.31.5+k3s1
• Operating system (run uname -a): Linux 212801065-522 5.15.0-161-generic doc: added features list and gitter chat. #171-Ubuntu SMP Sat Oct 11 08:17:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
• OpenResty / Nginx version (run openresty -V or nginx -V): NA
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info): NA
• APISIX Dashboard version, if relevant: NA
• Plugin runner version, for issues related to plugin runners: NA
• LuaRocks version, for installation issues (run luarocks --version): NA

[Baoyuantop]

upstream.tls.verify currently only supports Kafka upstream. You can refer to https://apisix.apache.org/docs/apisix/admin-api/#request-body-parameters-4