bug: Health check state lost and checker not working after upstream node changes

[solosky]

Current Behavior

When upstream nodes change (e.g., Kubernetes pod scaling, service discovery update, or DNS resolution change), the health checker has two critical issues:

1. Health check status lost: Previously detected unhealthy nodes reset to healthy after node changes
2. Health check not running: There is a probability that the health checker stops checking after node changes

Root Cause

APISIX uses a full destroy-and-rebuild strategy for health checkers when upstream nodes change. The core flow is:

1. Node change → _nodes_ver increments → resource_version changes
2. fetch_checker() detects version mismatch → adds to waiting_pool, returns nil (no checker during this period)
3. Timer (1s interval) destroys old checker → calls delayed_clear() to clear all health status from shared dict
4. Creates brand new checker → all nodes start as healthy

Impact

• Traffic routed to unhealthy nodes during the window between checker rebuild and next active check cycle
• Removed nodes remain in the health checker's target list, consuming resources and potentially affecting health check results
• In high-frequency node change scenarios, the checker may never be successfully created due to version race conditions

Suggested Fix

Implement incremental target update instead of full rebuild:

1. When nodes change but checks config remains the same, only add/remove targets on the existing checker
2. Use target.hostname from get_target_list() when calling remove_target() to ensure the correct target is matched
3. Only do full rebuild when checks configuration changes

Key changes in healthcheck_manager.lua:

• Add update_checker_targets(): incrementally adds new targets and removes stale ones
• Add checks_config_equal(): compares checks config to decide incremental vs full rebuild
• Fix remove_target() hostname: use stored target.hostname instead of checks.active.host
• Save checks config in working pool for later comparison

Expected Behavior

1. Existing nodes should retain their health status (healthy/unhealthy) when nodes are added/removed
2. Removed nodes should be properly cleaned up from the health checker
3. Health checking should not have gaps during node changes

Error Logs

No response

Steps to Reproduce

1. Create a route with health check enabled and multiple upstream nodes
2. Wait for one node to be detected as unhealthy
3. Add a new node to the upstream (or trigger service discovery update)
4. Observe that the previously unhealthy node resets to healthy
5. Check health checker target list — removed nodes may still be present

Environment

Environment
APISIX version: 3.16.0
lua-resty-healthcheck-api7: 3.2.1-0

[Baoyuantop]

Hi @solosky, could you provide more detailed steps to reproduce the issue? Please include the specific APISIX configuration and routes configuration details.

[solosky]

Reproducing Health Check State Lost After Upstream Node Changes

Prerequisites

• minikube
• helm
• kubectl
• Docker (or use minikube docker-env)

Step 1: Start Minikube and Install APISIX

# Start minikube
minikube start --cpus=4 --memory=8192 --driver=docker

# Add APISIX Helm chart repo
helm repo add apisix https://charts.apiseven.com
helm repo update

# Install APISIX and APISIX Ingress Controller
helm install apisix apisix/apisix \
  --namespace ingress-apisix \
  --create-namespace \
  --set gateway.type=NodePort \
  --set gateway.http.nodePort=30080 \
  --set etcd.persistence.enabled=false \
  --set ingress-controller.enabled=true

# Wait for all pods ready
kubectl -n ingress-apisix wait --for=condition=Ready pods --all --timeout=120s

Step 2: Build and Load the Test Image

The test server is a simple Node.js static file server that returns X-Pod-IP and X-Pod-Name headers on every response. This allows us to identify which pod served each request.

server.js

const http = require('http');
const fs = require('fs');
const path = require('path');
const os = require('os');

const PORT = process.env.PORT || 8080;
const STATIC_DIR = path.join(__dirname, 'static');

function getLocalIP() {
  const interfaces = os.networkInterfaces();
  for (const name of Object.keys(interfaces)) {
    for (const iface of interfaces[name]) {
      if (iface.family === 'IPv4' && !iface.internal) {
        return iface.address;
      }
    }
  }
  return '127.0.0.1';
}

function getPodName() {
  return process.env.HOSTNAME || 'unknown';
}

const POD_IP = getLocalIP();
const POD_NAME = getPodName();

const MIME_TYPES = {
  '.html': 'text/html',
  '.css': 'text/css',
  '.js': 'application/javascript',
  '.json': 'application/json',
  '.png': 'image/png',
  '.jpg': 'image/jpeg',
  '.gif': 'image/gif',
  '.svg': 'image/svg+xml',
  '.ico': 'image/x-icon',
  '.txt': 'text/plain',
};

function getMimeType(filePath) {
  const ext = path.extname(filePath).toLowerCase();
  return MIME_TYPES[ext] || 'application/octet-stream';
}

const server = http.createServer((req, res) => {
  const startTime = Date.now();
  const requestIP = req.socket.remoteAddress || 'unknown';
  const userAgent = req.headers['user-agent'] || 'unknown';

  res.setHeader('X-Pod-IP', POD_IP);
  res.setHeader('X-Pod-Name', POD_NAME);

  let filePath = req.url === '/' ? '/index.html' : req.url;
  filePath = path.normalize(filePath).replace(/^(\.\.[\/\\])+/, '');
  const fullPath = path.join(STATIC_DIR, filePath);

  fs.readFile(fullPath, (err, data) => {
    let statusCode = 200;
    if (err) {
      if (err.code === 'ENOENT') {
        statusCode = 404;
        res.writeHead(404, { 'Content-Type': 'text/plain' });
        res.end('404 Not Found');
      } else {
        statusCode = 500;
        res.writeHead(500, { 'Content-Type': 'text/plain' });
        res.end('500 Internal Server Error');
      }
    } else {
      const mimeType = getMimeType(fullPath);
      res.writeHead(200, { 'Content-Type': mimeType });
      res.end(data);
    }
    const elapsed = Date.now() - startTime;
    const time = new Date().toISOString();
    console.log(`[${time}] "${req.method} ${req.url}" ${statusCode} ${elapsed}ms ${requestIP} "${userAgent}"`);
  });
});

server.listen(PORT, '0.0.0.0', () => {
  console.log(`Server running on port ${PORT}`);
  console.log(`Pod IP: ${POD_IP}`);
});

Dockerfile

FROM node:20-alpine

WORKDIR /app

COPY server.js .

ADD static ./static

EXPOSE 8080

CMD ["node", "server.js"]

static/ files — Create the following files in a static/ directory:

• static/index.html — any HTML content
• static/healthcheck.html — any content (used by APISIX active health check)
• static/readiness.html — any content (used by Kubernetes readiness probe)
• static/liveness.html — any content (used by Kubernetes liveness probe)

Build and load the image into minikube:

# Build inside minikube's Docker daemon
eval $(minikube docker-env)
docker build -t httpbin-test:latest .

Step 3: Deploy the Test Application (2 Replicas)

httpbin-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  replicas: 2
  selector:
    matchLabels:
      app: httpbin
  template:
    metadata:
      labels:
        app: httpbin
    spec:
      containers:
        - name: httpbin
          image: httpbin-test:latest
          imagePullPolicy: Never
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /readiness.html
              port: 8080
            initialDelaySeconds: 3
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /liveness.html
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10

kubectl apply -f httpbin-deployment.yaml

# Wait for both pods to be Running and Ready
kubectl get pods -l app=httpbin -w
# You should see 2/2 READY, e.g.:
# httpbin-5f55f9686c-abcde   1/1     Running   0          30s
# httpbin-5f55f9686c-fghij   1/1     Running   0          30s

Step 4: Create Service, ApisixRoute, and ApisixUpstream

httpbin-resources.yaml

apiVersion: v1
kind: Service
metadata:
  name: httpbin-svc
spec:
  selector:
    app: httpbin
  ports:
    - port: 8080
      targetPort: 8080
---
apiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: httpbin-route
spec:
  http:
    - name: httpbin
      match:
        hosts:
          - httpbin.local
        paths:
          - "/*"
      backends:
        - serviceName: httpbin-svc
          servicePort: 8080
---
apiVersion: apisix.apache.org/v1
kind: ApisixUpstream
metadata:
  name: httpbin-svc
spec:
  healthCheck:
    active:
      type: http
      httpPath: /healthcheck.html
      healthy:
        interval: 2s
        httpStatuses:
          - 200
        successes: 1
      unhealthy:
        interval: 2s
        httpFailures: 1
        httpStatuses:
          - 40
          - 500
          - 502
          - 503
          - 504
        tcpFailures: 1
        timeouts: 1

kubectl apply -f httpbin-resources.yaml

Step 5: Verify Health Check is Working

Get the APISIX NodePort:

APISIX_PORT=$(kubectl get svc -n ingress-apisix apisix-gateway -o jsonpath='{.spec.ports[?(@.name=="http")].nodePort}")
APISIX_IP=$(minikube ip)
echo "APISIX endpoint: http://${APISIX_IP}:${APISIX_PORT}"

# Send requests and observe X-Pod-Name header to see which pod serves each request
curl -s -H "Host: httpbin.local" http://${APISIX_IP}:${APISIX_PORT}/ -D -
# You should see X-Pod-Name and X-Pod-IP headers in the response

# Trigger health check by sending a few requests
for i in $(seq 1 10); do
  curl -s -H "Host: httpbin.local" http://${APISIX_IP}:${APISIX_PORT}/ -o /dev/null -w "%{http_code}\n"
done

Step 6: Identify a Target Pod and Set Up Monitoring

Pick one of the two pods and start a monitoring loop:

# List pods
kubectl get pods -l app=httpbin
# Example output:
# httpbin-5f55f9686c-abcde   1/1     Running   0          5m
# httpbin-5f55f9686c-fghij   1/1     Running   0          5m

# Replace <POD_NAME> with one of the pod names from above
TARGET_POD=<POD_NAME>  # e.g., httpbin-5f55f9686c-abcde

# Start monitoring loop — this prints a line whenever the TARGET_POD serves a request
while true; do
  curl -s -I -H "Host: httpbin.local" http://${APISIX_IP}:${APISIX_PORT}/ | grep "${TARGET_POD}";
done

You should see the target pod name printed periodically as requests are load-balanced to it.

Step 7: Make the Target Pod Unhealthy

Open a shell into the target pod and remove/rename the health check file:

# Exec into the target pod
kubectl exec -it ${TARGET_POD} -- sh

# Inside the pod: rename the healthcheck file so APISIX active check gets a 404
mv /app/static/healthcheck.html /app/static/healthcheck.html.bak

# Exit the pod
exit

Wait for APISIX to detect the node as unhealthy (within ~4 seconds based on the check interval). The monitoring loop should stop printing output — confirming no traffic is being routed to the unhealthy pod.

# Verify via APISIX health check API (optional)
kubectl exec -n ingress-apisix <APISIX_POD> -- curl -s http://127.0.0.1:9180/apisix/admin/upstreams -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1'

Step 8: Scale Up Replicas to 3 (Trigger Node Change)

In a new terminal:

kubectl scale deployment httpbin --replicas=3

Step 9: Observe the Bug

Go back to the monitoring loop terminal. You will see the target pod name start printing again, meaning traffic is being routed to the unhealthy pod despite its health check still failing.

# Example of buggy output:
# (before scaling: no output — correctly excluded from load balancing)
# httpbin-5f55f9686c-abcde    <-- BUG: traffic resumed to unhealthy pod after scale-up
# httpbin-5f55f9686c-abcde
# httpbin-5f55f9686c-abcde

This confirms the health check status was lost during the upstream node change.

Expected Behavior

After scaling to 3 replicas:

• The existing unhealthy pod should remain unhealthy — no traffic should be routed to it
• The 2 healthy pods (original healthy + new pod) should handle all traffic
• Only after the unhealthy pod's healthcheck file is restored should traffic resume to that pod

Root Cause

When upstream nodes change (triggered by the scale-up), APISIX:

1. Detects the resource_version change
2. Destroys the old health checker and calls delayed_clear() to clear all health status from the shared dict
3. Creates a new checker where all nodes start as healthy by default

This means previously detected unhealthy nodes reset to healthy, causing traffic to be routed to them until the next active health check cycle discovers they are unhealthy again.

Additionally, in the incremental update path, remove_target() may silently fail because the hostname parameter used for removal does not match the one used during add_target(). This causes stale targets to remain in the checker even after nodes are removed from the upstream configuration.

Environment

• APISIX: apache/apisix:3.16.0-ubuntu
• APISIX Ingress Controller: apache/apisix-ingress-controller:2.0.1
• lua-resty-healthcheck-api7: 3.2.1-0
• Kubernetes: minikube (any recent version)

apisix and apisix ingress controller installed via helm

helm repo add apisix https://apache.github.io/apisix-helm-chart
helm repo update
helm install apisix apisix/apisix --create-namespace  --namespace apisix
helm install apisix-dashboard apisix/apisix-dashboard --create-namespace --namespace apisix
helm install apisix-ingress-controller apisix/apisix-ingress-controller --namespace apisix --create-namespace