Consistent downtime during rolling update and scale-in events

[mwatkins-ld]

Describe the bug
During scale-in events and more importantly rolling updates, we can observe consistent 503s even when otherwise healthy targets exist in the lattice service target group.

To Reproduce
Steps to reproduce the behavior:

1. Given a "shared" lattice service network owned by another AWS account (not sure if relevant)
2. Create a gateway resource in a different namespace that will be referenced by other applications (not sure if relevant)
3. Create a Deployment, Service, and HTTPRoute
4. Wait for VPC Lattice service creation to be ready, service target registered as healthy, and a 200 response from the lattice service domain name
5. Attempt a rolling deployment or scale-in event.
6. Make period requests to the service endpoint and observe 503s (5 second response time when it returns a 503)
7. Can be observed sporadically for some time (~20-40 seconds maybe?) after last pod has terminated at which point all 200s again.
8. Can be reproduced 100% of the time

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: example-gateway
  namespace: gateway-system
spec:
  gatewayClassName: amazon-vpc-lattice
  listeners:
  - allowedRoutes:
      namespaces:
        from: Same
    name: http
    port: 80
    protocol: HTTP
  - allowedRoutes:
      namespaces:
        from: Same
    name: https
    port: 443
    protocol: HTTPS
    tls:
      mode: Terminate
      options:
        application-networking.k8s.aws/certificate-arn: arn:{cert}

apiVersion: apps/v1
kind: Deployment
metadata:
  name: podinfo
spec:
  replicas: 3
  minReadySeconds: 3
  revisionHistoryLimit: 5
  progressDeadlineSeconds: 60
  strategy:
    rollingUpdate:
      maxUnavailable: 0
    type: RollingUpdate
  selector:
    matchLabels:
      app: podinfo
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9797"
      labels:
        app: podinfo
    spec:
      containers:
      - name: podinfod
        image: ghcr.io/stefanprodan/podinfo:6.9.1
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 9898
          protocol: TCP
        - name: http-metrics
          containerPort: 9797
          protocol: TCP
        - name: grpc
          containerPort: 9999
          protocol: TCP
        command:
        - ./podinfo
        - --port=9898
        - --port-metrics=9797
        - --grpc-port=9999
        - --grpc-service-name=podinfo
        - --level=info
        - --random-delay=false
        - --random-error=false
        env:
        - name: PODINFO_UI_COLOR
          value: "#34577c"
        livenessProbe:
          exec:
            command:
            - podcli
            - check
            - http
            - localhost:9898/healthz
          initialDelaySeconds: 5
          timeoutSeconds: 5
        readinessProbe:
          exec:
            command:
            - podcli
            - check
            - http
            - localhost:9898/readyz
          initialDelaySeconds: 5
          timeoutSeconds: 5
        resources:
          limits:
            cpu: 2000m
            memory: 512Mi
          requests:
            cpu: 100m
            memory: 64Mi
        volumeMounts:
          - name: data
            mountPath: /data
      volumes:
        - name: data
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: podinfo
spec:
  type: ClusterIP
  selector:
    app: podinfo
  ports:
    - name: http
      port: 9898
      protocol: TCP
      targetPort: http
    - port: 9999
      targetPort: grpc
      protocol: TCP
      name: grpc
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: podinfo
spec:
  parentRefs:
  - name: example-gateway
    sectionName: http
    namespace: gateway-system
  rules:
  - backendRefs:
    - name: podinfo
      kind: Service
      port: 9898
      weight: 10

Expected behavior
Pod lifecycle termination event to happen. The controller to update the lattice target as DRAINING. The http server in the pod to start closing any in-flight connections and then terminate. The lattice service target that is draining and pointing to the terminating/terminated pod to have 0 in-flight connection to that target and not serve any new traffic to that target and only serve traffic to the healthy targets.

Additional context
v1.1.3, eks 1.33, AWS vpc-cni

• Using pod-readiness gate feature as well. Scaling out and testing the endpoint produce all 200s so it would be assumed that the healthy endpoints in lattice correspond to the healthy pods in the cluster.
• Same behavior seems to be present when not using the readiness gate feature.
• I also tested this outside of podinfo using a toy-app that does nothing but handle a single route responding 200 and close the http server on SIGTERM. Same exact behavior.
• Same behavior when incorporating health checks, using HTTP or HTTPS listeners, custom domains or direct lattice DNS name.
• Target group in the UI has the terminated pod IPs still visible as "DRAINING" in the UI for some 1 minute? or so after the pod has already been terminated.

Field some amount of consistent requests (1/sec) after (no in-flight requests) the rolling update is initiated or being fielding them and try an update/scale-in. Same behavior.

INFO[0005] 503: 5005.757126ms                            source=console
INFO[0013] 503: 5004.059355ms                            source=console
INFO[0021] 503: 5002.491885ms                            source=console
INFO[0027] 503: 5001.096097ms                            source=console
INFO[0034] 503: 5001.817081ms                            source=console
INFO[0040] 503: 4999.573067ms                            source=console
INFO[0046] 503: 4999.728199ms                            source=console
INFO[0052] 503: 5002.151223ms                            source=console
INFO[0059] 503: 5000.536234ms                            source=console
INFO[0065] 503: 5004.477808ms                            source=console
INFO[0072] 503: 5002.736388ms                            source=console
INFO[0081] 503: 5000.442743ms                            source=console
INFO[0090] 503: 5004.646548ms                            source=console
INFO[0099] 503: 5004.92498ms                             source=console


  █ TOTAL RESULTS

    checks_total.......................: 100    0.831667/s
    checks_succeeded...................: 72.00% 72 out of 100
    checks_failed......................: 28.00% 28 out of 100

    ✗ status is 200
      ↳  72% — ✓ 36 / ✗ 14
    ✗ response time < 500ms
      ↳  72% — ✓ 36 / ✗ 14

The logs during the update

{"level":"info","ts":"2025-07-23T20:15:46.372Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:15:46.372Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:15:46.372Z","logger":"controller.route","caller":"gwlog/metadata.go:76","msg":"reconcile_start","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"e5170b3a-c2ac-46d3-b3c0-ca1d7c42cd7a"}
{"level":"info","ts":"2025-07-23T20:15:46.372Z","logger":"controller.route","caller":"controllers/route_controller.go:293","msg":"reconcile, adding or updating","name":"podinfo","trace_id":"e5170b3a-c2ac-46d3-b3c0-ca1d7c42cd7a"}
{"level":"info","ts":"2025-07-23T20:15:46.388Z","logger":"controller.route","caller":"gateway/model_build_lattice_service.go:159","msg":"No custom-domain-name for route podinfo-gateway-sample","trace_id":"e5170b3a-c2ac-46d3-b3c0-ca1d7c42cd7a"}
{"level":"info","ts":"2025-07-23T20:15:47.038Z","logger":"controller.route","caller":"lattice/listener_manager.go:55","msg":"Upsert listener podinfo-gateway-sample","trace_id":"e5170b3a-c2ac-46d3-b3c0-ca1d7c42cd7a"}
{"level":"info","ts":"2025-07-23T20:15:47.522Z","logger":"controller.route","caller":"controllers/route_controller.go:162","msg":"reconcile error","name":"podinfo","message":"error during target post synthesis error during target post synthesis error during target post synthesis error during target post synthesis error during target post synthesis error during service synthesis failed ServiceManager.Upsert podinfo-gateway-sample due to error during tg synthesis error during target group synthesis, will retry: sn-service-association-id: snsa-0f52afe942e995c7c, non-active status: CREATE_IN_PROGRESS: target status still in pending: target status still in pending: target status still in pending: target status still in pending: target status still in pending","trace_id":"e5170b3a-c2ac-46d3-b3c0-ca1d7c42cd7a"}
{"level":"info","ts":"2025-07-23T20:15:47.522Z","logger":"controller.route","caller":"gwlog/metadata.go:82","msg":"reconcile_end","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"e5170b3a-c2ac-46d3-b3c0-ca1d7c42cd7a"}
{"level":"info","ts":"2025-07-23T20:16:07.523Z","logger":"controller.route","caller":"gwlog/metadata.go:76","msg":"reconcile_start","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"61710599-da86-4b46-97ef-e241b7993857"}
{"level":"info","ts":"2025-07-23T20:16:07.523Z","logger":"controller.route","caller":"controllers/route_controller.go:293","msg":"reconcile, adding or updating","name":"podinfo","trace_id":"61710599-da86-4b46-97ef-e241b7993857"}
{"level":"info","ts":"2025-07-23T20:16:07.537Z","logger":"controller.route","caller":"gateway/model_build_lattice_service.go:159","msg":"No custom-domain-name for route podinfo-gateway-sample","trace_id":"61710599-da86-4b46-97ef-e241b7993857"}
{"level":"info","ts":"2025-07-23T20:16:08.096Z","logger":"controller.route","caller":"lattice/listener_manager.go:55","msg":"Upsert listener podinfo-gateway-sample","trace_id":"61710599-da86-4b46-97ef-e241b7993857"}
{"level":"info","ts":"2025-07-23T20:16:08.626Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:08.626Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:08.660Z","logger":"controller.route","caller":"controllers/route_controller.go:376","msg":"reconciled","name":"podinfo","trace_id":"61710599-da86-4b46-97ef-e241b7993857"}
{"level":"info","ts":"2025-07-23T20:16:08.660Z","logger":"controller.route","caller":"gwlog/metadata.go:82","msg":"reconcile_end","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"61710599-da86-4b46-97ef-e241b7993857"}
{"level":"info","ts":"2025-07-23T20:16:08.660Z","logger":"controller.route","caller":"gwlog/metadata.go:76","msg":"reconcile_start","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"303b33f9-c77a-4394-902b-66e102f1698d"}
{"level":"info","ts":"2025-07-23T20:16:08.660Z","logger":"controller.route","caller":"controllers/route_controller.go:293","msg":"reconcile, adding or updating","name":"podinfo","trace_id":"303b33f9-c77a-4394-902b-66e102f1698d"}
{"level":"info","ts":"2025-07-23T20:16:08.677Z","logger":"controller.route","caller":"gateway/model_build_lattice_service.go:159","msg":"No custom-domain-name for route podinfo-gateway-sample","trace_id":"303b33f9-c77a-4394-902b-66e102f1698d"}
{"level":"info","ts":"2025-07-23T20:16:09.336Z","logger":"controller.route","caller":"lattice/listener_manager.go:55","msg":"Upsert listener podinfo-gateway-sample","trace_id":"303b33f9-c77a-4394-902b-66e102f1698d"}
{"level":"info","ts":"2025-07-23T20:16:09.931Z","logger":"controller.route","caller":"controllers/route_controller.go:376","msg":"reconciled","name":"podinfo","trace_id":"303b33f9-c77a-4394-902b-66e102f1698d"}
{"level":"info","ts":"2025-07-23T20:16:09.931Z","logger":"controller.route","caller":"gwlog/metadata.go:82","msg":"reconcile_end","name":"podinfo","namespace":"gateway-sample","type":"route","trace_id":"303b33f9-c77a-4394-902b-66e102f1698d"}
{"level":"info","ts":"2025-07-23T20:16:11.705Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:11.706Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:11.706Z","logger":"controller.route","caller":"gwlog/metadata.go:76","msg":"reconcile_start","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"872c4631-d072-4471-ab21-a59ffe1d9ba9"}
{"level":"info","ts":"2025-07-23T20:16:11.706Z","logger":"controller.route","caller":"controllers/route_controller.go:293","msg":"reconcile, adding or updating","name":"podinfo","trace_id":"872c4631-d072-4471-ab21-a59ffe1d9ba9"}
{"level":"info","ts":"2025-07-23T20:16:11.722Z","logger":"controller.route","caller":"gateway/model_build_lattice_service.go:159","msg":"No custom-domain-name for route podinfo-gateway-sample","trace_id":"872c4631-d072-4471-ab21-a59ffe1d9ba9"}
{"level":"info","ts":"2025-07-23T20:16:11.939Z","logger":"controller.route","caller":"lattice/targets_manager.go:72","msg":"Found 1 stale targets to deregister from target group tg-065b59dd14c366392","trace_id":"872c4631-d072-4471-ab21-a59ffe1d9ba9"}
{"level":"info","ts":"2025-07-23T20:16:12.464Z","logger":"controller.route","caller":"lattice/listener_manager.go:55","msg":"Upsert listener podinfo-gateway-sample","trace_id":"872c4631-d072-4471-ab21-a59ffe1d9ba9"}
{"level":"info","ts":"2025-07-23T20:16:12.966Z","logger":"controller.route","caller":"controllers/route_controller.go:376","msg":"reconciled","name":"podinfo","trace_id":"872c4631-d072-4471-ab21-a59ffe1d9ba9"}
{"level":"info","ts":"2025-07-23T20:16:12.966Z","logger":"controller.route","caller":"gwlog/metadata.go:82","msg":"reconcile_end","name":"podinfo","namespace":"gateway-sample","type":"route","trace_id":"872c4631-d072-4471-ab21-a59ffe1d9ba9"}
{"level":"info","ts":"2025-07-23T20:16:13.131Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:13.131Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:13.131Z","logger":"controller.route","caller":"gwlog/metadata.go:76","msg":"reconcile_start","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"26434d87-0aec-43bb-85ab-93854b4bdd2e"}
{"level":"info","ts":"2025-07-23T20:16:13.131Z","logger":"controller.route","caller":"controllers/route_controller.go:293","msg":"reconcile, adding or updating","name":"podinfo","trace_id":"26434d87-0aec-43bb-85ab-93854b4bdd2e"}
{"level":"info","ts":"2025-07-23T20:16:13.153Z","logger":"controller.route","caller":"gateway/model_build_lattice_service.go:159","msg":"No custom-domain-name for route podinfo-gateway-sample","trace_id":"26434d87-0aec-43bb-85ab-93854b4bdd2e"}
{"level":"info","ts":"2025-07-23T20:16:13.724Z","logger":"controller.route","caller":"lattice/listener_manager.go:55","msg":"Upsert listener podinfo-gateway-sample","trace_id":"26434d87-0aec-43bb-85ab-93854b4bdd2e"}
{"level":"info","ts":"2025-07-23T20:16:14.190Z","logger":"controller.route","caller":"controllers/route_controller.go:162","msg":"reconcile error","name":"podinfo","message":"error during target post synthesis error during target post synthesis error during target post synthesis error during target post synthesis error during target post synthesis error during target post synthesis error during service synthesis failed ServiceManager.Upsert podinfo-gateway-sample due to error during tg synthesis error during target group synthesis, will retry: sn-service-association-id: snsa-0f52afe942e995c7c, non-active status: CREATE_IN_PROGRESS: target status still in pending: target status still in pending: target status still in pending: target status still in pending: target status still in pending: target status still in pending","trace_id":"26434d87-0aec-43bb-85ab-93854b4bdd2e"}
{"level":"info","ts":"2025-07-23T20:16:14.190Z","logger":"controller.route","caller":"gwlog/metadata.go:82","msg":"reconcile_end","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"26434d87-0aec-43bb-85ab-93854b4bdd2e"}
{"level":"info","ts":"2025-07-23T20:16:14.835Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:14.835Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:14.835Z","logger":"controller.route","caller":"gwlog/metadata.go:76","msg":"reconcile_start","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"aa646cf8-3aad-4120-9e9f-fd3dfc529475"}
{"level":"info","ts":"2025-07-23T20:16:14.835Z","logger":"controller.route","caller":"controllers/route_controller.go:293","msg":"reconcile, adding or updating","name":"podinfo","trace_id":"aa646cf8-3aad-4120-9e9f-fd3dfc529475"}
{"level":"info","ts":"2025-07-23T20:16:14.848Z","logger":"controller.route","caller":"gateway/model_build_lattice_service.go:159","msg":"No custom-domain-name for route podinfo-gateway-sample","trace_id":"aa646cf8-3aad-4120-9e9f-fd3dfc529475"}
{"level":"info","ts":"2025-07-23T20:16:15.370Z","logger":"controller.route","caller":"lattice/listener_manager.go:55","msg":"Upsert listener podinfo-gateway-sample","trace_id":"aa646cf8-3aad-4120-9e9f-fd3dfc529475"}
{"level":"info","ts":"2025-07-23T20:16:15.926Z","logger":"controller.route","caller":"controllers/route_controller.go:376","msg":"reconciled","name":"podinfo","trace_id":"aa646cf8-3aad-4120-9e9f-fd3dfc529475"}
{"level":"info","ts":"2025-07-23T20:16:15.926Z","logger":"controller.route","caller":"gwlog/metadata.go:82","msg":"reconcile_end","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"aa646cf8-3aad-4120-9e9f-fd3dfc529475"}
{"level":"info","ts":"2025-07-23T20:16:25.233Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:25.233Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:25.233Z","logger":"controller.route","caller":"gwlog/metadata.go:76","msg":"reconcile_start","namespace":"gateway-sample","type":"route","name":"podinfo","trace_id":"5aa0f10c-230e-4373-8c96-10f905f1d70b"}
{"level":"info","ts":"2025-07-23T20:16:25.233Z","logger":"controller.route","caller":"controllers/route_controller.go:293","msg":"reconcile, adding or updating","name":"podinfo","trace_id":"5aa0f10c-230e-4373-8c96-10f905f1d70b"}
{"level":"info","ts":"2025-07-23T20:16:25.247Z","logger":"controller.route","caller":"gateway/model_build_lattice_service.go:159","msg":"No custom-domain-name for route podinfo-gateway-sample","trace_id":"5aa0f10c-230e-4373-8c96-10f905f1d70b"}
{"level":"info","ts":"2025-07-23T20:16:25.838Z","logger":"controller.route","caller":"lattice/listener_manager.go:55","msg":"Upsert listener podinfo-gateway-sample","trace_id":"5aa0f10c-230e-4373-8c96-10f905f1d70b"}
{"level":"info","ts":"2025-07-23T20:16:26.333Z","logger":"controller.route","caller":"controllers/route_controller.go:376","msg":"reconciled","name":"podinfo","trace_id":"5aa0f10c-230e-4373-8c96-10f905f1d70b"}
{"level":"info","ts":"2025-07-23T20:16:26.333Z","logger":"controller.route","caller":"gwlog/metadata.go:82","msg":"reconcile_end","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"5aa0f10c-230e-4373-8c96-10f905f1d70b"}
{"level":"info","ts":"2025-07-23T20:16:28.431Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:28.431Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:28.431Z","logger":"controller.route","caller":"gwlog/metadata.go:76","msg":"reconcile_start","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"2ccdd446-41f2-44c1-aa9e-96387b418abf"}
{"level":"info","ts":"2025-07-23T20:16:28.431Z","logger":"controller.route","caller":"controllers/route_controller.go:293","msg":"reconcile, adding or updating","name":"podinfo","trace_id":"2ccdd446-41f2-44c1-aa9e-96387b418abf"}
{"level":"info","ts":"2025-07-23T20:16:28.460Z","logger":"controller.route","caller":"gateway/model_build_lattice_service.go:159","msg":"No custom-domain-name for route podinfo-gateway-sample","trace_id":"2ccdd446-41f2-44c1-aa9e-96387b418abf"}
{"level":"info","ts":"2025-07-23T20:16:28.692Z","logger":"controller.route","caller":"lattice/targets_manager.go:72","msg":"Found 1 stale targets to deregister from target group tg-065b59dd14c366392","trace_id":"2ccdd446-41f2-44c1-aa9e-96387b418abf"}
{"level":"info","ts":"2025-07-23T20:16:29.128Z","logger":"controller.route","caller":"lattice/listener_manager.go:55","msg":"Upsert listener podinfo-gateway-sample","trace_id":"2ccdd446-41f2-44c1-aa9e-96387b418abf"}
{"level":"info","ts":"2025-07-23T20:16:29.312Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:29.312Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:29.816Z","logger":"controller.route","caller":"controllers/route_controller.go:376","msg":"reconciled","name":"podinfo","trace_id":"2ccdd446-41f2-44c1-aa9e-96387b418abf"}
{"level":"info","ts":"2025-07-23T20:16:29.816Z","logger":"controller.route","caller":"gwlog/metadata.go:82","msg":"reconcile_end","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"2ccdd446-41f2-44c1-aa9e-96387b418abf"}
{"level":"info","ts":"2025-07-23T20:16:29.816Z","logger":"controller.route","caller":"gwlog/metadata.go:76","msg":"reconcile_start","namespace":"gateway-sample","type":"route","name":"podinfo","trace_id":"51f11632-48c9-4527-8e92-97a637163e4a"}
{"level":"info","ts":"2025-07-23T20:16:29.816Z","logger":"controller.route","caller":"controllers/route_controller.go:293","msg":"reconcile, adding or updating","name":"podinfo","trace_id":"51f11632-48c9-4527-8e92-97a637163e4a"}
{"level":"info","ts":"2025-07-23T20:16:29.923Z","logger":"controller.route","caller":"gateway/model_build_lattice_service.go:159","msg":"No custom-domain-name for route podinfo-gateway-sample","trace_id":"51f11632-48c9-4527-8e92-97a637163e4a"}
{"level":"info","ts":"2025-07-23T20:16:30.472Z","logger":"controller.route","caller":"lattice/listener_manager.go:55","msg":"Upsert listener podinfo-gateway-sample","trace_id":"51f11632-48c9-4527-8e92-97a637163e4a"}
{"level":"info","ts":"2025-07-23T20:16:30.985Z","logger":"controller.route","caller":"controllers/route_controller.go:162","msg":"reconcile error","name":"podinfo","message":"error during target post synthesis error during target post synthesis error during target post synthesis error during target post synthesis error during target post synthesis error during target post synthesis error during target post synthesis error during service synthesis failed ServiceManager.Upsert podinfo-gateway-sample due to error during tg synthesis error during target group synthesis, will retry: sn-service-association-id: snsa-0f52afe942e995c7c, non-active status: CREATE_IN_PROGRESS: target status still in pending: target status still in pending: target status still in pending: target status still in pending: target status still in pending: target status still in pending: target status still in pending","trace_id":"51f11632-48c9-4527-8e92-97a637163e4a"}
{"level":"info","ts":"2025-07-23T20:16:30.985Z","logger":"controller.route","caller":"gwlog/metadata.go:82","msg":"reconcile_end","name":"podinfo","namespace":"gateway-sample","type":"route","trace_id":"51f11632-48c9-4527-8e92-97a637163e4a"}
{"level":"info","ts":"2025-07-23T20:16:31.539Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:31.539Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:31.540Z","logger":"controller.route","caller":"gwlog/metadata.go:76","msg":"reconcile_start","namespace":"gateway-sample","type":"route","name":"podinfo","trace_id":"41ee3e10-bf43-4805-aac3-9ab8416f4640"}
{"level":"info","ts":"2025-07-23T20:16:31.540Z","logger":"controller.route","caller":"controllers/route_controller.go:293","msg":"reconcile, adding or updating","name":"podinfo","trace_id":"41ee3e10-bf43-4805-aac3-9ab8416f4640"}
{"level":"info","ts":"2025-07-23T20:16:31.574Z","logger":"controller.route","caller":"gateway/model_build_lattice_service.go:159","msg":"No custom-domain-name for route podinfo-gateway-sample","trace_id":"41ee3e10-bf43-4805-aac3-9ab8416f4640"}
{"level":"info","ts":"2025-07-23T20:16:32.263Z","logger":"controller.route","caller":"lattice/listener_manager.go:55","msg":"Upsert listener podinfo-gateway-sample","trace_id":"41ee3e10-bf43-4805-aac3-9ab8416f4640"}
{"level":"info","ts":"2025-07-23T20:16:32.794Z","logger":"controller.route","caller":"controllers/route_controller.go:376","msg":"reconciled","name":"podinfo","trace_id":"41ee3e10-bf43-4805-aac3-9ab8416f4640"}
{"level":"info","ts":"2025-07-23T20:16:32.795Z","logger":"controller.route","caller":"gwlog/metadata.go:82","msg":"reconcile_end","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"41ee3e10-bf43-4805-aac3-9ab8416f4640"}
{"level":"info","ts":"2025-07-23T20:16:34.190Z","logger":"controller.route","caller":"gwlog/metadata.go:76","msg":"reconcile_start","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"b7eb56d3-f3f1-43c3-aea3-18ff20c6b1ec"}
{"level":"info","ts":"2025-07-23T20:16:34.190Z","logger":"controller.route","caller":"controllers/route_controller.go:293","msg":"reconcile, adding or updating","name":"podinfo","trace_id":"b7eb56d3-f3f1-43c3-aea3-18ff20c6b1ec"}
{"level":"info","ts":"2025-07-23T20:16:34.205Z","logger":"controller.route","caller":"gateway/model_build_lattice_service.go:159","msg":"No custom-domain-name for route podinfo-gateway-sample","trace_id":"b7eb56d3-f3f1-43c3-aea3-18ff20c6b1ec"}
{"level":"info","ts":"2025-07-23T20:16:34.769Z","logger":"controller.route","caller":"lattice/listener_manager.go:55","msg":"Upsert listener podinfo-gateway-sample","trace_id":"b7eb56d3-f3f1-43c3-aea3-18ff20c6b1ec"}
{"level":"info","ts":"2025-07-23T20:16:35.299Z","logger":"controller.route","caller":"controllers/route_controller.go:376","msg":"reconciled","name":"podinfo","trace_id":"b7eb56d3-f3f1-43c3-aea3-18ff20c6b1ec"}
{"level":"info","ts":"2025-07-23T20:16:35.299Z","logger":"controller.route","caller":"gwlog/metadata.go:82","msg":"reconcile_end","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"b7eb56d3-f3f1-43c3-aea3-18ff20c6b1ec"}
{"level":"info","ts":"2025-07-23T20:16:41.359Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:41.359Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:41.359Z","logger":"controller.route","caller":"gwlog/metadata.go:76","msg":"reconcile_start","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"15334da0-cea2-45bb-b62f-576efc3d8221"}
{"level":"info","ts":"2025-07-23T20:16:41.359Z","logger":"controller.route","caller":"controllers/route_controller.go:293","msg":"reconcile, adding or updating","name":"podinfo","trace_id":"15334da0-cea2-45bb-b62f-576efc3d8221"}
{"level":"info","ts":"2025-07-23T20:16:41.374Z","logger":"controller.route","caller":"gateway/model_build_lattice_service.go:159","msg":"No custom-domain-name for route podinfo-gateway-sample","trace_id":"15334da0-cea2-45bb-b62f-576efc3d8221"}
{"level":"info","ts":"2025-07-23T20:16:41.946Z","logger":"controller.route","caller":"lattice/listener_manager.go:55","msg":"Upsert listener podinfo-gateway-sample","trace_id":"15334da0-cea2-45bb-b62f-576efc3d8221"}
{"level":"info","ts":"2025-07-23T20:16:42.455Z","logger":"controller.route","caller":"controllers/route_controller.go:376","msg":"reconciled","name":"podinfo","trace_id":"15334da0-cea2-45bb-b62f-576efc3d8221"}
{"level":"info","ts":"2025-07-23T20:16:42.455Z","logger":"controller.route","caller":"gwlog/metadata.go:82","msg":"reconcile_end","name":"podinfo","namespace":"gateway-sample","type":"route","trace_id":"15334da0-cea2-45bb-b62f-576efc3d8221"}
{"level":"info","ts":"2025-07-23T20:16:44.442Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:44.442Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:44.442Z","logger":"controller.route","caller":"gwlog/metadata.go:76","msg":"reconcile_start","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"ea48d6b4-7e45-4a7e-90c0-d642e7da0dba"}
{"level":"info","ts":"2025-07-23T20:16:44.442Z","logger":"controller.route","caller":"controllers/route_controller.go:293","msg":"reconcile, adding or updating","name":"podinfo","trace_id":"ea48d6b4-7e45-4a7e-90c0-d642e7da0dba"}
{"level":"info","ts":"2025-07-23T20:16:44.461Z","logger":"controller.route","caller":"gateway/model_build_lattice_service.go:159","msg":"No custom-domain-name for route podinfo-gateway-sample","trace_id":"ea48d6b4-7e45-4a7e-90c0-d642e7da0dba"}
{"level":"info","ts":"2025-07-23T20:16:44.665Z","logger":"controller.route","caller":"lattice/targets_manager.go:72","msg":"Found 1 stale targets to deregister from target group tg-065b59dd14c366392","trace_id":"ea48d6b4-7e45-4a7e-90c0-d642e7da0dba"}
{"level":"info","ts":"2025-07-23T20:16:45.120Z","logger":"controller.route","caller":"lattice/listener_manager.go:55","msg":"Upsert listener podinfo-gateway-sample","trace_id":"ea48d6b4-7e45-4a7e-90c0-d642e7da0dba"}
{"level":"info","ts":"2025-07-23T20:16:45.628Z","logger":"controller.route","caller":"controllers/route_controller.go:376","msg":"reconciled","name":"podinfo","trace_id":"ea48d6b4-7e45-4a7e-90c0-d642e7da0dba"}
{"level":"info","ts":"2025-07-23T20:16:45.628Z","logger":"controller.route","caller":"gwlog/metadata.go:82","msg":"reconcile_end","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"ea48d6b4-7e45-4a7e-90c0-d642e7da0dba"}
{"level":"info","ts":"2025-07-23T20:16:47.595Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:47.595Z","logger":"controller.route","caller":"eventhandlers/service.go:76","msg":"Service impacting resource change triggered Route update","serviceName":"gateway-sample/podinfo","routeName":"gateway-sample/podinfo","routeType":"http"}
{"level":"info","ts":"2025-07-23T20:16:47.595Z","logger":"controller.route","caller":"gwlog/metadata.go:76","msg":"reconcile_start","namespace":"gateway-sample","type":"route","name":"podinfo","trace_id":"197ac55e-ab3c-488a-a0d7-08df99084092"}
{"level":"info","ts":"2025-07-23T20:16:47.595Z","logger":"controller.route","caller":"controllers/route_controller.go:293","msg":"reconcile, adding or updating","name":"podinfo","trace_id":"197ac55e-ab3c-488a-a0d7-08df99084092"}
{"level":"info","ts":"2025-07-23T20:16:47.610Z","logger":"controller.route","caller":"gateway/model_build_lattice_service.go:159","msg":"No custom-domain-name for route podinfo-gateway-sample","trace_id":"197ac55e-ab3c-488a-a0d7-08df99084092"}
{"level":"info","ts":"2025-07-23T20:16:48.120Z","logger":"controller.route","caller":"lattice/listener_manager.go:55","msg":"Upsert listener podinfo-gateway-sample","trace_id":"197ac55e-ab3c-488a-a0d7-08df99084092"}
{"level":"info","ts":"2025-07-23T20:16:48.681Z","logger":"controller.route","caller":"controllers/route_controller.go:376","msg":"reconciled","name":"podinfo","trace_id":"197ac55e-ab3c-488a-a0d7-08df99084092"}
{"level":"info","ts":"2025-07-23T20:16:48.681Z","logger":"controller.route","caller":"gwlog/metadata.go:82","msg":"reconcile_end","type":"route","name":"podinfo","namespace":"gateway-sample","trace_id":"197ac55e-ab3c-488a-a0d7-08df99084092"}

The pod rollout observed:

NAME                       READY   STATUS              RESTARTS   AGE    IP             NODE                           NOMINATED NODE   READINESS GATES
podinfo-8579986dbb-25kl5   1/1     Running             0          101s   10.32.22.10    ip-10-32-21-126.ec2.internal   <none>           1/1
podinfo-8579986dbb-jwxkm   1/1     Running             0          84s    10.32.21.32    ip-10-32-21-126.ec2.internal   <none>           1/1
podinfo-8579986dbb-scvbh   1/1     Running             0          58s    10.32.22.197   ip-10-32-21-126.ec2.internal   <none>           1/1
podinfo-b568754b8-c8hml    0/1     ContainerCreating   0          0s     <none>         ip-10-32-3-45.ec2.internal     <none>           0/1
podinfo-b568754b8-c8hml    0/1     Running             0          1s     10.32.0.148    ip-10-32-3-45.ec2.internal     <none>           0/1
podinfo-b568754b8-c8hml    0/1     Running             0          2s     10.32.0.148    ip-10-32-3-45.ec2.internal     <none>           0/1
podinfo-b568754b8-c8hml    1/1     Running             0          13s    10.32.0.148    ip-10-32-3-45.ec2.internal     <none>           0/1
podinfo-b568754b8-c8hml    1/1     Running             0          23s    10.32.0.148    ip-10-32-3-45.ec2.internal     <none>           1/1
podinfo-b568754b8-c8hml    1/1     Running             0          23s    10.32.0.148    ip-10-32-3-45.ec2.internal     <none>           1/1
podinfo-8579986dbb-scvbh   1/1     Terminating         0          84s    10.32.22.197   ip-10-32-21-126.ec2.internal   <none>           1/1
podinfo-b568754b8-7ljdp    0/1     Pending             0          0s     <none>         <none>                         <none>           0/1
podinfo-b568754b8-7ljdp    0/1     Pending             0          0s     <none>         ip-10-32-21-126.ec2.internal   <none>           0/1
podinfo-b568754b8-7ljdp    0/1     ContainerCreating   0          0s     <none>         ip-10-32-21-126.ec2.internal   <none>           0/1
podinfo-b568754b8-7ljdp    0/1     Running             0          2s     10.32.18.232   ip-10-32-21-126.ec2.internal   <none>           0/1
podinfo-b568754b8-7ljdp    0/1     Running             0          3s     10.32.18.232   ip-10-32-21-126.ec2.internal   <none>           0/1
podinfo-8579986dbb-scvbh   0/1     Completed           0          88s    10.32.22.197   ip-10-32-21-126.ec2.internal   <none>           1/1
podinfo-b568754b8-7ljdp    0/1     Running             0          4s     10.32.18.232   ip-10-32-21-126.ec2.internal   <none>           1/1
podinfo-b568754b8-7ljdp    1/1     Running             0          14s    10.32.18.232   ip-10-32-21-126.ec2.internal   <none>           1/1
podinfo-b568754b8-7ljdp    1/1     Running             0          14s    10.32.18.232   ip-10-32-21-126.ec2.internal   <none>           1/1
podinfo-8579986dbb-jwxkm   1/1     Terminating         0          2m7s   10.32.21.32    ip-10-32-21-126.ec2.internal   <none>           1/1
podinfo-b568754b8-hzm2w    0/1     Pending             0          0s     <none>         <none>                         <none>           0/1
podinfo-b568754b8-hzm2w    0/1     Pending             0          0s     <none>         ip-10-32-13-73.ec2.internal    <none>           0/1
podinfo-b568754b8-hzm2w    0/1     ContainerCreating   0          0s     <none>         ip-10-32-13-73.ec2.internal    <none>           0/1
podinfo-b568754b8-hzm2w    0/1     Running             0          1s     10.32.13.91    ip-10-32-13-73.ec2.internal    <none>           0/1
podinfo-b568754b8-hzm2w    0/1     Running             0          2s     10.32.13.91    ip-10-32-13-73.ec2.internal    <none>           0/1
podinfo-8579986dbb-jwxkm   0/1     Completed           0          2m11s   10.32.21.32    ip-10-32-21-126.ec2.internal   <none>           1/1
podinfo-b568754b8-hzm2w    0/1     Running             0          4s      10.32.13.91    ip-10-32-13-73.ec2.internal    <none>           1/1
podinfo-b568754b8-hzm2w    1/1     Running             0          13s     10.32.13.91    ip-10-32-13-73.ec2.internal    <none>           1/1
podinfo-8579986dbb-25kl5   1/1     Terminating         0          2m40s   10.32.22.10    ip-10-32-21-126.ec2.internal   <none>           1/1
podinfo-8579986dbb-25kl5   0/1     Completed           0          2m44s   10.32.22.10    ip-10-32-21-126.ec2.internal   <none>           1/1

Thank you for your feedback. I have updated our FAQs section outlining a solution to these 503s.