"Real-time notifications interrupted. Please refresh the page or check your connection."

[s-h-a-d-o-w]

My connection is fine and refreshing the page doesn't stop the error from getting shown. I see this error in the Chrome devtools:

GET https://prom.neev.dev/api/v1/notifications/live net::ERR_QUIC_PROTOCOL_ERROR 200 (OK)

If I disable QUIC, it's

GET https://prom.neev.dev/api/v1/notifications/live net::ERR_HTTP2_PROTOCOL_ERROR 200 (OK)

(By the way: The notification icon keeps disappearing and then re-appearing the next time the error happens. I suspect that the intention is that it should stick around until the user reads it?)

My setup

Deployed via pulumi:

  new k8s.helm.v3.Release(
    "kube-prometheus-stack",
    {
      chart: "kube-prometheus-stack",
      name: "kube-prometheus-stack",
      namespace,
      repositoryOpts: {
        repo: "https://prometheus-community.github.io/helm-charts",
      },
      values: {
        grafana: {
          adminPassword: grafanaAdminPassword,
          deploymentStrategy: {
            type: "Recreate",
          },
          ingress: {
            enabled: true,
            hosts: ["[REDACTED]"],
            ingressClassName: "nginx",
            path: "/",
            pathType: "Prefix",
          },
          persistence: {
            accessModes: ["ReadWriteOnce"],
            enabled: true,
            size: "5Gi",
          },
          replicas: 1,
        } satisfies DeepInput<GrafanaConfig>,
        prometheus: {
          ingress: {
            annotations: {
              "nginx.ingress.kubernetes.io/auth-realm":
                "Authentication Required",
              "nginx.ingress.kubernetes.io/auth-secret":
                "prometheus-ingress-auth",
              "nginx.ingress.kubernetes.io/auth-type": "basic",
            },
            enabled: true,
            hosts: ["[REDACTED]"],
            ingressClassName: "nginx",
            pathType: "Prefix",
            paths: ["/"],
          },
          prometheusSpec: {
            storageSpec: {
              volumeClaimTemplate: {
                spec: {
                  accessModes: ["ReadWriteOnce"],
                  resources: {
                    requests: {
                      storage: "10Gi",
                    },
                  },
                },
              },
            },
          },
        },
      },
      version: kubePrometheusStackVersion,
    },
    { dependsOn: [prometheusIngressAuthSecret, ...dependsOn], provider },
  );

DNS handled by cloudflare via wildcard. I tried temporarily switching to "DNS only" - no change.

[sueun-dev]

That 200 doesn't show the stream survived. Current main's handler flushes the SSE headers before waiting for notifications, without heartbeat writes. An in-memory test of that extracted handler, with a stub subscriber, produced 200 with no body until an injected notification; this wasn't a deployment reproduction.

Which Prometheus/ingress-controller versions are running, and how long after connecting does the failure occur? Could you compare /api/v1/notifications/live through ingress with the same path through a local pod port-forward? If only the ingress path closes it at a repeatable idle interval, check its effective proxy-read-timeout against that interval. NGINX defines the timeout between upstream reads. That's a diagnostic branch, not a confirmed explanation for your protocol errors.

The icon's flicker also fits current UI code: a new 200 clears notifications and the connection-error flag; the icon is hidden when both are empty/false, then returns on error. The deployed version still needs checking.

[s-h-a-d-o-w]

It's because of ingress. Thanks for the fast and thorough response!

[s-h-a-d-o-w]

Having looked into and thought about this more, I wonder why live doesn't send heartbeat messages. Is the Prometheus UI maybe supposed to be only used via port forwarding? After all, nginx (and also cloudflare and a load balancer that I have in front of k8s) have time outs to clear out possibly stalled connections and raising those timeouts (overall problematic impact aside) would only result in the error getting delayed.
Or is the intention maybe that when the connection to live drops, it's immediately reestablished anyway, making the connection drop negligible? If that's the case, then the UI error should probably only be shown if the connection to live can't be established within a certain duration?

[sueun-dev]

Port-forwarding was just a diagnostic comparison to isolate ingress, not a suggestion that the UI should only be accessed that way. For a stream that remains silent, I agree that increasing a finite idle timeout only postpones the disconnect.

I reran an isolated test of the [same handler](

prometheus/web/api/v1/api.go

Lines 2129 to 2173 in fffc15f

	func (api *API) notificationsSSE(w http.ResponseWriter, r *http.Request) {
	httputil.SetCORS(w, api.CORSOrigin, r)
	w.Header().Set("Content-Type", "text/event-stream")
	w.Header().Set("Cache-Control", "no-cache")
	w.Header().Set("Connection", "keep-alive")
	// Subscribe to notifications.
	notifications, unsubscribe, ok := api.notificationsSub()
	if !ok {
	w.WriteHeader(http.StatusNoContent)
	return
	}
	defer unsubscribe()
	// Set up a flusher to push the response to the client.
	flusher, ok := w.(http.Flusher)
	if !ok {
	http.Error(w, "Streaming unsupported", http.StatusInternalServerError)
	return
	}
	// Flush the response to ensure the headers are immediately and eventSource
	// onopen is triggered client-side.
	flusher.Flush()
	for {
	select {
	case notification := <-notifications:
	// Marshal the notification to JSON.
	jsonData, err := json.Marshal(notification)
	if err != nil {
	http.Error(w, err.Error(), http.StatusInternalServerError)
	continue
	}
	// Write the event data in SSE format with JSON content.
	fmt.Fprintf(w, "data: %s\n\n", jsonData)
	// Flush the response to ensure the data is sent immediately.
	flusher.Flush()
	case <-r.Context().Done():
	return
	}
	}
	}

), with a stub subscriber: it returned 200, sent no body bytes during a 3-second observation, then delivered an injected notification. That checks the handler behavior, not your proxy chain.

The [client code at that commit](

prometheus/web/ui/mantine-ui/src/components/NotificationsProvider.tsx

Lines 25 to 66 in fffc15f

	useEffect(() => {
	const controller = new AbortController();
	fetchEventSource(`${pathPrefix}/api/v1/notifications/live`, {
	signal: controller.signal,
	async onopen(response) {
	if (response.ok) {
	if (response.status === 200) {
	setNotifications([]);
	setIsConnectionError(false);
	} else if (response.status === 204) {
	controller.abort();
	setShouldFetchFromAPI(true);
	}
	} else {
	setIsConnectionError(true);
	throw new Error(
	`Unexpected response: ${response.status} ${response.statusText}`,
	);
	}
	},
	onmessage(event) {
	const notification: Notification = JSON.parse(event.data);
	setNotifications((prev: Notification[]) => {
	const updatedNotifications = [
	...prev.filter((n: Notification) => n.text !== notification.text),
	];
	if (notification.active) {
	updatedNotifications.push(notification);
	}
	return updatedNotifications;
	});
	},
	onclose() {
	throw new Error("Server closed the connection");
	},
	onerror() {
	setIsConnectionError(true);
	return 5000;
	},

) already retries after 5 seconds, but onerror sets the error flag immediately. The next successful 200 clears it. So reconnection is implemented, while the warning currently has no grace period.

I don't know why heartbeats were omitted or whether that was deliberate. Periodic server heartbeats to prevent idle disconnects, and a short grace period before showing a connection warning, seem like two useful improvements to discuss with the maintainers.