Minimal RestateCluster not starting on GKE Autopilot

[dpbrackin]

Hello again 👋 ,

I tried settings up a cluster using the minimal example found in the docs:

apiVersion: restate.dev/v1
kind: RestateCluster
metadata:
  name: restate-test
spec:
  compute:
    image: restatedev/restate:1.5
  storage:
    storageRequestBytes: 2147483648 # 2 GiB

But the cluster isn't starting up. Here are the logs:

{"timestamp":"2025-10-14T14:44:05.357821Z","level":"INFO","fields":{"message":"Starting Restate Server 1.5.2 (2adae94 x86_64-unknown-linux-gnu 2025-10-08)","node_name":"restate-0","config_source":"/config/config.toml","base_dir":"/restate-data/restate-0/"},"target":"restate_server"}
{"timestamp":"2025-10-14T14:44:05.549888Z","level":"INFO","fields":{"message":"Server listening"},"target":"restate_core::network::net_util","span":{"network.transport":"tcp","server.address":"0.0.0.0","server.port":5122,"server_name":"node-rpc-server","name":"server"},"spans":[{"network.transport":"tcp","server.address":"0.0.0.0","server.port":5122,"server_name":"node-rpc-server","name":"server"}]}
{"timestamp":"2025-10-14T14:44:05.550091Z","level":"INFO","fields":{"message":"Trying to join the cluster 'restate-test'"},"target":"restate_node::init"}
{"timestamp":"2025-10-14T14:44:16.457742Z","level":"INFO","fields":{"message":"Failed to join the cluster 'restate-test'. Has the cluster been provisioned, yet? Still trying to join..."},"target":"restate_node::init"}
{"timestamp":"2025-10-14T14:44:45.604159Z","level":"INFO","fields":{"message":"Failed to join the cluster 'restate-test'. Has the cluster been provisioned, yet? Still trying to join..."},"target":"restate_node::init"}
{"timestamp":"2025-10-14T14:45:16.475339Z","level":"INFO","fields":{"message":"Failed to join the cluster 'restate-test'. Has the cluster been provisioned, yet? Still trying to join..."},"target":"restate_node::init"}
{"timestamp":"2025-10-14T14:45:25.497597Z","level":"ERROR","fields":{"message":"failed with error client error (Connect)"},"target":"octocrab"}
{"timestamp":"2025-10-14T14:45:46.062868Z","level":"INFO","fields":{"message":"Failed to join the cluster 'restate-test'. Has the cluster been provisioned, yet? Still trying to join..."},"target":"restate_node::init"}
{"timestamp":"2025-10-14T14:46:16.519921Z","level":"WARN","fields":{"message":"Failed to join the cluster 'restate-test'. Has the cluster been provisioned, yet? Still trying to join..."},"target":"restate_node::init"}
{"timestamp":"2025-10-14T14:46:45.672458Z","level":"WARN","fields":{"message":"Failed to join the cluster 'restate-test'. Has the cluster been provisioned, yet? Still trying to join..."},"target":"restate_node::init"}
{"timestamp":"2025-10-14T14:47:16.167300Z","level":"WARN","fields":{"message":"Failed to join the cluster 'restate-test'. Has the cluster been provisioned, yet? Still trying to join..."},"target":"restate_node::init"}
{"timestamp":"2025-10-14T14:47:46.646677Z","level":"WARN","fields":{"message":"Failed to join the cluster 'restate-test'. Has the cluster been provisioned, yet? Still trying to join..."},"target":"restate_node::init"}
{"timestamp":"2025-10-14T14:48:16.024853Z","level":"WARN","fields":{"message":"Failed to join the cluster 'restate-test'. Has the cluster been provisioned, yet? Still trying to join..."},"target":"restate_node::init"}
{"timestamp":"2025-10-14T14:48:46.709058Z","level":"WARN","fields":{"message":"Failed to join the cluster 'restate-test'. Has the cluster been provisioned, yet? Still trying to join..."},"target":"restate_node::init"}
{"timestamp":"2025-10-14T14:49:05.551504Z","level":"ERROR","fields":{"message":"Shutting down: task 6 failed with: Giving up trying to initialize the node. Make sure that it can reach the metadata store and don't forget to provision the cluster on a fresh start\n\nCaused by:\n    deadline has elapsed","kind":"SystemBoot","name":"\"init\""},"target":"restate_core::task_center"}
{"timestamp":"2025-10-14T14:49:05.551563Z","level":"WARN","fields":{"message":"** Shutdown requested","reason":"task init(6) failed and requested a shutdown"},"target":"restate_core::task_center"}
{"timestamp":"2025-10-14T14:49:05.552066Z","level":"INFO","fields":{"message":"1 column families flushed in 26.863µs","db":"db"},"target":"restate_rocksdb::rock_access","span":{"db":"db","name":"shutdown"},"spans":[{"db":"db","name":"shutdown"}]}
{"timestamp":"2025-10-14T14:49:05.552139Z","level":"INFO","fields":{"message":"Rocksdb 'db' was gracefully closed"},"target":"restate_rocksdb::rock_access","span":{"db":"db","name":"shutdown"},"spans":[{"db":"db","name":"shutdown"}]}
{"timestamp":"2025-10-14T14:49:05.552244Z","level":"INFO","fields":{"message":"2 column families flushed in 36.552µs","db":"replicated-metadata-server"},"target":"restate_rocksdb::rock_access","span":{"db":"replicated-metadata-server","name":"shutdown"},"spans":[{"db":"replicated-metadata-server","name":"shutdown"}]}
{"timestamp":"2025-10-14T14:49:05.552379Z","level":"INFO","fields":{"message":"Rocksdb 'replicated-metadata-server' was gracefully closed"},"target":"restate_rocksdb::rock_access","span":{"db":"replicated-metadata-server","name":"shutdown"},"spans":[{"db":"replicated-metadata-server","name":"shutdown"}]}
{"timestamp":"2025-10-14T14:49:05.552519Z","level":"INFO","fields":{"message":"Stopped listening"},"target":"restate_core::network::net_util","span":{"network.transport":"tcp","server.address":"0.0.0.0","server.port":5122,"server_name":"node-rpc-server","name":"server"},"spans":[{"network.transport":"tcp","server.address":"0.0.0.0","server.port":5122,"server_name":"node-rpc-server","name":"server"}]}
{"timestamp":"2025-10-14T14:49:05.555014Z","level":"INFO","fields":{"message":"2 column families flushed in 16.798µs","db":"log-server"},"target":"restate_rocksdb::rock_access","span":{"db":"log-server","name":"shutdown"},"spans":[{"db":"log-server","name":"shutdown"}]}
{"timestamp":"2025-10-14T14:49:05.555246Z","level":"INFO","fields":{"message":"Rocksdb 'log-server' was gracefully closed"},"target":"restate_rocksdb::rock_access","span":{"db":"log-server","name":"shutdown"},"spans":[{"db":"log-server","name":"shutdown"}]}
{"timestamp":"2025-10-14T14:49:05.564757Z","level":"INFO","fields":{"message":"Rocksdb manager shutdown completed"},"target":"restate_rocksdb::db_manager"}
{"timestamp":"2025-10-14T14:49:05.564793Z","level":"INFO","fields":{"message":"Task center has stopped"},"target":"restate_core::task_center"}
{"timestamp":"2025-10-14T14:49:05.564812Z","level":"INFO","fields":{"message":"Restate has gracefully shutdown in 13.252985ms"},"target":"restate_core::task_center"}
{"timestamp":"2025-10-14T14:49:05.564931Z","level":"ERROR","fields":{"message":"Restate terminated with exit code 1!"},"target":"restate_server"}

Thanks again

[jackkleeman]

hello, i am not able to reproduce this im afraid. some ideas:

1. the pod isnt able tor each itself over http://restate-0.restate-cluster:5122. that would imply networking issues. you could exec in and try curling that
2. the pod previously got halfway through provisioning itself and panicked/was killed, and now it is stuck. scaling down, deleting pvc, scaling up, could let you try again
3. some kind of cpu scheduling bug perhaps, if there is only 1 or 2 cpus available?

otherwise please set env: RUST_LOG=restate=debug,info and we might see more

[dpbrackin]

Hello

It looks like there was a networking issue. For context, I'm running my cluster on GKE Autopilot. The pod could not reach kube-dns with the default network policies. It's running now after disabling network policies.

Thank you for your help @jackkleeman

[jackkleeman]

no problem. can you let me know what labels would have selected kube dns in your cluster? perhaps the operator has the wrong labels

[tillrohrmann]

Root Cause Identified: NodeLocal DNSCache

This issue was caused by the operator's DNS network policy only allowing traffic to kube-dns pods via label selector. On GKE Autopilot (and other clusters with NodeLocal DNSCache enabled), DNS queries go to 169.254.20.10 instead, which was blocked.

Affected Environments

• GKE Autopilot (NodeLocal DNSCache is always enabled, cannot be disabled)
• Any Kubernetes cluster with NodeLocal DNSCache enabled

Symptoms

• Pods show 0/1 Running with repeated restarts
• Logs contain dns error caused by: failed to lookup address information
• transport error when connecting to metadata store

Fix

The allow-egress-to-kube-dns network policy will be renamed to allow-egress-to-dns and updated to allow DNS traffic to both:

1. kube-dns pods in kube-system namespace (existing)
2. 169.254.20.10/32 - the standard NodeLocal DNSCache IP (new)

Workaround (until fix is released)

Set spec.security.disableNetworkPolicies: true in your RestateCluster, or manually create an additional NetworkPolicy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-to-nodelocal-dns
  namespace: <restate-cluster-namespace>
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/instance: <cluster-name>
      app.kubernetes.io/name: restate
  policyTypes:
  - Egress
  egress:
  - ports:
    - port: 53
      protocol: UDP
    - port: 53
      protocol: TCP
    to:
    - ipBlock:
        cidr: 169.254.20.10/32