K8SSAND-118 ⁃ When a restart of the C* nodes in a cluster occurs, Stargate is then unable to function #411

jdonenine · 2021-02-23T03:25:16Z

Bug Report

Describe the bug
Have observed a scenario where all of the C* nodes in the cluster restart while Stargate is up and running. In this example, going from cassandra.datacenters[0].size=3 down to cassandra.datacenters[0].size=1. When this happens a full rolling restart of the Cassandra pods can be observed. After the restart is completed the Stargate node is then unable to connect to the cluster and serve any requests with errors like:

{"description":"Server error: java.util.concurrent.ExecutionException: org.apache.cassandra.stargate.exceptions.UnavailableException: Cannot achieve consistency level ONE","code":500}

To Reproduce
Steps to reproduce the behavior:

Deploy a cluster with Traefik and using a configuration like this, where cassandra.datacenters[0].size=3

cassandra:
  version: "3.11.10"
  clusterName: k8ssandra-cluster
  heap:
   size: 500M
   newGenSize: 256M
  datacenters:
  - name: dc1
    size: 3
kube-prometheus-stack:
  prometheus:
    enabled: true
    prometheusSpec:
      externalUrl: http://localhost:9090/prometheus
      routePrefix: /prometheus
    ingress:
      enabled: true
      paths:
        - /prometheus
  grafana:
    enabled: true
    ingress:
      enabled: true
      path: /grafana
    adminUser: admin
    adminPassword: admin123
    grafana.ini:
      server:
        root_url: http://localhost:3000/grafana
        serve_from_sub_path: true
stargate:
  enabled: true
  replicas: 1
  ingress:
    enabled: true
reaper:
  ingress:
    enabled: true
    host: repair.127.0.0.1.nip.io

Observe the deployment

% kubectl get pods
NAME                                                  READY   STATUS      RESTARTS   AGE
k8ssandra-cass-operator-86958b76d5-dxqkb              1/1     Running     0          73m
k8ssandra-cluster-dc1-default-sts-0                   2/2     Running     0          4m44s
k8ssandra-cluster-dc1-default-sts-1                   2/2     Running     0          7m34s
k8ssandra-cluster-dc1-default-sts-2                   2/2     Running     0          10m
k8ssandra-dc1-stargate-5679468895-7kw9k               1/1     Running     0          4m40s
k8ssandra-grafana-b485f8785-5hp4r                     2/2     Running     0          73m
k8ssandra-kube-prometheus-operator-65d6cc8fd4-prpzh   1/1     Running     0          73m
k8ssandra-reaper-k8ssandra-676bbcd9c4-99snh           1/1     Running     8          63m
k8ssandra-reaper-k8ssandra-schema-7fc5b               0/1     Completed   0          63m
k8ssandra-reaper-operator-6d7fc46fcb-24zlf            1/1     Running     0          73m
prometheus-k8ssandra-kube-prometheus-prometheus-0     2/2     Running     1          73m
traefik-d5896685d-6fzkc                               1/1     Running     0          74m

Test a query via Stargate

% curl -L -X POST 'http://127.0.0.1:8081/v1/auth' -H 'Content-Type: application/json' --data-raw '{"username": "'$CASS_USERNAME'", "password": "'$CASS_PASSWORD'"}'

{"authToken":"659ab419-7f57-4ba8-b457-923002fef00b"}

STARGATE_AUTH_TOKEN="dc12c812-44bc-46c9-b048-4482a2171b48"

% curl --location --request GET 'http://stargate.127.0.0.1.nip.io:8080/v2/schemas/namespaces' \
--header "x-cassandra-token: $STARGATE_AUTH_TOKEN"

{"data":[{"name":"system_distributed","datacenters":[{"name":"dc1","replicas":3}]},{"name":"system"},{"name":"data_endpoint_auth"},{"name":"system_schema"},{"name":"stargate_system"},{"name":"reaper_db","datacenters":[{"name":"dc1","replicas":3}]},{"name":"system_auth","datacenters":[{"name":"dc1","replicas":3}]},{"name":"system_traces","datacenters":[{"name":"dc1","replicas":3}]}]}

Now upgrade the deployment setting cassandra.datacenters[0].size=1
Observe the deployment - watch it as it scales down and you will see that each of the C* pods will restart, the stargate pod will remain in place:

% kubecl get pods
NAME                                                  READY   STATUS             RESTARTS   AGE
k8ssandra-cass-operator-86958b76d5-dxqkb              1/1     Running            0          103m
k8ssandra-cluster-dc1-default-sts-0                   2/2     Running            0          13m
k8ssandra-dc1-stargate-5679468895-7kw9k               1/1     Running            0          34m
k8ssandra-grafana-b485f8785-5hp4r                     2/2     Running            0          103m
k8ssandra-kube-prometheus-operator-65d6cc8fd4-prpzh   1/1     Running            0          103m
k8ssandra-reaper-k8ssandra-676bbcd9c4-99snh           0/1     CrashLoopBackOff   14         93m
k8ssandra-reaper-k8ssandra-schema-7fc5b               0/1     Completed          0          93m
k8ssandra-reaper-operator-6d7fc46fcb-24zlf            1/1     Running            0          103m
prometheus-k8ssandra-kube-prometheus-prometheus-0     2/2     Running            1          103m
traefik-d5896685d-6fzkc                               1/1     Running            0          104m

Here you can see that the dc1-default pod has restarted more recently than the dc1-stargate

Test a request through Stargate again and observe the error:

% curl --location --request GET 'http://stargate.127.0.0.1.nip.io:8080/v2/schemas/namespaces' \
--header "x-cassandra-token: $STARGATE_AUTH_TOKEN"

{"description":"Server error: java.util.concurrent.ExecutionException: org.apache.cassandra.stargate.exceptions.UnavailableException: Cannot achieve consistency level ONE","code":500}

Exceptions like this can be seen in the stargate logs:

ERROR [dw-140 - GET /v2/schemas/namespaces] 2021-02-23 02:45:05,211 AuthnTableBasedService.java:276 - Failed to validate token
java.util.concurrent.ExecutionException: org.apache.cassandra.stargate.exceptions.UnavailableException: Cannot achieve consistency level ONE
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
        at io.stargate.auth.table.AuthnTableBasedService.validateToken(AuthnTableBasedService.java:251)
        at io.stargate.auth.AuthenticationService.validateToken(AuthenticationService.java:33)
        at io.stargate.web.resources.Db.getDataStoreForToken(Db.java:72)
        at io.stargate.web.docsapi.resources.NamespacesResource.lambda$getAllNamespaces$1(NamespacesResource.java:95)
        at io.stargate.web.resources.RequestHandler.handle(RequestHandler.java:36)
        at io.stargate.web.docsapi.resources.NamespacesResource.getAllNamespaces(NamespacesResource.java:92)
        at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
        at io.dropwizard.jetty.NonblockingServletHolder.handle(NonblockingServletHolder.java:50)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1631)
        at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
        at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47)
        at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
        at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:319)
        at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:273)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1369)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:489)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1284)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:249)
        at io.dropwizard.jetty.ContextRoutingHandler.handle(ContextRoutingHandler.java:37)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:767)
        at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54)
        at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:501)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:272)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.cassandra.stargate.exceptions.UnavailableException: Cannot achieve consistency level ONE
        at io.stargate.db.cassandra.impl.Conversion.toExternal(Conversion.java:276)
        at io.stargate.db.cassandra.impl.Conversion.convertInternalException(Conversion.java:472)
        at io.stargate.db.cassandra.impl.CassandraPersistence$CassandraConnection.lambda$executeRequestOnExecutor$0(CassandraPersistence.java:443)
        at io.stargate.db.cassandra.impl.CassandraPersistence.lambda$runOnExecutor$1(CassandraPersistence.java:268)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165)
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:113)
        ... 1 common frames omitted
        Suppressed: org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level ONE

Expected behavior
The system should be tolerant of restarts of the C* pods.

Environment (please complete the following information):

Helm charts version info

% helm ls -A
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
k8ssandra       default         10              2021-02-22 21:37:16.239715 -0500 EST    deployed        k8ssandra-0.55.0        3.11.10    
traefik         default         1               2021-02-22 20:15:39.848483 -0500 EST    deployed        traefik-9.14.3          2.4.5

Helm charts user-supplied values

% helm get values k8ssandra
USER-SUPPLIED VALUES:
cassandra:
  clusterName: k8ssandra-cluster
  datacenters:
  - name: dc1
    size: 1
  heap:
    newGenSize: 256M
    size: 500M
  version: 3.11.10
kube-prometheus-stack:
  grafana:
    adminPassword: admin123
    adminUser: admin
    enabled: true
    grafana.ini:
      server:
        root_url: http://localhost:3000/grafana
        serve_from_sub_path: true
    ingress:
      enabled: true
      path: /grafana
  prometheus:
    enabled: true
    ingress:
      enabled: true
      paths:
      - /prometheus
    prometheusSpec:
      externalUrl: http://localhost:9090/prometheus
      routePrefix: /prometheus
reaper:
  ingress:
    enabled: true
    host: repair.127.0.0.1.nip.io
stargate:
  enabled: true
  ingress:
    enabled: true
  replicas: 1

Kubernetes version information:

% kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.17", GitCommit:"f3abc15296f3a3f54e4ee42e830c61047b13895f", GitTreeState:"clean", BuildDate:"2021-01-22T22:20:06Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes cluster kind:

Kind

% k get nodes
NAME                      STATUS   ROLES    AGE    VERSION
k8ssandra-control-plane   Ready    master   132m   v1.17.17
k8ssandra-worker          Ready    <none>   132m   v1.17.17
k8ssandra-worker2         Ready    <none>   132m   v1.17.17
k8ssandra-worker3         Ready    <none>   132m   v1.17.17

Additional context

Forcing a restart of Stargate via something like helm upgrade k8ssandra k8ssandra/k8ssandra --reuse-values --set stargate.enabled=false followed by helm upgrade k8ssandra k8ssandra/k8ssandra --reuse-values --set stargate.enabled=true after the C* restart is complete will correct the problem.

┆Issue is synchronized with this Jira Bug by Unito
┆friendlyId: K8SSAND-118
┆priority: Medium

The text was updated successfully, but these errors were encountered:

burmanm · 2021-02-23T10:34:22Z

This also means that Stargate will not survive restore with stopped parameter (for example, schema restore).

jdonenine · 2021-02-23T16:24:53Z

It's worth mentioning so that it's clear, this seems to be a problem when ALL C* nodes is in play go down. Either because there was only a single C* node, or because they were forced down together at the same time for other reasons (failure, medusa restore w/ full shutdown, etc). I saw no issues during rolling restarts while at least one C* node was up and running.

jdonenine added bug Something isn't working needs-triage component:stargate labels Feb 23, 2021

jdonenine added this to the 1.0.0 milestone Feb 23, 2021

jdonenine assigned jsanda Feb 23, 2021

dimas-b mentioned this issue Feb 23, 2021

Automatically reload seeds from DNS when unable to connect to storage nodes stargate/stargate#710

Closed

jdonenine unassigned jsanda Feb 23, 2021

jdonenine modified the milestones: 1.0.0, Post-1.0.0, Expanded K8s Support + Automation, Post-1.0.0 Parking Lot Feb 23, 2021

jdonenine modified the milestones: Post-1.0.0 Parking Lot, Backlog Mar 20, 2021

This was referenced Apr 8, 2021

TestFullStackScenario fails on the Test_Stargate test #651

Closed

Draft release notes for 1.1.0 #653

Merged

jdonenine removed this from the Backlog milestone Jun 2, 2021

sync-by-unito bot changed the title ~~When a restart of the C* nodes in a cluster occurs, Stargate is then unable to function~~ K8SSAND-118 ⁃ When a restart of the C* nodes in a cluster occurs, Stargate is then unable to function Apr 1, 2022

adejanovski added the help-wanted Extra attention is needed label Jan 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

K8SSAND-118 ⁃ When a restart of the C* nodes in a cluster occurs, Stargate is then unable to function #411

K8SSAND-118 ⁃ When a restart of the C* nodes in a cluster occurs, Stargate is then unable to function #411

jdonenine commented Feb 23, 2021 •

edited by sync-by-unito bot

Loading

burmanm commented Feb 23, 2021

jdonenine commented Feb 23, 2021 •

edited

Loading

K8SSAND-118 ⁃ When a restart of the C* nodes in a cluster occurs, Stargate is then unable to function #411

K8SSAND-118 ⁃ When a restart of the C* nodes in a cluster occurs, Stargate is then unable to function #411

Comments

jdonenine commented Feb 23, 2021 • edited by sync-by-unito bot Loading

Bug Report

burmanm commented Feb 23, 2021

jdonenine commented Feb 23, 2021 • edited Loading

jdonenine commented Feb 23, 2021 •

edited by sync-by-unito bot

Loading

jdonenine commented Feb 23, 2021 •

edited

Loading