Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed to install Spinnaker on Kubernetes 1.4 data-redis-master CrashLoopBackOff #1354

Closed
BruceZu opened this issue Dec 30, 2016 · 5 comments
Closed

Failed to install Spinnaker on Kubernetes 1.4 data-redis-master CrashLoopBackOff #1354

BruceZu opened this issue Dec 30, 2016 · 5 comments

Comments

@BruceZu
Copy link

BruceZu commented Dec 30, 2016 •
edited

Title

Failed to install Spinnaker on Kubernetes
I read through all issues and did not find similar issues.
Cloud Provider

Kubernetes,

[root@k8s-09 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"3b417cc4ccd1b8f38ff9ec96bb50a81ca0ea9d56", GitTreeState:"clean", BuildDate:"2016-10-21T02:42:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
[root@k8s-09 ~]#

Environment

I follow the document at https://github.com/spinnaker/spinnaker/tree/master/experimental/kubernetes/simple
to deploy Spainnaker on Kubernetes cluster which is created on meta ( OS is CentOS 7)

Feature Area
I am new comer and just touch Spinnaker from yesterday.
Not sure which features the issue is related to
Description

I expect the pods running without error

Steps to Reproduce

Steps on Kubernate cluster master machine:

[root@k8s-09 ~]# pwd
/root
[root@k8s-09 ~]# mkdir spinnaker
[root@k8s-09 ~]# cd spinnaker/
[root@k8s-09 spinnaker]# git clone https://github.com/spinnaker/spinnaker.git
Cloning into 'spinnaker'...
< ....>
[root@k8s-09 spinnaker]# cd spinnaker/experimental/kubernetes/simple/
[root@k8s-09 simple]# bash scripts/startup-all.sh
namespace "spinnaker" created
Bootstrapping redis master...

service "data-redis-server" created
replicationcontroller "data-redis-master-v000" created
secret "spinnaker-config" created
secret "creds-config" created
secret "aws-config" created
Starting all services...

service "spin-clouddriver" created
service "spin-deck" created
service "spin-echo" created
service "spin-front50" created
service "spin-gate" created
service "spin-igor" created
service "spin-orca" created

Starting all replication controllers...

replicationcontroller "spin-clouddriver-v000" created
replicationcontroller "spin-deck-v000" created
replicationcontroller "spin-echo-v000" created
replicationcontroller "spin-front50-v000" created
replicationcontroller "spin-gate-v000" created
replicationcontroller "spin-igor-v000" created
replicationcontroller "spin-orca-v000" created
[root@k8s-09 simple]# kubectl get svc --namespace=spinnaker
NAME                CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
data-redis-server   10.96.108.67     <none>        6379/TCP   37s
spin-clouddriver    10.109.239.176   <none>        7002/TCP   35s
spin-deck           10.104.5.254     <none>        9000/TCP   34s
spin-echo           10.101.254.178   <none>        8089/TCP   34s
spin-front50        10.104.25.182    <none>        8080/TCP   34s
spin-gate           10.99.192.23     <none>        8084/TCP   33s
spin-igor           10.106.242.82    <none>        8088/TCP   33s
spin-orca           10.102.108.9     <none>        8083/TCP   32s
[root@k8s-09 simple]# kubectl get secret --namespace=spinnaker
NAME                  TYPE                                  DATA      AGE
aws-config            Opaque                                0         2m
creds-config          Opaque                                1         2m
default-token-pu8be   kubernetes.io/service-account-token   3         2m
spinnaker-config      Opaque                                18        2m
[root@k8s-09 simple]# kubectl get replicationcontroller  --namespace=spinnaker
NAME                     DESIRED   CURRENT   READY     AGE
data-redis-master-v000   1         1         0         2m
spin-clouddriver-v000    1         1         0         2m
spin-deck-v000           1         1         0         2m
spin-echo-v000           1         1         0         2m
spin-front50-v000        1         1         0         2m
spin-gate-v000           1         1         0         2m
spin-igor-v000           1         1         0         2m
spin-orca-v000           1         1         0         2m
[root@k8s-09 simple]# kubectl get replicationcontroller  --namespace=spinnaker
NAME                     DESIRED   CURRENT   READY     AGE
data-redis-master-v000   1         1         0         12m
spin-clouddriver-v000    1         1         1         12m
spin-deck-v000           1         1         1         12m
spin-echo-v000           1         1         0         12m
spin-front50-v000        1         1         1         12m
spin-gate-v000           1         1         0         12m
spin-igor-v000           1         1         0         12m
spin-orca-v000           1         1         0         12m
[root@k8s-09 simple]# kubectl get pod  --namespace=spinnaker
NAME                           READY     STATUS             RESTARTS   AGE
data-redis-master-v000-ynnnq   0/1       CrashLoopBackOff   7          12m
spin-clouddriver-v000-tnkze    1/1       Running            0          12m
spin-deck-v000-bqkg1           1/1       Running            0          12m
spin-echo-v000-6dfbs           0/1       Running            0          12m
spin-front50-v000-bfhx0        1/1       Running            0          12m
spin-gate-v000-b2uua           0/1       Running            0          12m
spin-igor-v000-ox26g           0/1       Running            0          12m
spin-orca-v000-bkqsf           0/1       Error              2          12m

[root@k8s-09 ~]# kubectl logs data-redis-master-v000-ynnnq  --namespace=spinnaker
[21] 30 Dec 01:34:35.952 # Can't open the append-only file: Permission denied
[root@k8s-09 ~]#

[root@k8s-09 simple]# kubectl attach spin-orca-v000-bkqsf --namespace=spinnaker

2016-12-30 01:42:24.624 ERROR 1 --- [           main] o.s.boot.SpringApplication               : Application startup failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'workerPool' defined in class com.netflix.spinnaker.orca.config.JesqueConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [net.lariverosc.jesquespring.SpringWorkerPool]: Factory method 'workerPool' threw exception; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Connection refused (Connection refused)

[root@k8s-09 ~]# kubectl logs spin-clouddriver-v000-tnkze  --namespace=spinnaker  
......
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
        at redis.clients.util.Pool.getResource(Pool.java:50)
        at redis.clients.jedis.JedisPool.getResource(JedisPool.java:99)
        at com.netflix.spinnaker.cats.redis.JedisPoolSource.getJedis(JedisPoolSource.java:31)
        at com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.acquireRunKey(ClusteredAgentScheduler.java:102)
        at com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.acquire(ClusteredAgentScheduler.java:63)
        at com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.runAgents(ClusteredAgentScheduler.java:84)
        at com.netflix.spinnaker.cats.redis.cluster.ClusteredAgentScheduler.run(ClusteredAgentScheduler.java:77)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Connection refused (Connection refused)
        at redis.clients.jedis.Connection.connect(Connection.java:164)
        at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:82)
        at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1641)
        at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:85)
        at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
        at redis.clients.util.Pool.getResource(Pool.java:48)
        ... 13 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at redis.clients.jedis.Connection.connect(Connection.java:158)
        ... 20 more

......
# kubectl logs spin-echo-v000-6dfbs  --namespace=spinnaker  |grep ERROR 
2016-12-30 00:25:22.484 ERROR 1 --- [           main] o.apache.cassandra.cql3.QueryProcessor   : Unable to initialize MemoryMeter (jamm not specified as javaagent).  This means Cassandra will be unable to measure object sizes accurately and may consequently OOM.
2016-12-30 00:25:22.669 ERROR 1 --- [           main] o.a.cassandra.service.CassandraDaemon    : Directory build/cassandra-test/data doesn't exist
2016-12-30 00:25:22.672 ERROR 1 --- [           main] o.a.cassandra.service.CassandraDaemon    : Directory build/cassandra-test/commitlog doesn't exist
2016-12-30 00:25:22.672 ERROR 1 --- [           main] o.a.cassandra.service.CassandraDaemon    : Directory build/cassandra-test/saved_caches doesn't exist
2016-12-30 00:25:23.217 ERROR 1 --- [           main] org.apache.cassandra.db.Memtable         : MemoryMeter uninitialized (jamm not specified as java agent); assuming liveRatio of 10.0.   Usually this means cassandra-env.sh disabled jamm because you are using a buggy JRE;  upgrade to the Sun JRE instead
2016-12-30 00:25:33.923 ERROR 1 --- [           main] o.s.boot.SpringApplication               : Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'historyController': Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire field: private com.netflix.spinnaker.echo.events.EventPropagator com.netflix.spinnaker.echo.history.HistoryController.propagator; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'propagator' defined in class path resource [com/netflix/spinnaker/echo/config/EchoCoreConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.netflix.spinnaker.echo.events.EventPropagator]: Factory method 'propagator' threw exception; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'orca.baseUrl' in string value "${orca.baseUrl}"

[root@k8s-09 ~]# kubectl logs spin-front50-v000-bfhx0  --namespace=spinnaker  |grep ERROR
2016-12-30 00:26:27.462 ERROR 1 --- [ost-startStop-1] o.apache.cassandra.cql3.QueryProcessor   : Unable to initialize MemoryMeter (jamm not specified as javaagent).  This means Cassandra will be unable to measure object sizes accurately and may consequently OOM.
2016-12-30 00:26:27.639 ERROR 1 --- [ost-startStop-1] o.a.cassandra.service.CassandraDaemon    : Directory build/cassandra-test/data doesn't exist
2016-12-30 00:26:27.642 ERROR 1 --- [ost-startStop-1] o.a.cassandra.service.CassandraDaemon    : Directory build/cassandra-test/commitlog doesn't exist
2016-12-30 00:26:27.642 ERROR 1 --- [ost-startStop-1] o.a.cassandra.service.CassandraDaemon    : Directory build/cassandra-test/saved_caches doesn't exist
2016-12-30 00:26:28.217 ERROR 1 --- [ost-startStop-1] org.apache.cassandra.db.Memtable         : MemoryMeter uninitialized (jamm not specified as java agent); assuming liveRatio of 10.0.   Usually this means cassandra-env.sh disabled jamm because you are using a buggy JRE;  upgrade to the Sun JRE instead
2016-12-30 00:26:28.225 ERROR 1 --- [ost-startStop-1] org.apache.cassandra.db.Memtable         : MemoryMeter uninitialized (jamm not specified as java agent); assuming liveRatio of 10.0.   Usually this means cassandra-env.sh disabled jamm because you are using a buggy JRE;  upgrade to the Sun JRE instead


[root@k8s-09 ~]# kubectl logs spin-gate-v000-b2uua  --namespace=spinnaker  |grep ERROR  
2016-12-30 00:25:46.079 ERROR 1 --- [           main] o.s.boot.SpringApplication               : Application startup failed
2016-12-30 00:26:24.986 ERROR 1 --- [readScheduler-1] c.n.s.gate.services.ApplicationService   : Unable to refresh application list, reason: Task java.util.concurrent.Futu
reTask@6a9ddee7 rejected from java.util.concurrent.ThreadPoolExecutor@970bbb7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]

spin-deck and spin-igor seems okay

[root@k8s-09 ~]# kubectl logs spin-deck-v000-bqkg1  --namespace=spinnaker  | grep ERROR
[root@k8s-09 ~]# kubectl logs spin-igor-v000-ox26g  --namespace=spinnaker  | grep ERROR
[root@k8s-09 ~]#
[root@k8s-09 ~]# kubectl get pod  --namespace=spinnaker
NAME                           READY     STATUS             RESTARTS   AGE
data-redis-master-v000-ynnnq   0/1       CrashLoopBackOff   202        17h
spin-clouddriver-v000-tnkze    1/1       Running            0          17h
spin-deck-v000-bqkg1           1/1       Running            0          17h
spin-echo-v000-6dfbs           0/1       Running            0          17h
spin-front50-v000-bfhx0        1/1       Running            0          17h
spin-gate-v000-b2uua           0/1       Running            0          17h
spin-igor-v000-ox26g           0/1       Running            0          17h
spin-orca-v000-bkqsf           0/1       Running            157        17h
[root@k8s-09 ~]#

**But from the status spin-igor seems still not ready
6 or 8 pods can not work. I am not clear the relation between them and from where to do trouble shooting firstly.

Any hints are welcome**

Additional Details
at the " Configuring your Docker registry"
I did not touch or update the ./config/clouddriver-local.yml file
I ever plan update it later once GCR account is ready as it need pay and I have to apply to manager.

at the " Configuring Pipeline Storage"
I did not touch or update the ./config/clouddriver-local.yml
I have neither GCS or AWS account and plan update it later

please correct me if these 2 -local.yml files must be updated before I run the initial script # bash scripts/startup-all.sh
@BruceZu BruceZu changed the title Failed to install Spinnaker on Kubernetes Failed to install Spinnaker on Kubernetes 1.4 Dec 30, 2016
@BruceZu
Copy link
Author

BruceZu commented Dec 30, 2016

@gregturn will you please shed some light here?

@BruceZu BruceZu changed the title Failed to install Spinnaker on Kubernetes 1.4 Failed to install Spinnaker on Kubernetes 1.4 data-redis-master CrashLoopBackOff Dec 30, 2016
@gregturn
Copy link
Contributor

gregturn commented Jan 2, 2017

@BruceZu Sorry, but I've never used K8.

@BruceZu
Copy link
Author

BruceZu commented Jan 7, 2017 •
edited

I did not find any official meaning of CrashLoopBackOff. This is a reference link http://stackoverflow.com/questions/36683240/kubernetes-replication-controller-in-crashloopbackoff-state
and
kubernetes/kubernetes#16507

@ismith
Copy link

ismith commented Jan 7, 2017

If you get a crashloopbackoff, I have found the relevant logs using kubectl describe - so kubectl describe pod data-redis-master-v000-ynnnq. I would guess most or all of your other errors connect to redis not being up.

(CrashLoopBackOff meaning - the container failed to start [crash], so k8s tried again [loop] with an exponential backoff.)

@lwander
Copy link
Member

lwander commented Jun 7, 2017

Closing since this is using a deprecated install path. If you want help, ping me (@lwander) on join.spinnaker.io.

@lwander lwander closed this as completed Jun 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ismith @gregturn @BruceZu @lwander