Failed to find instance to delegate to for agent #7133

nextrevision · 2016-12-19T20:01:31Z

Rancher Version:
v1.2.1

Docker Version:
1.11.3

OS and where are the hosts located? (cloud, bare metal, etc):
Ubuntu Trusty, AWS EC2

Setup Details: (single node rancher vs. HA rancher, internal DB vs. external DB)
Rancher HA w/ external DB

Environment Type: (Cattle/Kubernetes/Swarm/Mesos)
Cattle

Steps to Reproduce:
Upgraded from v1.1.4 to v1.2.1 and now the Rancher server is logging this error regularly. Possibly an orphaned agent?

Results:

2016-12-19 19:54:47,172 ERROR [:] [] [] [] [ecutorService-1] [i.c.p.agent.impl.AgentLocatorImpl   ] Failed to find instance to delegate to for agent [486] uri [delegate:///?vnetId=49&networkServiceProviderId=12]
2016-12-19 19:54:49,928 ERROR [:] [] [] [] [ecutorService-1] [.p.c.v.i.ConfigItemStatusManagerImpl] Error updating item(s) [configscripts, services, agent-instance-scripts, monit, agent-instance-startup, node-services, ipsec, ipsec-hosts, hosts, iptables, healthcheck] on [agent:486] com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Delegate [delegate:///?vnetId=49&networkServiceProviderId=12] has no instance associated
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache.get(LocalCache.java:3932) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4812) ~[guava-15.0.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl.lookupAgent(AgentLocatorImpl.java:85) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.call(ConfigUpdatePublisher.java:327) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.publishUpdate(ConfigUpdatePublisher.java:231) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.publish(ConfigUpdatePublisher.java:201) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.loopBody(ConfigUpdatePublisher.java:127) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.doRun(ConfigUpdatePublisher.java:100) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:15) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_72]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_72]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Delegate [delegate:///?vnetId=49&networkServiceProviderId=12] has no instance associated
	at io.cattle.platform.agent.impl.AgentLocatorImpl.getAgentForDelegate(AgentLocatorImpl.java:132) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl.buildDelegate(AgentLocatorImpl.java:104) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl$1.load(AgentLocatorImpl.java:52) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl$1.load(AgentLocatorImpl.java:47) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193) ~[guava-15.0.jar:na]
	... 23 common frames omitted

Expected:

The text was updated successfully, but these errors were encountered:

eltair · 2016-12-20T01:56:34Z

I have this issue as well after upgrading from 1.1.4 to 1.2.1. I'm running server on docker 1.11.2 on Amazon Linux AMI 2016.09. Hosts are running ubuntu.

chenapan · 2016-12-29T16:56:27Z

Same problem here, I thought first a problem with volumes while deploying a stack using them but even a clean restart of rancher server raises this exception that loops :

2016-12-29 16:55:08,337 ERROR [:] [] [] [] [ecutorService-1] [.p.c.v.i.ConfigItemStatusManagerImpl] Error updating item(s) [configscripts, services, agent-instance-scripts, monit, agent-instance-startup, node-services, ipsec, ipsec-hosts, hosts, iptables, healthcheck] on [agent:8] com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Delegate [delegate:///?vnetId=3&networkServiceProviderId=4] has no instance associated
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199) ~[guava-15.0.jar:na]

Rancher version : 1.2.2
Docker version 1.12.3, build 6b644ec

ephillipe · 2017-01-12T15:46:54Z

Same problem here:
Rancher version : 1.3.0
Docker version 1.12.6, build 78d1802

2017-01-12 15:46:39,051 ERROR [:] [] [] [] [ecutorService-7] [.p.c.v.i.ConfigItemStatusManagerImpl] Error updating item(s) [agent-instance-scripts, healthcheck, services, iptables, configscripts, hosts, ipsec-hosts, ipsec, node-services, agent-instance-startup, monit] on [agent:22404] com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Delegate [delegate:///?vnetId=594&networkServiceProviderId=1] has no instance associated

sra · 2017-01-23T12:28:45Z

Happening for us as well - the message is spamming as fast as it can print.

Rancher Server 1.3.2
Rancher OS 0.7.1

2017-01-22 19:47:37,962 ERROR [:] [] [] [] [ecutorService-6] [i.c.p.agent.impl.AgentLocatorImpl   ] Failed to find instance to delegate to for agent [180] uri [delegate:///?vnetId=19&networkServiceProviderId=3]
2017-01-22 19:47:37,966 ERROR [:] [] [] [] [ecutorService-6] [.p.c.v.i.ConfigItemStatusManagerImpl] Error updating item(s) [configscripts, services, agent-instance-scripts, monit, agent-instance-startup, node-services, ipsec, ipsec-hosts, hosts, iptables, healthcheck] on [agent:180] com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Delegate [delegate:///?vnetId=19&networkServiceProviderId=3] has no instance associated
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache.get(LocalCache.java:3932) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4812) ~[guava-15.0.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl.lookupAgent(AgentLocatorImpl.java:85) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.call(ConfigUpdatePublisher.java:327) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.publishUpdate(ConfigUpdatePublisher.java:231) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.publish(ConfigUpdatePublisher.java:201) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.loopBody(ConfigUpdatePublisher.java:127) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.doRun(ConfigUpdatePublisher.java:100) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:15) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_72]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_72]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Delegate [delegate:///?vnetId=19&networkServiceProviderId=3] has no instance associated
	at io.cattle.platform.agent.impl.AgentLocatorImpl.getAgentForDelegate(AgentLocatorImpl.java:132) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl.buildDelegate(AgentLocatorImpl.java:104) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl$1.load(AgentLocatorImpl.java:52) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl$1.load(AgentLocatorImpl.java:47) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193) ~[guava-15.0.jar:na]
	... 23 common frames omitted

2017-01-22 19:47:52,982 ERROR [:] [] [] [] [ecutorService-6] [i.c.p.agent.impl.AgentLocatorImpl   ] Failed to find instance to delegate to for agent [180] uri [delegate:///?vnetId=19&networkServiceProviderId=3]
2017-01-22 19:47:52,986 ERROR [:] [] [] [] [ecutorService-6] [.p.c.v.i.ConfigItemStatusManagerImpl] Error updating item(s) [configscripts, services, agent-instance-scripts, monit, agent-instance-startup, node-services, ipsec, ipsec-hosts, hosts, iptables, healthcheck] on [agent:180] com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Delegate [delegate:///?vnetId=19&networkServiceProvide

Ilya-Kuchaev · 2017-02-02T18:06:40Z

+1

tcdev0 · 2017-02-09T15:07:10Z

And again.

Rancher version : 1.4.0 / Cattle
Docker version 1.12.6
OS: Ubuntu 16.04

2017-02-09 15:00:10,354 ERROR [:] [] [] [] [ecutorService-5] [i.c.p.agent.impl.AgentLocatorImpl   ] Failed to find instance to delegate to for agent [317] uri [delegate:///?vnetId=11&networkServiceProviderId=5] 
2017-02-09 15:00:10,355 ERROR [:] [] [] [] [ecutorService-5] [.p.c.v.i.ConfigItemStatusManagerImpl] Error updating item(s) [services, agent-instance-scripts, monit, agent-instance-startup, node-services, ipsec, ipsec-hosts, hosts, iptables, healthcheck] on [agent:1919] com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Delegate [delegate:///?vnetId=21&networkServiceProviderId=8] has no instance associated
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache.get(LocalCache.java:3932) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4812) ~[guava-15.0.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl.lookupAgent(AgentLocatorImpl.java:85) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.call(ConfigUpdatePublisher.java:327) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.publishUpdate(ConfigUpdatePublisher.java:231) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.publish(ConfigUpdatePublisher.java:201) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.loopBody(ConfigUpdatePublisher.java:127) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.doRun(ConfigUpdatePublisher.java:100) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:15) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_72]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_72]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Delegate [delegate:///?vnetId=21&networkServiceProviderId=8] has no instance associated
	at io.cattle.platform.agent.impl.AgentLocatorImpl.getAgentForDelegate(AgentLocatorImpl.java:132) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl.buildDelegate(AgentLocatorImpl.java:104) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl$1.load(AgentLocatorImpl.java:52) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl$1.load(AgentLocatorImpl.java:47) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193) ~[guava-15.0.jar:na]
	... 23 common frames omitted

siosphere · 2017-02-14T23:03:55Z

Docker version: 1.12.1
Rancher version: 1.4.1 / Cattle
OS: Centos 7.2

2017-02-11 20:55:15,144 ERROR [:] [] [] [] [ecutorService-5] [.p.c.v.i.ConfigItemStatusManagerImpl] Error updating item(s) [agent-instance-startup, node-services, ipsec, ipsec-hosts, hosts, iptables, healthcheck, rancher-net] on [agent:405] com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Delegate [delegate:///?vnetId=15&networkServiceProviderId=3] has no instance associated
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache.get(LocalCache.java:3932) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4812) ~[guava-15.0.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl.lookupAgent(AgentLocatorImpl.java:85) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.call(ConfigUpdatePublisher.java:327) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.publishUpdate(ConfigUpdatePublisher.java:231) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.publish(ConfigUpdatePublisher.java:201) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.loopBody(ConfigUpdatePublisher.java:127) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.configitem.version.impl.ConfigUpdatePublisher.doRun(ConfigUpdatePublisher.java:100) [cattle-config-item-common-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:15) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_72]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_72]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_72]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_72]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Delegate [delegate:///?vnetId=15&networkServiceProviderId=3] has no instance associated
	at io.cattle.platform.agent.impl.AgentLocatorImpl.getAgentForDelegate(AgentLocatorImpl.java:132) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl.buildDelegate(AgentLocatorImpl.java:104) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl$1.load(AgentLocatorImpl.java:52) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.agent.impl.AgentLocatorImpl$1.load(AgentLocatorImpl.java:47) ~[cattle-iaas-agent-0.5.0-SNAPSHOT.jar:na]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278) ~[guava-15.0.jar:na]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193) ~[guava-15.0.jar:na]
	... 23 common frames omitted

This eventually causes Rancher to stop responding to web requests. Temporary fix is to stop the container, remove it, and re-create a new container (pointed to the same existing external DB), this proved ~2 days of up time, before it stops responding to web requests again.

CLEMARCx · 2017-02-20T08:54:36Z

Hi there,

I had the same error with rancher and a local database. The mysql database was corrupt.
After

docker exec -i -t containerID /bin/bash
mysqlrepair --all-databases --auto-repair

the errors went away and rancher was reachable again.

aemneina · 2017-03-17T17:58:48Z

@CLEMARCx I dont think you have the same issue as the others in this thread. It looks like there are orphaned agents, there might be an entry for the agent that doesnt match any existing host. Does this query return anything: select id from agent where id not in (select agent_id from host)

csbogdan · 2017-03-21T01:15:03Z

+1 for the error with rancher 1.3.3, upgraded sequentially from 1.1.4 (original installation).

tcdev0 · 2017-03-27T06:13:33Z

@aemneina executing your statement returned for me 246 rows / results

sankaet · 2017-03-27T07:21:58Z

Hi any resolution here? I am seeing something similar: #8308

sra · 2017-03-27T07:51:21Z

I made this quit by finding the row in the agent table in the rancher database (use the delegate message from your log messages):

select * from agent where uri like 'delegate:///?vnetId=19&networkServiceProviderId=3'

Then I marked the record as purged and restarted rancher server.

update agent set state = 'purged' where id = ID_FROM_PREVIOUS_SELECT

This stopped the spam, but I'm not sure if it had any other effect. So far so good.

sankaet · 2017-03-27T07:53:51Z

@sra in my case it returns an empty set

greenu · 2017-03-30T10:42:17Z

@sankaet try this:
update agent set removed = NOW(), state = 'removed' where removed is null and uri like 'delegate%'

snipking · 2017-03-31T10:28:11Z

@aemneina @sra

Thanks. That's worked for me.

Find out active agent records which didn't related to a host.

select id, kind, uuid, state, created, uri from agent where id not in (select agent_id from host) and state = 'active' and uri like 'delegate%';

Double check the uri with your rancher-server log. In my case I got 2 records which uri match the exception in server log.
Update the record state to purged

update agent set state = 'purged' where id = <ID_FROM_PREVIOUS_SELECT>

Restart rancher-server.

aemneina · 2017-05-17T17:06:27Z

This should have since been resolved in newer versions of rancher. Please file a new issue if there are any issues surfacing post upgrades.

aemneina closed this as completed May 17, 2017

aemneina mentioned this issue May 17, 2017

Failed to find instance to delegate to for agent #7166

Closed

robertoamoreno mentioned this issue Jul 12, 2017

Failed to find instance to delegate to for agent 1.4.3 #9316

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to find instance to delegate to for agent #7133

Failed to find instance to delegate to for agent #7133

nextrevision commented Dec 19, 2016 •

edited

eltair commented Dec 20, 2016

chenapan commented Dec 29, 2016

ephillipe commented Jan 12, 2017

sra commented Jan 23, 2017 •

edited

Ilya-Kuchaev commented Feb 2, 2017

tcdev0 commented Feb 9, 2017

siosphere commented Feb 14, 2017

CLEMARCx commented Feb 20, 2017 •

edited

aemneina commented Mar 17, 2017

csbogdan commented Mar 21, 2017

tcdev0 commented Mar 27, 2017

sankaet commented Mar 27, 2017

sra commented Mar 27, 2017 •

edited

sankaet commented Mar 27, 2017

greenu commented Mar 30, 2017 •

edited

snipking commented Mar 31, 2017 •

edited

aemneina commented May 17, 2017

Failed to find instance to delegate to for agent #7133

Failed to find instance to delegate to for agent #7133

Comments

nextrevision commented Dec 19, 2016 • edited

eltair commented Dec 20, 2016

chenapan commented Dec 29, 2016

ephillipe commented Jan 12, 2017

sra commented Jan 23, 2017 • edited

Ilya-Kuchaev commented Feb 2, 2017

tcdev0 commented Feb 9, 2017

siosphere commented Feb 14, 2017

CLEMARCx commented Feb 20, 2017 • edited

aemneina commented Mar 17, 2017

csbogdan commented Mar 21, 2017

tcdev0 commented Mar 27, 2017

sankaet commented Mar 27, 2017

sra commented Mar 27, 2017 • edited

sankaet commented Mar 27, 2017

greenu commented Mar 30, 2017 • edited

snipking commented Mar 31, 2017 • edited

aemneina commented May 17, 2017

nextrevision commented Dec 19, 2016 •

edited

sra commented Jan 23, 2017 •

edited

CLEMARCx commented Feb 20, 2017 •

edited

sra commented Mar 27, 2017 •

edited

greenu commented Mar 30, 2017 •

edited

snipking commented Mar 31, 2017 •

edited