High CPU usage after 1.2.0-rc #6857

miv · 2016-12-01T15:12:02Z

After upgrading to 1.2.0-rc1 and beyond (from 1.2.0-pre6) cpu usage skyrocketed: http://monosnap.com/image/tAsPZJZiVSNaSciz8dWUkOfaAb9H5K

Logs from server: https://gist.github.com/miv/fe7e088e9426d06662eaec08c4a2b43d

View from inside container:

root@67713b730bdb:/# ps auxww
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0    188     4 ?        Ss   15:01   0:00 /usr/bin/s6-svscan /service
root         6  0.0  0.0    188     4 ?        S    15:01   0:00 s6-supervise cattle
root         7  0.0  0.0    188     4 ?        S    15:01   0:00 s6-supervise mysql
root         8  167 15.9 6922908 1306484 ?     Ssl  15:01  13:19 java -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Xms128m -Xmx4g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/cattle/logs -Dlogback.bootstrap.level=WARN -cp /usr/share/cattle/e4431ee2b0e535064a4e3e6a7bf8dc67:/usr/share/cattle/e4431ee2b0e535064a4e3e6a7bf8dc67/etc/cattle io.cattle.platform.launcher.Main
root        76  4.2  0.2 207764 22820 ?        Sl   15:02   0:18 websocket-proxy
root        84  0.3  0.2 157860 16676 ?        Sl   15:02   0:01 rancher-catalog-service -configFile repo.json -refreshInterval 300
root        86  0.1  0.2 137528 17848 ?        Sl   15:02   0:00 go-machine-service
root        88  0.0  0.1  18248 12928 ?        Sl   15:02   0:00 rancher-compose-executor
root        92  0.1  0.2 149624 20232 ?        Sl   15:02   0:00 telemetry client
root        97  0.0  0.1 127248 14540 ?        Sl   15:02   0:00 rancher-auth-service
root       913  0.1  0.0  19768  3744 ?        Ss   15:08   0:00 /bin/bash
root       929  0.0  0.0  17172  2424 ?        R+   15:09   0:00 ps auxww

Is there a way to profile what's going?

Useful	Info
Versions	Rancher `v1.2.0-rc3` Cattle: `v0.174.2` UI: `v1.2.34`
Access	`localauth` `admin`
Orchestration	`Cattle`
Route	`service.containers`

The text was updated successfully, but these errors were encountered:

cjellick · 2016-12-01T16:23:59Z

@miv Are your logs continuously spitting out Unknown exception running process [mount.create:2409170] on [2605], canceled by [State [purged] is not valid for process [volume.activate:3068009] or has the calmed down over time?

What's your deployment look like (roughly)?

How many environments
How many total hosts
How many containers

miv · 2016-12-01T16:48:48Z

@cjellick It has not calmed down :( Here's full rancher server log - https://gist.github.com/miv/959145626e3c044eafd5212dcdc16b5f

We have 1 environment
15 hosts in rancher (in different datacenters)
Less than 100 containers probably.

EDIT: updated logs

miv · 2016-12-01T17:09:16Z

@cjellick This is a m4.large for instance and db.t2.medium for rds mysql.

I have a theory that using all those prereleases took a toll on database structure. I think to recreate everything from scratch using export of all environment, and using more potent instances.

aemneina · 2016-12-03T18:35:05Z

@miv were your hosts showing as reconnecting in the UI? looks like your hosts were having trouble... Understandably you might have scratched the deployment and started over. If thats the case lets close this for now.

miv · 2016-12-03T18:45:09Z

@aemneina Well, after upgrade to full release, it still was working weird. But, after deleting hosts and adding again to rancher (and rebooting them) they seem to be working fine and cpu is normal now. Thou it's still doing something weird every 5 minutes mark - http://monosnap.com/image/6sS0T2CzE5acnCE2VEOY8Vkh6IidtJ

aemneina · 2016-12-03T21:40:30Z

it would be good to see what logs correlate with those spikes. I'd imagine its something mundane, like a periodic host polling.

miv · 2016-12-04T16:14:50Z

@aemneina There is nothing in the logs on every 5 minutes mark but there is something,

BTW Is there a way do decipher who is this host 5d7c0945-216d-40c7-5e1e-e03ce2b1655b that mysteriously reconnected?

BTW2: Can i launch server in sql-debug mode? Such that it would show sql parameters, and would be able to debug this message Cannot delete or update a parent row: a foreign key constraint fails

Logs:

2016-12-04 14:00:03,585 ERROR [:] [] [] [] [TaskScheduler-1] [i.c.p.core.cleanup.TableCleanup     ] SQL [delete from `instance` where `instance`.`id` in (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; Cannot delete or update a parent row: a foreign key constraint fails (`RancherServer`.`network_service_provider_instance_map`, CONSTRAINT `fk_network_service_provider_instance_map__instance_id` FOREIGN KEY (`instance_id`) REFERENCES `instance` (`id`) ON DELETE N)
2016-12-04 14:00:03,746 WARN  [:] [] [] [] [TaskScheduler-1] [i.c.p.core.cleanup.TableCleanup     ] [Rows Skipped] healthcheck_instance=1 volume=100 host=6 instance=28 physical_host=6 service_index=100 service=100 environment=93 storage_pool=6 agent=100 account=100
time="2016-12-04T14:04:40Z" level=info msg="Shutting down backend 5d7c0945-216d-40c7-5e1e-e03ce2b1655b. Connection closed because: websocket: close 1006 unexpected EOF."
time="2016-12-04T14:04:40Z" level=info msg="Removed backend. Key: 5d7c0945-216d-40c7-5e1e-e03ce2b1655b. Session ID 08b33b19-42bb-4a8a-a95a-57a24ea0c965 ."
time="2016-12-04T14:04:45Z" level=info msg="Handling backend connection request."
time="2016-12-04T14:04:45Z" level=info msg="Registering backend for host 5d7c0945-216d-40c7-5e1e-e03ce2b1655b with session ID 174ec52d-d3b9-488d-bfab-a81ae2774ddf."
2016-12-04 14:25:22,260 ERROR [c18726a1-3d7c-4608-9f40-c203c463e9e5:3171191] [service:38] [service.update] [] [ecutorService-4] [c.p.e.p.i.DefaultProcessInstanceImpl] Unknown exception org.jooq.exception.DataChangedException: Database record has been changed
	at org.jooq.impl.UpdatableRecordImpl.checkIfChanged(UpdatableRecordImpl.java:550) ~[jooq-3.3.0.jar:na]
	at org.jooq.impl.UpdatableRecordImpl.storeUpdate0(UpdatableRecordImpl.java:291) ~[jooq-3.3.0.jar:na]
	at org.jooq.impl.UpdatableRecordImpl.access$200(UpdatableRecordImpl.java:90) ~[jooq-3.3.0.jar:na]
	at org.jooq.impl.UpdatableRecordImpl$3.operate(UpdatableRecordImpl.java:260) ~[jooq-3.3.0.jar:na]
	at org.jooq.impl.RecordDelegate.operate(RecordDelegate.java:123) ~[jooq-3.3.0.jar:na]
	at org.jooq.impl.UpdatableRecordImpl.storeUpdate(UpdatableRecordImpl.java:255) ~[jooq-3.3.0.jar:na]
	at org.jooq.impl.UpdatableRecordImpl.update(UpdatableRecordImpl.java:149) ~[jooq-3.3.0.jar:na]
	at io.cattle.platform.object.impl.JooqObjectManager.persistRecord(JooqObjectManager.java:223) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.object.impl.JooqObjectManager.setFieldsInternal(JooqObjectManager.java:130) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.object.impl.JooqObjectManager$3.execute(JooqObjectManager.java:118) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.idempotent.Idempotent.change(Idempotent.java:88) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.object.impl.JooqObjectManager.setFields(JooqObjectManager.java:115) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.object.impl.JooqObjectManager.setFields(JooqObjectManager.java:110) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.object.impl.AbstractObjectManager.setFields(AbstractObjectManager.java:135) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.activity.impl.ActivityLogImpl.close(ActivityLogImpl.java:81) ~[cattle-activity-log-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.activity.impl.EntryImpl.close(EntryImpl.java:23) ~[cattle-activity-log-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.activity.ActivityService.run(ActivityService.java:48) ~[cattle-activity-log-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.servicediscovery.process.ServiceUpdateActivate.handle(ServiceUpdateActivate.java:70) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:448) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:399) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:393) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandlers(DefaultProcessInstanceImpl.java:393) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runLogic(DefaultProcessInstanceImpl.java:495) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:326) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:243) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:240) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:182) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:155) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:114) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:111) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:111) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.server.impl.ProcessInstanceDispatcherImpl.processExecuteWithLock(ProcessInstanceDispatcherImpl.java:98) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.server.impl.ProcessInstanceDispatcherImpl$1$1.doWithLockNoResult(ProcessInstanceDispatcherImpl.java:71) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl$4.doWithLock(AbstractLockManagerImpl.java:50) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.tryLock(AbstractLockManagerImpl.java:25) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.tryLock(AbstractLockManagerImpl.java:47) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.server.impl.ProcessInstanceDispatcherImpl$1.doRun(ProcessInstanceDispatcherImpl.java:68) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:15) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]

2016-12-04 14:25:22,290 ERROR [:] [] [] [] [ecutorService-4] [.e.s.i.ProcessInstanceDispatcherImpl] Unknown exception running process [service.update:3171191] on [38] org.jooq.exception.DataChangedException: Database record has been changed
	at org.jooq.impl.UpdatableRecordImpl.checkIfChanged(UpdatableRecordImpl.java:550) ~[jooq-3.3.0.jar:na]
	at org.jooq.impl.UpdatableRecordImpl.storeUpdate0(UpdatableRecordImpl.java:291) ~[jooq-3.3.0.jar:na]
	at org.jooq.impl.UpdatableRecordImpl.access$200(UpdatableRecordImpl.java:90) ~[jooq-3.3.0.jar:na]
	at org.jooq.impl.UpdatableRecordImpl$3.operate(UpdatableRecordImpl.java:260) ~[jooq-3.3.0.jar:na]
	at org.jooq.impl.RecordDelegate.operate(RecordDelegate.java:123) ~[jooq-3.3.0.jar:na]
	at org.jooq.impl.UpdatableRecordImpl.storeUpdate(UpdatableRecordImpl.java:255) ~[jooq-3.3.0.jar:na]
	at org.jooq.impl.UpdatableRecordImpl.update(UpdatableRecordImpl.java:149) ~[jooq-3.3.0.jar:na]
	at io.cattle.platform.object.impl.JooqObjectManager.persistRecord(JooqObjectManager.java:223) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.object.impl.JooqObjectManager.setFieldsInternal(JooqObjectManager.java:130) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.object.impl.JooqObjectManager$3.execute(JooqObjectManager.java:118) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.idempotent.Idempotent.change(Idempotent.java:88) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.object.impl.JooqObjectManager.setFields(JooqObjectManager.java:115) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.object.impl.JooqObjectManager.setFields(JooqObjectManager.java:110) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.object.impl.AbstractObjectManager.setFields(AbstractObjectManager.java:135) ~[cattle-framework-object-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.activity.impl.ActivityLogImpl.close(ActivityLogImpl.java:81) ~[cattle-activity-log-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.activity.impl.EntryImpl.close(EntryImpl.java:23) ~[cattle-activity-log-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.activity.ActivityService.run(ActivityService.java:48) ~[cattle-activity-log-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.servicediscovery.process.ServiceUpdateActivate.handle(ServiceUpdateActivate.java:70) ~[cattle-iaas-service-discovery-server-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandler(DefaultProcessInstanceImpl.java:448) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:399) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$4.execute(DefaultProcessInstanceImpl.java:393) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.idempotent.Idempotent.execute(Idempotent.java:42) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runHandlers(DefaultProcessInstanceImpl.java:393) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runLogic(DefaultProcessInstanceImpl.java:495) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runWithProcessLock(DefaultProcessInstanceImpl.java:326) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$2.doWithLockNoResult(DefaultProcessInstanceImpl.java:243) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.acquireLockAndRun(DefaultProcessInstanceImpl.java:240) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.runDelegateLoop(DefaultProcessInstanceImpl.java:182) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.executeWithProcessInstanceLock(DefaultProcessInstanceImpl.java:155) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:114) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl$1.doWithLock(DefaultProcessInstanceImpl.java:111) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl$3.doWithLock(AbstractLockManagerImpl.java:40) ~[cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:13) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.lock(AbstractLockManagerImpl.java:37) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.process.impl.DefaultProcessInstanceImpl.execute(DefaultProcessInstanceImpl.java:111) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.server.impl.ProcessInstanceDispatcherImpl.processExecuteWithLock(ProcessInstanceDispatcherImpl.java:98) ~[cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.server.impl.ProcessInstanceDispatcherImpl$1$1.doWithLockNoResult(ProcessInstanceDispatcherImpl.java:71) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:7) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.LockCallbackNoReturn.doWithLock(LockCallbackNoReturn.java:3) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl$4.doWithLock(AbstractLockManagerImpl.java:50) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.LockManagerImpl.doLock(LockManagerImpl.java:33) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.tryLock(AbstractLockManagerImpl.java:25) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.lock.impl.AbstractLockManagerImpl.tryLock(AbstractLockManagerImpl.java:47) [cattle-framework-lock-0.5.0-SNAPSHOT.jar:na]
	at io.cattle.platform.engine.server.impl.ProcessInstanceDispatcherImpl$1.doRun(ProcessInstanceDispatcherImpl.java:68) [cattle-framework-engine-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.NoExceptionRunnable.runInContext(NoExceptionRunnable.java:15) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:108) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) [cattle-framework-managed-context-0.5.0-SNAPSHOT.jar:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]

2016-12-04 15:00:03,499 ERROR [:] [] [] [] [TaskScheduler-2] [i.c.p.core.cleanup.TableCleanup     ] SQL [delete from `instance` where `instance`.`id` in (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; Cannot delete or update a parent row: a foreign key constraint fails (`RancherServer`.`network_service_provider_instance_map`, CONSTRAINT `fk_network_service_provider_instance_map__instance_id` FOREIGN KEY (`instance_id`) REFERENCES `instance` (`id`) ON DELETE N)
2016-12-04 15:00:03,619 WARN  [:] [] [] [] [TaskScheduler-2] [i.c.p.core.cleanup.TableCleanup     ] [Rows Skipped] healthcheck_instance=1 volume=100 host=6 instance=28 physical_host=6 service_index=100 service=100 environment=93 storage_pool=6 agent=100 account=100
2016-12-04 16:00:03,889 ERROR [:] [] [] [] [TaskScheduler-1] [i.c.p.core.cleanup.TableCleanup     ] SQL [delete from `instance` where `instance`.`id` in (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)]; Cannot delete or update a parent row: a foreign key constraint fails (`RancherServer`.`network_service_provider_instance_map`, CONSTRAINT `fk_network_service_provider_instance_map__instance_id` FOREIGN KEY (`instance_id`) REFERENCES `instance` (`id`) ON DELETE N)
2016-12-04 16:00:04,011 WARN  [:] [] [] [] [TaskScheduler-1] [i.c.p.core.cleanup.TableCleanup     ] [Rows Skipped] healthcheck_instance=1 volume=100 host=6 instance=28 physical_host=6 service_index=100 service=100 environment=93 storage_pool=6 agent=100 account=100
2016-12-04 16:05:20,860 ERROR [7343f03d-8adf-4cc5-8142-f494e67a44f0:3172653] [service:38] [service.update] [] [ecutorService-2] [c.p.e.p.i.DefaultProcessInstanceImpl] Exiting with code [STATE_CHANGED] : STATE_CHANGED
2016-12-04 16:05:20,995 ERROR [:] [] [] [] [ecutorService-2] [.e.s.i.ProcessInstanceDispatcherImpl] Process [service.update:3172653] on [38] failed, exit [STATE_CHANGED] : io.cattle.platform.engine.process.impl.ProcessExecutionExitException: STATE_CHANGED

aemneina · 2016-12-04T18:02:25Z

you should be able to query the db, host or agent table, there you'll find an entry matching that uuid.

aemneina · 2016-12-04T19:17:24Z

@miv how do your hosts look, in the hosts tab? Are they all active?

deniseschannon · 2016-12-15T23:28:27Z

To test, on upgraded setup, any table prefixed with network_service_* should have the contents of the table to be empty. @sangeethah

sangeethah · 2016-12-16T01:06:01Z

Tested with upgrading from rancher-server version 1.1.4 - v1.2.1-rc3

After upgrade , all the existing entries from network_service_provider_instance_map and network_service_provider get deleted.

deniseschannon added this to the Release 1.2.1 milestone Dec 7, 2016

deniseschannon assigned ibuildthecloud Dec 7, 2016

deniseschannon mentioned this issue Dec 14, 2016

Deal with data inconsistency rancher/cattle#2319

Merged

deniseschannon added the status/resolved label Dec 14, 2016

ibuildthecloud assigned sangeethah Dec 15, 2016

ibuildthecloud added the status/to-test label Dec 15, 2016

ibuildthecloud removed their assignment Dec 15, 2016

tfiduccia mentioned this issue Dec 15, 2016

Incredibly high traffic between containers #7076

Closed

sangeethah closed this as completed Dec 16, 2016

EugenMayer mentioned this issue Jul 14, 2017

High CPU Load after upgrading to 1.6.3 ( from 1.6.1) #9346

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

High CPU usage after 1.2.0-rc #6857

High CPU usage after 1.2.0-rc #6857

miv commented Dec 1, 2016

cjellick commented Dec 1, 2016

miv commented Dec 1, 2016 •

edited

Loading

miv commented Dec 1, 2016

aemneina commented Dec 3, 2016

miv commented Dec 3, 2016

aemneina commented Dec 3, 2016

miv commented Dec 4, 2016

aemneina commented Dec 4, 2016

aemneina commented Dec 4, 2016

deniseschannon commented Dec 15, 2016 •

edited

Loading

sangeethah commented Dec 16, 2016

High CPU usage after 1.2.0-rc #6857

High CPU usage after 1.2.0-rc #6857

Comments

miv commented Dec 1, 2016

cjellick commented Dec 1, 2016

miv commented Dec 1, 2016 • edited Loading

miv commented Dec 1, 2016

aemneina commented Dec 3, 2016

miv commented Dec 3, 2016

aemneina commented Dec 3, 2016

miv commented Dec 4, 2016

aemneina commented Dec 4, 2016

aemneina commented Dec 4, 2016

deniseschannon commented Dec 15, 2016 • edited Loading

sangeethah commented Dec 16, 2016

miv commented Dec 1, 2016 •

edited

Loading

deniseschannon commented Dec 15, 2016 •

edited

Loading