cpu Isolation a problem.

[brucewoo]

warden running 105 container, As follows：
root 453 1 0 Aug09 ? 00:00:00 wshd: 17vkggc0120
root 563 1 0 Aug10 ? 00:00:00 wshd: 17vkggc0163
root 1021 1 0 Aug11 ? 00:00:00 wshd: 17vkggc0173
root 1081 1 0 Aug11 ? 00:00:00 wshd: 17vkggc017u
root 1083 1 0 Aug11 ? 00:00:00 wshd: 17vkggc017d
root 1929 1 0 Aug12 ? 00:00:00 wshd: 17vkggc01bf
root 1956 1 0 Aug09 ? 00:00:00 wshd: 17vkggc0121
root 2217 1 0 Aug09 ? 00:00:00 wshd: 17vkggc011q
root 3460 1 0 09:37 ? 00:00:00 wshd: 17vkggc01c4
root 3698 1 0 Aug11 ? 00:00:00 wshd: 17vkggc01a1
root 3919 1 0 Aug11 ? 00:00:00 wshd: 17vkggc019k
root 3943 1 0 Aug11 ? 00:00:00 wshd: 17vkggc019j
root 4014 1 0 Aug07 ? 00:00:00 wshd: 17vkggc00u8
root 4570 1 0 Aug11 ? 00:00:00 wshd: 17vkggc017e
root 4881 1 0 Aug10 ? 00:00:00 wshd: 17vkggc016a
root 5810 1 0 Aug11 ? 00:00:00 wshd: 17vkggc019h
root 6780 1 0 Aug10 ? 00:00:00 wshd: 17vkggc0168
root 7098 1 0 Aug10 ? 00:00:00 wshd: 17vkggc016o
root 7103 1 0 Aug09 ? 00:00:00 wshd: 17vkggc011t
root 7121 1 0 Aug11 ? 00:00:00 wshd: 17vkggc018k
root 8178 1 0 Aug11 ? 00:00:00 wshd: 17vkggc017b
root 8218 1 0 Aug11 ? 00:00:00 wshd: 17vkggc017o
root 8277 1 0 Aug12 ? 00:00:00 wshd: 17vkggc01bq
root 8859 1 0 Aug07 ? 00:00:00 wshd: 17vkggc00ul
root 9499 1 0 Aug11 ? 00:00:00 wshd: 17vkggc0189
root 10362 1 0 Aug12 ? 00:00:00 wshd: 17vkggc01b2
root 10463 1 0 Aug11 ? 00:00:00 wshd: 17vkggc016t
root 10806 1 0 Aug12 ? 00:00:00 wshd: 17vkggc01bi
root 11195 1 0 Aug12 ? 00:00:00 wshd: 17vkggc01ar
root 11412 1 0 Aug11 ? 00:00:00 wshd: 17vkggc0172
root 11483 1 0 09:04 ? 00:00:00 wshd: 17vkggc01c2
root 12241 1 0 Aug11 ? 00:00:00 wshd: 17vkggc0174
root 12408 1 0 Aug10 ? 00:00:00 wshd: 17vkggc016b
root 12647 1 0 Aug11 ? 00:00:00 wshd: 17vkggc0175
root 12670 1 0 Aug12 ? 00:00:00 wshd: 17vkggc01bk
root 12763 1 0 Aug08 ? 00:00:00 wshd: 17vkggc011d
root 13717 1 0 Aug10 ? 00:00:00 wshd: 17vkggc015j
root 13841 1 0 Aug10 ? 00:00:00 wshd: 17vkggc014n
root 13842 1 0 Aug06 ? 00:00:00 wshd: 17vkggc00q3
root 14385 1 0 Aug11 ? 00:00:00 wshd: 17vkggc01a3
root 14484 1 0 Aug10 ? 00:00:00 wshd: 17vkggc016r
root 14486 1 0 Aug06 ? 00:00:00 wshd: 17vkggc00sk
root 14655 1 0 Aug10 ? 00:00:00 wshd: 17vkggc015k
root 14987 1 0 Aug09 ? 00:00:00 wshd: 17vkggc0124
root 15020 1 0 Aug11 ? 00:00:00 wshd: 17vkggc01a4
root 15143 1 0 Aug10 ? 00:00:00 wshd: 17vkggc016m
root 15381 1 0 09:58 ? 00:00:00 wshd: 17vkggc01c9
root 15611 1 0 Aug10 ? 00:00:00 wshd: 17vkggc016i
root 15615 1 0 Aug11 ? 00:00:00 wshd: 17vkggc0188
root 15627 1 0 Aug04 ? 00:00:00 wshd: 17vkggc0001
root 15919 1 0 Aug10 ? 00:00:00 wshd: 17vkggc0169
root 16632 1 0 Aug12 ? 00:00:00 wshd: 17vkggc01br
root 16837 1 0 Aug08 ? 00:00:00 wshd: 17vkggc00vq
root 17385 1 0 Aug09 ? 00:00:00 wshd: 17vkggc013d
root 17425 1 0 Aug11 ? 00:00:00 wshd: 17vkggc01a5
root 17428 1 0 Aug08 ? 00:00:00 wshd: 17vkggc00vt
root 17692 1 0 Aug11 ? 00:00:00 wshd: 17vkggc01ae
root 18168 1 0 Aug12 ? 00:00:00 wshd: 17vkggc01b7
root 18625 1 0 Aug10 ? 00:00:00 wshd: 17vkggc0141
root 18736 1 0 Aug11 ? 00:00:00 wshd: 17vkggc017v
root 18991 1 0 Aug09 ? 00:00:00 wshd: 17vkggc011s
root 19797 1 0 Aug12 ? 00:00:00 wshd: 17vkggc01be
root 20343 1 0 Aug06 ? 00:00:00 wshd: 17vkggc00ru
root 20502 1 0 Aug11 ? 00:00:00 wshd: 17vkggc01a6
root 20810 1 0 Aug12 ? 00:00:00 wshd: 17vkggc01b6
root 21125 1 0 Aug11 ? 00:00:00 wshd: 17vkggc01an
root 21815 1 0 Aug11 ? 00:00:00 wshd: 17vkggc0187
root 21856 1 0 Aug12 ? 00:00:00 wshd: 17vkggc01bl
root 21971 1 0 Aug12 ? 00:00:00 wshd: 17vkggc01bg
root 21982 1 0 Aug11 ? 00:00:00 wshd: 17vkggc017a
root 22298 1 0 Aug11 ? 00:00:00 wshd: 17vkggc018c
root 22476 1 0 Aug10 ? 00:00:00 wshd: 17vkggc016f
root 22584 1 0 Aug02 ? 00:00:00 wshd: 17vkggbvv4u
root 22761 1 0 Aug09 ? 00:00:00 wshd: 17vkggc011r
root 23689 1 0 Aug09 ? 00:00:00 wshd: 17vkggc0122
root 23919 1 0 Aug08 ? 00:00:00 wshd: 17vkggc010u
root 24127 1 0 Aug11 ? 00:00:00 wshd: 17vkggc01a2
root 24331 1 0 Aug11 ? 00:00:00 wshd: 17vkggc0178
root 25270 1 0 Aug09 ? 00:00:00 wshd: 17vkggc0123
root 25724 1 0 Aug12 ? 00:00:00 wshd: 17vkggc01au
root 25917 1 0 Aug10 ? 00:00:00 wshd: 17vkggc015m
root 26098 1 0 Aug11 ? 00:00:00 wshd: 17vkggc019s
root 26405 1 0 Aug11 ? 00:00:00 wshd: 17vkggc01a8
root 26544 1 0 Aug12 ? 00:00:00 wshd: 17vkggc01bb
root 26617 1 0 Aug09 ? 00:00:00 wshd: 17vkggc011u
root 27000 1 0 Aug10 ? 00:00:00 wshd: 17vkggc0166
root 27002 1 0 Aug06 ? 00:00:00 wshd: 17vkggc00qn
root 27099 1 0 Aug12 ? 00:00:00 wshd: 17vkggc01av
root 27101 1 0 Aug11 ? 00:00:00 wshd: 17vkggc0191
root 27135 1 0 Aug12 ? 00:00:00 wshd: 17vkggc01aq
root 27546 1 0 Aug11 ? 00:00:00 wshd: 17vkggc01a7
root 27661 1 0 Aug04 ? 00:00:00 wshd: 17vkggc008j
root 27970 1 0 Aug11 ? 00:00:00 wshd: 17vkggc01ao
root 28162 1 0 Aug11 ? 00:00:00 wshd: 17vkggc019o
root 28433 1 0 10:20 ? 00:00:00 wshd: 17vkggc01ca
root 28750 1 0 Aug06 ? 00:00:00 wshd: 17vkggc00se
root 29962 1 0 Aug09 ? 00:00:00 wshd: 17vkggc011v
root 30166 1 0 Aug10 ? 00:00:00 wshd: 17vkggc016g
root 30453 1 0 09:52 ? 00:00:00 wshd: 17vkggc01c7
root 31184 1 0 Aug04 ? 00:00:00 wshd: 17vkggbvvvt
root 31202 1 0 Aug11 ? 00:00:00 wshd: 17vkggc019m
root 31346 1 0 Aug10 ? 00:00:00 wshd: 17vkggc016j
root 31781 1 0 02:04 ? 00:00:00 wshd: 17vkggc01bt
root 32219 1 0 Aug12 ? 00:00:00 wshd: 17vkggc01at
root 32392 1 0 Aug12 ? 00:00:00 wshd: 17vkggc01as
root 32708 1 0 Aug11 ? 00:00:00 wshd: 17vkggc017l

but has a container‘s process， cpu As follows：
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18277 10103 20 0 159m 15m 9780 R 100 0.1 2581:49 httpd
15847 10103 20 0 159m 16m 9952 R 99 0.1 2582:15 httpd
18371 10103 20 0 161m 15m 9716 R 99 0.1 2578:00 httpd
12705 10103 20 0 160m 17m 10m R 98 0.1 2578:03 httpd

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/76845176.

[sykesm]

@brucewoo,

Can you elaborate on what you think the "isolation problem" is? It looks like you have four processes that are aggressively consuming CPU, but you don't indicate how many processors there are, what the processes inside the container are doing, or what your expected results are...

What I can say is that warden relies heavily on Linux cgroup cpu accounting. We set a CPU fair-share limit that is based on the requested memory limits and those limits only get enforced where there is competition for compute resource across containers.

If you only have four processes aggressively consuming CPU, what you're seeing is expected; if all processes were trying to consume CPU, I'd expect different stats.

Thanks.

[brucewoo]

@sykesm,
yes,your analysis is correct,our server 8 core.but one container's processes 50% cpu(4 core),this is not what we expect.because this server also has other 104 container. I don't kown if other container need to consume cpu, that container consume CPU will reduce?

[sykesm]

@brucewoo, as configured today, linux cpu accounting should prevent processes from cpu starvation. If cpu quotas were enabled, starvation could be a problem but that's not what happens today.

cpu shares really only become relevant when there is competition for the cpu resource.

[dieucao]

Closing due to inactivity.