cc @quinton-hoole

lsof of kube-proxy:
http://paste.kismatic.io/view/db8afb49#L187

P1

On Fri, Mar 13, 2015 at 2:34 PM, Brian Grant notifications@github.com
wrote:

cc @quinton-hoole https://github.com/quinton-hoole

—
Reply to this email directly or view it on GitHub
#5461 (comment)
.

@thockin Done.

@jmreicha Any chance you could try a more recent version of Kubernetes?

@bgrant0607 that is what I am aiming for at this point. My cluster is basically dead. What version should I move to?

Try Kubernetes v0.12.0

@quinton-hoole I'm going to shoot for v0.12.0, I will update once the new version is up and running.

Just an update. Site traffic died off and I stopped getting the "too many open files" errors in the kube-proxy logs and I am not experiencing the slowness issues currently.

I think I just have to figure out where to increase the limits and how to not break anything else in the process.

Yes it sounds like we're either leaking file handles through not closing connections when necessary, or you're simply seeing more concurrent connections than your file open limits allow.

Given the following, it looks like you're simply hitting a 1024 file handle limit somewhere:

$ wc -l ~/Downloads/db8afb49.txt
1032 /Users/quinton/Downloads/db8afb49.txt

I don't know that part of the code very well, but can poke around a bit.

@dchen1107 What is the intended per-process file handle limit on the node, and where do we set that (if at all - it sounds like we might be hitting the 1024 default limit)?

@quinton-hoole we don't set them at all today. Arguable we should for kube-proxy. I will re-purpose this issue for that (if you don't mind :) ) assigning myself.

@jmreicha can you provide the output of the following:

$ cat /proc/sys/fs/file-max
$ cat /proc/<pid of kube-proxy>/limits

@timothysc won't that only fix systemd systems though?

Since it does address the issue at hand, I'm fine closing :)

Yes I'm okay with closing.

I checked a few other services but not all of them are respecting the 65k limit like the kube-proxy is, will that be an issue? For example I just checked the kube-apiserver and it still has the 4k limit.

Would it be worthwhile to update the getting started guides with the correct file limits or make a note about the file limit issue somewhere?

I'm working on a PR to have the kube-proxy monitor its limit and warn if its too low. We also need to start documenting the bare requirements for each Kube component, I think there is an issue out for that.

How sure are we that we don't leak FDs? An IRC (paddyforan) user of version 0.13 reported this problem.

Background: services stopped responding, was directed by @lavalamp to look into kube-proxy. I'm running v0.13.0 on GCE, using container-vm-v20150305.

I'm currently pulling all the files in /var/log off the nodes before restarting them to (hopefully) get things back online, but there are several gigabytes of log files (/var/log/kube-proxy.log seemed to spit out a E0410 23:32:01.350853 4891 proxier.go:104] Accept failed: accept tcp [::]:60995: too many open files message every .00001 seconds...) so if there's anything else I should grab, let me know.

I did test this some time ago, manually, probably should capture it as an
e2e.

ls -l /proc/pidof kube-proxy/fd

On Fri, Apr 10, 2015 at 5:00 PM, Paddy notifications@github.com wrote:

Background: services stopped responding, was directed by @lavalamp
https://github.com/lavalamp to look into kube-proxy. I'm running
v0.13.0 on GCE, using container-vm-v20150305.

I'm currently pulling all the files in /var/log off the nodes before
restarting them to (hopefully) get things back online, but there are
several gigabytes of log files (/var/log/kube-proxy.log seemed to spit
out a E0410 23:32:01.350853 4891 proxier.go:104] Accept failed: accept
tcp [::]:60995: too many open files message every .00001 seconds...) so
if there's anything else I should grab, let me know.

—
Reply to this email directly or view it on GitHub
#5461 (comment)
.

Jeepers. I'll try to manually rig up a test and see if I can find the
problem.

On Fri, Apr 10, 2015 at 5:19 PM, Paddy notifications@github.com wrote:

$ sudo ls -l /proc/4774/fd
total 0
lrwx------ 1 root root 64 Apr 11 00:18 0 -> socket:[1258422909]
l-wx------ 1 root root 64 Apr 11 00:18 1 -> /var/log/kube-proxy.log
lrwx------ 1 root root 64 Apr 11 00:18 10 -> socket:[15248]
lrwx------ 1 root root 64 Apr 11 00:18 100 -> socket:[15113608]
lrwx------ 1 root root 64 Apr 11 00:18 1000 -> socket:[156748126]
lrwx------ 1 root root 64 Apr 11 00:18 1001 -> socket:[168048812]
lrwx------ 1 root root 64 Apr 11 00:18 1002 -> socket:[131454811]
lrwx------ 1 root root 64 Apr 11 00:18 1003 -> socket:[124433259]
lrwx------ 1 root root 64 Apr 11 00:18 1004 -> socket:[123439313]
lrwx------ 1 root root 64 Apr 11 00:18 1005 -> socket:[125555672]
lrwx------ 1 root root 64 Apr 11 00:18 1006 -> socket:[145356381]
lrwx------ 1 root root 64 Apr 11 00:18 1007 -> socket:[125870197]
lrwx------ 1 root root 64 Apr 11 00:18 1008 -> socket:[241583618]
lrwx------ 1 root root 64 Apr 11 00:18 1009 -> socket:[145356323]
lrwx------ 1 root root 64 Apr 11 00:18 101 -> socket:[12602549]
lrwx------ 1 root root 64 Apr 11 00:18 1010 -> socket:[135420908]
lrwx------ 1 root root 64 Apr 11 00:18 1011 -> socket:[370707271]
lrwx------ 1 root root 64 Apr 11 00:18 1012 -> socket:[124046892]
lrwx------ 1 root root 64 Apr 11 00:18 1013 -> socket:[1258422910]
lrwx------ 1 root root 64 Apr 11 00:18 1014 -> socket:[130580724]
lrwx------ 1 root root 64 Apr 11 00:18 1015 -> socket:[125224884]
lrwx------ 1 root root 64 Apr 11 00:18 1016 -> socket:[131454812]
lrwx------ 1 root root 64 Apr 11 00:18 1017 -> socket:[123763362]
lrwx------ 1 root root 64 Apr 11 00:18 1018 -> socket:[2742531696]
lrwx------ 1 root root 64 Apr 11 00:18 1019 -> socket:[3475055362]
lrwx------ 1 root root 64 Apr 11 00:18 102 -> socket:[36542083]
lrwx------ 1 root root 64 Apr 11 00:18 1020 -> socket:[124385914]
lrwx------ 1 root root 64 Apr 11 00:18 1021 -> socket:[1022287418]
lrwx------ 1 root root 64 Apr 11 00:18 1022 -> socket:[3474838885]
lrwx------ 1 root root 64 Apr 11 00:18 1023 -> socket:[3475177736]
lrwx------ 1 root root 64 Apr 11 00:18 103 -> socket:[5969096]
lrwx------ 1 root root 64 Apr 11 00:18 104 -> socket:[24495329]
lrwx------ 1 root root 64 Apr 11 00:18 105 -> socket:[26078589]
lrwx------ 1 root root 64 Apr 11 00:18 106 -> socket:[3751902]
lrwx------ 1 root root 64 Apr 11 00:18 107 -> socket:[66126]
lrwx------ 1 root root 64 Apr 11 00:18 108 -> socket:[19893656]
lrwx------ 1 root root 64 Apr 11 00:18 109 -> socket:[19893657]
lrwx------ 1 root root 64 Apr 11 00:18 11 -> socket:[15269]
lrwx------ 1 root root 64 Apr 11 00:18 110 -> socket:[12602550]
lrwx------ 1 root root 64 Apr 11 00:18 111 -> socket:[37426267]
lrwx------ 1 root root 64 Apr 11 00:18 112 -> socket:[37723101]
lrwx------ 1 root root 64 Apr 11 00:18 113 -> socket:[35662864]
lrwx------ 1 root root 64 Apr 11 00:18 114 -> socket:[31453470]
lrwx------ 1 root root 64 Apr 11 00:18 115 -> socket:[10433675]
lrwx------ 1 root root 64 Apr 11 00:18 116 -> socket:[26045729]
lrwx------ 1 root root 64 Apr 11 00:18 117 -> socket:[28796052]
lrwx------ 1 root root 64 Apr 11 00:18 118 -> socket:[43294473]
lrwx------ 1 root root 64 Apr 11 00:18 119 -> socket:[24495330]
lrwx------ 1 root root 64 Apr 11 00:18 12 -> socket:[15287]
lrwx------ 1 root root 64 Apr 11 00:18 120 -> socket:[9843532]
lrwx------ 1 root root 64 Apr 11 00:18 121 -> socket:[40509223]
lrwx------ 1 root root 64 Apr 11 00:18 122 -> socket:[10714372]
lrwx------ 1 root root 64 Apr 11 00:18 123 -> socket:[30770420]
lrwx------ 1 root root 64 Apr 11 00:18 124 -> socket:[10714373]
lrwx------ 1 root root 64 Apr 11 00:18 125 -> socket:[35662865]
lrwx------ 1 root root 64 Apr 11 00:18 126 -> socket:[28338363]
lrwx------ 1 root root 64 Apr 11 00:18 127 -> socket:[28338364]
lrwx------ 1 root root 64 Apr 11 00:18 128 -> socket:[34093569]
lrwx------ 1 root root 64 Apr 11 00:18 129 -> socket:[19329338]
lrwx------ 1 root root 64 Apr 11 00:18 13 -> socket:[36545901]
lrwx------ 1 root root 64 Apr 11 00:18 130 -> socket:[38686331]
lrwx------ 1 root root 64 Apr 11 00:18 131 -> socket:[28796053]
lrwx------ 1 root root 64 Apr 11 00:18 132 -> socket:[15612375]
lrwx------ 1 root root 64 Apr 11 00:18 133 -> socket:[14746134]
lrwx------ 1 root root 64 Apr 11 00:18 134 -> socket:[14746156]
lrwx------ 1 root root 64 Apr 11 00:18 135 -> socket:[7821989]
lrwx------ 1 root root 64 Apr 11 00:18 136 -> socket:[7821990]
lrwx------ 1 root root 64 Apr 11 00:18 137 -> socket:[8036635]
lrwx------ 1 root root 64 Apr 11 00:18 138 -> socket:[35285698]
lrwx------ 1 root root 64 Apr 11 00:18 139 -> socket:[30770421]
lrwx------ 1 root root 64 Apr 11 00:18 14 -> socket:[169685]
lrwx------ 1 root root 64 Apr 11 00:18 140 -> socket:[24750052]
lrwx------ 1 root root 64 Apr 11 00:18 141 -> socket:[8036636]
lrwx------ 1 root root 64 Apr 11 00:18 142 -> socket:[15612758]
lrwx------ 1 root root 64 Apr 11 00:18 143 -> socket:[18655757]
lrwx------ 1 root root 64 Apr 11 00:18 144 -> socket:[29884829]
lrwx------ 1 root root 64 Apr 11 00:18 145 -> socket:[40215303]
lrwx------ 1 root root 64 Apr 11 00:18 146 -> socket:[24792810]
lrwx------ 1 root root 64 Apr 11 00:18 147 -> socket:[24750184]
lrwx------ 1 root root 64 Apr 11 00:18 148 -> socket:[34093570]
lrwx------ 1 root root 64 Apr 11 00:18 149 -> socket:[40215546]
lrwx------ 1 root root 64 Apr 11 00:18 15 -> socket:[9332230]
lrwx------ 1 root root 64 Apr 11 00:18 150 -> socket:[52552007]
lrwx------ 1 root root 64 Apr 11 00:18 151 -> socket:[35323433]
lrwx------ 1 root root 64 Apr 11 00:18 152 -> socket:[18655758]
lrwx------ 1 root root 64 Apr 11 00:18 153 -> socket:[48081781]
lrwx------ 1 root root 64 Apr 11 00:18 154 -> socket:[19589353]
lrwx------ 1 root root 64 Apr 11 00:18 155 -> socket:[18362931]
lrwx------ 1 root root 64 Apr 11 00:18 156 -> socket:[59852398]
lrwx------ 1 root root 64 Apr 11 00:18 157 -> socket:[18693906]
lrwx------ 1 root root 64 Apr 11 00:18 158 -> socket:[18693907]
lrwx------ 1 root root 64 Apr 11 00:18 159 -> socket:[42596586]
lrwx------ 1 root root 64 Apr 11 00:18 16 -> socket:[36815385]
lrwx------ 1 root root 64 Apr 11 00:18 160 -> socket:[18690410]
lrwx------ 1 root root 64 Apr 11 00:18 161 -> socket:[44679333]
lrwx------ 1 root root 64 Apr 11 00:18 162 -> socket:[33844690]
lrwx------ 1 root root 64 Apr 11 00:18 163 -> socket:[39600045]
lrwx------ 1 root root 64 Apr 11 00:18 164 -> socket:[42597078]
lrwx------ 1 root root 64 Apr 11 00:18 165 -> socket:[18362951]
lrwx------ 1 root root 64 Apr 11 00:18 166 -> socket:[19329339]
lrwx------ 1 root root 64 Apr 11 00:18 167 -> socket:[28985321]
lrwx------ 1 root root 64 Apr 11 00:18 168 -> socket:[50543440]
lrwx------ 1 root root 64 Apr 11 00:18 169 -> socket:[24792811]
lrwx------ 1 root root 64 Apr 11 00:18 17 -> socket:[3751901]
lrwx------ 1 root root 64 Apr 11 00:18 170 -> socket:[31162870]
lrwx------ 1 root root 64 Apr 11 00:18 171 -> socket:[38383062]
lrwx------ 1 root root 64 Apr 11 00:18 172 -> socket:[28985322]
lrwx------ 1 root root 64 Apr 11 00:18 173 -> socket:[37118128]
lrwx------ 1 root root 64 Apr 11 00:18 174 -> socket:[30537928]
lrwx------ 1 root root 64 Apr 11 00:18 175 -> socket:[35627976]
lrwx------ 1 root root 64 Apr 11 00:18 176 -> socket:[33844691]
lrwx------ 1 root root 64 Apr 11 00:18 177 -> socket:[29847566]
lrwx------ 1 root root 64 Apr 11 00:18 178 -> socket:[42551944]
lrwx------ 1 root root 64 Apr 11 00:18 179 -> socket:[27561856]
lrwx------ 1 root root 64 Apr 11 00:18 18 -> socket:[9843550]
lrwx------ 1 root root 64 Apr 11 00:18 180 -> socket:[27561857]
lrwx------ 1 root root 64 Apr 11 00:18 181 -> socket:[19589354]
lrwx------ 1 root root 64 Apr 11 00:18 182 -> socket:[31496242]
lrwx------ 1 root root 64 Apr 11 00:18 183 -> socket:[24449382]
lrwx------ 1 root root 64 Apr 11 00:18 184 -> socket:[29306676]
lrwx------ 1 root root 64 Apr 11 00:18 185 -> socket:[24449383]
lrwx------ 1 root root 64 Apr 11 00:18 186 -> socket:[29306677]
lrwx------ 1 root root 64 Apr 11 00:18 187 -> socket:[36497900]
lrwx------ 1 root root 64 Apr 11 00:18 188 -> socket:[31496243]
lrwx------ 1 root root 64 Apr 11 00:18 189 -> socket:[37426268]
lrwx------ 1 root root 64 Apr 11 00:18 19 -> socket:[17753447]
lrwx------ 1 root root 64 Apr 11 00:18 190 -> socket:[35285699]
lrwx------ 1 root root 64 Apr 11 00:18 191 -> socket:[19593625]
lrwx------ 1 root root 64 Apr 11 00:18 192 -> socket:[43766694]
lrwx------ 1 root root 64 Apr 11 00:18 193 -> socket:[51682492]
lrwx------ 1 root root 64 Apr 11 00:18 194 -> socket:[27205804]
lrwx------ 1 root root 64 Apr 11 00:18 195 -> socket:[59852399]
lrwx------ 1 root root 64 Apr 11 00:18 196 -> socket:[27205805]
lrwx------ 1 root root 64 Apr 11 00:18 197 -> socket:[36542036]
lrwx------ 1 root root 64 Apr 11 00:18 198 -> socket:[35666447]
lrwx------ 1 root root 64 Apr 11 00:18 199 -> socket:[30767812]
l-wx------ 1 root root 64 Apr 11 00:18 2 -> /var/log/kube-proxy.log
lrwx------ 1 root root 64 Apr 11 00:18 20 -> socket:[7796877]
lrwx------ 1 root root 64 Apr 11 00:18 200 -> socket:[65086061]
lrwx------ 1 root root 64 Apr 11 00:18 201 -> socket:[57165342]
lrwx------ 1 root root 64 Apr 11 00:18 202 -> socket:[35627977]
lrwx------ 1 root root 64 Apr 11 00:18 203 -> socket:[54070456]
lrwx------ 1 root root 64 Apr 11 00:18 204 -> socket:[77622803]
lrwx------ 1 root root 64 Apr 11 00:18 205 -> socket:[34447106]
lrwx------ 1 root root 64 Apr 11 00:18 206 -> socket:[34022237]
lrwx------ 1 root root 64 Apr 11 00:18 207 -> socket:[34022238]
lrwx------ 1 root root 64 Apr 11 00:18 208 -> socket:[34447107]
lrwx------ 1 root root 64 Apr 11 00:18 209 -> socket:[30279833]
lrwx------ 1 root root 64 Apr 11 00:18 21 -> socket:[4691175]
lrwx------ 1 root root 64 Apr 11 00:18 210 -> socket:[30279874]
lrwx------ 1 root root 64 Apr 11 00:18 211 -> socket:[70731580]
lrwx------ 1 root root 64 Apr 11 00:18 212 -> socket:[38383063]
lrwx------ 1 root root 64 Apr 11 00:18 213 -> socket:[29550032]
lrwx------ 1 root root 64 Apr 11 00:18 214 -> socket:[29550033]
lrwx------ 1 root root 64 Apr 11 00:18 215 -> socket:[40805892]
lrwx------ 1 root root 64 Apr 11 00:18 216 -> socket:[31162871]
lrwx------ 1 root root 64 Apr 11 00:18 217 -> socket:[38686332]
lrwx------ 1 root root 64 Apr 11 00:18 218 -> socket:[36271326]
lrwx------ 1 root root 64 Apr 11 00:18 219 -> socket:[39599492]
lrwx------ 1 root root 64 Apr 11 00:18 22 -> socket:[30533524]
lrwx------ 1 root root 64 Apr 11 00:18 220 -> socket:[29023041]
lrwx------ 1 root root 64 Apr 11 00:18 221 -> socket:[51369816]
lrwx------ 1 root root 64 Apr 11 00:18 222 -> socket:[41108246]
lrwx------ 1 root root 64 Apr 11 00:18 223 -> socket:[29019560]
lrwx------ 1 root root 64 Apr 11 00:18 224 -> socket:[29019561]
lrwx------ 1 root root 64 Apr 11 00:18 225 -> socket:[36271708]
lrwx------ 1 root root 64 Apr 11 00:18 226 -> socket:[42552048]
lrwx------ 1 root root 64 Apr 11 00:18 227 -> socket:[37723102]
lrwx------ 1 root root 64 Apr 11 00:18 228 -> socket:[30767782]
lrwx------ 1 root root 64 Apr 11 00:18 229 -> socket:[32274221]
lrwx------ 1 root root 64 Apr 11 00:18 23 -> socket:[4180807]
lrwx------ 1 root root 64 Apr 11 00:18 230 -> socket:[32274222]
lrwx------ 1 root root 64 Apr 11 00:18 231 -> socket:[54715864]
lrwx------ 1 root root 64 Apr 11 00:18 232 -> socket:[30237335]
lrwx------ 1 root root 64 Apr 11 00:18 233 -> socket:[35033014]
lrwx------ 1 root root 64 Apr 11 00:18 234 -> socket:[41108247]
lrwx------ 1 root root 64 Apr 11 00:18 235 -> socket:[37118209]
lrwx------ 1 root root 64 Apr 11 00:18 236 -> socket:[31205290]
lrwx------ 1 root root 64 Apr 11 00:18 237 -> socket:[45285856]
lrwx------ 1 root root 64 Apr 11 00:18 238 -> socket:[31205310]
lrwx------ 1 root root 64 Apr 11 00:18 239 -> socket:[40806113]
lrwx------ 1 root root 64 Apr 11 00:18 24 -> socket:[40513927]
lrwx------ 1 root root 64 Apr 11 00:18 240 -> socket:[31453471]
lrwx------ 1 root root 64 Apr 11 00:18 241 -> socket:[36815386]
lrwx------ 1 root root 64 Apr 11 00:18 242 -> socket:[45285857]
lrwx------ 1 root root 64 Apr 11 00:18 243 -> socket:[36497901]
lrwx------ 1 root root 64 Apr 11 00:18 244 -> socket:[52583035]
lrwx------ 1 root root 64 Apr 11 00:18 245 -> socket:[89323699]
lrwx------ 1 root root 64 Apr 11 00:18 246 -> socket:[51980042]
lrwx------ 1 root root 64 Apr 11 00:18 247 -> socket:[44417398]
lrwx------ 1 root root 64 Apr 11 00:18 248 -> socket:[54491739]
lrwx------ 1 root root 64 Apr 11 00:18 249 -> socket:[58606662]
lrwx------ 1 root root 64 Apr 11 00:18 25 -> socket:[19593688]
lrwx------ 1 root root 64 Apr 11 00:18 250 -> socket:[33504546]
lrwx------ 1 root root 64 Apr 11 00:18 251 -> socket:[33504547]
lrwx------ 1 root root 64 Apr 11 00:18 252 -> socket:[39004529]
lrwx------ 1 root root 64 Apr 11 00:18 253 -> socket:[35072339]
lrwx------ 1 root root 64 Apr 11 00:18 254 -> socket:[35072643]
lrwx------ 1 root root 64 Apr 11 00:18 255 -> socket:[47448152]
lrwx------ 1 root root 64 Apr 11 00:18 256 -> socket:[39004530]
lrwx------ 1 root root 64 Apr 11 00:18 257 -> socket:[41055319]
lrwx------ 1 root root 64 Apr 11 00:18 258 -> socket:[50815668]
lrwx------ 1 root root 64 Apr 11 00:18 259 -> socket:[51980344]
lrwx------ 1 root root 64 Apr 11 00:18 26 -> socket:[2259576]
lrwx------ 1 root root 64 Apr 11 00:18 260 -> socket:[41359779]
lrwx------ 1 root root 64 Apr 11 00:18 261 -> socket:[35033068]
lrwx------ 1 root root 64 Apr 11 00:18 262 -> socket:[36545902]
lrwx------ 1 root root 64 Apr 11 00:18 263 -> socket:[41055444]
lrwx------ 1 root root 64 Apr 11 00:18 264 -> socket:[64130024]
lrwx------ 1 root root 64 Apr 11 00:18 265 -> socket:[39864883]
lrwx------ 1 root root 64 Apr 11 00:18 266 -> socket:[41701244]
lrwx------ 1 root root 64 Apr 11 00:18 267 -> socket:[50543472]
lrwx------ 1 root root 64 Apr 11 00:18 268 -> socket:[36501580]
lrwx------ 1 root root 64 Apr 11 00:18 269 -> socket:[36501581]
lrwx------ 1 root root 64 Apr 11 00:18 27 -> socket:[44105960]
lrwx------ 1 root root 64 Apr 11 00:18 270 -> socket:[39276555]
lrwx------ 1 root root 64 Apr 11 00:18 271 -> socket:[42592771]
lrwx------ 1 root root 64 Apr 11 00:18 272 -> socket:[48701038]
lrwx------ 1 root root 64 Apr 11 00:18 273 -> socket:[39276556]
lrwx------ 1 root root 64 Apr 11 00:18 274 -> socket:[41701245]
lrwx------ 1 root root 64 Apr 11 00:18 275 -> socket:[57162108]
lrwx------ 1 root root 64 Apr 11 00:18 276 -> socket:[57162182]
lrwx------ 1 root root 64 Apr 11 00:18 277 -> socket:[38728192]
lrwx------ 1 root root 64 Apr 11 00:18 278 -> socket:[39324300]
lrwx------ 1 root root 64 Apr 11 00:18 279 -> socket:[39324301]
lrwx------ 1 root root 64 Apr 11 00:18 28 -> socket:[29847565]
lrwx------ 1 root root 64 Apr 11 00:18 280 -> socket:[47448520]
lrwx------ 1 root root 64 Apr 11 00:18 281 -> socket:[50765759]
lrwx------ 1 root root 64 Apr 11 00:18 282 -> socket:[52583130]
lrwx------ 1 root root 64 Apr 11 00:18 283 -> socket:[37473545]
lrwx------ 1 root root 64 Apr 11 00:18 284 -> socket:[37473546]
lrwx------ 1 root root 64 Apr 11 00:18 285 -> socket:[63776154]
lrwx------ 1 root root 64 Apr 11 00:18 286 -> socket:[46276979]
lrwx------ 1 root root 64 Apr 11 00:18 287 -> socket:[104506014]
lrwx------ 1 root root 64 Apr 11 00:18 288 -> socket:[46277011]
lrwx------ 1 root root 64 Apr 11 00:18 289 -> socket:[36855539]
lrwx------ 1 root root 64 Apr 11 00:18 29 -> socket:[7131239]
lrwx------ 1 root root 64 Apr 11 00:18 290 -> socket:[36855589]
lrwx------ 1 root root 64 Apr 11 00:18 291 -> socket:[38728172]
lrwx------ 1 root root 64 Apr 11 00:18 292 -> socket:[47193131]
lrwx------ 1 root root 64 Apr 11 00:18 293 -> socket:[43532788]
lrwx------ 1 root root 64 Apr 11 00:18 294 -> socket:[43532789]
lrwx------ 1 root root 64 Apr 11 00:18 295 -> socket:[60462103]
lrwx------ 1 root root 64 Apr 11 00:18 296 -> socket:[38027291]
lrwx------ 1 root root 64 Apr 11 00:18 297 -> socket:[38027292]
lrwx------ 1 root root 64 Apr 11 00:18 298 -> socket:[39603372]
lrwx------ 1 root root 64 Apr 11 00:18 299 -> socket:[39603398]
lrwx------ 1 root root 64 Apr 11 00:18 3 -> socket:[15161]
lrwx------ 1 root root 64 Apr 11 00:18 30 -> socket:[6190373]
lrwx------ 1 root root 64 Apr 11 00:18 300 -> socket:[40509224]
lrwx------ 1 root root 64 Apr 11 00:18 301 -> socket:[41972046]
lrwx------ 1 root root 64 Apr 11 00:18 302 -> socket:[41972101]
lrwx------ 1 root root 64 Apr 11 00:18 303 -> socket:[40513953]
lrwx------ 1 root root 64 Apr 11 00:18 304 -> socket:[50769290]
lrwx------ 1 root root 64 Apr 11 00:18 305 -> socket:[57165364]
lrwx------ 1 root root 64 Apr 11 00:18 306 -> socket:[54110450]
lrwx------ 1 root root 64 Apr 11 00:18 307 -> socket:[44728313]
lrwx------ 1 root root 64 Apr 11 00:18 308 -> socket:[44728405]
lrwx------ 1 root root 64 Apr 11 00:18 309 -> socket:[47155652]
lrwx------ 1 root root 64 Apr 11 00:18 31 -> socket:[9332532]
lrwx------ 1 root root 64 Apr 11 00:18 310 -> socket:[47155681]
lrwx------ 1 root root 64 Apr 11 00:18 311 -> socket:[54411422]
lrwx------ 1 root root 64 Apr 11 00:18 312 -> socket:[54715865]
lrwx------ 1 root root 64 Apr 11 00:18 313 -> socket:[49839044]
lrwx------ 1 root root 64 Apr 11 00:18 314 -> socket:[41697649]
lrwx------ 1 root root 64 Apr 11 00:18 315 -> socket:[41059289]
lrwx------ 1 root root 64 Apr 11 00:18 316 -> socket:[48696311]
lrwx------ 1 root root 64 Apr 11 00:18 317 -> socket:[47488454]
lrwx------ 1 root root 64 Apr 11 00:18 318 -> socket:[51365271]
lrwx------ 1 root root 64 Apr 11 00:18 319 -> socket:[51365296]
lrwx------ 1 root root 64 Apr 11 00:18 32 -> socket:[2259577]
lrwx------ 1 root root 64 Apr 11 00:18 320 -> socket:[41059338]
lrwx------ 1 root root 64 Apr 11 00:18 321 -> socket:[42592443]
lrwx------ 1 root root 64 Apr 11 00:18 322 -> socket:[68999318]
lrwx------ 1 root root 64 Apr 11 00:18 323 -> socket:[115591463]
lrwx------ 1 root root 64 Apr 11 00:18 324 -> socket:[41359721]
lrwx------ 1 root root 64 Apr 11 00:18 325 -> socket:[54070457]
lrwx------ 1 root root 64 Apr 11 00:18 326 -> socket:[47488455]
lrwx------ 1 root root 64 Apr 11 00:18 327 -> socket:[39868442]
lrwx------ 1 root root 64 Apr 11 00:18 328 -> socket:[39868443]
lrwx------ 1 root root 64 Apr 11 00:18 329 -> socket:[57441243]
lrwx------ 1 root root 64 Apr 11 00:18 33 -> socket:[34815238]
lrwx------ 1 root root 64 Apr 11 00:18 330 -> socket:[45025103]
lrwx------ 1 root root 64 Apr 11 00:18 331 -> socket:[45025224]
lrwx------ 1 root root 64 Apr 11 00:18 332 -> socket:[52285329]
lrwx------ 1 root root 64 Apr 11 00:18 333 -> socket:[74320119]
lrwx------ 1 root root 64 Apr 11 00:18 334 -> socket:[71957181]
lrwx------ 1 root root 64 Apr 11 00:18 335 -> socket:[52245874]
lrwx------ 1 root root 64 Apr 11 00:18 336 -> socket:[53844964]
lrwx------ 1 root root 64 Apr 11 00:18 337 -> socket:[44682515]
lrwx------ 1 root root 64 Apr 11 00:18 338 -> socket:[44682516]
lrwx------ 1 root root 64 Apr 11 00:18 339 -> socket:[56919022]
lrwx------ 1 root root 64 Apr 11 00:18 34 -> socket:[17753592]
lrwx------ 1 root root 64 Apr 11 00:18 340 -> socket:[53841219]
lrwx------ 1 root root 64 Apr 11 00:18 341 -> socket:[51644190]
lrwx------ 1 root root 64 Apr 11 00:18 342 -> socket:[46595699]
lrwx------ 1 root root 64 Apr 11 00:18 343 -> socket:[46595947]
lrwx------ 1 root root 64 Apr 11 00:18 344 -> socket:[59203207]
lrwx------ 1 root root 64 Apr 11 00:18 345 -> socket:[48696312]
lrwx------ 1 root root 64 Apr 11 00:18 346 -> socket:[50233317]
lrwx------ 1 root root 64 Apr 11 00:18 347 -> socket:[55019117]
lrwx------ 1 root root 64 Apr 11 00:18 348 -> socket:[50233318]
lrwx------ 1 root root 64 Apr 11 00:18 349 -> socket:[53844988]
lrwx------ 1 root root 64 Apr 11 00:18 35 -> socket:[17432112]
lrwx------ 1 root root 64 Apr 11 00:18 350 -> socket:[55019527]
lrwx------ 1 root root 64 Apr 11 00:18 351 -> socket:[41655177]
lrwx------ 1 root root 64 Apr 11 00:18 352 -> socket:[41655178]
lrwx------ 1 root root 64 Apr 11 00:18 353 -> socket:[65086739]
lrwx------ 1 root root 64 Apr 11 00:18 354 -> socket:[48652482]
lrwx------ 1 root root 64 Apr 11 00:18 355 -> socket:[52245289]
lrwx------ 1 root root 64 Apr 11 00:18 356 -> socket:[44417399]
lrwx------ 1 root root 64 Apr 11 00:18 357 -> socket:[48700753]
lrwx------ 1 root root 64 Apr 11 00:18 358 -> socket:[48652483]
lrwx------ 1 root root 64 Apr 11 00:18 359 -> socket:[45987504]
lrwx------ 1 root root 64 Apr 11 00:18 36 -> socket:[2597141]
lrwx------ 1 root root 64 Apr 11 00:18 360 -> socket:[45987505]
lrwx------ 1 root root 64 Apr 11 00:18 361 -> socket:[51644191]
lrwx------ 1 root root 64 Apr 11 00:18 362 -> socket:[59507937]
lrwx------ 1 root root 64 Apr 11 00:18 363 -> socket:[85955137]
lrwx------ 1 root root 64 Apr 11 00:18 364 -> socket:[50587309]
lrwx------ 1 root root 64 Apr 11 00:18 365 -> socket:[44105698]
lrwx------ 1 root root 64 Apr 11 00:18 366 -> socket:[50587310]
lrwx------ 1 root root 64 Apr 11 00:18 367 -> socket:[50812178]
lrwx------ 1 root root 64 Apr 11 00:18 368 -> socket:[54373953]
lrwx------ 1 root root 64 Apr 11 00:18 369 -> socket:[46894589]
lrwx------ 1 root root 64 Apr 11 00:18 37 -> socket:[34815339]
lrwx------ 1 root root 64 Apr 11 00:18 370 -> socket:[48086559]
lrwx------ 1 root root 64 Apr 11 00:18 371 -> socket:[46894590]
lrwx------ 1 root root 64 Apr 11 00:18 372 -> socket:[48086560]
lrwx------ 1 root root 64 Apr 11 00:18 373 -> socket:[50765631]
lrwx------ 1 root root 64 Apr 11 00:18 374 -> socket:[50812200]
lrwx------ 1 root root 64 Apr 11 00:18 375 -> socket:[45945232]
lrwx------ 1 root root 64 Apr 11 00:18 376 -> socket:[45945233]
lrwx------ 1 root root 64 Apr 11 00:18 377 -> socket:[47193132]
lrwx------ 1 root root 64 Apr 11 00:18 378 -> socket:[77899495]
lrwx------ 1 root root 64 Apr 11 00:18 379 -> socket:[46847158]
lrwx------ 1 root root 64 Apr 11 00:18 38 -> socket:[4985993]
lrwx------ 1 root root 64 Apr 11 00:18 380 -> socket:[46847159]
lrwx------ 1 root root 64 Apr 11 00:18 381 -> socket:[49640266]
lrwx------ 1 root root 64 Apr 11 00:18 382 -> socket:[58610372]
lrwx------ 1 root root 64 Apr 11 00:18 383 -> socket:[51412772]
lrwx------ 1 root root 64 Apr 11 00:18 384 -> socket:[49640267]
lrwx------ 1 root root 64 Apr 11 00:18 385 -> socket:[51413057]
lrwx------ 1 root root 64 Apr 11 00:18 386 -> socket:[57437908]
lrwx------ 1 root root 64 Apr 11 00:18 387 -> socket:[60195180]
lrwx------ 1 root root 64 Apr 11 00:18 388 -> socket:[54411423]
lrwx------ 1 root root 64 Apr 11 00:18 389 -> socket:[99707015]
lrwx------ 1 root root 64 Apr 11 00:18 39 -> socket:[30237561]
lrwx------ 1 root root 64 Apr 11 00:18 390 -> socket:[54374214]
lrwx------ 1 root root 64 Apr 11 00:18 391 -> socket:[50193637]
lrwx------ 1 root root 64 Apr 11 00:18 392 -> socket:[50193657]
lrwx------ 1 root root 64 Apr 11 00:18 393 -> socket:[63779115]
lrwx------ 1 root root 64 Apr 11 00:18 394 -> socket:[53841220]
lrwx------ 1 root root 64 Apr 11 00:18 395 -> socket:[53223575]
lrwx------ 1 root root 64 Apr 11 00:18 396 -> socket:[50547364]
lrwx------ 1 root root 64 Apr 11 00:18 397 -> socket:[53223576]
lrwx------ 1 root root 64 Apr 11 00:18 398 -> socket:[49838563]
lrwx------ 1 root root 64 Apr 11 00:18 399 -> socket:[59203208]
lrwx------ 1 root root 64 Apr 11 00:18 4 -> anon_inode:[eventpoll]
lrwx------ 1 root root 64 Apr 11 00:18 40 -> socket:[30495011]
lrwx------ 1 root root 64 Apr 11 00:18 400 -> socket:[47771004]
lrwx------ 1 root root 64 Apr 11 00:18 401 -> socket:[47771005]
lrwx------ 1 root root 64 Apr 11 00:18 402 -> socket:[50769200]
lrwx------ 1 root root 64 Apr 11 00:18 403 -> socket:[58068692]
lrwx------ 1 root root 64 Apr 11 00:18 404 -> socket:[50237840]
lrwx------ 1 root root 64 Apr 11 00:18 405 -> socket:[50237841]
lrwx------ 1 root root 64 Apr 11 00:18 406 -> socket:[51940221]
lrwx------ 1 root root 64 Apr 11 00:18 407 -> socket:[50547365]
lrwx------ 1 root root 64 Apr 11 00:18 408 -> socket:[56915386]
lrwx------ 1 root root 64 Apr 11 00:18 409 -> socket:[52551323]
lrwx------ 1 root root 64 Apr 11 00:18 41 -> socket:[1285022]
lrwx------ 1 root root 64 Apr 11 00:18 410 -> socket:[68999370]
lrwx------ 1 root root 64 Apr 11 00:18 411 -> socket:[53219706]
lrwx------ 1 root root 64 Apr 11 00:18 412 -> socket:[50815669]
lrwx------ 1 root root 64 Apr 11 00:18 413 -> socket:[56915387]
lrwx------ 1 root root 64 Apr 11 00:18 414 -> socket:[79182032]
lrwx------ 1 root root 64 Apr 11 00:18 415 -> socket:[53219707]
lrwx------ 1 root root 64 Apr 11 00:18 416 -> socket:[88671226]
lrwx------ 1 root root 64 Apr 11 00:18 417 -> socket:[92644516]
lrwx------ 1 root root 64 Apr 11 00:18 418 -> socket:[49878876]
lrwx------ 1 root root 64 Apr 11 00:18 419 -> socket:[49878877]
lrwx------ 1 root root 64 Apr 11 00:18 42 -> socket:[18933195]
lrwx------ 1 root root 64 Apr 11 00:18 420 -> socket:[80898705]
lrwx------ 1 root root 64 Apr 11 00:18 421 -> socket:[67995047]
lrwx------ 1 root root 64 Apr 11 00:18 422 -> socket:[60195181]
lrwx------ 1 root root 64 Apr 11 00:18 423 -> socket:[58026149]
lrwx------ 1 root root 64 Apr 11 00:18 424 -> socket:[80034816]
lrwx------ 1 root root 64 Apr 11 00:18 425 -> socket:[72689614]
lrwx------ 1 root root 64 Apr 11 00:18 426 -> socket:[57437909]
lrwx------ 1 root root 64 Apr 11 00:18 427 -> socket:[70731889]
lrwx------ 1 root root 64 Apr 11 00:18 428 -> socket:[58315427]
lrwx------ 1 root root 64 Apr 11 00:18 429 -> socket:[50591408]
lrwx------ 1 root root 64 Apr 11 00:18 43 -> socket:[6868645]
lrwx------ 1 root root 64 Apr 11 00:18 430 -> socket:[50591409]
lrwx------ 1 root root 64 Apr 11 00:18 431 -> socket:[57481039]
lrwx------ 1 root root 64 Apr 11 00:18 432 -> socket:[51682493]
lrwx------ 1 root root 64 Apr 11 00:18 433 -> socket:[57481276]
lrwx------ 1 root root 64 Apr 11 00:18 434 -> socket:[68700871]
lrwx------ 1 root root 64 Apr 11 00:18 435 -> socket:[68700872]
lrwx------ 1 root root 64 Apr 11 00:18 436 -> socket:[77002937]
lrwx------ 1 root root 64 Apr 11 00:18 437 -> socket:[51940222]
lrwx------ 1 root root 64 Apr 11 00:18 438 -> socket:[71333271]
lrwx------ 1 root root 64 Apr 11 00:18 439 -> socket:[65584009]
lrwx------ 1 root root 64 Apr 11 00:18 44 -> socket:[2288851]
lrwx------ 1 root root 64 Apr 11 00:18 440 -> socket:[53557006]
lrwx------ 1 root root 64 Apr 11 00:18 441 -> socket:[53557007]
lrwx------ 1 root root 64 Apr 11 00:18 442 -> socket:[54415116]
lrwx------ 1 root root 64 Apr 11 00:18 443 -> socket:[52285330]
lrwx------ 1 root root 64 Apr 11 00:18 444 -> socket:[54415117]
lrwx------ 1 root root 64 Apr 11 00:18 445 -> socket:[71336399]
lrwx------ 1 root root 64 Apr 11 00:18 446 -> socket:[77622804]
lrwx------ 1 root root 64 Apr 11 00:18 447 -> socket:[72689615]
lrwx------ 1 root root 64 Apr 11 00:18 448 -> socket:[71333272]
lrwx------ 1 root root 64 Apr 11 00:18 449 -> socket:[67764846]
lrwx------ 1 root root 64 Apr 11 00:18 45 -> socket:[4986288]
lrwx------ 1 root root 64 Apr 11 00:18 450 -> socket:[88638851]
lrwx------ 1 root root 64 Apr 11 00:18 451 -> socket:[54110320]
lrwx------ 1 root root 64 Apr 11 00:18 452 -> socket:[58931082]
lrwx------ 1 root root 64 Apr 11 00:18 453 -> socket:[67730596]
lrwx------ 1 root root 64 Apr 11 00:18 454 -> socket:[54377391]
lrwx------ 1 root root 64 Apr 11 00:18 455 -> socket:[54377593]
lrwx------ 1 root root 64 Apr 11 00:18 456 -> socket:[57441319]
lrwx------ 1 root root 64 Apr 11 00:18 457 -> socket:[55287618]
lrwx------ 1 root root 64 Apr 11 00:18 458 -> socket:[55287652]
lrwx------ 1 root root 64 Apr 11 00:18 459 -> socket:[60738679]
lrwx------ 1 root root 64 Apr 11 00:18 46 -> socket:[43294472]
lrwx------ 1 root root 64 Apr 11 00:18 460 -> socket:[77002938]
lrwx------ 1 root root 64 Apr 11 00:18 461 -> socket:[60738680]
lrwx------ 1 root root 64 Apr 11 00:18 462 -> socket:[81857593]
lrwx------ 1 root root 64 Apr 11 00:18 463 -> socket:[72357946]
lrwx------ 1 root root 64 Apr 11 00:18 464 -> socket:[54491740]
lrwx------ 1 root root 64 Apr 11 00:18 465 -> socket:[70505742]
lrwx------ 1 root root 64 Apr 11 00:18 466 -> socket:[77940638]
lrwx------ 1 root root 64 Apr 11 00:18 467 -> socket:[56919063]
lrwx------ 1 root root 64 Apr 11 00:18 468 -> socket:[57783220]
lrwx------ 1 root root 64 Apr 11 00:18 469 -> socket:[63779205]
lrwx------ 1 root root 64 Apr 11 00:18 47 -> socket:[7128361]
lrwx------ 1 root root 64 Apr 11 00:18 470 -> socket:[67995048]
lrwx------ 1 root root 64 Apr 11 00:18 471 -> socket:[63776155]
lrwx------ 1 root root 64 Apr 11 00:18 472 -> socket:[60465598]
lrwx------ 1 root root 64 Apr 11 00:18 473 -> socket:[58068665]
lrwx------ 1 root root 64 Apr 11 00:18 474 -> socket:[60465599]
lrwx------ 1 root root 64 Apr 11 00:18 475 -> socket:[58606663]
lrwx------ 1 root root 64 Apr 11 00:18 476 -> socket:[63413458]
lrwx------ 1 root root 64 Apr 11 00:18 477 -> socket:[61639820]
lrwx------ 1 root root 64 Apr 11 00:18 478 -> socket:[61639821]
lrwx------ 1 root root 64 Apr 11 00:18 479 -> socket:[63413459]
lrwx------ 1 root root 64 Apr 11 00:18 48 -> socket:[39864824]
lrwx------ 1 root root 64 Apr 11 00:18 480 -> socket:[55023039]
lrwx------ 1 root root 64 Apr 11 00:18 481 -> socket:[55023059]
lrwx------ 1 root root 64 Apr 11 00:18 482 -> socket:[58315428]
lrwx------ 1 root root 64 Apr 11 00:18 483 -> socket:[62878758]
lrwx------ 1 root root 64 Apr 11 00:18 484 -> socket:[72357947]
lrwx------ 1 root root 64 Apr 11 00:18 485 -> socket:[70505743]
lrwx------ 1 root root 64 Apr 11 00:18 486 -> socket:[62878759]
lrwx------ 1 root root 64 Apr 11 00:18 487 -> socket:[58353456]
lrwx------ 1 root root 64 Apr 11 00:18 488 -> socket:[58353457]
lrwx------ 1 root root 64 Apr 11 00:18 489 -> socket:[60462338]
lrwx------ 1 root root 64 Apr 11 00:18 49 -> socket:[39908]
lrwx------ 1 root root 64 Apr 11 00:18 490 -> socket:[73415259]
lrwx------ 1 root root 64 Apr 11 00:18 491 -> socket:[58026150]
lrwx------ 1 root root 64 Apr 11 00:18 492 -> socket:[70075486]
lrwx------ 1 root root 64 Apr 11 00:18 493 -> socket:[59199658]
lrwx------ 1 root root 64 Apr 11 00:18 494 -> socket:[57783464]
lrwx------ 1 root root 64 Apr 11 00:18 495 -> socket:[59199659]
lrwx------ 1 root root 64 Apr 11 00:18 496 -> socket:[70075487]
lrwx------ 1 root root 64 Apr 11 00:18 497 -> socket:[76733304]
lrwx------ 1 root root 64 Apr 11 00:18 498 -> socket:[59507938]
lrwx------ 1 root root 64 Apr 11 00:18 499 -> socket:[60156343]
lrwx------ 1 root root 64 Apr 11 00:18 5 -> socket:[29023067]
lrwx------ 1 root root 64 Apr 11 00:18 50 -> socket:[2599971]
lrwx------ 1 root root 64 Apr 11 00:18 500 -> socket:[60156344]
lrwx------ 1 root root 64 Apr 11 00:18 501 -> socket:[67765166]
lrwx------ 1 root root 64 Apr 11 00:18 502 -> socket:[60062145]
lrwx------ 1 root root 64 Apr 11 00:18 503 -> socket:[86294884]
lrwx------ 1 root root 64 Apr 11 00:18 504 -> socket:[80898981]
lrwx------ 1 root root 64 Apr 11 00:18 505 -> socket:[104827848]
lrwx------ 1 root root 64 Apr 11 00:18 506 -> socket:[71336339]
lrwx------ 1 root root 64 Apr 11 00:18 507 -> socket:[97904395]
lrwx------ 1 root root 64 Apr 11 00:18 508 -> socket:[64129326]
lrwx------ 1 root root 64 Apr 11 00:18 509 -> socket:[73415307]
lrwx------ 1 root root 64 Apr 11 00:18 51 -> socket:[19333254]
lrwx------ 1 root root 64 Apr 11 00:18 510 -> socket:[95267536]
lrwx------ 1 root root 64 Apr 11 00:18 511 -> socket:[58930703]
lrwx------ 1 root root 64 Apr 11 00:18 512 -> socket:[71957182]
lrwx------ 1 root root 64 Apr 11 00:18 513 -> socket:[94329121]
lrwx------ 1 root root 64 Apr 11 00:18 514 -> socket:[63416776]
lrwx------ 1 root root 64 Apr 11 00:18 515 -> socket:[105720099]
lrwx------ 1 root root 64 Apr 11 00:18 516 -> socket:[83629706]
lrwx------ 1 root root 64 Apr 11 00:18 517 -> socket:[61052916]
lrwx------ 1 root root 64 Apr 11 00:18 518 -> socket:[61052917]
lrwx------ 1 root root 64 Apr 11 00:18 519 -> socket:[63416803]
lrwx------ 1 root root 64 Apr 11 00:18 52 -> socket:[18933196]
lrwx------ 1 root root 64 Apr 11 00:18 520 -> socket:[75517131]
lrwx------ 1 root root 64 Apr 11 00:18 521 -> socket:[89281767]
lrwx------ 1 root root 64 Apr 11 00:18 522 -> socket:[68033106]
lrwx------ 1 root root 64 Apr 11 00:18 523 -> socket:[62554279]
lrwx------ 1 root root 64 Apr 11 00:18 524 -> socket:[62554280]
lrwx------ 1 root root 64 Apr 11 00:18 525 -> socket:[60062146]
lrwx------ 1 root root 64 Apr 11 00:18 526 -> socket:[62558603]
lrwx------ 1 root root 64 Apr 11 00:18 527 -> socket:[68033107]
lrwx------ 1 root root 64 Apr 11 00:18 528 -> socket:[85955138]
lrwx------ 1 root root 64 Apr 11 00:18 529 -> socket:[93186679]
lrwx------ 1 root root 64 Apr 11 00:18 53 -> socket:[26045728]
lrwx------ 1 root root 64 Apr 11 00:18 530 -> socket:[65583406]
lrwx------ 1 root root 64 Apr 11 00:18 531 -> socket:[92640692]
lrwx------ 1 root root 64 Apr 11 00:18 532 -> socket:[62558604]
lrwx------ 1 root root 64 Apr 11 00:18 533 -> socket:[104550450]
lrwx------ 1 root root 64 Apr 11 00:18 534 -> socket:[74315772]
lrwx------ 1 root root 64 Apr 11 00:18 535 -> socket:[66484470]
lrwx------ 1 root root 64 Apr 11 00:18 536 -> socket:[61678730]
lrwx------ 1 root root 64 Apr 11 00:18 537 -> socket:[61678731]
lrwx------ 1 root root 64 Apr 11 00:18 538 -> socket:[77899516]
lrwx------ 1 root root 64 Apr 11 00:18 539 -> socket:[123660864]
lrwx------ 1 root root 64 Apr 11 00:18 54 -> socket:[2016868]
lrwx------ 1 root root 64 Apr 11 00:18 540 -> socket:[64126106]
lrwx------ 1 root root 64 Apr 11 00:18 541 -> socket:[64126107]
lrwx------ 1 root root 64 Apr 11 00:18 542 -> socket:[61674315]
lrwx------ 1 root root 64 Apr 11 00:18 543 -> socket:[61674316]
lrwx------ 1 root root 64 Apr 11 00:18 544 -> socket:[80034840]
lrwx------ 1 root root 64 Apr 11 00:18 545 -> socket:[66484471]
lrwx------ 1 root root 64 Apr 11 00:18 546 -> socket:[77936010]
lrwx------ 1 root root 64 Apr 11 00:18 547 -> socket:[76733915]
lrwx------ 1 root root 64 Apr 11 00:18 548 -> socket:[75266247]
lrwx------ 1 root root 64 Apr 11 00:18 549 -> socket:[62837268]
lrwx------ 1 root root 64 Apr 11 00:18 55 -> socket:[5239846]
lrwx------ 1 root root 64 Apr 11 00:18 550 -> socket:[62837269]
lrwx------ 1 root root 64 Apr 11 00:18 551 -> socket:[97169154]
lrwx------ 1 root root 64 Apr 11 00:18 552 -> socket:[92161665]
lrwx------ 1 root root 64 Apr 11 00:18 553 -> socket:[64702130]
lrwx------ 1 root root 64 Apr 11 00:18 554 -> socket:[65049002]
lrwx------ 1 root root 64 Apr 11 00:18 555 -> socket:[63809600]
lrwx------ 1 root root 64 Apr 11 00:18 556 -> socket:[63809875]
lrwx------ 1 root root 64 Apr 11 00:18 557 -> socket:[65049003]
lrwx------ 1 root root 64 Apr 11 00:18 558 -> socket:[93138786]
lrwx------ 1 root root 64 Apr 11 00:18 559 -> socket:[105172198]
lrwx------ 1 root root 64 Apr 11 00:18 56 -> socket:[7128362]
lrwx------ 1 root root 64 Apr 11 00:18 560 -> socket:[75266655]
lrwx------ 1 root root 64 Apr 11 00:18 561 -> socket:[75261871]
lrwx------ 1 root root 64 Apr 11 00:18 562 -> socket:[65586434]
lrwx------ 1 root root 64 Apr 11 00:18 563 -> socket:[89285088]
lrwx------ 1 root root 64 Apr 11 00:18 564 -> socket:[65586435]
lrwx------ 1 root root 64 Apr 11 00:18 565 -> socket:[77936011]
lrwx------ 1 root root 64 Apr 11 00:18 566 -> socket:[64702174]
lrwx------ 1 root root 64 Apr 11 00:18 567 -> socket:[84464177]
lrwx------ 1 root root 64 Apr 11 00:18 568 -> socket:[93186741]
lrwx------ 1 root root 64 Apr 11 00:18 569 -> socket:[75262254]
lrwx------ 1 root root 64 Apr 11 00:18 57 -> socket:[5239847]
lrwx------ 1 root root 64 Apr 11 00:18 570 -> socket:[77593604]
lrwx------ 1 root root 64 Apr 11 00:18 571 -> socket:[66182693]
lrwx------ 1 root root 64 Apr 11 00:18 572 -> socket:[66182753]
lrwx------ 1 root root 64 Apr 11 00:18 573 -> socket:[82720975]
lrwx------ 1 root root 64 Apr 11 00:18 574 -> socket:[76768806]
lrwx------ 1 root root 64 Apr 11 00:18 575 -> socket:[67730597]
lrwx------ 1 root root 64 Apr 11 00:18 576 -> socket:[77895646]
lrwx------ 1 root root 64 Apr 11 00:18 577 -> socket:[92640944]
lrwx------ 1 root root 64 Apr 11 00:18 578 -> socket:[77895647]
lrwx------ 1 root root 64 Apr 11 00:18 579 -> socket:[96470331]
lrwx------ 1 root root 64 Apr 11 00:18 58 -> socket:[44679332]
lrwx------ 1 root root 64 Apr 11 00:18 580 -> socket:[72354163]
lrwx------ 1 root root 64 Apr 11 00:18 581 -> socket:[72354164]
lrwx------ 1 root root 64 Apr 11 00:18 582 -> socket:[84464479]
lrwx------ 1 root root 64 Apr 11 00:18 583 -> socket:[82720976]
lrwx------ 1 root root 64 Apr 11 00:18 584 -> socket:[97578216]
lrwx------ 1 root root 64 Apr 11 00:18 585 -> socket:[74319492]
lrwx------ 1 root root 64 Apr 11 00:18 586 -> socket:[124483382]
lrwx------ 1 root root 64 Apr 11 00:18 587 -> socket:[92644915]
lrwx------ 1 root root 64 Apr 11 00:18 588 -> socket:[77593652]
lrwx------ 1 root root 64 Apr 11 00:18 589 -> socket:[76768807]
lrwx------ 1 root root 64 Apr 11 00:18 59 -> socket:[58610397]
lrwx------ 1 root root 64 Apr 11 00:18 590 -> socket:[86295237]
lrwx------ 1 root root 64 Apr 11 00:18 591 -> socket:[97578217]
lrwx------ 1 root root 64 Apr 11 00:18 592 -> socket:[96506775]
lrwx------ 1 root root 64 Apr 11 00:18 593 -> socket:[93138604]
lrwx------ 1 root root 64 Apr 11 00:18 594 -> socket:[94677138]
lrwx------ 1 root root 64 Apr 11 00:18 595 -> socket:[94677139]
lrwx------ 1 root root 64 Apr 11 00:18 596 -> socket:[79181587]
lrwx------ 1 root root 64 Apr 11 00:18 597 -> socket:[74366875]
lrwx------ 1 root root 64 Apr 11 00:18 598 -> socket:[80894401]
lrwx------ 1 root root 64 Apr 11 00:18 599 -> socket:[80894402]
lrwx------ 1 root root 64 Apr 11 00:18 6 -> socket:[2288850]
lrwx------ 1 root root 64 Apr 11 00:18 60 -> socket:[10434184]
lrwx------ 1 root root 64 Apr 11 00:18 600 -> socket:[75517034]
lrwx------ 1 root root 64 Apr 11 00:18 601 -> socket:[100100220]
lrwx------ 1 root root 64 Apr 11 00:18 602 -> socket:[89285089]
lrwx------ 1 root root 64 Apr 11 00:18 603 -> socket:[102344481]
lrwx------ 1 root root 64 Apr 11 00:18 604 -> socket:[75829973]
lrwx------ 1 root root 64 Apr 11 00:18 605 -> socket:[75829974]
lrwx------ 1 root root 64 Apr 11 00:18 606 -> socket:[86972794]
lrwx------ 1 root root 64 Apr 11 00:18 607 -> socket:[92678117]
lrwx------ 1 root root 64 Apr 11 00:18 608 -> socket:[89281768]
lrwx------ 1 root root 64 Apr 11 00:18 609 -> socket:[74366876]
lrwx------ 1 root root 64 Apr 11 00:18 61 -> socket:[7131240]
lrwx------ 1 root root 64 Apr 11 00:18 610 -> socket:[84798850]
lrwx------ 1 root root 64 Apr 11 00:18 611 -> socket:[84798851]
lrwx------ 1 root root 64 Apr 11 00:18 612 -> socket:[82507234]
lrwx------ 1 root root 64 Apr 11 00:18 613 -> socket:[82507235]
lrwx------ 1 root root 64 Apr 11 00:18 614 -> socket:[74315790]
lrwx------ 1 root root 64 Apr 11 00:18 615 -> socket:[85111256]
lrwx------ 1 root root 64 Apr 11 00:18 616 -> socket:[82165379]
lrwx------ 1 root root 64 Apr 11 00:18 617 -> socket:[76999314]
lrwx------ 1 root root 64 Apr 11 00:18 618 -> socket:[81857573]
lrwx------ 1 root root 64 Apr 11 00:18 619 -> socket:[93452130]
lrwx------ 1 root root 64 Apr 11 00:18 62 -> socket:[142094]
lrwx------ 1 root root 64 Apr 11 00:18 620 -> socket:[82165688]
lrwx------ 1 root root 64 Apr 11 00:18 621 -> socket:[88049330]
lrwx------ 1 root root 64 Apr 11 00:18 622 -> socket:[83280743]
lrwx------ 1 root root 64 Apr 11 00:18 623 -> socket:[83629821]
lrwx------ 1 root root 64 Apr 11 00:18 624 -> socket:[88049641]
lrwx------ 1 root root 64 Apr 11 00:18 625 -> socket:[83280744]
lrwx------ 1 root root 64 Apr 11 00:18 626 -> socket:[96502732]
lrwx------ 1 root root 64 Apr 11 00:18 627 -> socket:[76999315]
lrwx------ 1 root root 64 Apr 11 00:18 628 -> socket:[93452131]
lrwx------ 1 root root 64 Apr 11 00:18 629 -> socket:[92678118]
lrwx------ 1 root root 64 Apr 11 00:18 63 -> socket:[1285040]
lrwx------ 1 root root 64 Apr 11 00:18 630 -> socket:[85111257]
lrwx------ 1 root root 64 Apr 11 00:18 631 -> socket:[96506776]
lrwx------ 1 root root 64 Apr 11 00:18 632 -> socket:[113221071]
lrwx------ 1 root root 64 Apr 11 00:18 633 -> socket:[97574372]
lrwx------ 1 root root 64 Apr 11 00:18 634 -> socket:[100059872]
lrwx------ 1 root root 64 Apr 11 00:18 635 -> socket:[77940618]
lrwx------ 1 root root 64 Apr 11 00:18 636 -> socket:[96473673]
lrwx------ 1 root root 64 Apr 11 00:18 637 -> socket:[79176552]
lrwx------ 1 root root 64 Apr 11 00:18 638 -> socket:[79176553]
lrwx------ 1 root root 64 Apr 11 00:18 639 -> socket:[100100269]
lrwx------ 1 root root 64 Apr 11 00:18 64 -> socket:[30537946]
lrwx------ 1 root root 64 Apr 11 00:18 640 -> socket:[97904396]
lrwx------ 1 root root 64 Apr 11 00:18 641 -> socket:[93142766]
lrwx------ 1 root root 64 Apr 11 00:18 642 -> socket:[100060091]
lrwx------ 1 root root 64 Apr 11 00:18 643 -> socket:[115941255]
lrwx------ 1 root root 64 Apr 11 00:18 644 -> socket:[94329390]
lrwx------ 1 root root 64 Apr 11 00:18 645 -> socket:[85069290]
lrwx------ 1 root root 64 Apr 11 00:18 646 -> socket:[80080928]
lrwx------ 1 root root 64 Apr 11 00:18 647 -> socket:[80080929]
lrwx------ 1 root root 64 Apr 11 00:18 648 -> socket:[95267537]
lrwx------ 1 root root 64 Apr 11 00:18 649 -> socket:[83625371]
lrwx------ 1 root root 64 Apr 11 00:18 65 -> socket:[6190728]
lrwx------ 1 root root 64 Apr 11 00:18 650 -> socket:[124662566]
lrwx------ 1 root root 64 Apr 11 00:18 651 -> socket:[91938235]
lrwx------ 1 root root 64 Apr 11 00:18 652 -> socket:[202977874]
lrwx------ 1 root root 64 Apr 11 00:18 653 -> socket:[93142767]
lrwx------ 1 root root 64 Apr 11 00:18 654 -> socket:[97941915]
lrwx------ 1 root root 64 Apr 11 00:18 655 -> socket:[95225347]
lrwx------ 1 root root 64 Apr 11 00:18 656 -> socket:[83625372]
lrwx------ 1 root root 64 Apr 11 00:18 657 -> socket:[104827849]
lrwx------ 1 root root 64 Apr 11 00:18 658 -> socket:[87187122]
lrwx------ 1 root root 64 Apr 11 00:18 659 -> socket:[83592456]
lrwx------ 1 root root 64 Apr 11 00:18 66 -> socket:[33242652]
lrwx------ 1 root root 64 Apr 11 00:18 660 -> socket:[83592576]
lrwx------ 1 root root 64 Apr 11 00:18 661 -> socket:[87187201]
lrwx------ 1 root root 64 Apr 11 00:18 662 -> socket:[95225348]
lrwx------ 1 root root 64 Apr 11 00:18 663 -> socket:[94633668]
lrwx------ 1 root root 64 Apr 11 00:18 664 -> socket:[98531027]
lrwx------ 1 root root 64 Apr 11 00:18 665 -> socket:[115937432]
lrwx------ 1 root root 64 Apr 11 00:18 666 -> socket:[81852601]
lrwx------ 1 root root 64 Apr 11 00:18 667 -> socket:[86633042]
lrwx------ 1 root root 64 Apr 11 00:18 668 -> socket:[81852602]
lrwx------ 1 root root 64 Apr 11 00:18 669 -> socket:[91938144]
lrwx------ 1 root root 64 Apr 11 00:18 67 -> socket:[30533135]
lrwx------ 1 root root 64 Apr 11 00:18 670 -> socket:[88638539]
lrwx------ 1 root root 64 Apr 11 00:18 671 -> socket:[87780798]
lrwx------ 1 root root 64 Apr 11 00:18 672 -> socket:[109294070]
lrwx------ 1 root root 64 Apr 11 00:18 673 -> socket:[99707193]
lrwx------ 1 root root 64 Apr 11 00:18 674 -> socket:[89874253]
lrwx------ 1 root root 64 Apr 11 00:18 675 -> socket:[86889476]
lrwx------ 1 root root 64 Apr 11 00:18 676 -> socket:[86889477]
lrwx------ 1 root root 64 Apr 11 00:18 677 -> socket:[87780799]
lrwx------ 1 root root 64 Apr 11 00:18 678 -> socket:[86633043]
lrwx------ 1 root root 64 Apr 11 00:18 679 -> socket:[88671227]
lrwx------ 1 root root 64 Apr 11 00:18 68 -> socket:[29884828]
lrwx------ 1 root root 64 Apr 11 00:18 680 -> socket:[104550476]
lrwx------ 1 root root 64 Apr 11 00:18 681 -> socket:[105968532]
lrwx------ 1 root root 64 Apr 11 00:18 682 -> socket:[85069291]
lrwx------ 1 root root 64 Apr 11 00:18 683 -> socket:[99095801]
lrwx------ 1 root root 64 Apr 11 00:18 684 -> socket:[101861709]
lrwx------ 1 root root 64 Apr 11 00:18 685 -> socket:[99095802]
lrwx------ 1 root root 64 Apr 11 00:18 686 -> socket:[98531028]
lrwx------ 1 root root 64 Apr 11 00:18 687 -> socket:[109293767]
lrwx------ 1 root root 64 Apr 11 00:18 688 -> socket:[104505872]
lrwx------ 1 root root 64 Apr 11 00:18 689 -> socket:[100982346]
lrwx------ 1 root root 64 Apr 11 00:18 69 -> socket:[41697815]
lrwx------ 1 root root 64 Apr 11 00:18 690 -> socket:[92161747]
lrwx------ 1 root root 64 Apr 11 00:18 691 -> socket:[120425936]
lrwx------ 1 root root 64 Apr 11 00:18 692 -> socket:[124431231]
lrwx------ 1 root root 64 Apr 11 00:18 693 -> socket:[115634271]
lrwx------ 1 root root 64 Apr 11 00:18 694 -> socket:[105119862]
lrwx------ 1 root root 64 Apr 11 00:18 695 -> socket:[86972700]
lrwx------ 1 root root 64 Apr 11 00:18 696 -> socket:[101277554]
lrwx------ 1 root root 64 Apr 11 00:18 697 -> socket:[105965072]
lrwx------ 1 root root 64 Apr 11 00:18 698 -> socket:[101501403]
lrwx------ 1 root root 64 Apr 11 00:18 699 -> socket:[100952662]
lrwx------ 1 root root 64 Apr 11 00:18 7 -> socket:[15166]
lrwx------ 1 root root 64 Apr 11 00:18 70 -> socket:[13280770]
lrwx------ 1 root root 64 Apr 11 00:18 700 -> socket:[88323125]
lrwx------ 1 root root 64 Apr 11 00:18 701 -> socket:[88323126]
lrwx------ 1 root root 64 Apr 11 00:18 702 -> socket:[96473774]
lrwx------ 1 root root 64 Apr 11 00:18 703 -> socket:[94633175]
lrwx------ 1 root root 64 Apr 11 00:18 704 -> socket:[101501474]
lrwx------ 1 root root 64 Apr 11 00:18 705 -> socket:[101277592]
lrwx------ 1 root root 64 Apr 11 00:18 706 -> socket:[89874392]
lrwx------ 1 root root 64 Apr 11 00:18 707 -> socket:[95553760]
lrwx------ 1 root root 64 Apr 11 00:18 708 -> socket:[95553761]
lrwx------ 1 root root 64 Apr 11 00:18 709 -> socket:[88675697]
lrwx------ 1 root root 64 Apr 11 00:18 71 -> socket:[97873]
lrwx------ 1 root root 64 Apr 11 00:18 710 -> socket:[88675747]
lrwx------ 1 root root 64 Apr 11 00:18 711 -> socket:[114111639]
lrwx------ 1 root root 64 Apr 11 00:18 712 -> socket:[89913709]
lrwx------ 1 root root 64 Apr 11 00:18 713 -> socket:[95522566]
lrwx------ 1 root root 64 Apr 11 00:18 714 -> socket:[89323700]
lrwx------ 1 root root 64 Apr 11 00:18 715 -> socket:[89913710]
lrwx------ 1 root root 64 Apr 11 00:18 716 -> socket:[123142057]
lrwx------ 1 root root 64 Apr 11 00:18 717 -> socket:[97573611]
lrwx------ 1 root root 64 Apr 11 00:18 718 -> socket:[93760069]
lrwx------ 1 root root 64 Apr 11 00:18 719 -> socket:[93760070]
lrwx------ 1 root root 64 Apr 11 00:18 72 -> socket:[4180808]
lrwx------ 1 root root 64 Apr 11 00:18 720 -> socket:[97941916]
lrwx------ 1 root root 64 Apr 11 00:18 721 -> socket:[100982347]
lrwx------ 1 root root 64 Apr 11 00:18 722 -> socket:[96811785]
lrwx------ 1 root root 64 Apr 11 00:18 723 -> socket:[98136950]
lrwx------ 1 root root 64 Apr 11 00:18 724 -> socket:[101546731]
lrwx------ 1 root root 64 Apr 11 00:18 725 -> socket:[105172199]
lrwx------ 1 root root 64 Apr 11 00:18 726 -> socket:[93191113]
lrwx------ 1 root root 64 Apr 11 00:18 727 -> socket:[93191114]
lrwx------ 1 root root 64 Apr 11 00:18 728 -> socket:[120714309]
lrwx------ 1 root root 64 Apr 11 00:18 729 -> socket:[113221252]
lrwx------ 1 root root 64 Apr 11 00:18 73 -> socket:[6868690]
lrwx------ 1 root root 64 Apr 11 00:18 730 -> socket:[95522567]
lrwx------ 1 root root 64 Apr 11 00:18 731 -> socket:[96502733]
lrwx------ 1 root root 64 Apr 11 00:18 732 -> socket:[100056133]
lrwx------ 1 root root 64 Apr 11 00:18 733 -> socket:[95855935]
lrwx------ 1 root root 64 Apr 11 00:18 734 -> socket:[97168681]
lrwx------ 1 root root 64 Apr 11 00:18 735 -> socket:[106297115]
lrwx------ 1 root root 64 Apr 11 00:18 736 -> socket:[100595543]
lrwx------ 1 root root 64 Apr 11 00:18 737 -> socket:[95855936]
lrwx------ 1 root root 64 Apr 11 00:18 738 -> socket:[98202351]
lrwx------ 1 root root 64 Apr 11 00:18 739 -> socket:[115634439]
lrwx------ 1 root root 64 Apr 11 00:18 74 -> socket:[3187722]
lrwx------ 1 root root 64 Apr 11 00:18 740 -> socket:[98202352]
lrwx------ 1 root root 64 Apr 11 00:18 741 -> socket:[144381309]
lrwx------ 1 root root 64 Apr 11 00:18 742 -> socket:[93456234]
lrwx------ 1 root root 64 Apr 11 00:18 743 -> socket:[93456235]
lrwx------ 1 root root 64 Apr 11 00:18 744 -> socket:[123142058]
lrwx------ 1 root root 64 Apr 11 00:18 745 -> socket:[124338223]
lrwx------ 1 root root 64 Apr 11 00:18 746 -> socket:[123578652]
lrwx------ 1 root root 64 Apr 11 00:18 747 -> socket:[105965073]
lrwx------ 1 root root 64 Apr 11 00:18 748 -> socket:[119189380]
lrwx------ 1 root root 64 Apr 11 00:18 749 -> socket:[95823849]
lrwx------ 1 root root 64 Apr 11 00:18 75 -> socket:[3187723]
lrwx------ 1 root root 64 Apr 11 00:18 750 -> socket:[99759621]
lrwx------ 1 root root 64 Apr 11 00:18 751 -> socket:[95823850]
lrwx------ 1 root root 64 Apr 11 00:18 752 -> socket:[99759622]
lrwx------ 1 root root 64 Apr 11 00:18 753 -> socket:[101546732]
lrwx------ 1 root root 64 Apr 11 00:18 754 -> socket:[100595544]
lrwx------ 1 root root 64 Apr 11 00:18 755 -> socket:[100952709]
lrwx------ 1 root root 64 Apr 11 00:18 756 -> socket:[97636893]
lrwx------ 1 root root 64 Apr 11 00:18 757 -> socket:[97636963]
lrwx------ 1 root root 64 Apr 11 00:18 758 -> socket:[123525185]
lrwx------ 1 root root 64 Apr 11 00:18 759 -> socket:[101861849]
lrwx------ 1 root root 64 Apr 11 00:18 76 -> socket:[35323432]
lrwx------ 1 root root 64 Apr 11 00:18 760 -> socket:[94374784]
lrwx------ 1 root root 64 Apr 11 00:18 761 -> socket:[94374785]
lrwx------ 1 root root 64 Apr 11 00:18 762 -> socket:[96811786]
lrwx------ 1 root root 64 Apr 11 00:18 763 -> socket:[108697687]
lrwx------ 1 root root 64 Apr 11 00:18 764 -> socket:[100056151]
lrwx------ 1 root root 64 Apr 11 00:18 765 -> socket:[114433606]
lrwx------ 1 root root 64 Apr 11 00:18 766 -> socket:[107826291]
lrwx------ 1 root root 64 Apr 11 00:18 767 -> socket:[110522221]
lrwx------ 1 root root 64 Apr 11 00:18 768 -> socket:[105720395]
lrwx------ 1 root root 64 Apr 11 00:18 769 -> socket:[104554787]
lrwx------ 1 root root 64 Apr 11 00:18 77 -> socket:[48082179]
lrwx------ 1 root root 64 Apr 11 00:18 770 -> socket:[96470229]
lrwx------ 1 root root 64 Apr 11 00:18 771 -> socket:[124347585]
lrwx------ 1 root root 64 Apr 11 00:18 772 -> socket:[113497663]
lrwx------ 1 root root 64 Apr 11 00:18 773 -> socket:[125044692]
lrwx------ 1 root root 64 Apr 11 00:18 774 -> socket:[109640054]
lrwx------ 1 root root 64 Apr 11 00:18 775 -> socket:[99755530]
lrwx------ 1 root root 64 Apr 11 00:18 776 -> socket:[101550525]
lrwx------ 1 root root 64 Apr 11 00:18 777 -> socket:[131544865]
lrwx------ 1 root root 64 Apr 11 00:18 778 -> socket:[101550526]
lrwx------ 1 root root 64 Apr 11 00:18 779 -> socket:[114115654]
lrwx------ 1 root root 64 Apr 11 00:18 78 -> socket:[17432113]
lrwx------ 1 root root 64 Apr 11 00:18 780 -> socket:[118910825]
lrwx------ 1 root root 64 Apr 11 00:18 781 -> socket:[115941256]
lrwx------ 1 root root 64 Apr 11 00:18 782 -> socket:[104554788]
lrwx------ 1 root root 64 Apr 11 00:18 783 -> socket:[110522465]
lrwx------ 1 root root 64 Apr 11 00:18 784 -> socket:[98137160]
lrwx------ 1 root root 64 Apr 11 00:18 785 -> socket:[112348251]
lrwx------ 1 root root 64 Apr 11 00:18 786 -> socket:[109901809]
lrwx------ 1 root root 64 Apr 11 00:18 787 -> socket:[112348302]
lrwx------ 1 root root 64 Apr 11 00:18 788 -> socket:[109596258]
lrwx------ 1 root root 64 Apr 11 00:18 789 -> socket:[103285078]
lrwx------ 1 root root 64 Apr 11 00:18 79 -> socket:[7796878]
lrwx------ 1 root root 64 Apr 11 00:18 790 -> socket:[105123284]
lrwx------ 1 root root 64 Apr 11 00:18 791 -> socket:[103285163]
lrwx------ 1 root root 64 Apr 11 00:18 792 -> socket:[99755742]
lrwx------ 1 root root 64 Apr 11 00:18 793 -> socket:[105123285]
lrwx------ 1 root root 64 Apr 11 00:18 794 -> socket:[118911226]
lrwx------ 1 root root 64 Apr 11 00:18 795 -> socket:[109901810]
lrwx------ 1 root root 64 Apr 11 00:18 796 -> socket:[114111640]
lrwx------ 1 root root 64 Apr 11 00:18 797 -> socket:[123410598]
lrwx------ 1 root root 64 Apr 11 00:18 798 -> socket:[108732472]
lrwx------ 1 root root 64 Apr 11 00:18 799 -> socket:[104875331]
lrwx------ 1 root root 64 Apr 11 00:18 8 -> socket:[15193]
lrwx------ 1 root root 64 Apr 11 00:18 80 -> socket:[35666302]
lrwx------ 1 root root 64 Apr 11 00:18 800 -> socket:[114433607]
lrwx------ 1 root root 64 Apr 11 00:18 801 -> socket:[104875432]
lrwx------ 1 root root 64 Apr 11 00:18 802 -> socket:[102344377]
lrwx------ 1 root root 64 Apr 11 00:18 803 -> socket:[123471848]
lrwx------ 1 root root 64 Apr 11 00:18 804 -> socket:[106604189]
lrwx------ 1 root root 64 Apr 11 00:18 805 -> socket:[106604190]
lrwx------ 1 root root 64 Apr 11 00:18 806 -> socket:[108732473]
lrwx------ 1 root root 64 Apr 11 00:18 807 -> socket:[100642900]
lrwx------ 1 root root 64 Apr 11 00:18 808 -> socket:[100642901]
lrwx------ 1 root root 64 Apr 11 00:18 809 -> socket:[102344378]
lrwx------ 1 root root 64 Apr 11 00:18 81 -> socket:[13280771]
lrwx------ 1 root root 64 Apr 11 00:18 810 -> socket:[123965915]
lrwx------ 1 root root 64 Apr 11 00:18 811 -> socket:[105968556]
lrwx------ 1 root root 64 Apr 11 00:18 812 -> socket:[102344384]
lrwx------ 1 root root 64 Apr 11 00:18 813 -> socket:[102344385]
lrwx------ 1 root root 64 Apr 11 00:18 814 -> socket:[124136040]
lrwx------ 1 root root 64 Apr 11 00:18 815 -> socket:[102344386]
lrwx------ 1 root root 64 Apr 11 00:18 816 -> socket:[103280838]
lrwx------ 1 root root 64 Apr 11 00:18 817 -> socket:[102344387]
lrwx------ 1 root root 64 Apr 11 00:18 818 -> socket:[119189381]
lrwx------ 1 root root 64 Apr 11 00:18 819 -> socket:[124344340]
lrwx------ 1 root root 64 Apr 11 00:18 82 -> socket:[18690533]
lrwx------ 1 root root 64 Apr 11 00:18 820 -> socket:[111477365]
lrwx------ 1 root root 64 Apr 11 00:18 821 -> socket:[113493487]
lrwx------ 1 root root 64 Apr 11 00:18 822 -> socket:[102344388]
lrwx------ 1 root root 64 Apr 11 00:18 823 -> socket:[119242564]
lrwx------ 1 root root 64 Apr 11 00:18 824 -> socket:[111477366]
lrwx------ 1 root root 64 Apr 11 00:18 825 -> socket:[124179028]
lrwx------ 1 root root 64 Apr 11 00:18 826 -> socket:[103280839]
lrwx------ 1 root root 64 Apr 11 00:18 827 -> socket:[111734003]
lrwx------ 1 root root 64 Apr 11 00:18 828 -> socket:[124430878]
lrwx------ 1 root root 64 Apr 11 00:18 829 -> socket:[109332791]
lrwx------ 1 root root 64 Apr 11 00:18 83 -> socket:[15113257]
lrwx------ 1 root root 64 Apr 11 00:18 830 -> socket:[107826292]
lrwx------ 1 root root 64 Apr 11 00:18 831 -> socket:[109640007]
lrwx------ 1 root root 64 Apr 11 00:18 832 -> socket:[102344389]
lrwx------ 1 root root 64 Apr 11 00:18 833 -> socket:[102344391]
lrwx------ 1 root root 64 Apr 11 00:18 834 -> socket:[111734004]
lrwx------ 1 root root 64 Apr 11 00:18 835 -> socket:[124352771]
lrwx------ 1 root root 64 Apr 11 00:18 836 -> socket:[119497010]
lrwx------ 1 root root 64 Apr 11 00:18 837 -> socket:[105119710]
lrwx------ 1 root root 64 Apr 11 00:18 838 -> socket:[108697852]
lrwx------ 1 root root 64 Apr 11 00:18 839 -> socket:[109030372]
lrwx------ 1 root root 64 Apr 11 00:18 84 -> socket:[30495099]
lrwx------ 1 root root 64 Apr 11 00:18 840 -> socket:[108419975]
lrwx------ 1 root root 64 Apr 11 00:18 841 -> socket:[119242565]
lrwx------ 1 root root 64 Apr 11 00:18 842 -> socket:[119551887]
lrwx------ 1 root root 64 Apr 11 00:18 843 -> socket:[108149143]
lrwx------ 1 root root 64 Apr 11 00:18 844 -> socket:[109599435]
lrwx------ 1 root root 64 Apr 11 00:18 845 -> socket:[108419918]
lrwx------ 1 root root 64 Apr 11 00:18 846 -> socket:[108737043]
lrwx------ 1 root root 64 Apr 11 00:18 847 -> socket:[109599517]
lrwx------ 1 root root 64 Apr 11 00:18 848 -> socket:[122227927]
lrwx------ 1 root root 64 Apr 11 00:18 849 -> socket:[113493507]
lrwx------ 1 root root 64 Apr 11 00:18 85 -> socket:[780573]
lrwx------ 1 root root 64 Apr 11 00:18 850 -> socket:[111770637]
lrwx------ 1 root root 64 Apr 11 00:18 851 -> socket:[109030373]
lrwx------ 1 root root 64 Apr 11 00:18 852 -> socket:[119551888]
lrwx------ 1 root root 64 Apr 11 00:18 853 -> socket:[130559331]
lrwx------ 1 root root 64 Apr 11 00:18 854 -> socket:[109944160]
lrwx------ 1 root root 64 Apr 11 00:18 855 -> socket:[109944161]
lrwx------ 1 root root 64 Apr 11 00:18 856 -> socket:[120714310]
lrwx------ 1 root root 64 Apr 11 00:18 857 -> socket:[106296757]
lrwx------ 1 root root 64 Apr 11 00:18 858 -> socket:[111188251]
lrwx------ 1 root root 64 Apr 11 00:18 859 -> socket:[105715053]
lrwx------ 1 root root 64 Apr 11 00:18 86 -> socket:[2597159]
lrwx------ 1 root root 64 Apr 11 00:18 860 -> socket:[105715187]
lrwx------ 1 root root 64 Apr 11 00:18 861 -> socket:[140870113]
lrwx------ 1 root root 64 Apr 11 00:18 862 -> socket:[123753916]
lrwx------ 1 root root 64 Apr 11 00:18 863 -> socket:[111188252]
lrwx------ 1 root root 64 Apr 11 00:18 864 -> socket:[108149144]
lrwx------ 1 root root 64 Apr 11 00:18 865 -> socket:[123790828]
lrwx------ 1 root root 64 Apr 11 00:18 866 -> socket:[109332851]
lrwx------ 1 root root 64 Apr 11 00:18 867 -> socket:[108737063]
lrwx------ 1 root root 64 Apr 11 00:18 868 -> socket:[110518700]
lrwx------ 1 root root 64 Apr 11 00:18 869 -> socket:[122227928]
lrwx------ 1 root root 64 Apr 11 00:18 87 -> socket:[6528626]
lrwx------ 1 root root 64 Apr 11 00:18 870 -> socket:[110518725]
lrwx------ 1 root root 64 Apr 11 00:18 871 -> socket:[124178272]
lrwx------ 1 root root 64 Apr 11 00:18 872 -> socket:[118705303]
lrwx------ 1 root root 64 Apr 11 00:18 873 -> socket:[123762662]
lrwx------ 1 root root 64 Apr 11 00:18 874 -> socket:[106648242]
lrwx------ 1 root root 64 Apr 11 00:18 875 -> socket:[106648243]
lrwx------ 1 root root 64 Apr 11 00:18 876 -> socket:[109595966]
lrwx------ 1 root root 64 Apr 11 00:18 877 -> socket:[115590963]
lrwx------ 1 root root 64 Apr 11 00:18 878 -> socket:[114115987]
lrwx------ 1 root root 64 Apr 11 00:18 879 -> socket:[119868283]
lrwx------ 1 root root 64 Apr 11 00:18 88 -> socket:[6528627]
lrwx------ 1 root root 64 Apr 11 00:18 880 -> socket:[131951849]
lrwx------ 1 root root 64 Apr 11 00:18 881 -> socket:[134261970]
lrwx------ 1 root root 64 Apr 11 00:18 882 -> socket:[108990587]
lrwx------ 1 root root 64 Apr 11 00:18 883 -> socket:[138247097]
lrwx------ 1 root root 64 Apr 11 00:18 884 -> socket:[108990588]
lrwx------ 1 root root 64 Apr 11 00:18 885 -> socket:[120425937]
lrwx------ 1 root root 64 Apr 11 00:18 886 -> socket:[122878893]
lrwx------ 1 root root 64 Apr 11 00:18 887 -> socket:[144491146]
lrwx------ 1 root root 64 Apr 11 00:18 888 -> socket:[122879176]
lrwx------ 1 root root 64 Apr 11 00:18 889 -> socket:[124046533]
lrwx------ 1 root root 64 Apr 11 00:18 89 -> socket:[19333255]
lrwx------ 1 root root 64 Apr 11 00:18 890 -> socket:[111140414]
lrwx------ 1 root root 64 Apr 11 00:18 891 -> socket:[111140415]
lrwx------ 1 root root 64 Apr 11 00:18 892 -> socket:[110200991]
lrwx------ 1 root root 64 Apr 11 00:18 893 -> socket:[110200992]
lrwx------ 1 root root 64 Apr 11 00:18 894 -> socket:[124238353]
lrwx------ 1 root root 64 Apr 11 00:18 895 -> socket:[125294050]
lrwx------ 1 root root 64 Apr 11 00:18 896 -> socket:[133383082]
lrwx------ 1 root root 64 Apr 11 00:18 897 -> socket:[118655524]
lrwx------ 1 root root 64 Apr 11 00:18 898 -> socket:[138675485]
lrwx------ 1 root root 64 Apr 11 00:18 899 -> socket:[111770638]
lrwx------ 1 root root 64 Apr 11 00:18 9 -> socket:[15229]
lrwx------ 1 root root 64 Apr 11 00:18 90 -> socket:[26078588]
lrwx------ 1 root root 64 Apr 11 00:18 900 -> socket:[118655567]
lrwx------ 1 root root 64 Apr 11 00:18 901 -> socket:[123661680]
lrwx------ 1 root root 64 Apr 11 00:18 902 -> socket:[116224977]
lrwx------ 1 root root 64 Apr 11 00:18 903 -> socket:[131548976]
lrwx------ 1 root root 64 Apr 11 00:18 904 -> socket:[133219649]
lrwx------ 1 root root 64 Apr 11 00:18 905 -> socket:[111144342]
lrwx------ 1 root root 64 Apr 11 00:18 906 -> socket:[116224997]
lrwx------ 1 root root 64 Apr 11 00:18 907 -> socket:[115937270]
lrwx------ 1 root root 64 Apr 11 00:18 908 -> socket:[120763977]
lrwx------ 1 root root 64 Apr 11 00:18 909 -> socket:[120764142]
lrwx------ 1 root root 64 Apr 11 00:18 91 -> socket:[164862]
lrwx------ 1 root root 64 Apr 11 00:18 910 -> socket:[111144343]
lrwx------ 1 root root 64 Apr 11 00:18 911 -> socket:[115896457]
lrwx------ 1 root root 64 Apr 11 00:18 912 -> socket:[112352367]
lrwx------ 1 root root 64 Apr 11 00:18 913 -> socket:[112352391]
lrwx------ 1 root root 64 Apr 11 00:18 914 -> socket:[158396134]
lrwx------ 1 root root 64 Apr 11 00:18 915 -> socket:[115896458]
lrwx------ 1 root root 64 Apr 11 00:18 916 -> socket:[125223556]
lrwx------ 1 root root 64 Apr 11 00:18 917 -> socket:[114430043]
lrwx------ 1 root root 64 Apr 11 00:18 918 -> socket:[113497152]
lrwx------ 1 root root 64 Apr 11 00:18 919 -> socket:[114430044]
lrwx------ 1 root root 64 Apr 11 00:18 92 -> socket:[33242888]
lrwx------ 1 root root 64 Apr 11 00:18 920 -> socket:[113829776]
lrwx------ 1 root root 64 Apr 11 00:18 921 -> socket:[113829777]
lrwx------ 1 root root 64 Apr 11 00:18 922 -> socket:[118914342]
lrwx------ 1 root root 64 Apr 11 00:18 923 -> socket:[123764342]
lrwx------ 1 root root 64 Apr 11 00:18 924 -> socket:[133376148]
lrwx------ 1 root root 64 Apr 11 00:18 925 -> socket:[122882250]
lrwx------ 1 root root 64 Apr 11 00:18 926 -> socket:[121962226]
lrwx------ 1 root root 64 Apr 11 00:18 927 -> socket:[121962271]
lrwx------ 1 root root 64 Apr 11 00:18 928 -> socket:[122882251]
lrwx------ 1 root root 64 Apr 11 00:18 929 -> socket:[123665214]
lrwx------ 1 root root 64 Apr 11 00:18 93 -> socket:[780574]
lrwx------ 1 root root 64 Apr 11 00:18 930 -> socket:[114744802]
lrwx------ 1 root root 64 Apr 11 00:18 931 -> socket:[118914343]
lrwx------ 1 root root 64 Apr 11 00:18 932 -> socket:[115594449]
lrwx------ 1 root root 64 Apr 11 00:18 933 -> socket:[195699581]
lrwx------ 1 root root 64 Apr 11 00:18 934 -> socket:[124431037]
lrwx------ 1 root root 64 Apr 11 00:18 935 -> socket:[119496415]
lrwx------ 1 root root 64 Apr 11 00:18 936 -> socket:[114744852]
lrwx------ 1 root root 64 Apr 11 00:18 937 -> socket:[114983842]
lrwx------ 1 root root 64 Apr 11 00:18 938 -> socket:[114983843]
lrwx------ 1 root root 64 Apr 11 00:18 939 -> socket:[115594537]
lrwx------ 1 root root 64 Apr 11 00:18 94 -> socket:[2599747]
lrwx------ 1 root root 64 Apr 11 00:18 940 -> socket:[128584968]
lrwx------ 1 root root 64 Apr 11 00:18 941 -> socket:[134334526]
lrwx------ 1 root root 64 Apr 11 00:18 942 -> socket:[123764889]
lrwx------ 1 root root 64 Apr 11 00:18 943 -> socket:[118705304]
lrwx------ 1 root root 64 Apr 11 00:18 944 -> socket:[124107882]
lrwx------ 1 root root 64 Apr 11 00:18 945 -> socket:[121921512]
lrwx------ 1 root root 64 Apr 11 00:18 946 -> socket:[130311368]
lrwx------ 1 root root 64 Apr 11 00:18 947 -> socket:[123410593]
lrwx------ 1 root root 64 Apr 11 00:18 948 -> socket:[123146804]
lrwx------ 1 root root 64 Apr 11 00:18 949 -> socket:[160619490]
lrwx------ 1 root root 64 Apr 11 00:18 95 -> socket:[43766693]
lrwx------ 1 root root 64 Apr 11 00:18 950 -> socket:[123146805]
lrwx------ 1 root root 64 Apr 11 00:18 951 -> socket:[124238832]
lrwx------ 1 root root 64 Apr 11 00:18 952 -> socket:[119236964]
lrwx------ 1 root root 64 Apr 11 00:18 953 -> socket:[119914908]
lrwx------ 1 root root 64 Apr 11 00:18 954 -> socket:[119236965]
lrwx------ 1 root root 64 Apr 11 00:18 955 -> socket:[119914909]
lrwx------ 1 root root 64 Apr 11 00:18 956 -> socket:[124433512]
lrwx------ 1 root root 64 Apr 11 00:18 957 -> socket:[132972664]
lrwx------ 1 root root 64 Apr 11 00:18 958 -> socket:[121094711]
lrwx------ 1 root root 64 Apr 11 00:18 959 -> socket:[125225195]
lrwx------ 1 root root 64 Apr 11 00:18 96 -> socket:[51369067]
lrwx------ 1 root root 64 Apr 11 00:18 960 -> socket:[124433547]
lrwx------ 1 root root 64 Apr 11 00:18 961 -> socket:[134729516]
lrwx------ 1 root root 64 Apr 11 00:18 962 -> socket:[130859382]
lrwx------ 1 root root 64 Apr 11 00:18 963 -> socket:[133219415]
lrwx------ 1 root root 64 Apr 11 00:18 964 -> socket:[133881723]
lrwx------ 1 root root 64 Apr 11 00:18 965 -> socket:[123105855]
lrwx------ 1 root root 64 Apr 11 00:18 966 -> socket:

I've still got the clusters up and am happy to run tests or grant SSH access if someone wants to poke around a bit with a live version of the problem. It's not the end of the world for me to just spin up a new cluster and update DNS, especially if it helps.

[edit] restarting the cluster now, but still have ~12GB of log files, if anyone's interested.

Interestingly I can not repro this against head.

root@kubernetes-minion-qup8:/home/thockin# pidof kube-proxy
423
root@kubernetes-minion-qup8:/home/thockin# ls -l /proc/423/fd | wc -l
24
root@kubernetes-minion-qup8:/home/thockin# for i in `seq 1 100`; do
nslookup kubernetes.default.kubernetes.local 10.0.0.10 > /dev/null; done
root@kubernetes-minion-qup8:/home/thockin# ls -l /proc/423/fd | wc -l
127
root@kubernetes-minion-qup8:/home/thockin# sleep 60; ls -l /proc/423/fd |
wc -l
25

What you're seeing here is the UDP timeout of 1 minute (which may be too
long - bug filed).

{
  "paths": [
    "/api",
    "/api/v1beta1",
    "/api/v1beta2",
    "/api/v1beta3",
    "/healthz",
    "/healthz/ping",
    "/logs/",
    "/metrics",
    "/static/",
    "/swagger-ui/",
    "/swaggerapi/",
    "/validate",
    "/version"
  ]
}
root@kubernetes-minion-qup8:/home/thockin# ls -l /proc/423/fd | wc -l
25
root@kubernetes-minion-qup8:/home/thockin# for i in`seq 1 100`; do curl
10.0.0.1 > /dev/null 2>&1; done
root@kubernetes-minion-qup8:/home/thockin# ls -l /proc/423/fd | wc -l
24

What you see here is that the endpoint exists, but hitting it 100 times
does not leak FDs.

Can you try these experiments on a freshly killed-and-restarted kube-proxy?

On Fri, Apr 10, 2015 at 5:29 PM, Paddy notifications@github.com wrote:

I've still got the clusters up and am happy to run tests or grant SSH
access if someone wants to poke around a bit with a live version of the
problem. It's not the end of the world for me to just spin up a new cluster
and update DNS, especially if it helps.

—
Reply to this email directly or view it on GitHub
#5461 (comment)
.

FWIW we don't see leaking fd's on the proxy, going form 0 to 100 pods per node then back again. Depending on how they user starts, they will need to up the ulimit, we only changed the systemd unit files... If folks start another way, then we'll need to doc.

I've been trying to reproduce this on and off again for a while now, and haven't been able to. There is a non-zero chance that it was at the application level, not at the kubernetes level (e.g., the influxdb monitoring tool went berserk, or something.)

I am experiencing leaked filehandles with 0.17.1's kube-proxy:

i tuned up logging on kube-proxy to --v=4 and found a reproducable pattern. We are hitting an endpoint through an apache reverse proxy, which is exposed via publicIP. With every http request we see 2 filehandles created, which are never returned. Correlating to the request & fd openings are entries in the log (1 per request): Mapped service "default/datahub-backend:" to endpoint 10.244.93.5:8080

There seems to be some throttling to this: the 5th request will neither trigger increase the fd count permanently nor trigger "Mapped service". After you wait a while, the leaking starts again.

The animation clarifies maybe what i'm talking about:

@jayunit100 ^ could you please verify.

@timothysc @mkulke okay, interesting !

• I see the exact same thing w/ my soak test... which does polling on a public ip similar to your use case... which uses the public ip and makes requests every second.
• We can modify the pending E2e k8bps (take #2) + Update to V1 #8145 to detect for this in the e2e's.
  Screenshot below.
• To reproduce , you can just directly run ./examples/k8petstore/k8petstore.sh , and lsof on the kube-proxy process. Youll see the increments in creasing in twos.

also, confirmed that its periodic, so, not sure what the precise bug is.. i thought it was monotonically increasing over time, but appears to be capped .

This looks like sessions being kept alive at @mkulke's rev-proxy or at @jayunit100's public IP balancer. I can not repro this directly.

root@kubernetes-minion-2re0:/home/thockin# > /var/log/kube-proxy.log; /etc/init.d/kube-proxy restart
[ ok ] Restarting The Kubernetes network proxy: kube-proxy.
root@kubernetes-minion-2re0:/home/thockin# ls -l /proc/`pidof kube-proxy`/fd | wc -l
16
root@kubernetes-minion-2re0:/home/thockin# for in in $(seq 1 10); do curl -s 10.0.138.45:8675 >/dev/null; done
root@kubernetes-minion-2re0:/home/thockin# ls -l /proc/`pidof kube-proxy`/fd | wc -l
16

For stronger evidence:

root@kubernetes-minion-2re0:/home/thockin# ls -l /proc/`pidof kube-proxy`/fd | wc -l
16
root@kubernetes-minion-2re0:/home/thockin# for in in $(seq 1 100); do curl -s 10.0.138.45:8675; echo; done | sort | uniq -c
     20 hostnames-24br0
     20 hostnames-708ex
     20 hostnames-avup4
     20 hostnames-b6oty
     20 hostnames-b9b7i
root@kubernetes-minion-2re0:/home/thockin# ls -l /proc/`pidof kube-proxy`/fd | wc -l
16
root@kubernetes-minion-2re0:/home/thockin# for in in $(seq 1 1000); do curl -s 10.0.138.45:8675; echo; done | sort | uniq -c
    200 hostnames-24br0
    200 hostnames-708ex
    200 hostnames-avup4
    200 hostnames-b6oty
    200 hostnames-b9b7i
root@kubernetes-minion-2re0:/home/thockin# ls -l /proc/`pidof kube-proxy`/fd | wc -l
16
root@kubernetes-minion-2re0:/home/thockin# for in in $(seq 1 10000); do curl -s 10.0.138.45:8675; echo; done | sort | uniq -c
   2000 hostnames-24br0
   2000 hostnames-708ex
   2000 hostnames-avup4
   2000 hostnames-b6oty
   2000 hostnames-b9b7i
root@kubernetes-minion-2re0:/home/thockin# ls -l /proc/`pidof kube-proxy`/fd | wc -l
16

yup, shall we close? from what i can see.

• deleting the namespace cleans up all connections
• at worst, there are periodic bursts where some extra connections linger for a short period.
• net maximum open fds doesnt increase globally even under high request load

at worst, i think there is some possibility that, in some cases, if you have a persistent service, and you keep pollling it, and then you stop - and keep the service around - maybe some open files persist? but it doesnt seem like a flagrant file descripter leak is occuring in side kubeproxy.

of course, i have only been looking at this for a few hours, but thats my first guess.

hi there,

plz have a look at this one, it is related i guess: #8891

the leaked handles are not returned in our setup, i tried to meditate over the kube-proxy code, but i could not find out why the handles are not closed. it appears that connections between a keepalive'd tomcat and kube-proxy are not closed, but they're not closed even on the deletion of the tomcat pod.

@jayunit100 This issue is already closed. :) The only fix here would be to force-close sockets after a timeout, which seems sort of hostile since we are "just" a proxy.

@mkulke can you repro my results without your extra reverse-proxy?

@thokin: i'll try that later, but i suspect everything will be fine. The problems seem to be related to keepalive connections from proxy to backend. While those can be turned off, it this is a pretty common scenario (vanilla apache conf on rhel7 + vanilla tomcat7) and users can eventually break/slow down a cluster by submitting misconfigured pods.

ACK. We could close connections from the middle (the kube-proxy) but I am
worried about the semantic change that introduces. If someone worked up
that PR we could at least test it. We have that for UDP (because no
connections) so it should mostly be following the same pattern.

On Mon, Jun 1, 2015 at 10:16 AM, Magnus Kulke notifications@github.com
wrote:

@thokin https://github.com/thokin: i'll try that later, but i suspect
everything will be fine. The problems seem to be related to keepalive
connections from proxy to backend. While those can be turned off, it this
is a pretty common scenario (vanilla apache conf on rhel7 + vanilla
tomcat7) and users can eventually break/slow down a cluster by submitting
misconfigured pods.

—
Reply to this email directly or view it on GitHub
#5461 (comment)
.