FreeBSD CI workers got ENOMEM randomly

[iblislin]

ERROR (unhandled task failure): write: not enough memory (ENOMEM)
Stacktrace:
 [1] try_yieldto at ./event.jl:189 [inlined]
 [2] wait() at ./event.jl:248
 [3] uv_write(::Base.PipeEndpoint, ::Ptr{UInt8}, ::UInt64) at ./stream.jl:811
 [4] unsafe_write(::Base.PipeEndpoint, ::Ptr{UInt8}, ::UInt64) at ./stream.jl:832
 [5] write(::Pipe, ::String) at ./strings/string.jl:71
 [6] write_cookie(::Base.Process) at ./distributed/cluster.jl:1120
 [7] launch(::Base.Distributed.LocalManager, ::Dict{Any,Any}, ::Array{WorkerConfig,1}, ::Condition) at ./distributed/managers.jl:328
 [8] (::Base.Distributed.##39#42{Base.Distributed.LocalManager,Dict{Any,Any}})() at ./event.jl:73

But the available memory observed from top on each work is enough.

Not sure it's bsd only issue or not.

Ref builds:

1. https://julia.iblis.cnmc.tw/#/builders/1/builds/903
2. https://julia.iblis.cnmc.tw/#/builders/1/builds/887
3. https://julia.iblis.cnmc.tw/#/builders/1/builds/891
4. https://julia.iblis.cnmc.tw/#/builders/1/builds/772
   ...

[ararslan]

I believe @fredrikekre said he's seen this happen locally on Ubuntu as well. This is triggered by the parallel tests, right?

[fredrikekre]

I think it was the same, can not give any more info until getting back to work in couple of weeks though, since I have only seen it there.

[iblislin]

I bumped JULIA_TEST_MAXRSS_MB to 1000 now.
hope it's helpful. 🙏

[iblislin]

some build reports don't contain the statistic testsuit

• https://julia.iblis.cnmc.tw/#/builders/1/builds/988
• https://julia.iblis.cnmc.tw/#/builders/1/builds/986
• https://julia.iblis.cnmc.tw/#/builders/1/builds/983

Any unreasonable memoy usage in that testsuit?

[ararslan]

This is odd, I haven't seen this locally on FreeBSD with JULIA_TEST_MAXRSS_MB=600 on a fairly limited memory machine. Seems like setting it to 600 would be more likely to trigger ENOMEM than 1000?

[iamed2]

I have seen this when SIGINTing while doing parallel work with Dispatcher. Maybe processes are being killed?

[iblislin]

dmesg keep showing this:

Aug  8 06:17:17 ionic kernel: kern.ipc.maxpipekva exceeded; see tuning(7)
Aug  8 06:17:19 ionic last message repeated 2 times
Aug  8 06:17:20 ionic kernel: pid 82904 (julia), uid 1001: exited on signal 6 (core dumped)
Aug  8 06:17:21 ionic kernel: kern.ipc.maxpipekva exceeded; see tuning(7)
Aug  8 07:34:07 ionic kernel: kern.ipc.maxpipekva exceeded; see tuning(7)
Aug  8 07:34:10 ionic last message repeated 3 times
Aug  8 08:52:07 ionic kernel: kern.ipc.maxpipekva exceeded; see tuning(7)
Aug  8 08:52:10 ionic last message repeated 3 times
Aug  8 10:10:26 ionic kernel: kern.ipc.maxpipekva exceeded; see tuning(7)
Aug  8 10:10:29 ionic last message repeated 3 times

My local machine also have these messages, but do not raise any ENOMEM.

I have seen this when SIGINTing while doing parallel work with Dispatcher. Maybe processes are being killed?

any suggestion/tools about how to obseving that?

[iblislin]

after bumped MAX_RSS up, the frequence of ENOMEM ⬇️ , but freezing ⬆️

• https://julia.iblis.cnmc.tw/#/builders/1/builds/1005
• https://julia.iblis.cnmc.tw/#/builders/1/builds/1001

this ENOMEM is related to freezing? 😱

[mbauman]

I don't think we're seeing this anymore — is this still relevant with the new buildbot infrastructure?

[ararslan]

We still get freezes occasionally, but not as often as we used to, and I haven't seen the ENOMEM in a long time, probably since we switched to the new buildbots.

[ViralBShah]

Reopen if necessary?

[Keno]

It's back: https://build.julialang.org/#/builders/26/builds/10373

[Keno]

The FreeBSD kernel behavior here seems odd to me. I'll reach out to the kernel devs.

[Keno]

https://lists.freebsd.org/pipermail/freebsd-hackers/2020-March/055714.html

[Keno]

@staticfloat tells me our current kern.ipc.maxpipekva is 512MB on the test machine, which seems like quite a bit, but maybe we can bump it to 1GB if we start seeing this again with some regularity. I also wouldn't be surprised if we leak pipes somewhere, which would also explain the fd leak we have seen on other platforms. (I still think FreeBSD should handle this more gracefully, but we can workaround this independent of any potential improvements to the FreeBSD kernel).

[staticfloat]

I've started graphing kern.ipc.pipekva, so we can check to see if the ENOMEM errors actually correspond to us running out of memory, whether it's a slow leak or a quick exhaustion, etc... If someone sees a bunch of ENOMEM errors, post the error logs and I'll look up the corresponding graphs.