Resource temporarily unavailable error

[natlg]

• Parity Ethereum version: v2.3.0-nightly-7f5e6b3a0-20181128/aarch64-linux-gnu/rustc1.30.1
• Operating system: Linux
• Installation: binary
• Fully synchronized: yes
• Network: Kovan
• Restarted: yes

Got this error on the archive trace node, then whole instance became inaccessible

Dec 03 03:33:24 ip-10-0-0-119.poa.internal parity[2223]: 2018-12-03 03:33:24 UTC   40/50 peers      7 MiB chain    2 MiB db  0 bytes queue   84 KiB sync  RPC:  1 conn,   56 req/s,  512 µs
Dec 03 03:33:32 ip-10-0-0-119.poa.internal parity[2223]: 2018-12-03 03:33:32 UTC Imported #9562666 0x5f67…b820 (6 txs, 0.72 Mgas, 208 ms, 3.98 KiB)
Dec 03 03:33:35 ip-10-0-0-119.poa.internal parity[2223]: 2018-12-03 03:33:34 UTC failed to spawn worker thread; err=Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }
Dec 03 03:33:37 ip-10-0-0-119.poa.internal parity[2223]: 2018-12-03 03:33:34 UTC failed to spawn worker thread; err=Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }
Dec 03 03:34:30 ip-10-0-0-119.poa.internal parity[2223]: 2018-12-03 03:34:29 UTC   40/50 peers      4 MiB chain    2 MiB db 736 bytes queue   84 KiB sync  RPC:  1 conn,    0 req/s, 23922 µs
-- Reboot --

Happened once, after reboot works fine (at least till now).
Don't you know what can be the case and how to prevent this error?

[jam10o-new]

It's hard to say without knowing the specifics of your setup, but it looks like some hardware component of your system was failing or disconnected temporarily. If this was on a cloud provider, it looks like your storage was temporarily unmounted for some reason and I'd take it up with them to make sure it doesn't happen again.

[ordian]

This warning
failed to spawn worker thread; err=Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }
originated from tokio-threadpool. The call to libc::pthread_create failed with error code 11 (EAGAIN), which means The system lacked the necessary resources to create another thread, or the system-imposed limit on the total number of threads in a process PTHREAD_THREADS_MAX would be exceeded. . Maybe you have too many running threads on your system?

[natlg]

There are some instance settings:

Architecture:        aarch64
Byte Order:          Little Endian
CPU(s):              4
On-line CPU(s) list: 0-3
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Model:               3
BogoMIPS:            166.66
NUMA node0 CPU(s):   0-3
Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid

[network]
allow_ips = "all"
warp = false

[rpc]
apis = ["web3","eth","net", "parity",  "traces" ]
processing_threads = 3
server_threads = 6
cors=["all"]

[websockets]
max_connections = 1000
apis = ["web3","eth","net","parity", "pubsub", "traces"]

[snapshots]
disable_periodic = true

[footprint]
tracing = "on"
pruning = "archive"
pruning_history = 1200
fat_db = "on"
cache_size_db = 12000

RAM consuming is slowly increasing after the restart, so almost no free is left in a day after restart. Maybe it could be related, but this error didn't happen again.

This node is used by BlockScout explorer, so it's under quite high load. We use a1.xlarge instanse type.

[ordian]

RAM consuming is slowly increasing after the restart, so almost no free is left in a day after restart. Maybe it could be related

Yes, running out of RAM was likely the reason, 8 GiB is probably not enough for these settings. I would suggest trying a1.2xlarge, which has 16 GiB, although it's costs twice as much, or try to decrease max_connections, cache_size_db, pruning_history.

The system lacked the necessary resources to create another thread

[natlg]

Thank you for answers! I'll watch it for some time and will let you know if the error happens with different setup.