Common Pitfalls

Describing some common pitfalls and resolutions to ensure smooth ServiceQ usage -

Socket: too many open files

After passing concurrent requests to ServiceQ, if you see an error like 'socket: too many open files' or DOWNSTREAM_TIMED_OUT (in serviceq_error.log) or 503 Request Buffered response, then it is possible that your system's open file descriptor limit has been reached. You can check this limit by running ulimit -Sn. The number shows soft limit for open files allowed by the system and can be changed by running below (for linux) -

sudo ulimit -n 16384

This will change the setting for current session.

If you want to persist it across reboots, edit /etc/security/limits.conf, uncomment below line and update the number -

* soft nofile 16384

You might need a reboot after making the changes.

For mac, please refer https://wilsonmar.github.io/maximum-limits/

General latency

While sending request to downstream nodes, ServiceQ considers OUTGOING_REQUEST_TIMEOUT (present in sq.properties) value as timeout. Default is set to 5s. If you are seeing latency or repeated timeouts, it is possible that this value is not set according to your cluster setup and it needs to be modified in line with average response time of the downstream nodes. Having lower value is preferable to allow ServiceQ to quickly retry the request on another node and improving overall turnaround time.

As an example, if average response time for the cluster is 1s, timeout can be configured as 1s or max 2s for best performance.