Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit concurrency on structure_pid calls #49

Closed
jbodah opened this issue Nov 29, 2017 · 2 comments
Closed

Limit concurrency on structure_pid calls #49

jbodah opened this issue Nov 29, 2017 · 2 comments

Comments

@jbodah
Copy link

jbodah commented Nov 29, 2017

We're on a fork of an earlier version of wobserver, but I thought I'd bring this issue up in the core repo as an issue.

We've been seeing consistent timeouts from this parallel_map call: https://github.com/shinyscorpion/wobserver/blob/master/lib/wobserver/util/application.ex#L97. We've also noticed that the BEAM load spikes pretty hard whenever we navigate to the wobserver page for one of our apps that has many processes (~100 Ranch acceptors under a single supervisor). I'm guessing this has to do with all of the core :erlang calls getting process info

Here's a stacktrace from our version:

14:40:38.116 [error] Task #PID<0.1046.0> started from #PID<0.1033.0> terminating
** (stop) exited in: Task.await(%Task{owner: #PID<0.1046.0>, pid: #PID<0.1056.0>, ref: #Reference<0.0.1.97714>}, 5000)
    ** (EXIT) time out
    (elixir) lib/task.ex:416: Task.await/2
    (elixir) lib/enum.ex:1229: Enum."-map/2-lists^map/1-0-"/2
    (wobserver) lib/wobserver/util/application.ex:101: Wobserver.Util.Application.structure_pid/1
    (elixir) lib/task/supervised.ex:85: Task.Supervised.do_apply/2
    (elixir) lib/task/supervised.ex:36: Task.Supervised.reply/5
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Function: #Function<1.105204156/0 in Wobserver.Util.Helper.parallel_map/2>
    Args: []
14:40:38.116 [error] Ranch protocol #PID<0.1024.0> (:cowboy_protocol) of listener TaskRunner.Endpoint.HTTP terminated
** (exit) exited in: Task.await(%Task{owner: #PID<0.1046.0>, pid: #PID<0.1056.0>, ref: #Reference<0.0.1.97714>}, 5000)
    ** (EXIT) time out

When this time out happens the entire wobserver page fails to load which has made it unusable in app without changes.

I'd suggest using a pool to limit concurrency or chunking the processes

@jbodah
Copy link
Author

jbodah commented Nov 30, 2017

Actually on further investigation, this is probably failing because we have a long-running message handler in one of our processes. Will report back when we get more info

@jbodah
Copy link
Author

jbodah commented Nov 30, 2017

Disregard, this was our own dumb fault for having a Supervisor.Spec.supervisor that should've been a worker

@jbodah jbodah closed this as completed Nov 30, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant