Tasks not completing for the last 8 days

[smoebot]

Hi,

I have a relatively new install of Scumblr 2. It was working fine up until about 8 days ago. It appears that tasks aren't completing.

I have 3 groups of tasks
Group 1 - 2 tasks that each do a google search for some domain names
Group 2 - 1 task that does a Twitter mention search
Group 3 - 2 tasks that each do a cURL query against results found in Group 1

Running the tasks manually shows:
Group 1: running (6bc03767376739cd9736f212)
for eternity

If I disable Group 1, the same behavior is replicated on Group 2.
If i disable Group 2, the same behavior is replicated on Group 3.

Web interface seems to be working fine

[sbehrens]

@ahoernecke This may be fixed by the the async-refactor branch?

[smoebot]

Hi, is there any more insight on this one?

[ahoernecke]

Hi @smoebot,

We have been working on a pretty large minor release in the async-refactor branch. The code in that branch isn't and is actively being worked on, but it's what we're using internally and has a number of enhancement and improvements, including to the task handling. You might want to try that branch and see if you get better results.

Have you tried anything that has resolved the issue? (For example, is the issue temporarily resolved if the sidekiq is restarted)? Can you give me an idea of how long each task/group ran before they stopped working properly?

Thanks

[smoebot]

Hi @ahoernecke.

Tried pulling that branch. No dice.
Nothing is resolved if sidekiq is restarted.

The task groups seemed to be running fine for about a month until they stopped working.

[sbehrens]

Hi @smoebot,

Did you happen to notice any memory issues (rails or sidekiq taking up most system memory) when the task groups stopped executing?