Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Performance on Probot 9 #970
We've been trying to upgrade the GitHub Slack integration from Probot 7.5.3 to Probot 9, so that we can make use of all the awesome new features specifically the ability to cache installation tokens in redis. Pull request is here: integrations/slack#877
Yesterday, we tried to deploy this pull request, but we saw a severe performance degradation so we had to roll back the change. We saw memory usage triple, requests started taking much longer and some timed out.
I'm opening this issue because we're not super sure what the underlying cause is, but we did have some initial thoughts:
cc @scepticulous since you might be looking into this more
cc @gimenete as the author of https://github.com/integrations/slack/blob/master/lib/activity/processor.js
In my opinion the work done in
But I'll keep an eye on this issue to see if there's something we can change to improve things.
The new throttle plugin does more nuance throttling than the previous 1s / request. The Bottleneck settings are not configurable at this point but we could change that: https://github.com/octokit/plugin-throttling.js/blob/b2ac75fde6b862c39710d758adc8cfa306871098/lib/index.js#L14-L38 so you could fine-tune it to your requirements.
You can also use Redis for throttling data persistence instead of memory by passing your own Bottleneck instance: https://github.com/octokit/plugin-throttling.js#clustering
Hope that helps
I'd be curious to know the length of the Bottleneck queue while memory usage is high. I recently heavily refactored Bottleneck's core engine to avoid closures and allocations by reusing prototype functions and that reduced memory usage by 40%.
The throttling plugin is not a simple machine. It has to compose a lot of closures and it's possible that they lead to a high memory usage while the plugin is juggling a lot of requests.
Could you try disabling the throttling plugin to measure the difference that makes?
After, could you please do the same for the retry plugin and then both plugins?
Neither of them is particularly inefficient, but the octokit plugin system is like an onion. Each layer compounds the memory usage of the underlying layers.
@gr2m yep, we ended up upgrading to Probot 9 a few weeks ago after disabling all the throttling we could. There was still an increase in memory usage and GC activity (presumably due to some of the built in throttling we can't disable), but to a level that was acceptable to us