New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add 10 minute sleep to infinite loops #1

Open
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
2 participants
@bd808

bd808 commented Apr 11, 2016

During investigation of T128838 it was discovered that 30% of all edit
activity on Wikimedia wikis are null edits attributable to a small
number of bot accounts. Null edits are edits that do not change the
existing page content. @atdt investigated further and found that several
SoxBot scripts are using while(true) loops around their business
logic. This means that the scripts will do their work on the wikis and
then immediately start over to do the work again. The bot tasks are
important, but not so important that they should stress the Wikimedia
production servers and the Tool Labs servers by spinning in tight
infinite loops.

This patch introduces a 10 minute sleep to each while(true) loop to
reduce the frequency of operations from "as fast as the hardware and
software will support" to "approximately 144 times per day".

@bd808

This comment has been minimized.

bd808 commented Apr 11, 2016

I didn't try really hard to understand the relative importance of each script modified here. It may well be the case that one or more of the tasks is time sensitive enough that 10 minutes between runs is more than the wikis can abide. I do however hope that @cyberpower678 will consider the relative importance of the on-wiki changes vs the resource costs that are incurred and introduce a one to ten minute rest period between each iteration of a script's business logic. If some scripts can search and edit even less often that would be appreciated as well.

@cyberpower678

This comment has been minimized.

Owner

cyberpower678 commented Apr 12, 2016

I introduced a different which makes the rest period unneeded. By caching the old page content from the previous pass, and comparing it to the current pass's intended page edit, it can then tell whether it needs to edit or not, thus reducing null edits from many to 1 per script. The one null edit is upon script startup. You should now notice a significant drop in edit submissions.

Add 10 minute sleep to infinite loops
Several SoxBot scripts are using `while(true)` loops around their
business logic. This means that the scripts will do their work on the
wikis and then immediately start over to do the work again. The bot
tasks are important, but not so important that they should stress the
Tool Labs servers by spinning in tight infinite loops.

This patch introduces a 10 minute sleep to each `while(true)` loop to
reduce the frequency of operations from "as fast as the hardware and
software will support" to "approximately 144 times per day".
@bd808

This comment has been minimized.

bd808 commented Apr 13, 2016

@cyberpower678 I've rebased my patch on your changes from 84f056b. Your addition of change validation is a nice improvement for the production server side of things. I still think that introducing the sleep(600) breaks in the code will be of benefit for cpu utilization and logging bandwidth on the Tool Labs servers.

@cyberpower678

This comment has been minimized.

Owner

cyberpower678 commented Apr 25, 2016

Sorry. I forgot about this. Cyberbot doesn't run on the tool labs server, but rather on its own project on labs. CPU utilization is at a reasonable rate right now, so I don't think the sleeps are necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment