Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

autopurge: add some logging and lower the parallel purges #7

Merged
merged 1 commit into from Mar 14, 2024

Conversation

david-caro
Copy link

Also my editor autoformatted the code.

Also my editor autoformatted the code.

Signed-off-by: David Caro <me@dcaro.es>
@david-caro
Copy link
Author

Example of run:

==> autopurge-hourly.err <==
Skipped '/workspace/user-config.py': owned by someone else.
2024-03-14 14:41:41,386 INFO     Starting hourly purge...
2024-03-14 14:41:45,595 INFO     Starting purge: Processing 1984 members for category К:Википедия:Страницы с ежечасно очищаемым кэшем
Sleeping for 5.5 seconds, 2024-03-14 14:41:45
2024-03-14 14:41:51,446 INFO         purged from 0 to 50
Sleeping for 9.5 seconds, 2024-03-14 14:41:51
2024-03-14 14:42:01,393 INFO         purged from 50 to 100
Sleeping for 9.7 seconds, 2024-03-14 14:42:01
2024-03-14 14:42:11,403 INFO         purged from 100 to 150
Sleeping for 9.7 seconds, 2024-03-14 14:42:11
2024-03-14 14:42:21,430 INFO         purged from 150 to 200
Sleeping for 9.7 seconds, 2024-03-14 14:42:21
2024-03-14 14:42:31,395 INFO         purged from 200 to 250
Sleeping for 9.7 seconds, 2024-03-14 14:42:31
2024-03-14 14:42:41,436 INFO         purged from 250 to 300
Sleeping for 9.7 seconds, 2024-03-14 14:42:41
2024-03-14 14:42:51,413 INFO         purged from 300 to 350
Sleeping for 9.7 seconds, 2024-03-14 14:42:51
2024-03-14 14:43:01,424 INFO         purged from 350 to 400
Sleeping for 9.7 seconds, 2024-03-14 14:43:01
2024-03-14 14:43:11,419 INFO         purged from 400 to 450
Sleeping for 9.7 seconds, 2024-03-14 14:43:11
2024-03-14 14:43:21,425 INFO         purged from 450 to 500
Sleeping for 9.7 seconds, 2024-03-14 14:43:21
2024-03-14 14:43:31,619 INFO         purged from 500 to 550
Sleeping for 9.5 seconds, 2024-03-14 14:43:31
2024-03-14 14:43:41,667 INFO         purged from 550 to 600
Sleeping for 9.4 seconds, 2024-03-14 14:43:41
2024-03-14 14:43:52,400 INFO         purged from 600 to 650
Sleeping for 8.7 seconds, 2024-03-14 14:43:52
2024-03-14 14:44:08,300 INFO         purged from 650 to 700
2024-03-14 14:44:19,944 INFO         purged from 700 to 750
2024-03-14 14:44:27,351 INFO         purged from 750 to 800
Sleeping for 3.8 seconds, 2024-03-14 14:44:27
2024-03-14 14:44:38,654 INFO         purged from 800 to 850
2024-03-14 14:44:47,787 INFO         purged from 850 to 900
Sleeping for 3.3 seconds, 2024-03-14 14:44:47
2024-03-14 14:44:58,198 INFO         purged from 900 to 950
2024-03-14 14:45:06,855 INFO         purged from 950 to 1000
Sleeping for 4.3 seconds, 2024-03-14 14:45:06
2024-03-14 14:45:16,147 INFO         purged from 1000 to 1050
Sleeping for 5.0 seconds, 2024-03-14 14:45:16
2024-03-14 14:45:29,565 INFO         purged from 1050 to 1100
2024-03-14 14:45:37,046 INFO         purged from 1100 to 1150
Sleeping for 4.1 seconds, 2024-03-14 14:45:37
2024-03-14 14:45:50,702 INFO         purged from 1150 to 1200
2024-03-14 14:45:57,665 INFO         purged from 1200 to 1250
Sleeping for 3.5 seconds, 2024-03-14 14:45:57
2024-03-14 14:46:10,185 INFO         purged from 1250 to 1300
2024-03-14 14:46:16,272 INFO         purged from 1300 to 1350
Sleeping for 4.8 seconds, 2024-03-14 14:46:16
2024-03-14 14:46:26,017 INFO         purged from 1350 to 1400
Sleeping for 5.1 seconds, 2024-03-14 14:46:26
2024-03-14 14:46:37,922 INFO         purged from 1400 to 1450
Sleeping for 3.2 seconds, 2024-03-14 14:46:37
2024-03-14 14:46:46,054 INFO         purged from 1450 to 1500
Sleeping for 5.1 seconds, 2024-03-14 14:46:46
2024-03-14 14:46:56,418 INFO         purged from 1500 to 1550
Sleeping for 4.7 seconds, 2024-03-14 14:46:56
2024-03-14 14:47:05,789 INFO         purged from 1550 to 1600
Sleeping for 5.3 seconds, 2024-03-14 14:47:05
2024-03-14 14:47:16,650 INFO         purged from 1600 to 1650
Sleeping for 4.5 seconds, 2024-03-14 14:47:16
2024-03-14 14:47:27,181 INFO         purged from 1650 to 1700
Sleeping for 3.9 seconds, 2024-03-14 14:47:27
2024-03-14 14:47:38,592 INFO         purged from 1700 to 1750
2024-03-14 14:47:49,445 INFO         purged from 1750 to 1800
2024-03-14 14:47:59,386 INFO         purged from 1800 to 1850
2024-03-14 14:48:12,402 INFO         purged from 1850 to 1900
2024-03-14 14:48:18,435 INFO         purged from 1900 to 1950
Sleeping for 4.0 seconds, 2024-03-14 14:48:18
2024-03-14 14:48:28,720 INFO         purged from 1950 to 2000
2024-03-14 14:48:28,721 INFO     Purged 1984 members
2024-03-14 14:48:28,725 INFO     Finished

I think that the high parallel rate was what was making it fail, making it go a bit slower seems to make it run without issues.

@Saisengen Saisengen merged commit 449e667 into wikisaurus:master Mar 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants