Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing (hopefully) faster urlchecker updated with multiprocessing #768

Merged
merged 2 commits into from
Mar 28, 2022

Conversation

vsoch
Copy link
Member

@vsoch vsoch commented Mar 26, 2022

Changes here! urlstechie/urlchecker-python#63 and the associated action is a development / temporary branch

Signed-off-by: vsoch vsoch@users.noreply.github.com

Signed-off-by: vsoch <vsoch@users.noreply.github.com>
@vsoch
Copy link
Member Author

vsoch commented Mar 26, 2022

Major improvement! 4-5 minutes down to... 44 seconds!! ping @SuperKogito! @mrmundt!

So for next steps, we need to review / merge urlstechie/urlchecker-python#63 and then I can do an official release for the action, and update the branch test/multiprocessing to the release here.

Signed-off-by: vsoch <vsoch@users.noreply.github.com>
@vsoch
Copy link
Member Author

vsoch commented Mar 27, 2022

@USRSE/usrse-admin this is ready for review! The tributors and urlchecker actions are both now releases: #768

  • urlchecker uses multiprocessing for the speedup mentioned above
  • tributors is fixed so it won't open two PRs given a new contributor

@mrmundt
Copy link
Contributor

mrmundt commented Mar 28, 2022

@vsoch - Woot woot!!

Copy link
Member

@cosden cosden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an awesome improvement! I would always get distracted waiting for the URL checker to finish and then forget to come back and check.

@vsoch
Copy link
Member Author

vsoch commented Mar 28, 2022

Awesome!! And @SuperKogito and I have other ideas for improvement - we have a plan to make the changes, test and release, and then I can watch for a few weeks here and (hopefully) slowly get better. As the site has gotten bigger I've personally found the time urlchecker takes to be unacceptable (hence the recent work here!)

@vsoch
Copy link
Member Author

vsoch commented Mar 28, 2022

This is also adding an optimization to tributors to not need to merge two PRs (both sets of changes come with the first).

@vsoch vsoch merged commit c43486c into main Mar 28, 2022
@vsoch vsoch deleted the test/faster-urlchecker branch March 28, 2022 18:50
nicole-brewer pushed a commit to nicole-brewer/usrse.github.io that referenced this pull request Apr 7, 2022
…SRSE#768)

* testing (hopefully) faster urlchecker updated with multiprocessing
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants