Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Urlparse benchmarker #26

Merged
merged 5 commits into from May 22, 2018

Conversation

Projects
None yet
3 participants
@nctl144
Copy link
Member

commented May 21, 2018

Hey guys, I made a benchmarker for functions in w3lib that heavily use urlparse library. I got the following result when using this benchmarker:
Python2:

Total number of items extracted = 32799
Time taken = 1.52635908127
Rate of link extraction : 21488.3905121 items/second

Python3:

Total number of items extracted = 32799
Time taken = 2.80625425944163
Rate of link extraction : 11687.821903395927 items/second

Which is as I expected. If you have any suggestion, please let me know :)

@lopuhin
Copy link
Member

left a comment

Looks good to me! The only concern is that it's not clear which of these functions takes more time (it could be that one of them takes the majority of the time), but this is something that can be seen via profiling, and improved later if needed.

@Parth-Vader

This comment has been minimized.

Copy link
Member

commented May 21, 2018

I agree with @lopuhin. Can you try to run vmprof to check what exactly is the distribution of time taken by the functions?

@nctl144

This comment has been minimized.

Copy link
Member Author

commented May 21, 2018

Hey guys, I just added the result for each tested function :) It should be clearer now 😃

@lopuhin lopuhin merged commit a3ce337 into scrapy:master May 22, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@lopuhin

This comment has been minimized.

Copy link
Member

commented May 22, 2018

Looks good, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.