[MRG+1] Show elapsed time in statscollector #3638
Conversation
Codecov Report
@@ Coverage Diff @@
## master #3638 +/- ##
=========================================
+ Coverage 84.52% 84.73% +0.2%
=========================================
Files 167 168 +1
Lines 9410 9464 +54
Branches 1397 1407 +10
=========================================
+ Hits 7954 8019 +65
+ Misses 1199 1188 -11
Partials 257 257
|
|
@lopuhin , can you look into this? |
There was a problem hiding this comment.
Hey @maramsumanth adding elapsed_time_seconds looks like a useful addition to me, would be interesting to hear what others think.
Added some comments regarding the implementation. Also I think we need to have tests for the new feature, and update the docs.
|
Hey @kmike @whalebot-helmsman , what do you think of adding this feature? |
scrapy/extensions/corestats.py
Outdated
| self.stats.set_value('finish_time', datetime.datetime.utcnow(), spider=spider) | ||
| finish_time = datetime.datetime.utcnow() | ||
| elapsed_time = finish_time - self.stats.get_value('start_time') | ||
| elapsed_time_seconds = elapsed_time.seconds + elapsed_time.microseconds/1000000 |
There was a problem hiding this comment.
In Python 2 elapsed_time.microseconds/1000000 is always 0
There was a problem hiding this comment.
I think this is failing for python 2.7, what should I modify then @kmike ?
|
Hey @maramsumanth! I think having It'd be good to add a test for this feature (e.g. modify test_closespider_timeout - which is incorrect, by the way, as it adds seconds and microseconds) |
|
@kmike, can you please review this. Should I change close_on , since it is failing for version 2.7 |
Co-Authored-By: maramsumanth <maram.sumanth@gmail.com>
|
Thanks @Gallaecio :) |
Gallaecio
changed the title
|
Thanks @maramsumanth and @Gallaecio! |
Showing elapsed time in seconds in stats, also changing from utc to local time for better user experience.
(In my opinion time&date should be displayed based on the local time)