Adding a fallback option for failed actions #365
Conversation
This reverts commit e109920.
There was a problem hiding this comment.
@further-reading added a few comments, suggestions and questions :-)
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## master #365 +/- ##
==========================================
+ Coverage 76.00% 76.41% +0.41%
==========================================
Files 76 76
Lines 3188 3193 +5
Branches 376 378 +2
==========================================
+ Hits 2423 2440 +17
+ Misses 695 683 -12
Partials 70 70
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report at Codecov. |
There was a problem hiding this comment.
If I create a monitor suite like the following and don't provide any setting needed by SendSESEmail, when I run the spider, I get an spidermon.exceptions.NotConfigured and the spider is not executed:
from spidermon.contrib.actions.email.ses import SendSESEmail
from spidermon.contrib.scrapy.monitors import ItemCountMonitor
class SpiderCloseMonitorSuite(MonitorSuite):
monitors = [ItemCountMonitor]
monitors_failed_actions = [SendSESEmail]
2022-12-15 16:18:12 [scrapy.utils.log] INFO: Scrapy 2.6.2 started (bot: tutorial)
2022-12-15 16:18:12 [scrapy.utils.log] INFO: Versions: lxml 4.9.1.0, libxml2 2.9.14, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 22.4.0, Python 3.9.13 (main, Jun 7 2022, 17:52:43) - [GCC 9.4.0], pyOpenSSL 22.0.0 (OpenSSL 3.0.5 5 Jul 2022), cryptography 37.0.4, Platform Linux-5.15.0-56-generic-x86_64-with-glibc2.35
2022-12-15 16:18:12 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'tutorial',
'NEWSPIDER_MODULE': 'tutorial.spiders',
'ROBOTSTXT_OBEY': True,
'SPIDER_MODULES': ['tutorial.spiders'],
'STATS_CLASS': 'spidermon.contrib.stats.statscollectors.local_storage.LocalStorageStatsHistoryCollector'}
2022-12-15 16:18:12 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.epollreactor.EPollReactor
2022-12-15 16:18:12 [scrapy.extensions.telnet] INFO: Telnet Password: 67bc8af971bd713f
Traceback (most recent call last):
... removed traceback for brevity
File "/home/renne/projects/rennerocha/spidermon/spidermon/contrib/actions/email/__init__.py", line 58, in __init__
raise NotConfigured(
spidermon.exceptions.NotConfigured: You must provide a value for SPIDERMON_EMAIL_TO setting.
However, if SendSESEmail is a fallback action, the spider is executed even if we don't provide any setting for SendSESEmail.
from spidermon.contrib.actions.email.ses import SendSESEmail
from spidermon.contrib.scrapy.monitors import ItemCountMonitor
class MyCustomAction(Action):
fallback = SendSESEmail
def run_action(self):
raise ValueError("This action failed!")
class SpiderCloseMonitorSuite(MonitorSuite):
monitors = [ItemCountMonitor]
monitors_failed_actions = [MyCustomAction]
We should keep the action behavior consistent, so we should raise NotConfigured for fallback actions in the beginning of the execution too, not allowing the spider to be executed if it is not configured properly.
|
@rennerocha Great catch! I've updated the code to initialise the fallback in the Action's init, so this should raise errors like that before the spider runs. |
|
Hello @further-reading , @rennerocha Is there anything else pending on this ? |
Not from my end - the last changes should incorporate Renne's feedback. |
VMRuiz
dismissed
rennerocha’s stale review
Requested change seems to be implemented correctly
Adds the ability to run a different action when an unhandled exception occurs during an action.
Fixes #355