Exceptions are not helpful

[kmike]

See #1759 (comment).

[kmike]

This is caused by changes here: #1423. With defer.fail() traceback points to Scrapy code, e.g.

Traceback (most recent call last):
  File "/Users/kmike/svn/scrapy/scrapy/commands/crawl.py", line 57, in run
    self.crawler_process.crawl(spname, **opts.spargs)
  File "/Users/kmike/svn/scrapy/scrapy/crawler.py", line 152, in crawl
    return self._crawl(crawler, *args, **kwargs)
  File "/Users/kmike/svn/scrapy/scrapy/crawler.py", line 156, in _crawl
    d = crawler.crawl(*args, **kwargs)
  File "/Users/kmike/envs/dl/lib/python3.5/site-packages/Twisted-15.5.0-py3.5.egg/twisted/internet/defer.py", line 1274, in unwindGenerator
    return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
  File "/Users/kmike/envs/dl/lib/python3.5/site-packages/Twisted-15.5.0-py3.5.egg/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/Users/kmike/envs/dl/lib/python3.5/site-packages/Twisted-15.5.0-py3.5.egg/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/Users/kmike/svn/scrapy/scrapy/crawler.py", line 80, in crawl
    yield exc
builtins.AttributeError: 'NoneType' object has no attribute 'slot'

if the code is changed to an initial @jdemaeyer's workaround

        except Exception as e:
            self.crawling = False
            if self.engine is not None:
                yield self.engine.close()
            raise e

traceback makes more sense:

Traceback (most recent call last):
  File "/Users/kmike/svn/scrapy/scrapy/commands/crawl.py", line 57, in run
    self.crawler_process.crawl(spname, **opts.spargs)
  File "/Users/kmike/svn/scrapy/scrapy/crawler.py", line 152, in crawl
    return self._crawl(crawler, *args, **kwargs)
  File "/Users/kmike/svn/scrapy/scrapy/crawler.py", line 156, in _crawl
    d = crawler.crawl(*args, **kwargs)
  File "/Users/kmike/envs/dl/lib/python3.5/site-packages/Twisted-15.5.0-py3.5.egg/twisted/internet/defer.py", line 1274, in unwindGenerator
    return _inlineCallbacks(None, gen, Deferred())
--- <exception caught here> ---
  File "/Users/kmike/envs/dl/lib/python3.5/site-packages/Twisted-15.5.0-py3.5.egg/twisted/internet/defer.py", line 1128, in _inlineCallbacks
    result = g.send(result)
  File "/Users/kmike/svn/scrapy/scrapy/crawler.py", line 80, in crawl
    raise e
  File "/Users/kmike/svn/scrapy/scrapy/crawler.py", line 70, in crawl
    self.spider = self._create_spider(*args, **kwargs)
  File "/Users/kmike/svn/scrapy/scrapy/crawler.py", line 83, in _create_spider
    return self.spidercls.from_crawler(self, *args, **kwargs)
  File "/Users/kmike/svn/adaptive-crawler/acrawler/acrawler/spiders/adaptive.py", line 41, in from_crawler
    spider.dupefilter = crawler.engine.slot.scheduler.dupefilter
builtins.AttributeError: 'NoneType' object has no attribute 'slot'

[kmike]

There are many ways to trigger this issue; one is to have an exception in start_requests:

class Bad(scrapy.Spider):
    name = 'bad'
    def start_requests(self):
        raise Exception('oops')

Exceptions in spider.from_crawler and spider.__init__ also got worse.

[jdemaeyer]

if the code is changed to an initial @jdemaeyer's workaround

    except Exception as e:
        self.crawling = False
        if self.engine is not None:
            yield self.engine.close()
        raise e

traceback makes more sense:

iirc the problem with that was that it didn't preserve the traceback at all in Python 2?

[Digenis]

http://bugs.python.org/issue7563 (wontfix for py2)
Python2 workaround:

# Example, yielding partial result
try:
    result[key] = stuff()
except:
    import sys
    exc_cls, exc, tb = sys.exc_info()
    yield result
    raise exc_cls, exc, tb
else:
    yield result

[kmike]

@Digenis thanks for the reference, it was useful.