defer.inlineCallbacks in spider ?

[poupryc]

Hi,

I'm trying to use this feature of Twisted to run the following code:

# -*- coding: utf-8 -*-
import scrapy
import re
from twisted.internet.defer import inlineCallbacks

from sherlock import utils, items, regex


class PagesSpider(scrapy.spiders.SitemapSpider):
    name = 'pages'
    allowed_domains = ['thing.com']
    sitemap_follow = [r'sitemap_page']

    def __init__(self, site=None, *args, **kwargs):
        super(PagesSpider, self).__init__(*args, **kwargs)

    @inlineCallbacks
    def parse(self, response):
        # things
        response = yield scrapy.Request("https://google.com")
        # Twisted execute the request and resume the generator here with the response
        print(response.text)

Is this possible ? I'm trying to use this to dispense with the inline-request module.

Thanks

[poupryc]

After reading the code, I think I've found the solution. Perhaps documenting it would be beneficial?

# -*- coding: utf-8 -*-
import scrapy
import re
from twisted.internet.defer import inlineCallbacks

from sherlock import utils, items, regex


class PagesSpider(scrapy.spiders.SitemapSpider):
    name = 'pages'
    allowed_domains = ['thing.com']
    sitemap_follow = [r'sitemap_page']

    def __init__(self, site=None, *args, **kwargs):
        super(PagesSpider, self).__init__(*args, **kwargs)

    @inlineCallbacks
    def parse(self, response):
        # things
        request = scrapy.Request("https://google.com")
        response = yield self.crawler.engine.download(request, self) 
        # Twisted execute the request and resume the generator here with the response
        print(response.text)

It's a little verbose, but it works. Correct me if I'm wrong

[elacuesta]

Relevant: #542 (comment). Not sure if we want to document these ExecutionEngine methods though. Any thoughts @dangra?

If, on the other hand, this issue is motivated by the above one (#3500), the "Scrapy way" of achieving such result would be Passing additional data to callback functions

[wRAR]

The current way to do the same is to declare the callback async def, which is already documented. I don't want us to describe any inlineCallbacks uses where the alternatives exists, so I'm closing this.