URL required? why not use start_urls?

[tercerapersona]

Is it possible to use start_urls from the spider instead requiring URL?

[chekunkov]

Hi @john-bonachon

ScrapyRT is a HTTP server that allows to reuse spider code to crawl items from a specified URL. You give it spider name and URL - it gives you back whatever spider produced for this URL. It's useful when you want provide some sort of HTTP API to e.g. your customers and don't want to duplicate your spider code.

ScrapyRT supports running requests from start_urls list to properly initialize crawl (e.g. login) - for that one needs to override scrapyrt.core.CrawlManager to provide a condition which should be satisfied to enable requests from start_urls:

from scrapyrt.core import CrawlManager

class CustomCrawlManager(CrawlManager):

    def __init__(self, *args, **kwargs):
        super(CustomCrawlManager, self).__init__(*args, **kwargs)
        # enable start requests for spiders which names end with '_blah'
        if self.spider_name.endswith('_blah'):
            self.start_requests = True

Then one needs to set new class in CRAWL_MANAGER setting (for more details and options read Configuration):

$ scrapyrt -s yourmodule.CustomCrawlManager

After these easy modifications spider will crawl URLs from start_urls list first and then will crawl URL specified in HTTP request.

However if you want spider to crawl start_urls and always return the same set of collected items in the response - ScrapyRT is probably not the right tool to do that. But still you can do that if you really want - ScrapyRT is easy to override and tweak.

[pawelmhm]

@chekunkov since this issue pops up second time (other issue was #6 ) maybe we should consider making url optional?

[chekunkov]

Let's wait for @john-bonachon response - he may provide some useful details about his use case. I'm keen not to make such changes without good reason and strong demand from users. 1 questions/year is not a strong demand.

[pawelmhm]

currently url is required because we create scrapy.http.Request here and url is required argument for Request. We allow users to override request creation with modify_realtime_request, but we dont have spider method create_realtime_request that would be responsible for generating Request class. I think if there is demand from users we could easily add method like this, it would accept kwargs which would contain parameters passed to api and it would have to return valid scrapy.http.Request. If returned object is not valid Request we would throw 400 response to client. This solution could be more flexible than current one. User could define create_realtime_request on spider class without creating custom crawl manager.

EDIT:

re

1 questions/year is not a strong demand.

2 questions since project beginning still make it most commonly requested feature ever 😉

[tercerapersona]

In my case, I just need to retrieve items from my spider. I was thinking about making an API to do so, but found your great lib.
I'm using the same URL I use on the spider's start_urls, so I should maintain both if it breaks. That's why I wonder why this was required, although I understand the purpose of the library is to crawl different urls using the same spider.

[pawelmhm]

ok so we can solve this in following way: add start_requests_enabled parameter to api, and if this parameter is True, url argument will not be required. But if there is no URL and no start_requests argument api will still throw 400 Bad Request (because by default we cannot execute start requests for all spiders).

Will implement this and create PR.

[tercerapersona]

hey,great! that seems to be a nice way to do it without changing the original purpose of scrapyrt? thanks

(another naive example) Using scrapyrt as an API for a mobile app means that I should hardcode the url that's been scrapped in the crawl.json endpoint. If that url changes, I should update the mobile app build, whereas reading from start_urls, I could just update that on the server where the spider lives..

[jonicong]

Hi, any news for this feature? I am also waiting for optional url feature :)

[pawelmhm]

@jonicong we've cleared up project docs, released to pypi and now are good to go with adding features and improving project, so stay tuned

[jonicong]

@pawelmhm ok. wait for it, thanks!

[somada141]

+1 on the "why force the URL" thingo. All in all while I love that you guys made ScrapyRT (managing spiders with the abyssal Twisted reactor is a pain) but I feel like you've precluded the users from using half the smartness in Scrapy since we can no longer use the spider instantiation parameters.

E.g., through ScrapyRT all the URL composition logic is now pushed to whoever calls the API, who now has to know how the URLs for a given site are put together, something that could easily be abstracted over within the spider

[pennylliang]

Hi @pawelmhm , is this feature ready? Thanks.

[pawelmhm]

merged now #38 we'll make new release to pip with this change see updated docs for details on how to use this https://github.com/scrapinghub/scrapyrt/blob/master/docs/source/api.rst