…asyncio` (#390) ### Description - Apify (asyncio) and Scrapy (Twisted) now run on a single event loop. - `nest-asyncio` has been completely removed. - It seems that this change also improved the performance. - The `ApifyScheduler`, which is synchronous, now executes asyncio coroutines (communication with RQ) in a separate thread with its own asyncio event loop. - Logging setup has to be adjusted and I moved to a dedicated file in the SDK. - The try-import functionality for optional dependecies from Crawlee was added to `scrapy` subpackage. - A new integration test for Scrapy Actor has been added. ### Issues - Closes: #148 - Closes: #176 - Closes: #392 - Relates: apify/actor-templates#303 - This issue will be closed once the corresponding PR in `actor-templates` is merged. ### Tests - A new integration test for Scrapy Actor has been added. - And of course, it was tested manually using the Actor from guides/templates. ### Next steps - Update Scrapy Actor template in `actor-templates`. - Update [Actor Scrapy Books Example](https://github.com/apify/actor-scrapy-books-example). - Add HTTP cache storage for KVS, @honzajavorek will provide his implementation. ### Follow-up issues - There are still a few issues to be resolved. - #391 - #395

Add integration test for Scrapy integration #148

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions