Skip to content

issues Search Results · repo:rmax/scrapy-redis language:Python

Filter by

197 results
 (55 ms)

197 results

inrmax/scrapy-redis (press backspace or delete to remove)

import asyncio class RedisMixin: ... async def start(self): Asynchronous start method compatible with new Scrapy versions. for req in await asyncio.to_thread(self._collect_start_requests): ...
  • lexxai
  • Opened 
    28 days ago
  • #305

Is Valkey support being considered as an alternative to Redis in the future?
  • abebus
  • Opened 
    on May 10
  • #304

Problem: File .../lib/python3.11/site-packages/pydispatch/robustapply.py , line 55, in robustApply return receiver(*arguments, **named) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File .../lib/python3.11/site-packages/scrapy/extensions/logstats.py ...
  • lexxai
  • Opened 
    on Feb 10
  • #303

Description Today, I use web scraping and LLM models to extract data from URLs. If, for some reason, I encounter an error with the LLM, my plan is to retry processing during the next scheduled run (I ...
  • matheus-rossi
  • 5
  • Opened 
    on Nov 18, 2024
  • #302

Description AttributeError: BossSpider object has no attribute settings scrapy v2.11.2 scrapy-redis v0.9.1 python v3.12 Step to Reproduce `from scrapy import signals import scrapy from DrissionPage._pages.chromium_page ...
  • gaye746560359
  • Opened 
    on Sep 29, 2024
  • #301

https://github.com/rmax/scrapy-redis/blob/48a7a8921ae064fe7b4202b130f1054ede9103d6/src/scrapy_redis/scheduler.py#L136 calls from_spider in a dupefilter class. However, the from_spider ONLY implements ...
  • HairlessVillager
  • Opened 
    on Jun 22, 2024
  • #293

Description Let me introduce my python program: First, I get the data I need to crawl from MySQL and then process the data into a URL through image Submit to redis and wait for crawling. If crawling fails, ...
  • Hao1617
  • 7
  • Opened 
    on May 9, 2024
  • #291

Description I ve been using Scrapy for web scraping tasks and frequently pass additional information using response.meta for further processing within the spider. My current project requires integrating ...
  • SteliosGiannatos
  • 1
  • Opened 
    on Mar 19, 2024
  • #290

I encountered a new problem。 Program error interrupt, I am using scrapy==2.11.1, scrapy-redis==0.7.3 image In addition, since the latest package of scrapy-redis that is compatible with the new version ...
  • georgeJzzz
  • 13
  • Opened 
    on Feb 23, 2024
  • #289

I suggest adding a feature to Scrapy Redis for seamless integration with Kafka, enabling users to publish crawled items directly to a Kafka topic. I m willing to implement this feature and welcome feedback ...
  • hamed21q
  • 1
  • Opened 
    on Jan 18, 2024
  • #288
Issue origami icon

Learn how you can use GitHub Issues to plan and track your work.

Save views for sprints, backlogs, teams, or releases. Rank, sort, and filter issues to suit the occasion. The possibilities are endless.Learn more about GitHub Issues
ProTip! 
Press the
/
key to activate the search input again and adjust your query.
Issue origami icon

Learn how you can use GitHub Issues to plan and track your work.

Save views for sprints, backlogs, teams, or releases. Rank, sort, and filter issues to suit the occasion. The possibilities are endless.Learn more about GitHub Issues
ProTip! 
Press the
/
key to activate the search input again and adjust your query.
Issue search results · GitHub