Skip to content

Commit

Permalink
fix scrapy squeue tests after recent changes to queuelib
Browse files Browse the repository at this point in the history
  • Loading branch information
@dangra
dangra committed Sep 9, 2015
1 parent 600f27c commit 9a64d8f
Showing 1 changed file with 4 additions and 4 deletions.
8 changes: 4 additions & 4 deletions tests/test_squeues.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ class MarshalFifoDiskQueueTest(t.FifoDiskQueueTest):
chunksize = 100000

def queue(self):
return MarshalFifoDiskQueue(self.qdir, chunksize=self.chunksize)
return MarshalFifoDiskQueue(self.qpath, chunksize=self.chunksize)

def test_serialize(self):
q = self.queue()
Expand Down Expand Up @@ -54,7 +54,7 @@ class PickleFifoDiskQueueTest(MarshalFifoDiskQueueTest):
chunksize = 100000

def queue(self):
return PickleFifoDiskQueue(self.qdir, chunksize=self.chunksize)
return PickleFifoDiskQueue(self.qpath, chunksize=self.chunksize)

def test_serialize_item(self):
q = self.queue()
Expand Down Expand Up @@ -99,7 +99,7 @@ class ChunkSize4PickleFifoDiskQueueTest(PickleFifoDiskQueueTest):
class MarshalLifoDiskQueueTest(t.LifoDiskQueueTest):

def queue(self):
return MarshalLifoDiskQueue(self.path)
return MarshalLifoDiskQueue(self.qpath)

def test_serialize(self):
q = self.queue()
Expand All @@ -120,7 +120,7 @@ def test_nonserializable_object(self):
class PickleLifoDiskQueueTest(MarshalLifoDiskQueueTest):

def queue(self):
return PickleLifoDiskQueue(self.path)
return PickleLifoDiskQueue(self.qpath)

def test_serialize_item(self):
q = self.queue()
Expand Down

6 comments on commit 9a64d8f

@foresightyj
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I encountered errors with both "FifoDiskQueueandFifoSQLiteQueue`.

image

request_to_dict is not a real serializer because it returns a dict.

 reqd = request_to_dict(request, self.spider)
 self.dqs.push(reqd, -request.priority)

So a dict is pushed into a queue instead of bytes.

@dangra
Copy link
Member Author

@dangra dangra commented on 9a64d8f Sep 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scrapy can't use FifoDiskQueueu or FifoSQLiteQueue directly, it wraps queuelib queues with a serializer, see https://github.com/scrapy/scrapy/blob/master/scrapy/squeues.py#L31-L34

@dangra
Copy link
Member Author

@dangra dangra commented on 9a64d8f Sep 11, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the settingi s https://github.com/scrapy/scrapy/blob/master/scrapy/settings/default_settings.py#L227

@foresightyj
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh. I completely overlooked this file and set SCHEDULER_DISK_QUEUE to FifoSQLiteQueue directly . Thanks.

I manually added the following two lines into squeues.py:

PickleFifoSQLiteQueue = _serializable_queue(queue.FifoSQLiteQueue,
                                        _pickle_serialize, pickle.loads)
PickleLifoSQLiteQueue = _serializable_queue(queue.LifoSQLiteQueue,
                                        _pickle_serialize, pickle.loads)

And used SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoSQLiteQueue' in the settings.py.

@dangra
Copy link
Member Author

@dangra dangra commented on 9a64d8f Sep 11, 2015 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@foresightyj
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is a better idea. Thanks.

Please sign in to comment.