KeyError with CoopConsumer and pgq_lazy_fetch=0

[epeld]

I get a
Traceback (most recent call last):
File "/home/avtech/skytools/python/skytools/scripting.py", line 568, in run_func_safely
return func()
File "/home/avtech/skytools/python/pgq/baseconsumer.py", line 260, in work
self._finish_batch(curs, batch_id, ev_list)
File "/home/avtech/skytools/python/pgq/coopconsumer.py", line 76, in _finish_batch
self._flush_retry(curs, batch_id, list)
File "/home/avtech/skytools/python/pgq/consumer.py", line 76, in _flush_retry
if ev._status == EV_RETRY:
File "/home/avtech/skytools/python/pgq/event.py", line 49, in getattr
return self._event_row[_fldmap[key]]
KeyError: '_status'
when running a CoopConsumer with pgq_lazy_fetch=0.

I believe that the cause of the problem is that pgq.baseconsumer.py:_load_batch_events_old() instantiates events directly using the Event-constructor, but CoopConsumer expects "RetriableEvent"s instead. That is, the Event-instances do not have a _status-field whereas RetriableEvents do.

When pgq_lazy_fetch > 1 the events are instantiated by the _batch_walker_class-field in _load_batch_events() so there is no crash (because CoopConsumer supplies its own BatchWalker which creates RetriableEvents).

Maybe events should always be created by some function _make_event that subclasses can override?

[markokr]

Thanks for report and good analysis. Fixed in GIT with _make_event() in consumer classes.

Alternative would be to have _event_class, but that is not good unless shared somehow with walker.
So I kept things simple ATM.

Please test.

[epeld]

Hi, thanks for the quick fix.

Unfortunately there are still problems:
Traceback (most recent call last):
File "/usr/local/lib64/python2.6/site-packages/skytools/scripting.py", line 568, in run_func_safely
return func()
File "/usr/local/lib64/python2.6/site-packages/pgq/baseconsumer.py", line 257, in work
self._launch_process_batch(db, batch_id, ev_list)
File "/usr/local/lib64/python2.6/site-packages/pgq/baseconsumer.py", line 286, in _launch_process_batch
self.process_batch(db, batch_id, list)
File "/usr/local/lib64/python2.6/site-packages/pgq/baseconsumer.py", line 233, in process_batch
self.process_event(db, ev)
File "./av2te_consumer.py", line 166, in process_event
ev.tag_done()
File "/usr/local/lib64/python2.6/site-packages/pgq/consumer.py", line 23, in tag_done
self._walker.tag_event_done(self)
AttributeError: 'AV2TEConsumer' object has no attribute 'tag_event_done'

I believe the problem is due to this line:
https://github.com/markokr/skytools/blob/master/python/pgq/consumer.py#L64
which attempts to create a RetriableWalkerEvent by passing self as the walker-parameter.
However self here is a Consumer-instance so does not have a tag_event_done method.
I think you meant to pass an instance of RetriableBatchWalker instead of self?

[markokr]

we don't have walker there. The right fix is to use RetryableEvent.

I did some reorg too, to make stuff more obvious next time.

Please test.