Show pending items in queue counter on dashboard

[mback2k]

Based upon ArchiveTeam/wpull#169

[hannahwhy]

This looks okay to me.

I'd like to hold off merging it until the hooks make it into a new wpull release. That way, we can change requirements.txt accordingly, and the usual pip install --upgrade bit will get the wpull we need.

One thing: for pipelines that don't have this code, we'll see zero for queued and downloaded items. I don't think this is a big problem: it will eventually be solved as all pipelines are updated, and we can adapt our brains in the meantime.

[mback2k]

@yipdw Okay, sounds reasonable.

[mback2k]

I don't understand why you think they are always being reset to 0. They are reset to 0 after their value has been added to the global counter in redis.
See the following lines:

• Increment the global redis counter by the internal value: https://github.com/ArchiveTeam/ArchiveBot/pull/96/files#diff-65851251e9f03eb14665490c14cf88c4R141 and https://github.com/ArchiveTeam/ArchiveBot/pull/96/files#diff-65851251e9f03eb14665490c14cf88c4R151
• Reset the internal counter value: https://github.com/ArchiveTeam/ArchiveBot/pull/96/files#diff-65851251e9f03eb14665490c14cf88c4R142 and https://github.com/ArchiveTeam/ArchiveBot/pull/96/files#diff-65851251e9f03eb14665490c14cf88c4R152

[ivan]

I mean, why are the counters even there in the first place, instead of just

                self.redis.hincrby(ident, 'items_downloaded', count)

[mback2k]

I created those internal counters just like bytes_outstanding for the global bytes_downloaded counter. Please see update_bytes_downloaded at: https://github.com/mback2k/ArchiveBot/blob/topic/queue_counter/pipeline/archivebot/control.py#L127

I guess the idea is to retry adding the value in case something goes wrong during the redis operation.

[ivan]

Ah, okay. @yipdw what's the point of bytes_outstanding?

[hannahwhy]

Yeah, the idea behind self.bytes_outstanding was to maintain a running tally that was reset once we had confirmation that the job record had been updated.

I'm not sure it's necessary, though -- losing a record here or there might not be worth the odd code flow. It may not actually fulfill its intended purpose: part of with conn(self) involves checking scripts, which in turn requires an active connection.

[hannahwhy]

wpull 0.1000 has been released and is being used by ArchiveBot, so I'm merging this.

[hannahwhy]

This seems to work pretty well; however, I've noticed that items_downloaded tends to be substantially less (a few dozen) than the sum of all responses. I'll push the dashboard changes and start up a pipeline so we can all see what's going on.

For the sort of coarse-grained "are we there yet" information we're looking for, this isn't a problem; however, from an accounting perspective, it'd be nice for all quantities to match up 😄