scrape mode should not duplicate tweets

[nichtich]

After running twarc for two days I analyzed the output and found that it downloads the same tweets over and over again. The script should hold a set of known tweet ids and only emit tweets that have not been written before.

[edsu]

That's weird, it was designed to not behave that way, and I've seen it working properly in the past. Can you share your log file?

[ruebot]

Hey! I've been noticing this same exact problem. You can see a whole bunch of duplicate tweets here. I've also gist'd up my twarc.log, and this is how I ran it: ./twarc.py --scrape "#freedaleaskey" Finally, here is the output of a grep query on the file for a particular line.

[edsu]

@ruebot thanks for the replication of the bug ; I haven't had time to look into it yet, but will do shortly.

[ruebot]

Thanks! @edsu++

[edsu]

I just ran a simple test with twarc.py "#code4lib" and did not see duplication. @nichtich were you using --scrape mode by any chance, like @ruebot?

[nichtich]

Yes, my query was `./twarc.py --scrape '@nichtich'. I just confirmed the bug at a new installation.