bianjiang/tweetf0rm

replace import futures with import concurrent.futures

Allow redis connection without password …

- Fixed formatting

allow crawling without proxy

refactor, add search_for function, need more work to create crawler d… …

…ynamically per request

stream apis

Added behavior for crawling specific tweet IDs both manually and in …

batch mode.

Fixed two bugs. In one, the node queue size used an incorrect variable …

name. In the other, the command to redistribute to other nodes wasn't
being populated.

add support to clear a node's queues to get you a fresh start

load balancing rules update

shouldn't clear out the crawler queue after restart...

bug in recreating failed crawler

continue migrating local multiprocessing.Queue to redis queue

rewrite local crawler queue using redis; using lifo queue so that con… …

…trol commands can be grab asap;

handle dead crawler gracefully: cleanup residue resources; and schedu… …

…le it to restart after 30 mins

clean up leftover, if a crawler gets restarted

close redis connection when its done; otherwise, it causes error

add in function to get the full user object from a list of user ids; … …

…and add in GET_USERS_FROM_IDS command; see tweetf0rm/client.py;

killall.sh script

use app_key as id, and add-in support for restart the entire crawler

save empty file as well so we can track progress

adding auto-gzip to archived folder support

timestamp on logging

more debug code to diagnosis

show full stack when error occurs, so we can catch what's wrong..

debug message to find out bad status error

remove noses

add hash value for cmd

cmd_handlers specific for control adding new cmds based on the depth … …

…of the network you want to crawl

get rid of duplicated handlers

duplicates

user_id is not necessary always be numbers in the code

fix depth number problem

fix depth number problem

handle non-numbers

handle non-numbers