Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
The output folder is empty after I run the codes(no error occurs) #14
Comments
|
Did you send in a command from the client script? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
thenewcomer commentedMar 23, 2017
Hi~
I am trying to use your code to collect data from twitter. But after I have followed all the steps and strat to run ./bootstrap.sh -c config.json without proxies, the output folder is always emputy and I still can not collect any data. I don't know which part is going wrong, hope you can help me. My platform is Macbook and my python version is 2.7, here are the messages from terminal:
➜ tweetf0rm-master ./bootstrap.sh -c config.json
INFO-[2017-03-23 16:56:20,127][bootstrap][start_server][99]: output to /Users/xiangyuanxin/Developer/collectTwitterData/data
INFO-[2017-03-23 16:56:20,128][bootstrap][start_server][100]: archived to /Users/xiangyuanxin/Developer/collectTwitterData/data/archived
INFO-[2017-03-23 16:56:21,037][scheduler][init][46]: number of crawlers: 1
INFO-[2017-03-23 16:56:21,040][twitter_api][init][26]: {'apikeys': {u'oauth_token_secret': u'jT4RL4a11zT4Zyb4icbEc1dp5rr3odrtVoeyNV1****', u'app_secret': u'oMbssGKczQxkXgs34z9Tus6wRebkfK5qBSg7AD0Bv2q5o****', u'oauth_token': u'783543426411294720-XSyiiaxAuY5723pGT3GRSujTAPIthPR', u'app_key': u'mcqIwVZq4xwi6vWw9oSuW1G9y'}, 'client_args': {'timeout': 300}}
INFO-[2017-03-23 16:56:22,474][scheduler][init][63]: number of crawlers: 1 created
INFO-[2017-03-23 16:56:22,475][bootstrap][start_server][108]: starting node_id: b54b5c1ecdff3f6f8cadefe4f28bae7c
INFO-[2017-03-23 17:02:22,524][bootstrap][start_server][122]: [{'alive?': True,
'crawler_id': u'mcqIwVZq4xwi6vWw9oSuW1G9y',
'crawler_queue_key': u'queue:b54b5c1ecdff3f6f8cadefe4f28bae7c:mcqIwVZq4xwi6vWw9oSuW1G9y',
'qsize': 0}]
INFO-[2017-03-23 17:08:22,546][bootstrap][start_server][122]: [{'alive?': True,
'crawler_id': u'mcqIwVZq4xwi6vWw9oSuW1G9y',
'crawler_queue_key': u'queue:b54b5c1ecdff3f6f8cadefe4f28bae7c:mcqIwVZq4xwi6vWw9oSuW1G9y',
'qsize': 0}]
INFO-[2017-03-23 17:14:22,589][bootstrap][start_server][122]: [{'alive?': True,
'crawler_id': u'mcqIwVZq4xwi6vWw9oSuW1G9y',
'crawler_queue_key': u'queue:b54b5c1ecdff3f6f8cadefe4f28bae7c:mcqIwVZq4xwi6vWw9oSuW1G9y',
'qsize': 0}]
INFO-[2017-03-23 17:20:22,634][bootstrap][start_server][122]: [{'alive?': True,
'crawler_id': u'mcqIwVZq4xwi6vWw9oSuW1G9y',
'crawler_queue_key': u'queue:b54b5c1ecdff3f6f8cadefe4f28bae7c:mcqIwVZq4xwi6vWw9oSuW1G9y',
'qsize': 0}]
INFO-[2017-03-23 17:26:22,702][bootstrap][start_server][122]: [{'alive?': True,
'crawler_id': u'mcqIwVZq4xwi6vWw9oSuW1G9y',
'crawler_queue_key': u'queue:b54b5c1ecdff3f6f8cadefe4f28bae7c:mcqIwVZq4xwi6vWw9oSuW1G9y',
'qsize': 0}]
INFO-[2017-03-23 17:26:22,704][scheduler][balancing_load][128]: crawler with max_qsize: mcqIwVZq4xwi6vWw9oSuW1G9y (0)
INFO-[2017-03-23 17:26:22,704][scheduler][balancing_load][129]: crawler with min_qsize: mcqIwVZq4xwi6vWw9oSuW1G9y (0)
INFO-[2017-03-23 17:26:22,704][scheduler][balancing_load][130]: max_qsize - min_qsize > 0.5 * min_qsize ?: False
INFO-[2017-03-23 17:32:22,863][bootstrap][start_server][122]: [{'alive?': True,
'crawler_id': u'mcqIwVZq4xwi6vWw9oSuW1G9y',
'crawler_queue_key': u'queue:b54b5c1ecdff3f6f8cadefe4f28bae7c:mcqIwVZq4xwi6vWw9oSuW1G9y',
'qsize': 0}]
INFO-[2017-03-23 17:38:23,006][bootstrap][start_server][122]: [{'alive?': True,
'crawler_id': u'mcqIwVZq4xwi6vWw9oSuW1G9y',
'crawler_queue_key': u'queue:b54b5c1ecdff3f6f8cadefe4f28bae7c:mcqIwVZq4xwi6vWw9oSuW1G9y',
'qsize': 0}]
INFO-[2017-03-23 17:44:23,161][bootstrap][start_server][122]: [{'alive?': True,
'crawler_id': u'mcqIwVZq4xwi6vWw9oSuW1G9y',
'crawler_queue_key': u'queue:b54b5c1ecdff3f6f8cadefe4f28bae7c:mcqIwVZq4xwi6vWw9oSuW1G9y',
'qsize': 0}]