Breaking out of the iterator / django integration

[th3o6a1d]

Hi there,

I'm trying to integrate your twitterAPI with django. I would like to call a function that downloads 200 tweets. When I run the following code as a script, I'm able to break out of the loop after downloading the tweets. When I run it as a django view, I cannot get out of the get_iterator loop. Is there a way to break the iterator reliably? Or iterate until an endpoint?

def tweetmap(request):
stream = api.request('statuses/filter', {'locations':'-122,36,-121,37'})
count = 0
for tweet in stream.get_iterator():
try:
cleaned_tweet = (tweet['text'], tweet['coordinates']['coordinates'])
count += 1
print cleaned_tweet
if count > 200:
return HttpResponse("Complete")
except:
pass

[geduldig]

Hi,
I have not written a Django view, so I will have to rely on your knowledge for that. I would like to help you make this work, but it's not clear to my why running your code as a view would not work while it does work as a stand-alone script. The fact that you are able to break out of the loop in the latter case would suggest that it meets your requirement of breaking out of the iterator reliably. It seems like that functionality does work as it should. Could the issue be in the Django view? If you were to replace your loop code in the view with a simple iteration, does that work reliably? For example,

count = 0
while True:
    count+=1
    if count>200: return HttpResponse("Complete")

My other thought is, are there other HTTP requests in your Django code that may conflict with the Twitter requests?

Jonas

[th3o6a1d]

Hey Jonas,

Thanks for your response. I suspect it may have been the latter. After re-tooling it a little, the following seems to work. It's not where I where I want it to be yet (trying to stream geocoded tweets to a map for a public health app http://github.com/th3o6a1d/suicide_watch . My version of django doesn't support streaming yet, and I'm trying to avoid an upgrade), but the issue is resolved as far as I'm concerned. If I encounter it again, I'll refer back to this post.

for tweet in stream:
            cleaned_tweet = (tweet['text'], [tweet['coordinates']['coordinates'][1], tweet['coordinates']['coordinates'][0]])
            print cleaned_tweet
            for i in wordlist:
                if cleaned_tweet[0].find(i)>0:
                    with open('redflags.json','a') as output_file:
                        json.dump(cleaned_tweet,output_file)
                    f = open('redflags.json')
                    print "ALERT ALERT ALERT"
                    return HttpResponse(f.read())
    except:
        print "PASSED"

Jason

[geduldig]

Interesting project!
If the problem is streaming, I wonder if my class TwitterRestPager would help you. It makes a Twitter REST API call appear stream-like. You would use it like this:

pager = TwitterRestPager(api, 'search/tweets', {'q':'suicide', 'geocode':'37.78,-122.39,1mi'});        
for item in pager.get_iterator(wait=5, new_tweets=True):
  print(item['text'] if 'text' in item else item)

The pager downloads consecutive pages of results (default:10 tweets) with a short pause (default:5 seconds) between pages.

[th3o6a1d]

Thanks for the great suggestion. I'm not quite sure which will be better at this point, so I coded the rest pager into a django view. Also remade the repository as I had some issues with hiding my credentials (I'm new to this GitHub thing. So cool). http://github.com/th3o6a1d/suicide_watch. Thanks for watching my repo...stay in touch.