Version 1.1 of Twitter API, migration

[QuotesUK]

Apologies if I am asking this in the wrong place but I am looking for a helping hand in upgrading a couple of python scripts to use the new Twitter API v1.1. The scripts that I have are working with v1 but they were developed by someone else that is no longer able to work on them because they are unwell.

In a nutshell what the scripts are designed to do is connect to https://stream.twitter.com/1/statuses/filter.json and check for tweets sent to specific @usernames, retrieve the tweet data and insert them into a MySQL database. Afterwards another script does some post-processing to move desired data into other tables and then clean up. There are a couple of other non-Twitter scripts that do some general MySQL housekeeping for stats etc.

I am sure this is quite trivial for some people but I'm stumbling around in the dark because I can't find a working example of doing something similar that uses v1.1. What I need to do is go through the scripts and replace the old JSON syntax with the new one.

For example v1...
messagelog = "tweet_id=%s:screen_name=%s:time_created=%s in twitter stream (tweet.py)" % (content['id_str'],content['user']['screen_name'],msgtime)

The scripts are only needed to fetch public tweets. I don't have to worry about any user logins or authentication issues. The original author has created a working method that ensures the scripts don't make too many API connections. Everything works fine and dandy in API v1 but needs minor tweaks for v1.1.

Please point me towards somewhere else to ask for help if this is the wrong place. I've already tried dev-twitter. Any help you can offer would be appreciated. Thank you.

regards James (QuotesUK)

[QuotesUK]

If anyone is using the branch api_v1.1 can they tell me if upgrading the libraries caused any problems.

If you can help me with the syntax changes I will be very grateful.

[bear]

That branch is for the person who offered to help work on it - I have not
changed it in anyway just yet for v1.1 api.

It's very much a work in progress with nothing committed yet.

On Sun, Feb 17, 2013 at 11:37 AM, QuotesUK notifications@github.com wrote:

If anyone is using the branch api_v1.1 can they tell me if upgrading the
libraries caused any problems.

If you can help me with the syntax changes I will be very grateful.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-13688597.

Bear

bear@xmpp.org (email)
bear42@gmail.com (xmpp, email)
bear@code-bear.com (xmpp, email)
http://code-bear.com/bearlog (weblog)

PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29

[QuotesUK]

Thank you bear. Are you able to tell me whether v0.8.7 of twitter.py has been tweaked to work with the new JSON syntax of API v1.1... should I be upgrading twitter.py at this stage or should I wait for a while?

[bear]

I have not yet updated it to v1.1 of the API - I was giving one contributor
some time as he was working on that. But with the deadline happening this
month I may have to make the change myself and then tweak/enhance it later.

It will probably be a week or so before I have anything even for testing.

On Tue, Mar 5, 2013 at 11:54 AM, QuotesUK notifications@github.com wrote:

Thank you bear. Are you able to tell me whether v0.8.7 of twitter.py has
been tweaked to work with the new JSON syntax of API v1.1... should I be
upgrading twitter.py at this stage or should I wait for a while?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-14450939
.

Bear

bear@xmpp.org (email)
bear42@gmail.com (xmpp, email)
bear@code-bear.com (xmpp, email)
http://code-bear.com/bearlog (weblog)

PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29

[QuotesUK]

Thank you for the quick reply... I will keep an eye open for the update. And another thank you for all the work you do to keep twitter.py running for us. It is very much appreciated.

[bear]

your welcome!

I'm glad that the tech/problem itch DeWitte and myself needed to scratch oh
so many years ago is still considered useful

On Tue, Mar 5, 2013 at 12:10 PM, QuotesUK notifications@github.com wrote:

Thank you for the quick reply... I will keep an eye open for the update.
And another thank you for all the work you do to keep twitter.py running
for us. It is very much appreciated.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-14451971
.

Bear

bear@xmpp.org (email)
bear42@gmail.com (xmpp, email)
bear@code-bear.com (xmpp, email)
http://code-bear.com/bearlog (weblog)

PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29

[domenkozar]

It seems twitter dropped support for 1.0 api today.

[bear]

they are "testing" the dropping of support by having planned outages - they
are trying to poke/prod users into dropping apps and moving over to their
official stuff IMO

I'm tracking the tests and working on the v1.1 api code for python-twitter

On Tue, Mar 5, 2013 at 12:39 PM, Domen Kožar notifications@github.comwrote:

It seems twitter dropped support for 1.0 api today.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-14453617
.

Bear

bear@xmpp.org (email)
bear42@gmail.com (xmpp, email)
bear@code-bear.com (xmpp, email)
http://code-bear.com/bearlog (weblog)

PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29

[sebastianw]

So as time runs out, how should we do this? Do you need help?

[bear]

oh yea, do I !

my thought is that we need to move to a more modern requests library to pick up a lot of other features, but for sure we need the first step of going thru the current code and finding out what the v1.1 API method would be.

that way we can find out if we need new code or not or what the heck is required.

there is a v1.1 branch, please do just start working on that to start making any parts of it work.

i'll add you as a contributor now to make that part easier

[QuotesUK]

I'm a newcomer to Python. I would be very grateful if someone could help me understand how to adapt to the new structure of JSON responses. I understand this is more of an implementation issue rather than API development but I'm stuck because of a lack of documentation and examples, and don't know whether the new syntax requires me to:
1/ make script changes
2/ just update twitter.py that is being worked on for api 1.1
3/ or both

What I would like to see is a worked example that shows me how to work with https://stream.twitter.com/1.1/statuses/filter.json and grab selected fields to be inserted into a mysql database. I already have this working in api v1.

[bear]

this is why I added you as a contributor :)

in the v1.1 branch, make a copy of twitter.py and then start hacking on one of the methods and add as many print's debug stuff or whatever

then we can all see the code example and start commenting on the checking or just making changes

but yea, the core issue will be translating the result to the internal structures that we already have. we need to maintain the same class structure we have, so most will be changing the json-to-dict methods

[sebastianw]

I updated the GetSearch() function. I already committed it to avoid duplicate work. I would suggest we commit every function as a commit so that we keep up-to-date with each other as complete as possible. What do you think?