Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Version 1.1 of Twitter API, migration #31

Closed
QuotesUK opened this issue Feb 6, 2013 · 38 comments
Closed

Version 1.1 of Twitter API, migration #31

QuotesUK opened this issue Feb 6, 2013 · 38 comments
Milestone

Comments

@QuotesUK
Copy link

QuotesUK commented Feb 6, 2013

Apologies if I am asking this in the wrong place but I am looking for a helping hand in upgrading a couple of python scripts to use the new Twitter API v1.1. The scripts that I have are working with v1 but they were developed by someone else that is no longer able to work on them because they are unwell.

In a nutshell what the scripts are designed to do is connect to https://stream.twitter.com/1/statuses/filter.json and check for tweets sent to specific @usernames, retrieve the tweet data and insert them into a MySQL database. Afterwards another script does some post-processing to move desired data into other tables and then clean up. There are a couple of other non-Twitter scripts that do some general MySQL housekeeping for stats etc.

I am sure this is quite trivial for some people but I'm stumbling around in the dark because I can't find a working example of doing something similar that uses v1.1. What I need to do is go through the scripts and replace the old JSON syntax with the new one.

For example v1...
messagelog = "tweet_id=%s:screen_name=%s:time_created=%s in twitter stream (tweet.py)" % (content['id_str'],content['user']['screen_name'],msgtime)

The scripts are only needed to fetch public tweets. I don't have to worry about any user logins or authentication issues. The original author has created a working method that ensures the scripts don't make too many API connections. Everything works fine and dandy in API v1 but needs minor tweaks for v1.1.

Please point me towards somewhere else to ask for help if this is the wrong place. I've already tried dev-twitter. Any help you can offer would be appreciated. Thank you.

regards James (QuotesUK)

@QuotesUK
Copy link
Author

If anyone is using the branch api_v1.1 can they tell me if upgrading the libraries caused any problems.

If you can help me with the syntax changes I will be very grateful.

@bear
Copy link
Owner

bear commented Feb 17, 2013

That branch is for the person who offered to help work on it - I have not
changed it in anyway just yet for v1.1 api.

It's very much a work in progress with nothing committed yet.

On Sun, Feb 17, 2013 at 11:37 AM, QuotesUK notifications@github.com wrote:

If anyone is using the branch api_v1.1 can they tell me if upgrading the
libraries caused any problems.

If you can help me with the syntax changes I will be very grateful.


Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-13688597.

Bear

bear@xmpp.org (email)
bear42@gmail.com (xmpp, email)
bear@code-bear.com (xmpp, email)
http://code-bear.com/bearlog (weblog)

PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29

@QuotesUK
Copy link
Author

QuotesUK commented Mar 5, 2013

Thank you bear. Are you able to tell me whether v0.8.7 of twitter.py has been tweaked to work with the new JSON syntax of API v1.1... should I be upgrading twitter.py at this stage or should I wait for a while?

@bear
Copy link
Owner

bear commented Mar 5, 2013

I have not yet updated it to v1.1 of the API - I was giving one contributor
some time as he was working on that. But with the deadline happening this
month I may have to make the change myself and then tweak/enhance it later.

It will probably be a week or so before I have anything even for testing.

On Tue, Mar 5, 2013 at 11:54 AM, QuotesUK notifications@github.com wrote:

Thank you bear. Are you able to tell me whether v0.8.7 of twitter.py has
been tweaked to work with the new JSON syntax of API v1.1... should I be
upgrading twitter.py at this stage or should I wait for a while?


Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-14450939
.

Bear

bear@xmpp.org (email)
bear42@gmail.com (xmpp, email)
bear@code-bear.com (xmpp, email)
http://code-bear.com/bearlog (weblog)

PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29

@QuotesUK
Copy link
Author

QuotesUK commented Mar 5, 2013

Thank you for the quick reply... I will keep an eye open for the update. And another thank you for all the work you do to keep twitter.py running for us. It is very much appreciated.

@bear
Copy link
Owner

bear commented Mar 5, 2013

your welcome!

I'm glad that the tech/problem itch DeWitte and myself needed to scratch oh
so many years ago is still considered useful

On Tue, Mar 5, 2013 at 12:10 PM, QuotesUK notifications@github.com wrote:

Thank you for the quick reply... I will keep an eye open for the update.
And another thank you for all the work you do to keep twitter.py running
for us. It is very much appreciated.


Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-14451971
.

Bear

bear@xmpp.org (email)
bear42@gmail.com (xmpp, email)
bear@code-bear.com (xmpp, email)
http://code-bear.com/bearlog (weblog)

PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29

@domenkozar
Copy link
Contributor

It seems twitter dropped support for 1.0 api today.

@bear
Copy link
Owner

bear commented Mar 5, 2013

they are "testing" the dropping of support by having planned outages - they
are trying to poke/prod users into dropping apps and moving over to their
official stuff IMO

I'm tracking the tests and working on the v1.1 api code for python-twitter

On Tue, Mar 5, 2013 at 12:39 PM, Domen Kožar notifications@github.comwrote:

It seems twitter dropped support for 1.0 api today.


Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-14453617
.

Bear

bear@xmpp.org (email)
bear42@gmail.com (xmpp, email)
bear@code-bear.com (xmpp, email)
http://code-bear.com/bearlog (weblog)

PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29

@sebastianw
Copy link
Collaborator

So as time runs out, how should we do this? Do you need help?

@bear
Copy link
Owner

bear commented Mar 13, 2013

oh yea, do I !

my thought is that we need to move to a more modern requests library to pick up a lot of other features, but for sure we need the first step of going thru the current code and finding out what the v1.1 API method would be.

that way we can find out if we need new code or not or what the heck is required.

there is a v1.1 branch, please do just start working on that to start making any parts of it work.

i'll add you as a contributor now to make that part easier

@QuotesUK
Copy link
Author

I'm a newcomer to Python. I would be very grateful if someone could help me understand how to adapt to the new structure of JSON responses. I understand this is more of an implementation issue rather than API development but I'm stuck because of a lack of documentation and examples, and don't know whether the new syntax requires me to:
1/ make script changes
2/ just update twitter.py that is being worked on for api 1.1
3/ or both

What I would like to see is a worked example that shows me how to work with https://stream.twitter.com/1.1/statuses/filter.json and grab selected fields to be inserted into a mysql database. I already have this working in api v1.

@bear
Copy link
Owner

bear commented Mar 13, 2013

this is why I added you as a contributor :)

in the v1.1 branch, make a copy of twitter.py and then start hacking on one of the methods and add as many print's debug stuff or whatever

then we can all see the code example and start commenting on the checking or just making changes

but yea, the core issue will be translating the result to the internal structures that we already have. we need to maintain the same class structure we have, so most will be changing the json-to-dict methods

@sebastianw
Copy link
Collaborator

I updated the GetSearch() function. I already committed it to avoid duplicate work. I would suggest we commit every function as a commit so that we keep up-to-date with each other as complete as possible. What do you think?

@ghost ghost assigned sebastianw and bear Mar 14, 2013
@bear
Copy link
Owner

bear commented Mar 14, 2013

That sounds like a great idea. We may need to create a list of APIs that need working on - then we can make note on who is working on what.

@sebastianw
Copy link
Collaborator

I made a page in the Wiki with all twitter.Api function that use an URL. We can mark a function when we start working on it. https://github.com/bear/python-twitter/wiki/ApiUpdate

@0x0ece
Copy link
Collaborator

0x0ece commented Apr 5, 2013

Hi, at @theneeds we're using the python-twitter library and we'll be happy to contribute to v1.1 as May 7th is approaching :)

Is the above list still maintained? Can we sync with you guys to avoid duplicated work?

Unless you stop us, we will start with GetFollowers and PostDirectMessage. We expect to send you some code in the next few days.

@bear
Copy link
Owner

bear commented Apr 5, 2013

That page is current - i've been adding to it as I fix issues and convert items.

I welcome any help you can give - work and life has been busy so my focus has not been steady on python-twitter.

should I enabled your account for access or another one on your team?

@0x0ece
Copy link
Collaborator

0x0ece commented Apr 5, 2013

Me and @git-matrix will work on this

@bear
Copy link
Owner

bear commented Apr 5, 2013

ok, you both have been added to the contributors list. thanks!

@sebastianw sebastianw reopened this Apr 5, 2013
@sebastianw
Copy link
Collaborator

Yes, I've been unable to work on it the last few days, also busy. :( I hope I can add a few things again soon but every help is appreciated.

@dunvi
Copy link

dunvi commented May 22, 2013

i'm assuming this is still open because you still aren't on the v1.1 api? suddenly started getting errors while testing today and twitter says they're doing another blackout test so... :P i might take a look at this and see if i can't help move it to the new api when i have time.

@sebastianw
Copy link
Collaborator

I think all the API calls that were in there are done now. There are a hand full of new ones that are missing and the json data structures for the objects are not all checked/updated. But the basic functionality should be there with 1.1.

@dunvi
Copy link

dunvi commented May 22, 2013

yeah - discovered that i just needed to switch to the v1.1 branch. i did that and rebuild/reinstalled and everything stopped breaking. what's the timeline on moving the v1.1 branch to master?

@bear
Copy link
Owner

bear commented May 22, 2013

sometime this weekend (it's a long holiday weekend and I tend to "catch up"
with my personal projects during them) I will review all of the great work
the others have done and make a decision to push it out the door.

then I will take the current trunk and make that a branch for safety and
merge the v1.1 branch to trunk

I will probably do some heavy testing before then, but yea, expect it to be
published this weekend

On Wed, May 22, 2013 at 5:17 PM, dunvi notifications@github.com wrote:

yeah - more accurately, what i meant to say is that i need to switch to
the v1.1 branch. i did that and rebuild/reinstalled and everything stopped
breaking. what's the timeline on moving the v1.1 branch to master?


Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-18308615
.

Bear

bear@xmpp.org (email)
bear42@gmail.com (xmpp, email)
bear@code-bear.com (xmpp, email)
http://code-bear.com/bearlog (weblog)

PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29

@QuotesUK
Copy link
Author

Has it been four months since my original post? Wow, how time flies!

I am still using v1 of the API for my site and looking forward to the upgrade when it is ready.

If it's not too much trouble to ask, when the new trunk is ready, please could someone give me a pointer on comparing how to reference the old JSON structure with the new v1.1 JSON structure... I'm pretty good at figuring things out if I'm given a nudge in the right direction but working with Python is new for me and I welcome any help that is offered.

@bear
Copy link
Owner

bear commented May 22, 2013

the old to new json documentation/translation will be the hardest part IMO
because, to be honest, I don't store the returned json but rather transform
it to something else - but I do get why it would be nice to have

so yea, we should offer something

On Wed, May 22, 2013 at 6:04 PM, QuotesUK notifications@github.com wrote:

Has it been four months since my original post? Wow, how time flies!

I am still using v1 of the API for my site and looking forward to the
upgrade when it is ready.

If it's not too much trouble to ask, when the new trunk is ready, please
could someone give me a pointer on comparing how to reference the old JSON
structure with the new v1.1 JSON structure... I'm pretty good at figuring
things out if I'm given a nudge in the right direction but working with
Python is new for me and I welcome any help that is offered.


Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-18311147
.

Bear

bear@xmpp.org (email)
bear42@gmail.com (xmpp, email)
bear@code-bear.com (xmpp, email)
http://code-bear.com/bearlog (weblog)

PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29

@sebastianw
Copy link
Collaborator

I'm also a bit scared of updating the whole json structure. Perhaps just store the decoded json dict and only translate certain values into objects?

I'm thinking about utilizing getattr to get all the standard values out of the json by default and make exceptions for the values we want to handle in a special way (e.g. making objects out of them). What do you think? That would also take care of the eq comparison, we could probably just compare the original json dicts.

@bear
Copy link
Owner

bear commented May 22, 2013

Storing the original and exposing it as a source property makes a lot of
sense - that would allow folks to get to items that we may be slow at
including in our object - I like it.

Could we hash the json and use that for comparison? I guess I'm worried
about storing the JSON as a dictionary and then trying to compare that - or
are you talking about the string-ified JSON?

On Wed, May 22, 2013 at 6:18 PM, Sebastian Wiesinger <
notifications@github.com> wrote:

I'm also a bit scared of updating the whole json structure. Perhaps just
store the decoded json dict and only translate certain values into objects?

I'm thinking about utilizing getattr to get all the standard values out
of the json by default and make exceptions for the values we want to handle
in a special way (e.g. making objects out of them). What do you think? That
would also take care of the eq comparison, we could probably just
compare the original json dicts.


Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-18311859
.

Bear

bear@xmpp.org (email)
bear42@gmail.com (xmpp, email)
bear@code-bear.com (xmpp, email)
http://code-bear.com/bearlog (weblog)

PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29

@sebastianw
Copy link
Collaborator

Yeah, we could hash it, sounds good. I was talking about the dictionary but your idea sounds better. :)

@bear
Copy link
Owner

bear commented May 22, 2013

cool!

On Wed, May 22, 2013 at 6:32 PM, Sebastian Wiesinger <
notifications@github.com> wrote:

Yeah, we could hash it, sounds good. I was talking about the dictionary
but your idea sounds better. :)


Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-18312543
.

Bear

bear@xmpp.org (email)
bear42@gmail.com (xmpp, email)
bear@code-bear.com (xmpp, email)
http://code-bear.com/bearlog (weblog)

PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29

@sebastianw
Copy link
Collaborator

I'm wondering is if the JSON is actually stable or if it's structure changes between requests.

@bear
Copy link
Owner

bear commented May 23, 2013

over time the JSON is not stable, Twitter will add/remove items at will

but is it stable across a session - that is where the "use for comparison"
part is the most valid (or at least that is my working assumption.)

On Thu, May 23, 2013 at 2:47 PM, Sebastian Wiesinger <
notifications@github.com> wrote:

I'm wondering is if the JSON is actually stable or if it's structure
changes between requests.


Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-18363751
.

Bear

bear@xmpp.org (email)
bear42@gmail.com (xmpp, email)
bear@code-bear.com (xmpp, email)
http://code-bear.com/bearlog (weblog)

PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29

@domenkozar
Copy link
Contributor

When is new release scheduled?

@bear
Copy link
Owner

bear commented Jun 3, 2013

very soon - I am torn between waiting as long as possible to get more bug
reports or just releasing it now

On Mon, Jun 3, 2013 at 7:37 AM, Domen Kožar notifications@github.comwrote:

When is new release scheduled?


Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-18835589
.

Bear

bear@xmpp.org (email)
bear42@gmail.com (xmpp, email)
bear@code-bear.com (xmpp, email)
http://code-bear.com/bearlog (weblog)

PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29

@domenkozar
Copy link
Contributor

Release early, release often :)

@JulienD
Copy link

JulienD commented Jun 13, 2013

Hi guys,

So, since yesterday I'm not able to use push tweets on Twitter via the librarie:

The Twitter REST API v1 is no longer active. Please migrate to API v1.1. https://dev.twitter.com/docs/api/1.1/overview.

@bear, can we have the status of the api_v1.1 branch ?
Seems we need the new version :)

@bear
Copy link
Owner

bear commented Jun 13, 2013

I pushed out v1.0 of Python-Twitter to PyPI and it's now the version that
is the master branch on github.

https://pypi.python.org/pypi/python-twitter/1.0

so status is "please use it and let me know what broke" :)

one thing we do know is that a lot of the GetFollower() and GetFriends()
calls are being very ratelimited so I have to change how they work and
document that.

On Thu, Jun 13, 2013 at 3:18 AM, Julien Dubreuil
notifications@github.comwrote:

Hi guys,

So, since yesterday I'm not able to use push tweets on Twitter via the
librarie:

The Twitter REST API v1 is no longer active. Please migrate to API v1.1.
https://dev.twitter.com/docs/api/1.1/overview.

@bear https://github.com/bear, can we have the status of the api_v1.1
branch ?
Seems we need the new version :)


Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-19375894
.

Bear

bear@xmpp.org (email)
bear42@gmail.com (xmpp, email)
bear@code-bear.com (xmpp, email)
http://code-bear.com/bearlog (weblog)

PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29

@bear
Copy link
Owner

bear commented Jun 13, 2013

I'm going to close this issue so people won't think that 1.1 is still pending.

@bear bear closed this as completed Jun 13, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants