New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

280 characters via REST API only works for full_text and tweet_mode=extended #880

Open
craigthusiast opened this Issue Nov 13, 2017 · 8 comments

Comments

Projects
None yet
6 participants
@craigthusiast
Copy link

craigthusiast commented Nov 13, 2017

With Twitter's recent change to 280 characters, fetching tweets via the Search REST API endpoints only returns 280 characters when fetched via the tweet_mode=extended option. The default compatibility mode truncates the tweet text at 140 characters.

Here's the documentation:
https://developer.twitter.com/en/docs/tweets/tweet-updates
And here's is Andy Piper's confirmation of this:
https://twittercommunity.com/t/280-characters/94200/6

Note: that the full_text attribute replaces the text attribute in extended mode.

I have many features in my application that fetch tweets via the Search API and now that people are tweeting with 280 characters, a large number of them are being truncated.

Also, with the way Twitter no longer counts @replies in the character limit, people are often replying to many people in their tweets. But since the gem doesn't support full_text mode, it's not possible to parse out those replies and display only the actual text of the tweet. When there are many @replies, my application doesn't display any of the tweet text, which is really annoying to my users.

Can you please add support for the full_text attribute via tweet_mode=extended so that applications built with this gem can keep pace with Twitter's changes to the API?

@craigthusiast

This comment has been minimized.

Copy link

craigthusiast commented Nov 15, 2017

Update: after doing some testing and experimenting, it appears that the gem is in fact pulling down the right data when fetching tweets from the Twitter Search API in tweet_mode=extended.

https://github.com/twitterdev/tweet-updates/blob/master/samples/initial/compatibility_extended_13996.json

After fetching a tweet via tweet_mode=extended, I'm able to access the various attributes via the .attrs method. Here are some examples of tweet attributes I've tested (note: they all work properly):

tweet.attrs[:full_text]
tweet.attrs[:user][:screen_name]
tweet.attrs[:in_reply_to_screen_name]
tweet.attrs[:entities][:user_mentions]
tweet.attrs[:display_text_range]

This means that we can properly render 280 character tweets.

It also means that we can at least access all the "components" of a tweet. But it's still a lot of work to parse the full_text of each tweet and somehow remove the user_mentions from the text and display them in a "Replying to:" div, similar to the way Twitter does it in their apps.

It would be super awesome if the gem did that for us! 💯

@jm3

This comment has been minimized.

Copy link

jm3 commented Jan 17, 2018

FYI, you can easily fetch full 280 character statuses with the gem today by passing extended mode via the opts hash:

long_tweet = TwitterClient.status(953020927792631809, tweet_mode: 'extended')
fields = long_tweet.to_hash.slice(:full_text, :truncated)

...

[1] pry > fields
=> {:truncated=>false, :full_text=>
     "it's my understanding that some people are very angry that i jokingly used 
     the phrase \"mayonnaise sucka\". i want to offer my sincerest apologies to all 
     those who i have hurt. i've always tried to live my life according to the 
     strictest of moral codes. i will be seeing a therapist"
   }
[2] pry > fields[:full_text].size
   => 280
@craigthusiast

This comment has been minimized.

Copy link

craigthusiast commented Jan 18, 2018

Good tip, @jm3. Thanks.

@cmirnow

This comment has been minimized.

Copy link

cmirnow commented Jan 26, 2018

Hi, I'm asking for advice.
In two words;

if

@arr = Set.new
client.search("@#{params[:twitter_acc]}", result_type: "recent", tweet_mode: "extended")
            .take(params[:number_of_tweets].to_i)
            .collect do |tweet|
	@arr << tweet
                .full_text
                .gsub("@#{params[:twitter_acc]}", "")
                .gsub("RT :", "")
                .gsub("RT", "")
                puts tweet.full_text
end

, I get:

Started GET "/tweet_parser/index?number_of_tweets=10&twitter_acc=SkyNews" for 127.0.0.1 at 2018-01-26 18:51:24 +0300
Processing by TweetParserController#index as HTML
  Parameters: {"number_of_tweets"=>"10", "twitter_acc"=>"SkyNews"}




  Rendering tweet_parser/index.html.erb within layouts/application
  Rendered tweet_parser/index.html.erb within layouts/application (3.0ms)
  Rendered layouts/_navigation.html.erb (4.2ms)
Completed 200 OK in 2161ms (Views: 345.0ms | ActiveRecord: 0.0ms)

elsif:

@arr = Set.new
client.search("@#{params[:twitter_acc]}", result_type: "recent")
            .take(params[:number_of_tweets].to_i)
            .collect do |tweet|
	@arr << tweet
                .text
                .gsub("@#{params[:twitter_acc]}", "")
                .gsub("RT :", "")
                .gsub("RT", "")
                puts tweet.text
end

, I get:

Started GET "/tweet_parser/index?number_of_tweets=10&twitter_acc=SkyNews" for 127.0.0.1 at 2018-01-26 18:52:52 +0300
Processing by TweetParserController#index as HTML
  Parameters: {"number_of_tweets"=>"10", "twitter_acc"=>"SkyNews"}
@SkyNews Most media are a pile of biased shite anyway.
RT @jbwhitesnake: #FlagGate @SkyNews @BBCNews https://t.co/wF8ZG8TwKj
@SkyNews Not doing the Nutella brand any harm is it?
RT @SkyNews: Drone flew so close to passenger jet over Kew that crew thought it had hit plane  https://t.co/fAG0ZB7kqn
@Siubhan_H @MSANNCOOPER @estocker8 @SledgeH90949539 @Tochdg @rockyrose @IncognitoKarma @JeffersonArty… https://t.co/uE3InvTzbU
@WalidPhares @Reuters @AP @FoxNews @SkyNews @BBCWorld https://t.co/gxeoHJF7aH
@SkyNews This idiot should not talk as it seems he is an arsehole.
@Pardus_UK @SkyNews The show business world is full of them...Just waiting for a couple of rock bands to hit the he… https://t.co/KfQIS4WI22
RT @jbwhitesnake: #FlagGate @SkyNews @BBCNews https://t.co/wF8ZG8TwKj
@IsraelMFA @francediplo_EN @foreignoffice @KremlinRussia_E @markrutte @MinPres @DutchMFA @KatiPiri @KenRoth… https://t.co/9QdcRX6WSy
  Rendering tweet_parser/index.html.erb within layouts/application
  Rendered tweet_parser/index.html.erb within layouts/application (3.2ms)
  Rendered layouts/_navigation.html.erb (3.7ms)
Completed 200 OK in 2664ms (Views: 379.6ms | ActiveRecord: 0.0ms)

How to solve this problem?

@dtabares

This comment has been minimized.

Copy link

dtabares commented Feb 15, 2018

Hi! Has anyone made this work for Retweets? I still get RTs truncated while using tweet_mode: "extended". Original and Replies look fine.

Thanks.


Nevermind, I found that the solution was to get the full_text from raw_tweet.attrs[:retweeted_status][:full_text] instead of raw_tweet.attrs[:full_text]

@reaper-1991

This comment has been minimized.

Copy link

reaper-1991 commented Nov 7, 2018

Hi! Has anyone made this work for Retweets? I still get RTs truncated while using tweet_mode: "extended". Original and Replies look fine.

Thanks.

Nevermind, I found that the solution was to get the full_text from raw_tweet.attrs[:retweeted_status][:full_text] instead of raw_tweet.attrs[:full_text]

Hi @dtabares, can you please let me know where to do the changes which you have mentioned.

Thank you.

@dtabares

This comment has been minimized.

Copy link

dtabares commented Nov 7, 2018

Hi @reaper-1991 , in the object where you are storing the re-tweet.

@luizcarvalho

This comment has been minimized.

Copy link

luizcarvalho commented Dec 27, 2018

Hello @dtabares you solve the problem?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment