280 characters via REST API only works for full_text and tweet_mode=extended

[craigthusiast]

With Twitter's recent change to 280 characters, fetching tweets via the Search REST API endpoints only returns 280 characters when fetched via the tweet_mode=extended option. The default compatibility mode truncates the tweet text at 140 characters.

Here's the documentation:
https://developer.twitter.com/en/docs/tweets/tweet-updates
And here's is Andy Piper's confirmation of this:
https://twittercommunity.com/t/280-characters/94200/6

Note: that the full_text attribute replaces the text attribute in extended mode.

I have many features in my application that fetch tweets via the Search API and now that people are tweeting with 280 characters, a large number of them are being truncated.

Also, with the way Twitter no longer counts @replies in the character limit, people are often replying to many people in their tweets. But since the gem doesn't support full_text mode, it's not possible to parse out those replies and display only the actual text of the tweet. When there are many @replies, my application doesn't display any of the tweet text, which is really annoying to my users.

Can you please add support for the full_text attribute via tweet_mode=extended so that applications built with this gem can keep pace with Twitter's changes to the API?

[craigthusiast]

Update: after doing some testing and experimenting, it appears that the gem is in fact pulling down the right data when fetching tweets from the Twitter Search API in tweet_mode=extended.

https://github.com/twitterdev/tweet-updates/blob/master/samples/initial/compatibility_extended_13996.json

After fetching a tweet via tweet_mode=extended, I'm able to access the various attributes via the .attrs method. Here are some examples of tweet attributes I've tested (note: they all work properly):

tweet.attrs[:full_text]
tweet.attrs[:user][:screen_name]
tweet.attrs[:in_reply_to_screen_name]
tweet.attrs[:entities][:user_mentions]
tweet.attrs[:display_text_range]

This means that we can properly render 280 character tweets.

It also means that we can at least access all the "components" of a tweet. But it's still a lot of work to parse the full_text of each tweet and somehow remove the user_mentions from the text and display them in a "Replying to:" div, similar to the way Twitter does it in their apps.

It would be super awesome if the gem did that for us! 💯

[jm3]

FYI, you can easily fetch full 280 character statuses with the gem today by passing extended mode via the opts hash:

long_tweet = TwitterClient.status(953020927792631809, tweet_mode: 'extended')
fields = long_tweet.to_hash.slice(:full_text, :truncated)

...

[1] pry > fields
=> {:truncated=>false, :full_text=>
     "it's my understanding that some people are very angry that i jokingly used 
     the phrase \"mayonnaise sucka\". i want to offer my sincerest apologies to all 
     those who i have hurt. i've always tried to live my life according to the 
     strictest of moral codes. i will be seeing a therapist"
   }
[2] pry > fields[:full_text].size
   => 280

[craigthusiast]

Good tip, @jm3. Thanks.

[cmirnow]

Hi, I'm asking for advice.
In two words;

if

@arr = Set.new
client.search("@#{params[:twitter_acc]}", result_type: "recent", tweet_mode: "extended")
            .take(params[:number_of_tweets].to_i)
            .collect do |tweet|
	@arr << tweet
                .full_text
                .gsub("@#{params[:twitter_acc]}", "")
                .gsub("RT :", "")
                .gsub("RT", "")
                puts tweet.full_text
end

, I get:

Started GET "/tweet_parser/index?number_of_tweets=10&twitter_acc=SkyNews" for 127.0.0.1 at 2018-01-26 18:51:24 +0300
Processing by TweetParserController#index as HTML
  Parameters: {"number_of_tweets"=>"10", "twitter_acc"=>"SkyNews"}




  Rendering tweet_parser/index.html.erb within layouts/application
  Rendered tweet_parser/index.html.erb within layouts/application (3.0ms)
  Rendered layouts/_navigation.html.erb (4.2ms)
Completed 200 OK in 2161ms (Views: 345.0ms | ActiveRecord: 0.0ms)

elsif:

@arr = Set.new
client.search("@#{params[:twitter_acc]}", result_type: "recent")
            .take(params[:number_of_tweets].to_i)
            .collect do |tweet|
	@arr << tweet
                .text
                .gsub("@#{params[:twitter_acc]}", "")
                .gsub("RT :", "")
                .gsub("RT", "")
                puts tweet.text
end

, I get:

Started GET "/tweet_parser/index?number_of_tweets=10&twitter_acc=SkyNews" for 127.0.0.1 at 2018-01-26 18:52:52 +0300
Processing by TweetParserController#index as HTML
  Parameters: {"number_of_tweets"=>"10", "twitter_acc"=>"SkyNews"}
@SkyNews Most media are a pile of biased shite anyway.
RT @jbwhitesnake: #FlagGate @SkyNews @BBCNews https://t.co/wF8ZG8TwKj
@SkyNews Not doing the Nutella brand any harm is it?
RT @SkyNews: Drone flew so close to passenger jet over Kew that crew thought it had hit plane  https://t.co/fAG0ZB7kqn
@Siubhan_H @MSANNCOOPER @estocker8 @SledgeH90949539 @Tochdg @rockyrose @IncognitoKarma @JeffersonArty… https://t.co/uE3InvTzbU
@WalidPhares @Reuters @AP @FoxNews @SkyNews @BBCWorld https://t.co/gxeoHJF7aH
@SkyNews This idiot should not talk as it seems he is an arsehole.
@Pardus_UK @SkyNews The show business world is full of them...Just waiting for a couple of rock bands to hit the he… https://t.co/KfQIS4WI22
RT @jbwhitesnake: #FlagGate @SkyNews @BBCNews https://t.co/wF8ZG8TwKj
@IsraelMFA @francediplo_EN @foreignoffice @KremlinRussia_E @markrutte @MinPres @DutchMFA @KatiPiri @KenRoth… https://t.co/9QdcRX6WSy
  Rendering tweet_parser/index.html.erb within layouts/application
  Rendered tweet_parser/index.html.erb within layouts/application (3.2ms)
  Rendered layouts/_navigation.html.erb (3.7ms)
Completed 200 OK in 2664ms (Views: 379.6ms | ActiveRecord: 0.0ms)

How to solve this problem?

[dtabares]

Hi! Has anyone made this work for Retweets? I still get RTs truncated while using tweet_mode: "extended". Original and Replies look fine.

Thanks.

---

Nevermind, I found that the solution was to get the full_text from raw_tweet.attrs[:retweeted_status][:full_text] instead of raw_tweet.attrs[:full_text]

[reaper-1991]

Hi! Has anyone made this work for Retweets? I still get RTs truncated while using tweet_mode: "extended". Original and Replies look fine.

Thanks.

Nevermind, I found that the solution was to get the full_text from raw_tweet.attrs[:retweeted_status][:full_text] instead of raw_tweet.attrs[:full_text]

Hi @dtabares, can you please let me know where to do the changes which you have mentioned.

Thank you.

[dtabares]

Hi @reaper-1991 , in the object where you are storing the re-tweet.

[luizcarvalho]

Hello @dtabares you solve the problem?