No (extended) entities

[Brawl345]

When I use api.GetStatus(ID) I don't get extended_entities or even normal entities. All I get is "media", but that doesn't include video information (for example).

E.g. for this tweet:

print(api.GetStatus(778307811012780032))

and I only get this back:

{
   "created_at":"Tue Sep 20 19:00:17 +0000 2016",
   "favorite_count":212,
   "hashtags":[
      {
         "text":"amiibo"
      },
      {
         "text":"Picross"
      }
   ],
   "id":778307811012780032,
   "id_str ":"778307811012780032",
   "lang":"en",
   "media":[
      {
         "display_url":"pic.twitter.com/ibou4buFxe",
         "expanded_url":"https://twitter.com/NintendoAmerica/status/778307811012780032/video/1",
         "id":778025997606105089,
         "media_url":"http://pbs.twimg.com/media/CswaoY4UAAA8-Zj.jpg",
         "media_url_https":"https://pbs.twimg.com/media/CswaoY4UAAA8-Zj.jpg",
         "type":"video",
         "url":"https://t.co/ibou4buFxe"
      }
   ],
   "ret weet_count":39,
   "source":"<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>",
   "text":"Puzzled on how to use #amiibo in #Picross 3D Round 2? Just follow these six simple steps!\nhttps://t.co/MjciohRcuW https://t.co/ibou4buFxe",
   "urls":[
      {
         "expanded_url":"http://www.nintendo.com/games/detail/picross-3d-round-2-3ds",
         "url":"https://t.co/MjciohRcuW"
      }
   ],
   "user":{
      "created_at":" Wed Apr 18 22:43:15 +0000 2007",
      "description":"Welcome to the official Nintendo profile for gaming news! We\u2019re listening, too. For ESRB ratings go to https://t.co/OgSR65P8OY",
      "favourites_count":252,
      "followers_count":5042478,
      "fri ends_count":1348,
      "id":5162861,
      "lang":"en",
      "listed_count":10196,
      "name":" Nintendo of America",
      "profile_background_color":"ACDED6",
      "profile_background_ image_url":"http://pbs.twimg.com/profile_background_images/623621309210083328/e9ZICp8d.jpg",
      "profile_banner_url":"https://pbs.twimg.com/profile_banners/5162861/1465781592",
      "profile_image_url":"http://pbs.twimg.com/profile_images/745752686780387333/wsjpSx2K_normal.jpg",
      "profile_link_color":"038543",
      "profile_side bar_fill_color":"F6F6F6",
      "profile_text_color":"333333",
      "screen_name":"NintendoAmerica",
      "statuses_count":11812,
      "time_zone":"Pacific Time (US & Canada)",
      "url":"https://t.co/cMLmFbyXaL",
      "utc_offset":-25200,
      "verified":true
   },
   "user_mentions":[

   ]
}

Is this a problem on my end, did I have missed something or...?

[jeremylow]

Not just you. This was working before. See issue #338 - those changes just went into effect yesterday.

[jeremylow]

Ok, so a few things are going on here:

• The tweet isn't a normal tweet because it's from Nintendo so it gets all sorts of extra stuff included.
• In terms of whether you'll get extended_entities or entities: you won't. Those are sort of elided in favor of giving direct, property-based access to the "relevant" objects. For instance, the user portion of the JSON that you got above is really a twitter.User object.
• The media node that you seen above is the entities[media] node of the original tweet and I think you're not getting the extended_entities version of that because [I don't know - still trying to work this out].

This doesn't have anything to do with the changes apparently; I just don't think we saw a tweet like this before, so all our tests passed without noticing anything different.

[jeremylow]

I'm still trying to work out what's different from normal, video tweets. Thanks for the report!

[Brawl345]

Thank you for your answers! Just want to add that when I contact the API with LUA (should be the same as a simple GET request, I just use the OAuth lib for that), I get entities, extended entities as well as media. You can look at the JSON here (ignore the e+17 at the id integer, seems like LUA doesn't like overly large integers :D)

[tgrrtt]

In terms of whether you'll get extended_entities or entities: you won't. Those are sort of elided in favor of giving direct, property-based access to the "relevant" objects. For instance, the user portion of the JSON that you got above is really a twitter.User object.

What is the purpose of having include_entities as a param on anything that returns a twitter.User (or collection of them)? Example IMO it creates a very confusing interface if passing it along looks like it should do something, but doesn't actually have any effect.

[jeremylow]

It gives you access to URLs, Hashtags, Mentions, and Media in any returned Status. We try to mirror the interface of the endpoint; if there is an include_entities parameter, even if we process it slightly differently, we include that option in the method call.

[jeremylow]

Perhaps that's explaining it poorly; so I think your issue with the parameter is that it looks like it doesn't have an effect on the final output because there's no entities attribute on the twitter.User object. That's true - there isn't; as I said before, the entities node is deconstructed into its constituent sub-nodes and then attached as various Url, Media, [whatever] attributes onto the parent object.

That doesn't mean it doesn't do anything though: if include_entities is False then the twitter.User object will have a twitter.Status sub-object (or whatever it's properly called) which doesn't contain any: twitter.Url, twitter.Hashtags, or user_mentions attributes. If the parameter is True all of those are included.

Although I realize the translation from Twitter's documentation to ours is a little maladroit, given what we're working with in terms of the ecosystem, I think overall our approach maintains a level of fidelity to Twitter's API that allows people to adapt to the way we've done things and is consistent with the other methods that we expose.

I've attached the AsJsonString output for both.

without_entities.txt
with_entities.txt

[tgrrtt]

I mean, I could argue that it still doesn't do anything, from the User instance's perspective ... the User is still missing entity information about itself.

I see that the Status does infact contain entity information, but it seems strange to me thatGetUser would modify a Status sub-object on the User, but still leave out arguably important entity information about the User itself (which is probably what i'm actually interested in, and expecting, because I'm querying for a User, and passing that argument on a fn called GetUser).

Here's all the info that's actually getting dropped about the User:

    "url": {
      "urls": [
        {
          "url": "http://t.co/wtg3XzqQTX",
          "indices": [
            0,
            22
          ],
          "expanded_url": "http://iseverythingstilltheworst.com",
          "display_url": "iseverythingstilltheworst.com"
        }
      ]
    },
    "description": {
      "urls": []
    }

[jchen123]

One significant thing that's missing is "video_info" which exists inside extended_entities. Because it calls Media.NewFromJsonDict, "video_info" is excluded. So we have no way of getting the mp4 files from a video call, when the raw REST API call includes it.

[jeremylow]

Lands with PR #398

[bcb]

The api needs tweet_mode=extended to return media.

https://dev.twitter.com/overview/api/upcoming-changes-to-tweets

[jeremylow]

@bcb Partially, yeah. If len(tweet) + len(media) < 140, where len(media) is the number of characters per media reserved, then extended_entities will be returned in compat mode. If len(tweet) + len(media) > 140 then extended_entities will only be returned in extended mode.

It's confusing and I don't like it, but that's how the cookie crumbles. According to Twitter:

Any entities only include those in the available 140 text range

You can see this in action with the following tweets. The first tweet (here - it's hilarious, check it out) is only 26 characters long, which, plus the animated gif (24 characters reserved as of today) is within the 140 character limit. The second tweet (here) is 136 characters without media, but with the 24 characters reserved, it's over the 140 limit, which means that it doesn't return the extended_entities node.

>>> api.tweet_mode
>>> 'compat'
>>> st1 = api.GetStatus(793361910632738817)
>>> st1.media
>>> [Media(ID=793361349661302784, Type=animated_gif, DisplayURL='pic.twitter.com/7FPUEcsln8')]
>>> # But see:
>>> st2 = api.GetStatus(790462511162195968)
>>> st2.media
>>> # None