Handling Collections (with code)

[DaveSteadman]

I've been able to add some Collection handling to Tweepy with the following code.

For an overview, collections can be viewed in a different (and to my mind better) grid format than a usual timeline: https://developer.twitter.com/en/docs/twitter-for-websites/timelines/overview

The api.py changes are:

@property
  def list_collections_by_screenname(self):
      """ :reference: https://developer.twitter.com/en/docs/tweets/curate-a-collection/api-reference/get-collections-list
      """
      return bind_api(
          api=self,
          path='/collections/list.json',
          method='GET',           
          payload_type='json', 
          allowed_param=['screen_name', 'user_id'],
          require_auth=True
      )

  @property
  def create_collection(self):
      """ :reference: https://developer.twitter.com/en/docs/tweets/curate-a-collection/api-reference/post-collections-create
          :allowed_param:'name', 'timeline_order', 'description'
      """
      return bind_api(
          api=self,
          path='/collections/create.json',
          method='POST',
          payload_type='json',
          allowed_param=['name', 'description', 'timeline_order'],
          require_auth=True
      )


  @property
  def add_tweet_to_collection(self):
      """ :reference: https://developer.twitter.com/en/docs/tweets/curate-a-collection/api-reference/post-collections-entries-add
      """
      return bind_api(
          api=self,
          path='/collections/entries/add.json',
          method='POST',
          payload_type='json',
          allowed_param=['id', 'tweet_id'],
          require_auth=True
      )

  @property
  def remove_tweet_from_collection(self):
      """ :reference: https://developer.twitter.com/en/docs/tweets/curate-a-collection/api-reference/post-collections-entries-remove
      """
      return bind_api(
          api=self,
          path='/collections/entries/remove.json',
          method='POST',
          payload_type='json',
          allowed_param=['id', 'tweet_id'],
          require_auth=True
      )
      
  @property
  def list_tweets_in_collection(self):
      """ :reference: https://developer.twitter.com/en/docs/tweets/curate-a-collection/api-reference/get-collections-entries
      """
      return bind_api(
          api=self,
          path='/collections/entries.json',
          method='GET',
          payload_type='json',
          allowed_param=['id'],
          require_auth=True
      )

My changed python on top of this has been:

# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 

def getCollectionId(screenname, collname):
    
    api = getAPI()
    
    # Find the collection id of <collname> or create it
    responsejson = api.list_collections_by_screenname(screenname)
    for currcoll in responsejson['objects']['timelines']:
        if responsejson['objects']['timelines'][currcoll]['name'] == collname:
            coll_id = currcoll
        else:
            responsejson = api.create_collection(collname)
            coll_id = responsejson['response']['timeline_id']
    return coll_id

# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
 
def removeOldTweetsFromCollection(screenname, collname, ageLimitSeconds):

    api = getAPI()
    
    coll_id = getCollectionId(screenname, collname)

    # Get the collection tweets
    colljson = api.list_tweets_in_collection(coll_id)

    # Drop out if nothing in collection
    if len(colljson['objects']['tweets']) == 0:
        return

    # loop over all tweets
    for currtweet in colljson['objects']['tweets']:
        
        # Get tweet info
        tweetobj     = api.get_status(currtweet)
        tweetagesecs = TweetAgeSeconds(tweetobj)
        #print(currtweet, tweetagedays)
       
        # Remove those over age limit
        if tweetagesecs > ageLimitSeconds:
            api.remove_tweet_from_collection(coll_id, tweetobj.id_str)

# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 

def addTweetListToCollection(screenname, collname, newtweetlist):

    api = getAPI()

    coll_id = getCollectionId(screenname, collname)

    # Create list of collection tweet ids
    coll_twids = []
    colljson = api.list_tweets_in_collection(coll_id)
    for currtweet in colljson['objects']['tweets']:
        coll_twids.append(currtweet)

    # Create list of new tweet ids
    new_twids = []
    for currtweet in newtweetlist:
        id_str = currtweet.id_str
        new_twids.append(id_str)

    # Create list of unique tweet ids to add
    coll_twids_set        = set(coll_twids)
    new_twids_set         = set(new_twids)
    unique_new_twids_set  = new_twids_set - coll_twids_set
    unique_new_twids_list = list(unique_new_twids_set)

    # Add tweets to collection
    for currtweet in unique_new_twids_list:
        api.add_tweet_to_collection(coll_id, currtweet)

# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 

[karanlyons]

We should probably add the rest of the methods as well? The full list is:

• GET collections/entries
• GET collections/list
• GET collections/show
• POST collections/create
• POST collections/destroy
• POST collections/entries/add
• POST collections/entries/curate
• POST collections/entries/move
• POST collections/entries/remove
• POST collections/update

It looks like with bind_api this is fairly straightforward, there’s only one concern from the docs:

Differences with other methods

Those familiar with Twitter’s family of REST APIs may notice some differences in object structure compared to typical APIs.

Pay close attention to the differences in how collections are presented — often they will be decomposed, efficient objects with information about users, Tweets, and timelines grouped, simplified, and stripped of unnecessary repetition.

Navigating collections also differs from the other APIs in that the collection is not strictly creation-time oriented. Navigating by since_id and last_id has been replaced with a position-based pagination system that should still be familiar.

See Response structures for a deeper overview of these differences.

It’s unclear to me on what the differences actually shape up to be here, but it seems like we probably just need a simple Collections model. The shape of a response out of collections/entries looks to be:

{
  "objects": {
    "users": {
      "{user_id}" : {
        ...
      }
    }
    "timelines": {
      "custom-{id}": {
        "name": string,
        "description": string,
        "user_id": "{user_id}",
        "collection_url": "https://twitter.com/{user_id}/timelines/{id}",
        "collection_type": "user", // Other enum members?
        "timeline_order": "curation_reverse_chron" | "tweet_chron" | "tweet_reverse_chron",
        "url": string,
        "visibility": "public" | "private",
      }
    },
    "tweets": {
      "{tweet_id}" {
        ...
      }
    }
  }
  "response": {
    "position": {
      "max_position": "{index}",
      "min_position": "{index}",
      "was_truncated": boolean
    },
    "timeline": [
      {
        "feature_context": "{feature_id?}",
        "tweet": {
          "id": "{tweet_id}",
          "sort_index": "{index}"
        }
      },
      ...
    ],
    "timeline_id": "custom-{id}"
  }
}

collections/show omits objects/tweets, response/position and response/timeline.

collections/create and collections/update do the same, and also omit everything in objects/timelines/custom-{id} except for name and user-id.

The rest of the methods are, I think, fairly normal, except for collections/entries/curate, which takes a request that looks like:

{
  "id": "custom-{id}",
  "changes": [
    {
      "op": "add" | "move" | "remove",
      "tweet_id": "{tweet_id}"
    },
    ...
  ]
}

I'm not sure if that's a common pattern anywhere else. The response looks like it's a common pattern, though.