Skip to content

Commit

Permalink
twitter bug fix for scraping favorites (they changed their HTML a bit)
Browse files Browse the repository at this point in the history
  • Loading branch information
snarfed committed Mar 24, 2017
1 parent 93c8c9f commit 914fd58
Show file tree
Hide file tree
Showing 3 changed files with 8 additions and 4 deletions.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,7 @@ Changelog
### 1.8 - unreleased
* Twitter:
* Bug fix for creating replies, favorites, or retweets of video URLs, e.g. https://twitter.com/name/status/123/video/1 .
* Bug fix for parsing favorites HTML to handle a small change on Twitter's side.
* Instagram:
* Update scraping to handle new home page (ie news feed) JSON schema, which changed sometime around 2017-02-27. (Profile pages and individual photo/video permalinks still haven't changed yet.)
* microformats2:
Expand Down
9 changes: 6 additions & 3 deletions granary/test/test_twitter.py
Original file line number Diff line number Diff line change
Expand Up @@ -411,13 +411,15 @@ def tag_uri(name):
<ol class="activity-popup-users">
<li class="js-stream-item stream-item stream-item
" data-item-id="353" id="stream-item-user-353" data-item-type="user">
<!-- snipped <div class="account"... -->
<!-- snipped <div class="account"... -->
<!-- new HTML, as of at least 2017-03-23, wraps username in <b> -->
<div class="content">
<div class="stream-item-header">
<a class="account-group js-user-profile-link" href="/ge" >
<img class="avatar js-action-profile-avatar " src="https://twimg/353" alt="" data-user-id="353"/>
<strong class="fullname js-action-profile-name">George</strong>
<span class="username js-action-profile-name">@ge</span></a>
<span class="username u-dir" dir="ltr">@<b>ge</b></span></a>
</div>
</div>
</div>
Expand All @@ -428,11 +430,12 @@ def tag_uri(name):
<div class="content">
<div class="stream-item-header">
<a class="account-group js-user-profile-link" href="/ge" >
<span class="username js-action-profile-name">@jo</span></a>
<span class="username u-dir" dir="ltr">@<b>jo</b></span></a>
</div>
</div>
</li>
<!-- old HTML, has js-action-profile-name -->
<li class="js-stream-item stream-item stream-item">
<div class="content">
<div class="stream-item-header">
Expand Down
2 changes: 1 addition & 1 deletion granary/twitter.py
Original file line number Diff line number Diff line change
Expand Up @@ -1359,7 +1359,7 @@ def favorites_html_to_likes(self, tweet, html):
username = user.find(class_='username')
if not username:
continue
username = unicode(username.string)
username = unicode(username.get_text(''))
if username.startswith('@'):
username = username[1:]

Expand Down

0 comments on commit 914fd58

Please sign in to comment.