[ie/twitter] Add fallback, improve error handling #7621
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pukkandan
requested changes
Jul 20, 2023
yt_dlp/extractor/twitter.py
Outdated
Comment on lines
1194
to
1208
| 'cards_platform': 'Web-12', | ||
| 'include_cards': 1, | ||
| 'include_reply_count': 1, | ||
| 'include_user_entities': 0, | ||
| 'tweet_mode': 'extended', | ||
| }), 'retweeted_status', None) | ||
| elif not self.is_logged_in: | ||
| status = self._graphql_to_legacy( | ||
| self._call_graphql_api('2ICDjqPd81tulZcYrtpTuQ/TweetResultByRestId', twid), twid) | ||
| else: | ||
| if self.is_logged_in: | ||
| status = self._graphql_to_legacy( | ||
| self._call_graphql_api('zZXycP0V6H7m-2r0mOnFcA/TweetDetail', twid), twid) | ||
| else: | ||
| try: | ||
| status = self._graphql_to_legacy( | ||
| self._call_graphql_api('2ICDjqPd81tulZcYrtpTuQ/TweetResultByRestId', twid), twid) | ||
| except ExtractorError as e: | ||
| if self._login_hint() in e.msg or bug_reports_message() not in e.msg: | ||
| raise # Do not try fallback when tweet is expected to be unavailable | ||
| self.report_warning(e.msg, video_id=twid) | ||
| self.report_warning('Falling back to syndication endpoint; some metadata may be missing') | ||
| status = self._download_json( | ||
| 'https://cdn.syndication.twimg.com/tweet-result', twid, 'Downloading syndication JSON', | ||
| headers={'User-Agent': 'Googlebot'}, query={'id': twid}) |
There was a problem hiding this comment.
might be worth splitting this into a function, but not mandatory
|
The issue with the guest token was caused by a core regression (fixed by #7648), not by any change that Twitter made. I only realized this last night after I discovered the regression. I do think some of the changes in this PR are still valuable, like checking for protected tweets and adding the syndication fallback. Should I revert the unnecessary changes and continue working with this PR as a site-enhancement? |
|
Sure. Go with whichever impl you think is better |
bashonly
changed the title
pukkandan
approved these changes
Jul 22, 2023
pukkandan
reviewed
Jul 22, 2023
aalsuwaidi
pushed a commit
to aalsuwaidi/yt-dlp
that referenced
this pull request
Apr 21, 2024
Closes yt-dlp#7579, Closes yt-dlp#7625 Authored by: bashonly
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds syndication fallback for tweet extraction and improves error handling all over
Closes #7579, Closes #7625
Outdated description
(EDIT: The guest token extraction was actually broken by a core regression which has been fixed)
Twitter has apparently decommissioned the guest token API endpoint, and the browser now gets the token from the webpage html instead.
_perform_loginwas already doing this, so this patch updates_fetch_guest_tokento always do this. No more guest token endpoint means that "legacy API" tweet extraction is now broken, so the dead code has been removed.with master:
with patch:
Template
Before submitting a pull request make sure you have:
In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:
What is the purpose of your pull request?
Copilot Summary
🤖 Generated by Copilot at b13f71b
Summary
🔄🚀🔢
Improve the performance and reliability of the
twitter.pyextractor by streamlining the authentication process. Use the webpage to get the guest token and reduce redundant requests.Walkthrough