Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[extractor/iwara] Fix and improve authentication #7137

Merged
merged 3 commits into from Jun 15, 2023

Conversation

toomyzoom
Copy link
Contributor

@toomyzoom toomyzoom commented May 26, 2023

IMPORTANT: PRs without the template will be CLOSED

Description of your pull request and other information

This is the same issue in #6671 where access token is needed for some NSFW video. Update to allow user profile and playlist extractors to use the same access token.

Also has fix to renew expired user token in the cache (#7207)

Fixes #7035, Fixes #7207

Template

Before submitting a pull request make sure you have:

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:

  • I am the original author of this code and I am willing to release it under Unlicense
  • I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

Copilot Summary

🤖 Generated by Copilot at 64c1ecb

Summary

🔒📦✅

Improve IwaraIE and related extractors. This pull request enhances the authentication and extraction logic for the IwaraIE, IwaraUserIE, and IwaraPlaylistIE extractors, which handle videos from the Iwara website. It also adds a new test case and a helper method for the API headers.

We are the extractors of Iwara
We defy the limits of the media
We forge the headers for the API
We renew the tokens before they die

Walkthrough

  • Add error handling and debug message for media token request failure (link)
  • Simplify header construction for API requests by introducing a new helper method _prepare_auth_header in IwaraIE class (link, link)
  • Use IwaraIE class as the parent class for IwaraUserIE and IwaraPlaylistIE classes to inherit the new helper method (link, link)
  • Add headers from _prepare_auth_header method to API requests for user, playlist, and playlist information extraction in yt_dlp/extractor/iwara.py to support private and restricted videos (link, link, link)
  • Add a new test case for IwaraUserIE class to extract videos from a user profile page (link)

@toomyzoom toomyzoom mentioned this pull request Jun 5, 2023
11 tasks
@bashonly bashonly linked an issue Jun 5, 2023 that may be closed by this pull request
11 tasks
@bashonly bashonly added NSFW site-bug Issue with a specific website labels Jun 5, 2023
Copy link
Member

@bashonly bashonly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not subclass IwaraUserIE and IwaraPlaylistIE from the concrete IwaraIE class. Revert to subclassing from IwaraBaseIE, and instead move the _perform_login and _prepare_auth_header methods into the base class.

After moving the methods into the base class, I think _perform_login should be changed like so:

     def _perform_login(self, username, password):
+        if self._USERTOKEN and self._MEDIATOKEN:
+            return
-        if self.cache.load(self._NETRC_MACHINE, username) and self._get_media_token():
+        elif self.cache.load(self._NETRC_MACHINE, username) and self._get_media_token():
             self.write_debug('Skipping logging in')
             return

This will prevent a duplicate login/media token request when going from a playlist extractor to the video extractor

yt_dlp/extractor/iwara.py Outdated Show resolved Hide resolved
yt_dlp/extractor/iwara.py Outdated Show resolved Hide resolved
yt_dlp/extractor/iwara.py Outdated Show resolved Hide resolved
yt_dlp/extractor/iwara.py Outdated Show resolved Hide resolved
@bashonly
Copy link
Member

bashonly commented Jun 5, 2023

You could also check after loading the user token from cache if it is expired:

diff --git a/yt_dlp/extractor/iwara.py b/yt_dlp/extractor/iwara.py
index 1749dcd1a..64105c39b 100644
--- a/yt_dlp/extractor/iwara.py
+++ b/yt_dlp/extractor/iwara.py
@@ -2,15 +2,18 @@
 import urllib.parse
 import hashlib
 import json
+import time
 
 from .common import InfoExtractor
 from ..utils import (
     ExtractorError,
     OnDemandPagedList,
     int_or_none,
+    jwt_decode_hs256,
     mimetype2ext,
     qualities,
     traverse_obj,
+    try_call,
     unified_timestamp,
 )
 
@@ -27,7 +30,8 @@ def _get_user_token(self, invalidate=False):
 
         username, password = self._get_login_info()
         IwaraBaseIE._USERTOKEN = username and self.cache.load(self._NETRC_MACHINE, username)
-        if not IwaraBaseIE._USERTOKEN or invalidate:
+        expiry = try_call(lambda: jwt_decode_hs256(IwaraBaseIE._USERTOKEN)['exp']) or 0
+        if not IwaraBaseIE._USERTOKEN or invalidate or expiry <= time.time():
             IwaraBaseIE._USERTOKEN = self._download_json(
                 'https://api.iwara.tv/user/login', None, note='Logging in',
                 data=json.dumps({

@bashonly bashonly added the pending-fixes PR has had changes requested label Jun 5, 2023
@toomyzoom
Copy link
Contributor Author

Do not subclass IwaraUserIE and IwaraPlaylistIE from the concrete IwaraIE class. Revert to subclassing from IwaraBaseIE, and instead move the _perform_login and _prepare_auth_header methods into the base class.

After moving the methods into the base class, I think _perform_login should be changed like so:

     def _perform_login(self, username, password):
+        if self._USERTOKEN and self._MEDIATOKEN:
+            return
-        if self.cache.load(self._NETRC_MACHINE, username) and self._get_media_token():
+        elif self.cache.load(self._NETRC_MACHINE, username) and self._get_media_token():
             self.write_debug('Skipping logging in')
             return

This will prevent a duplicate login/media token request when going from a playlist extractor to the video extractor

This should be applied in the last commit.

@bashonly
Copy link
Member

bashonly commented Jun 15, 2023

@toomyzoom I pushed some changes to improve and simplify the authentication code. Now tokens are always invalidated based on expiration time, and the extractor will refresh both tokens as needed. I've tested the changes with and without an account and haven't had any issue. Let me know if you have any problems with it

@bashonly bashonly added pending-review PR needs a review and removed pending-fixes PR has had changes requested labels Jun 15, 2023
@bashonly bashonly changed the title [iwara] Update playlist extractor to use access token [extractor/iwara] Fix and Improve authentication Jun 15, 2023
@bashonly bashonly changed the title [extractor/iwara] Fix and Improve authentication [extractor/iwara] Fix and improve authentication Jun 15, 2023
@bashonly bashonly merged commit 0a5d7c3 into yt-dlp:master Jun 15, 2023
11 checks passed
@bashonly bashonly removed the pending-review PR needs a review label Jun 15, 2023
aalsuwaidi pushed a commit to aalsuwaidi/yt-dlp that referenced this pull request Apr 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NSFW site-bug Issue with a specific website
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Iwara login cache expires [iwara.tv] Videos with "blocked" tags don't appear in profile list
2 participants