[teachable] ERROR: Unable to find Video URL https://edu.ecomsuccess.pk/courses/enrolled/903296 #3564
Comments
pukkandan
added
account-needed
|
Amm, do I give account credentials here? |
pukkandan
added
the
can-share-account
|
No. The labels are just a hint for maintainers/contributors that account is needed to work on the issue. If any developer needs account details for working on this, they will let you know how to share it privately. |
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as resolved.
This comment was marked as resolved.
rsuchwani
changed the title
This comment was marked as resolved.
This comment was marked as resolved.
pukkandan
changed the title
This comment was marked as resolved.
This comment was marked as resolved.
|
yt-dlp/yt_dlp/extractor/teachable.py Lines 164 to 173 in 468f104 Should use |
|
Amm, is this a solution or suggestion for the developers?? Apologies in advance.. |
|
@rsuchwani, And sadly it's not any kind of solution, I've just looked at the source of the problem and shared my thoughts. Btw, is your cookiejar fresh? |
Hello, Thanks for taking the time to respond. I tried again with a fresh set of cookies but the same output. |
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as resolved.
This comment was marked as resolved.
|
See ytdl-org/youtube-dl#30929 (comment) for why no |
|
There a few issues here:
So even if you deal with the first two issues you still have the last one. How practical a solution is would depend on how often the video provider changes their EXT-X-KEY encryption strategy and how easy those changes are to determine. |
|
# 1 is straightforward; # 2 is tricky but using cookies should work around it; # 3 is the real meat. |
|
Regarding #2, perhaps cookies are required to get the m3u8 playlist URL but requesting the playlist, segments and key does not require cookies. The auth is passed in request params. Regarding #3, this requires extracting the client side JavaScript code and porting it to Python. I have extracted the code and verified that it does indeed decrypt the key URI. There's about 50 lines of code to port to Python. The encryption used for key URIs is AES-128. I don't have much motivation to implement this as I have an acceptable workaround. I only ever download individual videos. For this I'm already looking in Developer Tools to grab the m3u8 playlist URL, it's not much trouble to also grab the decrypted key URI from there at the same time and then download with streamlink's segment key URI override feature: |
Even if you don't intend to continue, sharing your findings in detail could be helpfull to devs trying to fix this |
Full source is available. You just need to locate the hotmart code in your browser Developer Tools where the decryption is happening (for example, by searching for the custom key URL protocol string "chave://" or "key://") and set a breakpoint there. |
|
I've been using yt-dlp to consume some Teachable content (cantrill for AWS certs is great) on my phone at high speed (with skip silence feature, I love lectures now, I rec the AntennaPod app), and I've come back to discover it not working. So I'm giving my shot at seeing if I can fix it. Right now, I've been able to do what @svenop5 says. That is, downloading something manually by grabbing stuff via Web Dev Tools after whatever of their necessary JS runs in the browser. Firstly, some of his content is public on his site, that you can view (as a trial I guess) without buying the course. I'll use it as an example and for tests. This is a good example link. Teachable SiteNow, when you visit that link in e.g. curl, it won't provide the hotmart iframe. Instead, you get this: Yes, the html download via curl did have that shoddy indentation. No clue where that It looks like those two outer divs are remnants from Teachable embedding Wistia, but they just replaced the player itself ergo the inner hotmart div. Now, when I visit the page in an actual browser, it injects an iframe into that IFrame on the Teachable SiteThat inner iframe looks like this: That link should be expired when you use it, so you'll have to generate a new one yourself by visiting the Teachable page. Things to note:
One interesting side note: there is only one flavor of link, but I've encountered two different video content sources, which I'll get to. One has an HLS playlist link of qualities you can plug directly into streamlink, and another has the seperate key link you need to need to give to streamlink separately. The latter actually causes Firefox to think it's a tracker and so it actually gets blocked, but both work when you visit the iframe src directly. I've been able to download both via streamlink. The former was what I initially encountered, but now I'm only getting the latter. I'm not sure if we'll want to support both or just the latter (i.e. the separate key link one). Hotmart Player SiteSo let's visit that src link now. Again, you'll need to grab your own, so I'm not going to put it in here again. Now open the Network section of the Web Dev tools -- preferably open it before you go to the src link, so you can see all requests without needing to refresh the page. There's some html, js, css, a favicon, and a thumbnail jpg (maybe it'll be fun to keep this later, too, like the youtube extractor). Also for me, several blocked POST calls for data collection, though unfortunately one goes through. Besides that, the one we care about. The easy type has the following interesting requests (I was only able to get this for a different paid video, titled "Public Introduction (Release v1)", which sounds like it's mistakenly private): The same video when loaded under the hard type looks like this: All of these when queried are expired, so it should be safe to put them here. Though that token is the same between the two -- hopefully that doesn't leak anything from my account. I'll try relogging later. Or perhaps it's actually an app id. I've also saved all their contents, but I won't upload them now just in case -- though the response from HLS Segment Key URI is the same between the two, though not between videos. Encrypted KeysNow, I was able to download these using streamlink, as I said before. It's a different sha1sum from the old video that used to be there (which I believe just served whatever video was uploaded, since those even had year old timestamps from when they were made), but now are just HLS streams. As @svenop5 said, the hard version needs The Which is the url of the key which yt-dlp and streamlink are able to get and decode the videos. Again, that result of that key uri is different for different sizes of video, but is the same for easy/hard mode for the same size of the video. Also, I noticed that the files downloaded are the same between streamlink and yt-dlp, regardless of easy/hard, for the same video size, as long as you have ConclusionThat should be enough information for anybody to easily write e.g. a selenium bot to get all this information to automate a download (I quickly did something similar to this in the past for something unsupported by yt-dlp, but with easy download mp4 source files, this was for logging in and getting a list of videos, instead of not needing to write a decryptor). Otherwise, this is what we need to do:
I might try and do these things myself. Only thing is, I've already spent a few hours fiddling around, figuring this out, and documenting this. And I don't know much about the yt-dlp codebase and contributing to it the best -- though my day job is Python. I'm mostly just trying to get a semi-quick and dirty way of fixing this, then maybe one of you can finish and fix it. IDK. It's been cool learning about all this, though. (And I do need these videos downloaded to do training for my job efficiently.) Last thought: I wonder if Hotmart is just using someone else's HLS implementation, in particular, their encryption code, so I wonder if there is another extractor already written which shared the |
Google didn't think so. Probably this thread and the earlier threads at yt-dl and streamlink are the only hits now. |
Sure. Changing the class-names would have meant changing JS and CSS that used those names; perhaps they didn't even have to change the associated CSS styles. |
|
Alright, I've figured out how to do #1 (my own #1, not the other set of 3 hotmart stuff). Take The other one is similar, but it has the Then, I found one API call that gets made. So run Here's the thing. Uh. This is actually very insecure. If anyone gives you the attachment id for any nonpublic video, plugging it into the API will give you the signature data and stuff, example of a public video shown above. But for the private video. It doesn't need the user id or anything. Boom, very public. The only thing that hides anything is the fact that a page won't load that div with the Anyway. Formatting the above: So all you need (in python fstring formatting) is That's step one, really the whole Teachable extractor (excluding metadata instructions and Hotmart instructions) complete. In summary:
So now someone just needs to code that. If it ends up being me, I'll probably do it after I figure out the later steps. I'd also like to see someone can confirm the above summary pattern we can just straight forwardly code for other Teachable sites. Does the same |
|
The beta version still appears to be working well with Teachable. However, the main branch doesn't appear to the have the fix. Would it be possible to get it added? |
|
@hubdows Yes, but I've been busy without much time to program in my free time. (Meanwhile there's plenty of free phone time, which isn't usable for programming 😢) If anyone else wants to take up the torch, they may. I just did it because I can't bear consuming lectures without skip silence, and I gotta be able to do it on my phone, often without consuming data, preferably in a better app than the web. But the current code isn't suitable for merging yet. It's not on master yet, because of some refactor I hadn't finished spending time understanding. And tbh I don't wholly understand some of the extractor stuff, or at least, the way I should be going about doing it. There's no architecture manual or anything. Besides bringing it up to date, it doesn't support the modern course list page. It supports the old one, which usecase uses at the very least. But not the main Teachable site afaik. It also doesn't support multiple videos on a page properly (and I don't know the best way to support this without it being wonky, which was the case earlier), nor is username and password signin working fully. Difference between Teachable SSO and same site signin. Cookies from browser also doesn't work, but I don't know if that's a general yt-dlp bug or not, nor if it's solved on recent versions or not. Testing isn't good or ready either. I'm sure I'm missing stuff. Ummm... There was definitely some other bug that occasionally popped up, something to do with title or something not showing up in attributes. I can't remember. There is a Discord. Might join that to ask questions when I next get around to trying to fix shit. |
Same issue, only the first video downloads fine, on the other ones there's an error: [debug] [generic] Extracting URL: https://
[generic] 8971587: Requesting header
WARNING: [generic] Falling back on generic information extractor.
[generic] 8971587: Downloading webpage
[generic] 8971587: Extracting information
[debug] Looking for video embeds
[debug] [Teachable] Extracting URL: teachable:https://
[Teachable] 8971587: Downloading webpage
ERROR: [Teachable] 8971587: Lecture contents locked. Use --cookies, --cookies-from-browser, --username and --password, or --netrc (teachable) to provide account credentials
File "/mnt/c/Users/Syed Umar/Downloads/Compressed/yt-dlp-teachable-fix-add-hotmart/yt-dlp-teachable-fix-add-hotmart/yt_dlp/extractor/common.py", line 642, in extract
ie_result = self._real_extract(url)
File "/mnt/c/Users/Syed Umar/Downloads/Compressed/yt-dlp-teachable-fix-add-hotmart/yt-dlp-teachable-fix-add-hotmart/yt_dlp/extractor/teachable.py", line 200, in _real_extract
hotmart_url = self._create_hotmart_url(webpage, video_id, site)
File "/mnt/c/Users/Syed Umar/Downloads/Compressed/yt-dlp-teachable-fix-add-hotmart/yt-dlp-teachable-fix-add-hotmart/yt_dlp/extractor/teachable.py", line 166, in _create_hotmart_url
self.raise_login_required('Lecture contents locked')
File "/mnt/c/Users/Syed Umar/Downloads/Compressed/yt-dlp-teachable-fix-add-hotmart/yt-dlp-teachable-fix-add-hotmart/yt_dlp/extractor/common.py", line 1100, in raise_login_required
raise ExtractorError(msg, expected=True)It seems it can't use the session cookies file for other videos. |
This comment was marked as spam.
This comment was marked as spam.
|
Until this gets fixed I have written my own script for downloading courses |
This comment was marked as outdated.
This comment was marked as outdated.
pukkandan
removed
the
patch-available
|
Thanks to everybody trying to help with this problem. Unfortunately my knowledge about this stuff is limited. I basically read the IDs needed for every video from the plain html and then downloaded them one by one.
Details
The script will create a folder with the name you provided and in there have a folder for each chapter, while prefixing every video with a number in the order they were read from the page. #!/bin/bash
# command to run: ./thisScript.sh <file with your cookies> <url of the first video in the course> <name of the course>
# example command: ./thisScript.sh cookies.txt https://domainOfYourCourse.com/courses/1234567/lectures/7654321 "My course"
# this also worked for me when the course has a string in the URL instead of a number
mkdir "$3"
# the base url is needed to add ids one by one later -> https://domainOfYourCourse.com/courses/1234567/
baseUrl=$(echo "$2" | awk -F "lectures" '{print $1}')
# sending a curl request to the site with your cookies, grepping all available video IDs and removing duplicates from the list
listOfVideoIds=$(curl -b "$1" "$2" 2>/dev/null | grep -Po "(?<=data-lecture-id=\")\d*" | awk '!visited[$0]++') #
# adding a complete link for every video to a text file to. With this the "-a" option of yt-dlp can be used, allowing to give each video in the list a unique number
for id in $listOfVideoIds
do
echo "$baseUrl"lectures/"$id" >> allVideos.txt
done
# the actual downloading
yt-dlp-teachable-fix-add-hotmart/yt-dlp.sh --cookies "$1" --verbose -N 32 -o "$3/%(chapter_number)s-%(chapter)s/%(autonumber)03d-%(title)s.%(ext)s" -a allVideos.txt
rm allVideos.txt |
This comment was marked as spam.
This comment was marked as spam.
|
Just chiming in to say this workaround is still working: You need:
And run @Green0Photon's hotfix: |
This comment was marked as off-topic.
This comment was marked as off-topic.
|
@g33khubber Have a look at my curl command in this response: |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
|
Ask support for third-party tools in their repo |
|
Re #3564 (comment): While not yet standardized, the best way to handle this seems to be to define a custom protocol handler. See AbemaTVIE |
|
Great stuff, @Green0Photon! I've been trying to download some Hotmart videos from Teachable with your fork: Getting this error (3rd line): |
|
@bruhcephalus the changes made by @Green0Photon are being integrated here: #7650 Could you please test if this fork works and give some feedback? : https://github.com/Abdess/yt-dlp/tree/teachable-fix-add-hotmart Any feedback is welcome to find out if the changes are working properly. The only remaining issue is the login which causes a 403 error, which is due to Cloudflare's security. :) |
|
@Abdess Your fork worked for me 😃 |
|
@Abdess how can I run your fork on windows? Do I need to install bash? |
|
@Abdess -- Looking to see how to build it as well to work on windows please :). EDIT --- I downloaded and installed cygwin64 and run the shell with it. I was able to download 200 courses successfully from the teachable site. |
|
hello greetings to all and please could someone help me how to use the script from https://github.com/Abdess/yt-dlp/tree/teachable-fix-add-hotmart to download courses from hotmart, thank you very much |
|
@Abdess 's repo doesn't work for me. Is this the Cloudflare issue mentioned above? |
and others
Checklist
Region
No response
Description
Just a few days ago I was able to download videos from the 'member-only' site, but not now. For every course link, yt-dlp issues the error 'unable to find video URL'. Please help.
Verbose log
The text was updated successfully, but these errors were encountered: