Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upGitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
USA Networks Issues #22105
USA Networks Issues #22105
Comments
|
Same issue - gets stuck on "ERROR: An extractor error has occurred. (caused by KeyError('data-mpx-guid',));" |
|
Is there a way to manually download from site? Dumping m3u8 from playback is erroring out at about 30 seconds. Something about ads |
|
I'm seeing the same issue and I am running the program under Ubuntu Linux, not Windows, but I get nearly identical errors. It worked last week but not last night, and if there's a way to manually d/l without it erroring out I sure haven't been able to figure out how. It plays fine in a browser but any method I know of to d/l seems totally broken. |
|
F4ckLogins, I could not find a valid m3u8. The m3u8 I did find did not contain any ts video links or embedded link to the real m3u8 |
|
Hi everyone, I found a work around method that allows downloading the video with the generic Youtube-dl extractor. I am not sure if the method would violate the forum rules or not. It involves modifying the m3u8 file, then using a web server, like IIS, to download the video. Let me know if anyone is interested. |
|
I just uninstalled the youtube-dl 2019.08.13 and then installed youtube-dl 2019.08.02. I still get the same error. Must not version related. See below. [debug] System config: [] During handling of the above exception, another exception occurred: Traceback (most recent call last):
|
|
How did you find the m3u? I tried to locate it in the Firefox console, but could only locate the advertisement. |
|
I used Video DownloadHelper (https://www.downloadhelper.net/) to get the m3u8 link, then just paste the link into the browser's address bar and save the m3u8. Click Details on the video you want and page will be displayed with the correct m3u8 link in the section called mediaManifest. BTW, the Video DownloadHelper does not correctly download the video either. Let me know if you need any more help. |
|
So I compared the current video page to the old video page from archive.org wayback machine. It looks like they moved 'data-mpx-guid' from class="player-wrapper" to class="video-player-wrapper section-player". The youtube-dl extractor is searching for 'data-usa-tve-player-container', but 'data-mpx-guid' is no longer in that class. |
|
Sounds like you figured out the issue. Do you have any idea how to fix the youtube-dl code so it works correctly? I have no clue. |
I'd be interested to know your technique, though I don't get the part about needing a web server! |
In looking at the source code, I see this: But I see no reference here or anywhere else to "player-wrapper". I do see references to both "data-usa-tve-player-container" and "data-mpx-guid" but can't figure out what would need to be changed to reference these in the new location (class?). Then again, I am not a programmer, so I'm kind of stumbling in the dark here. Any ideas? |
|
Hi everyone, here is a workaround for downloading video from USA Networks using the youtube-dl generic extractor. Step 1: Find and download m3u8 file for the video that you wish to download. Copied from above: I used Video DownloadHelper (https://www.downloadhelper.net/) to get the m3u8 link, then just paste the link into the browser's address bar and save the m3u8. Click Details on the video you want and page will be displayed with the correct m3u8 link in the section called mediaManifest. BTW, the Video DownloadHelper does not correctly download the video either. Step 2: Open the m3u8 file any text editor. Delete every line beginning with #EXTINF that has /video/ in the URL, stop deleting lines when you find the next #EXT-X-DISCONTINUITY. Do not delete the next #EXT-X-DISCONTINUITY. Repeat for the whole m3u8 file. Delete every line beginning with #EXTINF that has /ads/ in the URL, stop deleting lines when you find the next #EXT-X-DISCONTINUITY. Do not delete the next #EXT-X-DISCONTINUITY. Repeat for the whole m3u8 file. Keep the sections with #EXTINF that has /tve-adstitch/ in the URL. Step 3: With IIS installed move the modified m3u8 file to C:\inetpub\wwwroot. Now, run youtube-dl.exe --hls-prefer-native -v http://localhost/yourfile.m3u8 to download the video. I choose IIS since it is included in Windows 10 Professional. I am not sure if IIS is available in Windows 10 Home, but if IIS is available, the following web site give details on how to install IIS. https://helpdeskgeek.com/windows-10/install-and-setup-a-website-in-iis-on-windows-10/ IIS (or whichever web server chosen) must have the m3u8 mime type configured properly. For IIS see https://forums.iis.net/post/1973054.aspx and https://docs.microsoft.com/en-us/iis/configuration/system.webserver/staticcontent/mimemap. On the https://docs.microsoft.com/en-us/iis/configuration/system.webserver/staticcontent/mimemap only follow the instructions up to the Configure section. Let me know if you need any more help. |
The source code of the web page the video is on. |
|
Does the project staff plan to address this issue? I do not understand why the issue has not been acknowledged by the project staff. It has been 8 days since the issue has been open. |
|
@netuser0, nice write-up, but you do not need a webserver to complete the download. You can just use ffmpeg directly.
|
Thanks for the tip. I did not know that ffmpeg could do that. Should have known, since ffmpeg does a lot of cool things. |
I did use ffmpeg as you suggested; however, ffmpeg did not process the video correctly. When ffmpeg finished downloading the video, it only processed 12 minutes of the video; however, the video is supposed to be 42 minutes. It gave a lot of errors like: [mp4 @ 000001ca694237c0] Non-monotonous DTS in output stream 0:1; previous: 20225562, current: 20184064; changing to 20225563. This may result in incorrect timestamps in the output file. The native youtube-dl downloader did process the video correctly. So, at least for me, using youtube-dl and the web server works best. |
|
No need to have or install a web server, If you have Dropbox installed, save the *.m3u8 file in your Dropbox folder. Right click the file and select "Copy Dropbox link" (the Dropbox servers already have the MIME type for *.m3u8 configured correctly). Paste in the URL, to your YouTube-DL command-line. Change ?dl=0 to ?dl=1 Run YouTube-DL as usual. |
i tried but te file is only 11m lenght.. its a big file but playing only apears 11minutes. regards |
|
netuser0 thanks for the writeup, your method does appear to work but the shortcut using ffmpeg didn't for me, and I didn't try Dropbox as I don't use it. I found that under Linux you can use wget -O temporary.m3u8 -nc "filename in quotation marks.m3u8" to get the m3u8 file (saved to "temporary.m3u8" in this example), no need to open it in a browser. There should be some way to create a script to remove the lines in the m3u8 file so you don't have to do it manually, if they were just single lines you could use sed I think, but the lines come in pairs with the first line being "#EXTINF:" (followed by one or more numbers) and then the following line is the url and I'm assuming you have to remove both. I'm not well-versed enough in the ways of Linux to know how to do that off the top of my head, but theoretically you could create a script that would download the m3u8 (using wget), and discard the lines containing the patterns and the EXTINF lines that precede them. EDIT: I could not figure out how to make sed work but here is what for an actual programmer would have been a quick and dirty bash script, for me it kept me up half the night because I had to look up how to do every single thing and still kept getting errors. It assumes you have used the wget command I mentioned above and now have a temporary.m3u8 file. I'm sure there far are better ways to do this, but this example bash script looks for lines that begin with "#EXTINF:", then looks to see if the following line begins with "httos://" and if it does then it looks to see if that line contains either "/video/" or "/ads/", if all those are true then it drops those two lines, otherwise it echoes everything else to the screen, so use redirection to send the output to another file. So for example, ./scriptname.sh > newtemp.m3u8 and then use newtemp.m3u8 on the web server. Be sure to make the script executable or you'll get a permission denied error. No guarantees or warranties, this is an example script only and may or may not work as intended - I am not a programmer and don't know what I'm doing!!! |
Did you modify the m3u8 file per Step 2 of the workaround? If you do not remove the lines that contains "/video/" and "/ads/," then the video will be corrupted. |
|
tv21, Thanks posting your script. I would try the script, but I do not use Linux. I have no clue how to script something equivalent in Windows. If someone wants to make something to do as your Linux script does for Windows, then that certainly would be helpful. |
|
Does anyone have any idea how to draw the attention of project coders to this issue? |
|
netuser0, I don't use Windows so we are not really on the same platform. But, I was under the impression that Windows could run bash scripts. I searched on "How to run a bash script in Windows" and got many results, for example https://www.howtogeek.com/261591/how-to-create-and-run-bash-shell-scripts-on-windows-10/ Depending on what version of Windows you are running you may need to install additional software such as Cygwin but I was under the impression that Bash is available natively in Windows 10. Look it up in a search engine and you will likely find a method that works in your particular version. Do be aware of the difference in line ending characters between Windows and Linux, although I think if you copy text from this page in your browser it will use the correct line endings for Windows if you're running Windows, at least I'd hope so! |
|
@netuser0 and @tv21, python is a little more supported under both OSes. An equivalent python script attached. Run with something like the following: |
@ajfredrickson15, your python script worked perfectly. I did noticed it did leave multiple #EXT-X-DISCONTINUITY lines behind, but I guess it does not matter since it was exactly the same file size as the one I downloaded before when I deleted the line by hand. @tv21, I appreciate your help regarding running bash scripts in Windows, but, for me, ajfredrickson15's python script is much easier to implement. Thanks a lot to both of you. We now have a Linux and a Windows solution. |
|
@ajfredrickson15, could you write a python script for Fox? I looked the m3u8 file and it seem like the code for script would need more scripting than than just changing the "bad_words" per your usa script. If you do not want do this, then I completely understand, but I have to manually delete ads from the m3u8 on Fox videos when a script could easily do delete the ad lines. I just need to learn to script. :-) |
@ajfredrickson15, never mind I just figured out that youtube-dl will remove the Fox ads automatically. I just had to paste in the link to the m3u8. :-) |
|
ajfredrickson15 thanks but the bash script I wound up with after some more work yesterday eliminates the need to use Video DownloadHelper. The way it works is you need to open the Developer Tools console network tab and filter on m3u8 and begin playing the video, then copy the URL of first link that appears. Using that as an argument to the script, it will find the link by downloading that m3u8 (using wget -O temporary.m3u8 -nc -q "$1"), and searching the downloaded file for a line that begins with "#EXT-X-STREAM-INF:BANDWIDTH=" and also contains the string "RESOLUTION=1920x1080". When it finds that line it reads the next line, checks to make sure it begins with "https://", and then uses that as the target of another wget (after deleting the original temporary.m3u8 file). I believe (but haven fully tested) that this gives you the same file that you would get using Video DownloadHelper. You do have to be fairly quick about copying and pasting the link from the Developer Tools because it seems to expire very quickly (possibly one minute?). I am sure you could do this in Python also but I don't know Python well enough to modify your script. The lines I added (between "#!/bin/bash" and "linepre=''") were: (EDITED a couple times after posting to add error trap if the link has expired or the desired stream isn't present, and then fix it because I did it wrong, and also fix a couple other bits of bad code. I'm still not certain it's 100% correct.) This addition to the script has not been fully tested yet, and yes I know this is probably a horrible way to do it, using the case statement in this way, but when the only tool you have is a hammer... as I said I am not a programmer. If you would care to add something like this to your Python script I'd definitely try it. I just don't like the idea of having to leave Video DownloadHelper installed because I would only need it for this one thing, and I don't know if it is "phoning home" with any information about the sites I visit. With that problem solved the only other thing would be to figure out how to use the modified m3u8 without needing to use an external server. There are so many options available in youtube-dl that I'm wondering if any would allow it to read the m3u8 file directly, and to assume the correct m3u8 mime type if that is the issue. You would THINK there would be some way to pass it a link to a file rather than a URL; I know in a browser you can use a link that starts with file:// followed by a link but if you try that it gives an error message like... WARNING: Could not send HEAD request to file:///home/username/filename.m3u8: <urlopen error file:// scheme is explicitly disabled in youtube-dl for security reasons> Apparently this is disallowed because of the security vulnerability discussed at #8227 so I am not sure if there is any solution that doesn't involve using an outside server. Personally I kind of wish there were a command line option to override this check and allow using file:// in instances where you know what you are doing, but then again that would require actually reading every m3u8 file you create to make sure it doesn't contain anything malicious, and in any case I don't know if that would actually work. The code that disallows the use of file:// is in YoutubeDL.py (search for "scheme is explicitly disabled in youtube-dl for security reasons" to find it) and it looks like if you removed that check it might work, and if you read down a few lines there is a comment about using the --encoding option which (according to the help text) will "Force the specified encoding (experimental)" which MIGHT allow specifying the mime type in some way??? But, since this is all Python code I don't really understand it, so if anyone who does get Python wants to experiment, that might be a place to start. I can certainly understand, and very much appreciate the program developers wanting to protect us from security risks, but on the other hand I am really curious what would happen if that check for file:// weren't there, and we could tell the program to use the m3u8 directly without having to put it on an external server. |
|
@tv21, I think what you outlined in your last post is the ideal way to tackle the issue. All we need now is someone with python experience to work with you to come up with a solution. Thank you for all your efforts. |
|
This may or may not be helpful to anyone trying to make further progress: Tech Tip: Really Simple HTTP Server with Python If it doesn't work it may be because of the mime type thing, which I really don't understand at all. But my thought was that maybe in Python you can create a mini-http server on the fly, then it would be removed when the script ends, after youtube-dl is run? No idea, just a thought. |
|
I wonder if @dstftw (a major youtube-dl project contributor) has seen this thread? |
|
netuser0, you may as well give it up trying to get the attention of anyone that can fix this - I'm sure they've already seen this thread and my suspicion is that they don't care and you're just annoying them. There are far more longstanding bugs than this one, in some cases for which people have contributed the exact code needed to fix them, and still they remain unfixed (example see #11258 which affects every Comcast user). I suspect that youtube-dl is one of those pieces of software where the developers are probably burned out and tired of making changes, since sites are continuously changing the layout of their download pages and their download methods, and because it is free software and the developers aren't getting paid to do this, they feel no obligation or incentive to hurry up and fix something just because a user may want it. If you wait patiently they may eventually fix it, but if you go out of your way to annoy them, that could have the opposite effect of what you want. How would you feel if you had authored a software project, and every time you had a spare moment there was a whole pile of things that needed to be fixed, and people complaining that you weren't fixing them fast enough? You might just be inclined to go take a nice long walk in the sunshine and let all the ingrates fend for themselves! Just be happy that they do make fixes at their convenience, even if it takes a while! |
|
I'll repeat it 1001 time: for any work on such issues you must provide tv account credentials. Until you do that it neither possible for us to fix nor review and check ones contributions. |
Checklist
Verbose log
Description
Did USA Networks change something so Youtube-dl no longer works? See log below. It worked on 8/8/2019 with the youtube-dl version 2019.08.02.
Any ideas?