Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`extract_info` freezing after a lot of usage. #12815

Closed
Phxntxm opened this issue Apr 22, 2017 · 7 comments
Closed

`extract_info` freezing after a lot of usage. #12815

Phxntxm opened this issue Apr 22, 2017 · 7 comments
Labels

Comments

@Phxntxm
Copy link

@Phxntxm Phxntxm commented Apr 22, 2017

Please follow the guide below

  • You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
  • Put an x into all the boxes [ ] relevant to your issue (like that [x])
  • Use Preview tab to see how your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2017.04.17. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

  • [ x ] I've verified and I assure that I'm running youtube-dl 2017.04.17

Before submitting an issue make sure you have:

  • [ x ] At least skimmed through README and most notably FAQ and BUGS sections
  • [ x ] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

  • [ x ] Bug report (encountered problems with youtube-dl)
  • Site support request (request for adding support for a new site)
  • Feature request (request for a new functionality)
  • Question
  • Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue


If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

$ youtube-dl -v <your command line>
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2017.04.17
[debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
[debug] Proxy map: {}
...
<end of log>

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.


Description of your issue, suggested solution and other information

Explanation of your issue in arbitrary form goes here. Please make sure the description is worded well enough to be understood. Provide as much context and examples as possible.
If work on your issue requires account credentials please provide them or explain how one can obtain them.

I understand as I start this, that I do not have the relevant info in order to solve this issue, however let me explain because I need help getting that relevant info.

Here's my code I'm using that causes the issue: https://github.com/Phxntxm/Bonfire/blob/master/cogs/voice_utilities/downloader.py#L82

Basically, the issue is after lots of usage (what I'm using this for, about 20-30 calls are made every minute or so), this will seem to just freeze right here. If I add a print call before and after that coroutine call, the one before prints, the one after does not. It does not error, it simply seems to be an infinite call that never finishes, and that is where I'm stuck, trying to figure out why this happens.

I probably need a debugger, but as far as I can tell pdb does not handle coroutines so I can't use it. I also can't replicate this from command line, because it's only replicateable after a ton of usage in the same process. Additionally, no matter what I query this is the same result, if I query something that shouldn't work, that should, etc.

This is either an issue with youtube-dl or asyncio, however at this point I'm leaning towards youtube-dl, and I need help figuring out exactly where since there's no output and it will never continue past that point no matter how long I wait.

The only additional information I can probably add at this point is that I've asked a few people about this, and have heard that they've experienced it too, and believe it is location based. With this in mind, this issue did only start occurring (with the same exact relevant code), when I moved this from a server in the US to one in Paris.

@yan12125
Copy link
Collaborator

@yan12125 yan12125 commented Apr 22, 2017

Could you insert codes in extract_info and see what's the exact line that hangs? It's unlikely that extract_info runs forever, but it may run for a very long time if the network is unstable or the target URL contains a long playlist. Removing 'quiet': True, and add 'verbose': True, can help debugging, too.

@Phxntxm
Copy link
Author

@Phxntxm Phxntxm commented Apr 22, 2017

Thank you, I really should have been able to figure that out but I think this issue was frustrating me so much I wasn't thinking right. I've done both of these suggestions and will wait till the next time it stops working.

@Phxntxm
Copy link
Author

@Phxntxm Phxntxm commented Apr 22, 2017

Okay. This printing helped, here's what happened (in a pastebin link because it's pretty large):
https://pastebin.com/raw/hcvtfDJh
Here's the method extract_info again just for reference:
https://github.com/rg3/youtube-dl/blob/master/youtube_dl/YoutubeDL.py#L734

I printed ies after line 748, it's None
I printed ie after line 754, it's None
I printed ie_result after line 760 and that's the large dictionary of info in the pastebin.
Then where it says "We made it to process_ie_result" is right before line 771
I then have a "We made it past process_ie_result" after this return (changed it to a variable, printed, then returned obviously)

This is where it got stuck, when calling process_ie_result and you can see all the info at the end. For some odd reason, ffmpeg is waiting for input...this is from a script, I can't give it input, this is not desired....this then causes it to not function again because it is endlessly waiting for input. So the main issue here is why is this waiting for input? That should not happen.

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Apr 22, 2017

ffmpeg does not wait for input, it downloads what you asked it to download, i.e. Lchg2dGUbO0 that is a live stream 24/7. That's why it never ends.

@dstftw dstftw closed this Apr 22, 2017
@dstftw dstftw added invalid and removed clarification-needed labels Apr 22, 2017
@Phxntxm
Copy link
Author

@Phxntxm Phxntxm commented Apr 22, 2017

How do I disallow this then.

Whether or not it's ffmpeg waiting for input, this is still an issue because input is being requested by something before it can continue. Closing this with a very generic answer is a little bit annoying because this in no way resolved the issue. It may lead towards the right direction, but no issue has been solved.

This is not me downloading these videos, my bot is used by about 8k people at this time, and if youtube_dl does not disallow livestreams by default I need to disallow this somehow. How do I do so? I'm not finding an option for it as I look.

@dstftw
Copy link
Collaborator

@dstftw dstftw commented Apr 22, 2017

--match-filter '!is_live'.

@Phxntxm
Copy link
Author

@Phxntxm Phxntxm commented Apr 22, 2017

Perfect, that worked thank you for the help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.