Adding source instead of URL

[Alavar]

Is it possible to enter the source code and have the extractor continue (additional requests, etc) from there?

[jaimeMF]

The issue is missing some info, explain with detail what do you want to do.

What I understand is that you want to pass the webpage source code and youtube-dl should process it and extract all the required info. Am I right?

[Alavar]

You are exactly right.
I want to use cURL to receive the page source and then send it to youtube-dl.

Is this possible?

[Alavar]

Thanks for adding the label.

Is there a simple way for me to do this by myself before a parameter is added (or if a parameter even ever is added)?

[jaimeMF]

See #4551 for the reasons why we won't add support for this option. Relevant comments: #4551 (comment) and #4551 (comment).

[Alavar]

The reason to why it wasn't added was because multiple requests were made.
For example, I know that the DASH manifest needs to be requested as well, however the URL of the DASH manifest is retrieved from the main page (the first request).

In other words, I only input the source of the first page and youtube-dl will go from there (retrieving any additional files like the DASH manifest).

The reason to why I want to input the source instead of the URL is that I need to log the page source.

[jaimeMF]

• It would unnecessarily complicate the code, we would need to modify the extractors so that they work both by giving an url and the webpage. (I gess that you are problably only interested on youtube, but we support more sites).
• What happens if we decide to use another page for getting the info? You would get error messages and we would need to explain what has changed, and this is internal behaviour so we shouldn't have to.
• You would need to specify an extractor, which is another source of bug reports because people would use the wrong extractor and we sometimes split extractors or add a special one for handling some urls.

The reason to why I want to input the source instead of the URL is that I need to log the page source.

Use --write-pages.

[Alavar]

I'll have to look more into --write-pages, thanks.

It would unnecessarily complicate the code, we would need to modify the extractors so that they work both by giving an url and the webpage.

This won't be a problem as users could specify both the URL and the source in order to have the software use the right extractor.

I was also assuming that the first request was sent in a globally and depending on the site the further requests would be made in the extractor. If this is the case then a simple if statement checking for the parameter would solve the trick, wouldn't it?

[jaimeMF]

I was also assuming that the first request was sent in a globally and depending on the site the further requests would be made in the extractor

No, we iterate over all the extractors and check if the url can be handled, the webpage requests are made by the extractors. Although for urls that don't have a matching extractor, the GenericIE will look into webpage source and try to find embedded videos.

[Alavar]

That explains why it would be complicated to add.

Is there also an easy way of knowing if a YouTube video had an encrypted signature or not?

[jaimeMF]

Is there also an easy way of knowing if a YouTube video had an encrypted signature or not?

I think that looking for \u0026s= in the webpage would handle most cases. But this is off-topic.

[Alavar]

I think that looking for \u0026s= in the webpage would handle most cases. But this is off-topic.

But is this possible through youtube-dl?
I am currently outputting the JSON (with -j), but I'm not able to find any information about the signature in the output.

[jaimeMF]

There's no explicit info about it, since you don't need it to use the json output (handling the signatures is youtube-dl's job). You can try looking for player_url in the formats, I think that for videos with encrypted signatures they are something like: `"player_url": "//s.ytimg.com/yts/jsbin/html5player-en_US-vflCbZ69_/html5player.js" (for http://www.youtube.com/watch?v=UxxajLWwzqY).

It could probably added, but I don't see why it would be useful. Feel free to open a new issue of pull reuest

[Alavar]

You can try looking for player_url in the formats, I think that for videos with encrypted signatures they are something like: `"player_url": "//s.ytimg.com/yts/jsbin/html5player-en_US-vflCbZ69_/html5player.js"

I just tried this and there wasn't a difference between an encrypted signature and a regular one.

The parameter --dump-pages would help me tremendously, but when I run the command I only get a lot of randomly repeating alphanumeric characters starting with slashes.

Example:
/IHl0Y2ZnLmQoKVtrXSA6IG87fSxzZXQ6IGZ1bmN0aW9uKCkge3ZhciBhID0gYXJndW1lbnRzO2lmIChhLmxlbmd0aCA+IDEpIHt5dGNmZy5kKClbYVswXV0gPSBhWzFdO30gZWxzZSB7Zm9yICh2YXIgayBpbiBhWzBdKSB7eXRjZmcuZCgpW2tdID0gYVswXVtrXTt9fX19Ozwvc2NyaXB0PiAgPHNjcmlwdD4KICAgIHl0Y2ZnLnNldCgiRVhQX0xBQ1RfTU9VU0UiLCBmYWxzZSk7CiAgICB5dGNmZy5zZXQoIkVYUF9MQUNUX1JFU0laRSIsIGZhbHNlKTsKICAgIHl0Y2ZnLnNldCgiRVhQX0xBQ1RfU0NST0xMIiwgZmFsc2UpOwogIDwvc2NyaXB0PgogIDxzY3JpcHQ+eXRjZmcuc2V0KCJMQUNUIiwgbnVsbCk7PC9zY3JpcHQ+CiAgCgoKCgogIDxzY3JpcHQ+CiAgICAgICAgKGZ1bmN0aW9uKCl7dmFyIGI9e2Y6ImNvbnRlbnQtc25hcC13aWR0aC0xIixoOiJjb250ZW50LXNuYXAtd2lkdGgtMiIsajoiY29udGVudC1zbmFwLXdpZHRoLTMiLGM6ImNvbnRlbnQtc25hcC13aWR0aC1za2lubnktbW9kZSJ9O2Z1bmN0aW9uIGcoKXt2YXIgYT1bXSxjO2ZvcihjIGluIGIpYS5wdXNoKGJbY10pO3JldHVybiBhfWZ1bmN0aW9uIGgoYSl7dmFyIGM9ZygpLmNvbmNhdChbImd1aWRlLXBpbm5lZCIsInNob3ctZ3VpZGUiXSksZT1jLmxlbmd0aCxmPVtdO2EucmVwbGFjZSgvXFMrL2csZnVuY3Rpb24oYSl7Zm9yKHZhciBkPTA7ZDxlO2QrKylpZihhPT1jW2RdKXJldHVybjtmLnB1c2goYSl9KTtyZXR1cm4gZn07ZnVuY3Rpb24gbChhLGMsZSl7dmFyIGY9ZG9jdW1lbnQuZ2V0RWxlbWVudHNCeVRhZ05hbWUoImh0bWwiKVswXSxrPWgoZi5jbGFzc05hbWUpO2EmJjEyNTE8PSh3aW5kb3cuaW5uZXJXaWR0aHx8ZG9jdW1lbnQuZG9jdW1lbnRFbGVtZW50LmNsaWVudFdpZHRoKSYmKGsucHVzaCgiZ3VpZGUtcGlubmVkIiksYyYmay5wdXNoKCJzaG93LWd1aWRlIikpO2lmKGUpe2U9d2luZG93LmlubmVyV2lkdGh8fGRvY3VtZW50LmRvY3VtZW50RWxlbWVudC5jbGllbnRXaWR0aDt2YXIgZD1lLTIxLTUwOzEyNTE8PSh3aW5kb3cuaW5uZXJXaWR0aHx8ZG9jdW1lbnQuZG9jdW1lbnRFbGVtZW50LmNsaWVudFdpZHRoKSYmYSYmYyYmKGQtPTIzMCk7ay5wdXNoKDY0MD49ZT8iY29udGVudC1zbmFwLXdpZHRoLXNraW5ueS1tb2RlIjoxMjYyPD1kPyJjb250ZW50LXNuYXAtd2lkdGgtMyI6MTA1Njw9ZD8iY29udGVudC1zbmFwLXdpZHRoLTIiOiJjb250ZW50LXNuYXAtd2lkdGgtMSIpfWYuY2xhc3NOYW1lPQprLmpvaW4oIiAiKX12YXIgbT1bInl0Iiwid3d3IiwibWFzdGhlYWQiLCJzaXppbmciLCJydW5CZWZvcmVCb2R5SXNSZWFkeSJdLG49dGhpczttWzBdaW4gbnx8IW4uZXhlY1NjcmlwdHx8bi5leGVjU2NyaXB0KCJ2YXIgIittWzBdKTtmb3IodmFyIHA7bS5sZW5ndGgmJihwPW0uc2hpZnQoKSk7KW0ubGVuZ3RofHx2b2lkIDA9PT1sP25bcF0

[Alavar]

I opened a new issue as this is getting off-topic.
#5781