Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add login to Newgrounds extractor #9334

Closed
wants to merge 3 commits into from
Closed

Add login to Newgrounds extractor #9334

wants to merge 3 commits into from

Conversation

mrmedieval
Copy link

IMPORTANT: PRs without the template will be CLOSED

Description of your pull request and other information

Add a login function to Newgrounds extractor to fix issues with adult media.

Fixes #9324

Template

Before submitting a pull request make sure you have:

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:

  • [x ] I am the original author of this code and I am willing to release it under Unlicense
  • I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

@bashonly bashonly added site-bug Issue with a specific website site-enhancement Feature request for some website labels Mar 1, 2024
@bashonly
Copy link
Member

bashonly commented Mar 1, 2024

Thanks for the PR. Before reviewing, I have some questions:

@mrmedieval
Copy link
Author

  1. Passing cookies works on my pull request, but not on the current version, because of [newgrounds] Cookies causes video to download profile page #6488
  2. Removing media extractor fixes [newgrounds] Cookies causes video to download profile page #6488. I believe this issue has to do with the media extractor's regex. Videos still download with the code removed.
  3. Without cookies, or with an invalid login, the download will fail with "Unable to download webpage: HTTP Error 401: Unauthorized".

@Grub4K
Copy link
Member

Grub4K commented Mar 2, 2024

While trying to test this extractor I found that some of the old tests are no longer up to date, and went down deeper and deeper the rabbithole. I went on to fix the tests first, which shows us that the removed code is required to download audio and flash content.

The reason why the download with login was failing is that the "url":\s*"[^"]+" matches the json for the user info. Specifying a more locked down embedController\(\[{ before manages to only capture the audio. After a lot of experimentation I found that this is easier than to parse the whole thing as JSON since that contains callback code and inline function invocations.

  1. Without cookies, or with an invalid login, the download will fail with "Unable to download webpage: HTTP Error 401: Unauthorized".

I think the question was more about what response you get when failing to log in. I eventually managed to get a JSON response by passing the X-Requested-With headers as gets done by the media api call.

See master...Grub4K:yt-dlp:fix/newgrounds for what I eventually ended up with

@mrmedieval
Copy link
Author

See master...Grub4K:yt-dlp:fix/newgrounds for what I eventually ended up with

I am very impressed with that solution. I'm still a novice, so it took a lot of trial and error to find out what I did about the extractor. I'm very happy to see the issues with the extractor fixed, and I hope some of what I did was helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
site-bug Issue with a specific website site-enhancement Feature request for some website
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Newgrounds] Error 401, Adult-rated video NSFW
3 participants