-
Notifications
You must be signed in to change notification settings - Fork 629
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RedgifsRipper: Implemented redgifs.com Ripper #1672
Conversation
- Using similar functionality found in the gfycat ripper the same was done for the redgifs ripper - A user's profile can be ripped, by retrieving all gfycats from their user api call - The categories and search ripper is also implemented but uses and incrementing start and count in the api url paramters as the cursor requires cookies (I believe) to work and this one just increments until there are no gfycats and returns error (there are no more gfycats) - Differentiated between getting profile and search urls using regex matching
Seems like there are build issues with other rippers, which is breaking maven Addresses - #1654 |
@cyian-1756 @felixaufreisen could you guys take a look at my PR and let me know, if there's anything I can fix. Thanks! |
Someone implemented your solution on a fork apparently. I compiled it and it work (but some files are corrupted but it seems it's redgif fault). |
@revanwolf Thanks, for checking it out, yeah it seems that person just added a RipUtils logging commit not sure what that does, aside from logging. Also, it seems that redgifs has updated some of their api urls, so I'll update my PR to reflect that, thanks for catching that. I'm still not sure who merges the pull requests into master, I'm assuming its @cyian-1756 as that's the name usually associated with the merged PRs, so hopefully he see's this and can merge it. |
I think it's for the reddit ripper so it can handle redgif link and he only copy/paste the Gfycat part. |
Some gfy links redirect to gifdeliverynetwork.com. Which makes RipMe to fail. |
@BestRestUp3300 Could you give me an example? |
Warning, NSFW: Reddit thread: Link: |
- Changed redgifs api URL to reflect updated change - Added check in getNextPage to see if page has any URLs else finishes rather than throwing error - Added gifdeliverynetwork.com as knwon host as it redirects to redgifs page - Added reddit compatibility by adding as known host in RipUtils reddit ripper - Unit test to cover above changes
I've updated the PR, if anyone wants to test it out. Also @BestRestUp3300 it looks like the gfy redirect issue should be another PR related more towards gfycat rather than redgifs. |
I've tested with the Reddit ripper and it seems to work most of the time but when the download fail it seem to be Redgifs fault. |
Yeah it seems that the redgifs api isn't very stable so it'll kick you out if you make too many calls, which I think is what ends up happening as they just start rejecting all the api calls we are making. (i.e. - you'll notice it reject one vid with a 403 error and then every video after will also give a 403). Not much we can do about that atm, until redgifs gets a more stable api we can scrape through. |
One solution is to download by default the webm version of the file (the file is usually lighter than the mp4). With Chrome i got" network error" randomly and a lot of retry on ripme(no 403 error only corrupt file). |
LGTM |
Is there a possibility of this not being added due to the API issue outlined above? |
@avekifes It isn't currently in the newest release https://github.com/ripmeapp/ripme/releases but it is in the updated master of the repo, so if you want to use it atm then you can clone the repo and create a jar file by running the application. @cyian-1756 When will you be adding a new release for the ripper? |
@borderline232 Regarding your pull request and implementation, I was unable to rip redgif links from a redditor's profile (Individual redgif links do work however). I built and tested the functionality on the current master branch. Have you come across this issue as well? |
@borderline232 What's the profile? Seems to be working fine for me. |
I'm experiencing this issue with this account (NSFW): https://www.reddit.com/user/supersecretself/ The url's direct to Did you build from the master or from borderline's branch? |
Doesn't work for me either. I wasn't even aware that older NSFW Gfycat stuff would automatically redirect to Redgifs. |
Thanks for validating the issue @avekifes @borderline232 is this something you'll be able to fix? There's actually a current issue related to this that's open. #1680 |
@CaptRussia So this issue is that gfycat redirects the links to gifdeliverynetwork.com which I have an individual redirect for (i.e. you rip using the gifdeliverynetowrk link and it downloads the associated redgifs video), the thing with ripping a reddit profile which contains gfycat links that redirect to gifdeliverynetwork requires somehow modifying either RipUtils.java file or the DownloadVideoThread.java file where I believe? it handles the redirects when downloading videos, not too sure tbh. I can take a look at it again to see if there's a fix for it. |
Funny thing is the old version 1.7.67 can download gifdeliverynetwork.com link without much issues but this version can't download Gfycat(change on the API/format IIRC) and of course RedGifs. |
Oh damn, you're right. Thanks I'll take a look at it. New fix available in #1695 depending on how long @cyian-1756 takes to merge it, it's best to clone my fork https://github.com/borderline232/ripme if you want the changes immediately. Let me know if there's any issues. |
It works and the only issues is their website is running on a potato. |
Thanks for rolling out a fix so quickly @borderline232 I just compiled your fork and am sadly getting errors. I tried the same URL (reddit profile) I posted before and managed to download only 2 videos. The logs seems to suggest it's still trying to download from redgifs. I assume it redirects to redgif and then redirects again to gifdeliverynetwork.com. Here's the output:
|
It's Redgif fault, their servers is very bad, try to open the link with your internet browser (Chrome/Firefox/etc) and it won't load the video without issues and the download will stop with "network error". |
Oh, you're correct! My bad for not checking the links. 😬 I also verified that the links on the reddit profile don't work so it's completely on their end. |
I've also noticed that sometimes the .mp4 redgifs URL doesn't work as well as the .webm URL in the redirect (which you've mentioned before @revanwolf ) so I'll probably just do a quick change on that and it should hopefully work? I'll update this if it doesn't. Edit: It's able to scrape a couple of the vids that still render that weren't able to show the mp4, however it still misses a couple. I'm thinking about just scraping the mobile.mp4 files which is a bit more consistent but yields a lower resolution video. Thoughts? Keep mp4Source (mildly reliable + high res) |
@borderline232 Is there a solution for those of us who only want the highest-quality version (.mp4)? If I can't get a video in its highest quality, then I don't want it at all. How does your current fork behave, anyway? Does it download lower-quality .webm when it can't grab the .mp4? |
Actually it download MP4Source by default. for the format choice maybe use the option for "prefer MP4 over Gif" to give choice? or add an option on rip.properties ? |
can we not add some way for it to fall back once it fails ? Try mp4source, Doesnt work ? do webm instead. Doesnt work ? do mobile instead. |
I agree @BestRestUp3300 I suggested that solution in #1695 Hopefully @borderline232 will be able to implement such a solution! |
I don't think it will really work with the current state of RedGifs servers, if it want to fail it will fail with every type of file (WebM/MP4/MP4-mobile), the only change is with smaller file it has a higher chance to download a valid file. Redgifs admin said they will upgrade their architecture to better handle the traffic, until then this site is not reliable. |
I would say this is not entirely true. Sometimes it will always fail on others except MP4 Mobile. No matter how many times you try or you open in browser. The gif just doesn't exist other than in one version. So making the site handle more traffic won't fix this issue. |
For the past few days only thcf1.redgifs.com work for HD version of the video. |
Category
This change is exactly one of the following (please change
[ ]
to[x]
) to indicate which:Description
COMMIT-2
Testing
Required verification:
mvn test
(there are no new failures or errors).Optional but recommended: