Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incompletely found posts from ThisVid users, blank entries on missing posts #147

Closed
pingzoo opened this issue Jun 19, 2023 · 5 comments
Closed
Labels
help wanted Extra attention is needed

Comments

@pingzoo
Copy link

pingzoo commented Jun 19, 2023

https://thisvid.com/members/1169929/

User has:
91 public videos
53 private videos
1 album with 7 images

SCrawler found 48 public videos (page 1 and 2).
SCrawler downloaded 45 public videos -> Log sais "There are missing posts".
Missing posts form for this user shows:
image
I can't select a missing post here.

When I start download again SCrawler sais "All data downloaded".

What should I do in such a case?

I deleted the *_Posts.txt and restarted the download. SCrawler found 60 Posts (53 private + 7 images) and started downloading (currently running).

So after the download at least 46 public videos will be missing again.

I'll keep you posted, when the current download job is done.

@AAndyProgram
Copy link
Owner

Not confirmed. Look, I added this profile, SCrawler found 91 public videos (like on the site). This site is too slow. I've already downloaded 46 videos in 5 hours. I canceled the download because it's too long.

image

image

@pingzoo
Copy link
Author

pingzoo commented Jun 19, 2023

Hmm ok. Don't know why it happens on my end.
But in the last days I got more often error messages, on ThisVid, because of overload of the server (Bad Gateway, 502, 503, etc)
Maybe this happend, when SCrawler curling/invoking the site?

@pingzoo
Copy link
Author

pingzoo commented Jun 19, 2023

The Download of 53 private and 7 images is done now.
When i start the download again, SCrawler instantly responds with notification "All data downloaded"

I deleted Posts.txt again and startet the download again.
SCrawler now found the missing 46 public videos and startet downloading. (job currently running) ;D

@pingzoo
Copy link
Author

pingzoo commented Jun 19, 2023

Ok. Thank you again for checking this on your end.
So I guess, it's because of thisvids server load.

Is there any other way than deleting the Posts.txt and starting the download again, or is that ok in this case?

@AAndyProgram AAndyProgram added the help wanted Extra attention is needed label Jun 20, 2023
@AAndyProgram
Copy link
Owner

AAndyProgram commented Jun 20, 2023

But in the last days I got more often error messages, on ThisVid, because of overload of the server (Bad Gateway, 502, 503, etc)

Indeed. This is not my main site, but even I have seen these messages.

Maybe this happend, when SCrawler curling/invoking the site?

If the log is empty, then SCrawler didn't face these errors. To get the video download URL, SCrawler uses yt-dlp. Don't forget the cookies!

The Download of 53 private and 7 images is done now.
When i start the download again, SCrawler instantly responds with notification "All data downloaded"

I deleted Posts.txt again and startet the download again.
SCrawler now found the missing 46 public videos and startet downloading. (job currently running) ;D

Different numbers mean that something is wrong on your end. Try updating your cookies, then add this profile and download it separately. If you're using automation, pause it. Don't start downloading any other profiles until this one has finished downloading.

Ok. Thank you again for checking this on your end.

You're welcome

So I guess, it's because of thisvids server load.

Perhaps

Is there any other way than deleting the Posts.txt and starting the download again, or is that ok in this case?

No, there are no other ways. It's okay.

@pingzoo pingzoo closed this as completed Jun 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants