What is "overusage" exactly?

[Testaccount342]

Prerequisites

• [ Yes] Did you read FAQ section?
• [ Yes? ] Did you test with the latest releases or commit ?

Description

It's not really an issue with the program, which I have to say is a damn marvel, but rather about the download rate limits. In the FAQ it is described that overusage can lead to getting throttled by pixiv.

So my question is, what is a good rate? I'm aiming to download about 1k artists, and also leave the program running as long as needed, so if possible I'd like to skip testing of rates. Do you have this information? Thank you for your time

[Nandaka]

I believe it is more on the parallel running (e.g. opening multiple instance of the application).

For single instance, I'm also not quite sure, as my internet connection is not that fast that I never hit the server limit.

[photonometric]

I believe it is more on the parallel running (e.g. opening multiple instance of the application).

I'd think so too. Never seen an overusage warning in 5 years and always run only 1 instance. And since delay has been an option it's been set at 1.

But then, if the question is about downloading 1000 artists' galleries entirely in one stretch, that may lead to problems if they measure images or data downloaded over several days. If I have a list that I know will take more than a day, I like to break it up a bit to avoid the possibility of getting flagged (if it exists). Not automatic, but worth a little manual input at the outset to be able to keep the delay at minimum for the rest of the time.

My usual download speeds seem to be 0.3-0.85 MiB/s, where my practical usable speed is about 14mib/s. I assume that means I'm always saturating pixiv's speed limits (due to policy or traffic)? But again, never had any issue.

[scottyseng]

My own personal experience, I have yet to get a flag (I'm not sure if it's because I have Pixiv Premium). I've downloaded roughly 4,000 artists in 400 artist chunks (roughly 5-7 days 24/7) with around 8-10 hours between each 400 artist pass. It's been running for the past two months without any warnings. I believe as long as you're not trying to run multiple Pixiv Downloaders at the same time, you'll be plenty fine.

Usual download speeds per image are 0.3-0.85 MiB/s as well. My home internet speed is 18.75 MiB/s (150 Mb/s)

Do note that there is a size limit to Pixiv Downloader for member IDs. I think it was around 450ish that you would see the list get cut off. I just do 400 member ID chunks to be safe.

Edit: Nevermind, I just got an account suspension for too many requests. I believe running scans for images you might be more afraid of than downloading lots of images in one pass.

[ikkion]

Yes I got the same message today. It seems together with the new login system they have also severly limited the downloads from their site. I will have to reorganize the way I update and split my list.txt into several smaller lists that are updated at different times.

Does anyone have a good idea what the new limits are?

[zoram999]

Edit: Nevermind, I just got an account suspension for too many requests. I believe running scans for images you might be more afraid of than downloading lots of images in one pass.

Just happened the same to me. I've been using the app for many years, following over 4000 artists plus a few hundreds on another account, downloading their updates regularly (first account almost daily, the other once or twice a week), no warnings of sort until an hour ago, and now I've got a suspension. Pixiv may have introduced stricter measures along with the changes that have been just fixed in the newest version of the app. This sucks... I hope it can be circumvented in some way.

[scottyseng]

Yeah, I kind of feared that my suspension was related to the new system. I guess I'll have to make burner accounts and use a VPN (I don't want my main art account deleted).

I'll be breaking up my lists into probably 25 or 50 artist chunks now.

[zoram999]

I've just had the same idea. My second account has less than 500 follows and, of course, the volume of new posts in a span of 10 days is less than the other one can make in 36, 48 hours tops; anyway, I just got over 250 posts from the secound account without a hitch, so it may really boil down to the volume of downloads from a single account over a certain span of time, so the ones with a lot of follows get shafted like it just happened to me.
Handling several accounts, each with its own instance of the app, is going to be an hassle but still better than nothing...

[Hamuko]

I'm running PixivUtil2.py -s 8 1 3 -x (first three pages from bookmarked members page) every 600 seconds (10 minutes) with 1433 accounts being followed, but luckily haven't been suspended yet. Maybe there's not an issue there considering that I haven't received anything yet?

Are all the suspended people downloading from a list?

[scottyseng]

Are all the suspended people downloading from a list?

Yeah, I'm downloading from a member ID list. No delay. I believe what kicked it off for me was pixiv downloader scanning through already downloaded art (I wasn't downloading many new posts).

[bnick23]

I was also suspended yesterday, got the same suspension message as others here. Was just downloading the new illustrations from bookmark when it kicked me out. Been doing it almost daily for years now with no issue, so it suprised me quite a bit.

I still got my lists to use if I make another user, but I am going to have to re-follow almost 3.5k artists. Dont suppose anyone knows anyway to automate that using my exported bookmarks list?

[ikkion]

I tested using a secondary account and a shortened list. It doesn't take alot to get suspended. I get the feeling it takes less and less to get suspended. I will try later with a even shorter list.

[Nandaka]

Try this one #483 (comment), it will try to skip the OAuth api if possible.

[scottyseng]

Yeah, I was able to get my account back, but will be creating burner accounts. It really does seem like Pixiv cracked down on downloading.

Will try beta3

[Revemohl]

I'm still too afraid to run the tool for now, but technically shouldn't bumping downloaddelay up help a bit? Also, does this OAuth skip work for option 8? (edit: just checked on a second account, looks like it does)
And it looks like the linked compiled beta3 is lacking a PixivUtil2.exe.

[IPGxwcu]

I suspect that the suspensions are from frequency of requests rather than volume of images downloaded. I used option 8 from v20190423-beta1 on April 23rd and April 24th to parse/download roughly 15 pages of new images and 20 pages of old images. My config settings were retrywait = 10 and downloaddelay = 10; there were no issues. However, browsing and downloading various images from Pixiv's daily ranking gave me a suspension within about 5 minutes.

[bnick23]

However, browsing and downloading various images from Pixiv's daily ranking gave me a suspension within about 5 minutes.

Do you mean that you got suspended downloading from the daily ranking even with the added delay? Or was that without?

[IPGxwcu]

Do you mean that you got suspended downloading from the daily ranking even with the added delay? Or was that without?

Option 2 (Download by image_id) does not utilize downloaddelay, as far as I know. Instead, it'll download images as fast as you can input their id into PixivUtil2. Which mens that if you queue multiple images into PixivUtil2 via option 2, you'll be downloading at 0 downloaddealy even if your config.ini says otherwise.

At the time of my suspension, however, I wasn't inputting image ids so quickly that they were queuing up in PixivUtil2. I think the suspension was from browsing images/pages on my web browser while occasionally downloading images that I liked through option 2.

[reyaz006]

Since using several instances is somewhat trivial amongst seemingly most older users, would it make sense to implement some kind of global lock file? E.g. use it to make sure that only one instance of PixivUtil2 is making server requests at the moment. This can also help with finding exact limit after which account gets banned, even if connection is slow.

[zoram999]

I suspect that the suspensions are from frequency of requests rather than volume of images downloaded. I used option 8 from v20190423-beta1 on April 23rd and April 24th to parse/download roughly 15 pages of new images and 20 pages of old images. My config settings were retrywait = 10 and downloaddelay = 10; there were no issues. However, browsing and downloading various images from Pixiv's daily ranking gave me a suspension within about 5 minutes.

8 is the only option I've ever used over the last months, and always in separate sessions for each account; however, I never touched those config lines you mention, always leaving the default values. Had I realized sooner I may have avoided the suspension, who knows. I'll certainly try and modify them now.

[bnick23]

If you received the same suspension message as the one in the image above for your main account, like I myself did, you can still try to appeal it by filling the support ticket form mentioned. Just a heads up.

Going to make a temp account to try with downloaddelay as well, definitely not going to risk my main account for now.

[ikkion]

I have now tested Beta3 with a new secondary account and my full list and it worked. Beta1 and Beta2 made a new login for every request, which seems to caused the suspensions, not the amount downloaded. Beta3 seems to have solved this. In any case all my main downloads in the future will be conducted using secondary accounts.

Edit: It still can happen that Beta3 will log in from time to time when running longer lists, but it seems it does not happen often enough to trigger Pixivs systems.

Edit2: Having tested downloading from Pixiv fanbox using option f2, Beta3 still does a lot of OAuth calls, at least one for every page. Basically whatever you will put in the Max Page option when running option f2 that many OAuth calls it will make. Put in 10 pages it will make at least 10 OAuth calls. This is problematic because most have their fanbox subscriptions on their main account risking that account every time you download fanbox stuff. Also you cannot put any delay on fanbox downloads,

[zoram999]

User whinette has posted this in the other ongoing discussion:

I've used this api to reimport the list of followed user on my subaccount: https://github.com/alphasp/pixiv-api-client
I've made a dirty script (never wrote js until then, lucky me) to migrate the users exported from pixivutil2 (option e):
https://gist.github.com/whinette/4bde849d69a16356abdd995efff2f6aa

Knowing jack shit about programming I don't how to use them, the most I can gather is that the script from the second link has spaces to insert the list of UserIDs and the login info. If someone could tell what is needed exactly to make them work, it'd be really useful! Especially if this experimenting forces to sacrifice some throwaway accounts and redo them all over.

Before updating to beta3, all my new accounts ran fine except one which received a "stop the massive number of accesses" notification (I was also forced to reset the password). Checking its login history, it reported a constant string of 2 to 4 logins every single minute, and even without advices it happened with other accounts indeed. Now, with beta3, it reports a single access so far.

Nonetheless, even if the issue was now solved it's been a good chance to break my follows in more manageable chunks.

[IPGxwcu]

Before updating to beta3, all my new accounts ran fine except one which received a "stop the massive number of accesses" notification (I was also forced to reset the password). Checking its login history, it reported a constant string of 2 to 4 logins every single minute, and even without advices it happened with other accounts indeed. Now, with beta3, it reports a single access so far.

Nonetheless, even if the issue was now solved it's been a good chance to break my follows in more manageable chunks.

I know that this is a bit off topic, but how serious is Pixiv about their single account rule? I've thought about breaking my follows into multiple accounts before as well, but always hesitated because of article 10 in their terms of use.

Does Pixiv not track account usage/creation originating from a single ip address?

[zoram999]

I really don't know. All I've got over a single day has been going through some recaptcha stuff before I could register another account. Only time will tell; of course I don't intend to abuse it, only one instance of the downloader at a time, with smaller lists of less than 500 follows each, you can easily wait several days of new posts before getting the artist updates from each.

[reyaz006]

I've noticed "Rate limit" notices appear at some point during the execution. I have 9 instances running. Should I worry about it? If yes, what parameters should I touch? Otherwise, will it take more time to process everything (I mean, do these notices mean there is a further penalty for waiting time, compared to case where they do not appear)?

[Nandaka]

you can set download delay in config.ini

downloadDelay  ==> Set random delay up to n seconds for each image post.
                   Set to 0 to disable.

[reyaz006]

Do we know something specific about appropriate intervals at all? Or can we only use trial-and-error method to discover anything?