Downloader is creating folders and JSON files for about half of my library, then exiting

[RBZL]

I have three bundles which I have added to my itch.io library, resulting in about 4,000 games in my library. When using python -m itchiodl.downloader with my API key to download my library, the following occurs:

• The downloader says a number of "pages" are loaded
• The downloader starts going through the list of games, saying it is downloading them
• Publisher and game folders are created for these games on my hard drive in the CWD
• JSON files are also created for games, within publisher folders
• The downloader says games have been downloaded (x of 3918)
• Nothing is actually downloaded except for the JSON files
• Eventually the downloader simply stops, with the last item being that a game was downloaded (x of 3918)

Sometimes it makes it to about 1,000 games "downloaded", sometimes 1,500 before it quits. It leaves no error message or error text file with any other information. It doesn't seem to matter if I change the number of jobs (I've tried 4 jobs and 8 jobs), run with administrator privileges (using Python through Windows command prompt), or anything else. I'm not utilizing a VPN or anything else special. I'm running Python 3.9 64-bit on Windows 10.

[RBZL]

As a follow-up, I tried disabling 2FA and using my username and password without the API key. Same result as above.

I also tried running the downloader on another system to verify it wasn't something with the local environment on one machine (another Windows 10 system with Python 3.10). Same result.

Finally, I've noticed the downloader has been dying on the same item sometimes. I've seen Downloaded Star and Light 1.0 (918 of 3918) as the last item before it stops more than once now. It doesn't stop there every time, but enough that I've noticed it. I wonder if a few select game titles are causing problems. Without any errors or error logs, though, hard to track down.

[N64Core]

I was curious if I was alone in this regard, itch.io seems to have changed how their API works to some degree, essentially what changed as far as I can tell is now when it's grabbing the list of files from the server it instead grabs the list of free files, I'm looking into potential workarounds

[N64Core]

Okay the fix may be quite simple, what happened was they might have changed how download keys are presented in the api, when you query https://api.itch.io/profile/owned-keys there are a few fields labeled "id" and the top level one is what needs to be used for the download key, testing this theory shortly

update: yeah that was the issue, I've currently got a working branch here https://github.com/N64Core/itchio with some other additions I've been testing, it's not perfect yet (for some reason it won't grab specific titles that are in zip archives and misses the occasional pdf, but it's working fairly well at the moment on my end

[Emersont1]

I was curious if I was alone in this regard, itch.io seems to have changed how their API works to some degree, essentially what changed as far as I can tell is now when it's grabbing the list of files from the server it instead grabs the list of free files, I'm looking into potential workarounds

Oh dear, this looks like it was broken with @wertercatt 's PR

if you run with pip install itchiodl==2.0 does it work?

[wertercatt]

I had to do some workarounds to get the id from the free files to work properly, since the endpoint I'm using for them defines id differently. The way I wrote it should make it only trigger when actually using the free game downloaders, so I'm not sure what's going on with op's system.

[RBZL]

Oh dear, this looks like it was broken with @wertercatt 's PR

if you run with pip install itchiodl==2.0 does it work?

I tested this today, and no, it didn't work. I uninstalled the current version, installed 2.0, and deleted everything it created previously and started from scratch again using an API key.

• It creates folders for publishers and games, but no JSONs
• It rapidly says it's downloading hundreds or thousands of actual discernable files now, i.e. game-name.zip, some-document.pdf, etc
• It never actually downloads files, or says they were downloaded
• After about 1,100 publisher folders, it just stopped with no error output in the console or in a file

As before, happy to provide more info or diagnostics.

Edit: As a follow-up, I ran it again and it seems to be ending at the same point, which could be the end of my Itch.io library. The files are just not being actually downloaded.

Edit 2: Attempted on another system, and similar results - however, only 1,080 publisher folders were created. It seems that many things are silently failing: downloads, and something which is causing the program to terminate before getting through my entire library.

Edit 3: I spun up a WSL Ubuntu 22.04 box and set up Python3 and itchiodl there. Everything appears to be working with the current itchiodl release. It seems something might be up with itchiodl and Python on Windows.

[wertercatt]

Oh yeah I haven't had any luck with Windows support either, before or after my changes. Linux appears to be a requirement to run itchiodl properly.

[RBZL]

Oh yeah I haven't had any luck with Windows support either, before or after my changes. Linux appears to be a requirement to run itchiodl properly.

Good info for the README - would save some time! Fortunately WSL2 makes life pretty easy.

If Windows support is supposed to be present or will be added later and anyone needs help testing it out, let me know.

[Emersont1]

That's peculiar, I'll do some tests to try and get it working on windows

[RBZL]

@Emersont1 - I was just about to comment, lol.

I let this run for a while in the Ubuntu box, and it downloaded some things but it missed a lot of them too. I'd say maybe half of the game folders contain no files. A very small amount of items have errors, and they're usually for things like links to external sites in the download lists. However, many things are just not being downloaded, and not creating any sort of error.

It's also still stopping short - at 3710 of 3919 items. It's stopping there consistently on repeated runs. It says it downloaded that last game (Last Knight: Rogue Rider Edition) before it stops, but it did not download it. I'm trying to download the game's files manually and run it again to see if it'll get past that one.

Edit: Same result - downloads are stopping at the same point.

[N64Core]

@Emersont1 - I was just about to comment, lol.

I let this run for a while in the Ubuntu box, and it downloaded some things but it missed a lot of them too. I'd say maybe half of the game folders contain no files. A very small amount of items have errors, and they're usually for things like links to external sites in the download lists. However, many things are just not being downloaded, and not creating any sort of error.

It's also still stopping short - at 3710 of 3919 items. It's stopping there consistently on repeated runs. It says it downloaded that last game (Last Knight: Rogue Rider Edition) before it stops, but it did not download it. I'm trying to download the game's files manually and run it again to see if it'll get past that one.

Edit: Same result - downloads are stopping at the same point.

I figured this was worth investigating because I noticed I had downloads coming up short as well, between the above commit and the previous one it now finishes the script with the correct number of entries listed in the X of Y library counter. However the only thing that was added was a catch-all try/except/finally, what I suspect is happening is that any time an exception would occur anywhere inside the game.download function it was dropping that job entirely and neglecting to increment the counter.

It spits out each title that causes an exception to errors.txt, which may begin to shed light on the common thread between them, but currently it's so generic that all of the titles I've investigated were actually downloaded correctly. I'm going to look a little deeper into this phenomenon

[N64Core]

Okay, as of the latest commit, exceptions within game.download have dropped by perhaps 90%, I'm now noticing there's an issue with the handling of titles whose names end in '...' (A trait shared by multiple entries in my library, weirdly)

[RBZL]

Okay, as of the latest commit, exceptions within game.download have dropped by perhaps 90%, I'm now noticing there's an issue with the handling of titles whose names end in '...' (A trait shared by multiple entries in my library, weirdly)

Thanks - giving it a spin now, but getting some errors. I'll hop over to your repo and create an issue.

Edit: Or not, issues are disabled there. This is occurring for every single download:

  An error occurred while downloading: Penumbra- A Trophy Dark Incursion
 Traceback (most recent call last):
  File "/home/rbzl/.local/lib/python3.10/site-packages/itchiodl/library.py", line 88, in dl
    x = g.download(self.login, platform)
  File "/home/rbzl/.local/lib/python3.10/site-packages/itchiodl/game.py", line 99, in download
    self.load_downloads(token)
  File "/home/rbzl/.local/lib/python3.10/site-packages/itchiodl/game.py", line 88, in load_downloads
    if not self.skipping_large_entries:
AttributeError: 'Game' object has no attribute 'skipping_large_entries'

[N64Core]

Welp, that was just a silly typo on my part, try now

[RBZL]

Welp, that was just a silly typo on my part, try now

It happens!

It's working now and making progress. Things are actually downloading, and it's going slowly enough that I trust it's actually getting everything now. Thanks! I'll let you know if there are any errors once it finishes.

[N64Core]

There is an issue I'm aware of that involves the download uuid not existing, I'm unsure what's causing it but in a library of 2800 titles it occurred in eight

[RBZL]

Things worked well overall - one issue though, is that by default your fork uses the publisher and game display names, while the original does not. That created a bunch of duplicate folders and files as a result. I'll probably have to delete everything and start over, as I don't want to try to manually sort out 3,000+ folders and duplicates.

[N64Core]

That wasn't intended, but it should be fixed now, I hadn't done as much testing in the default configuration as I should have frankly