HTTP Error 503 with big game library (600+)

[shakeyourbunny]

gogrepo is running into GOG's IP block you do a

$ gogrepo update

during the MD5 hash download at game 600 or so.

Oh, the whole run did take about 2h (!).

09:12:12 | total time: 2:05:17.007290

Would be cool if you implement incremential updates and cut down the update time :-)

[eddie3]

Damn you gog! One thing you can currently do is change the delay between http get's..

Currently it's:
HTTP_FETCH_DELAY = 1 # in seconds

I would try 1.5 or 2. It will make your update longer, but it should avoid ban.

I have it set to 1 with 700+ games and I don't get banned... but who knows how their banning really works. It does take a long time, but that's because there's a fetch per game for the game data, and a fetch per file to get MD5 data. That's the killer.

I can add support for "resuming" updates... but ultimately I'd like to figure out a way to avoid ban.

I don't know of any way to "speed up update" without sacrificing fetching the MD5 info.

Any suggestions? :)