url cache is not used when offline

[matdillen]

Similar Requests
I searched for similar issues and there were some related ones (e.g. #4319) , that gave me an understanding of how I think the image downloading and caching currently works, but I could be mistaken.

Description of New Feature
Keep all the images downloaded on the fly, so they do not need to be ever redownloaded. A user can force redownload simply by deleting the image that he wants replaced.

Additional Context
Each of my Cockatrice sessions, the app redownloads all images of decks loaded into games. I can stop this behavior by downloading all images I need for my decks using an external script directly calling the scryfall API, storing these image files in the ../Cockatrice/pics/CUSTOM directory, and disabling the download on the fly option in settings/Card Sources. But then any image I may have missed is not downloaded and not shown. If I enable downloading on the fly, all images are always downloaded again (determined by the slower loading time and in the debug log) and the files in CUSTOM are ignored.

I'm not entirely clear on how the caching currently works, but why isn't there an option to keep any image that was downloaded on the fly? Or is there an issue with my installation and is this feature request how the app is supposed to work? Some issues mention the folder ../Cockatrice/pics/downloadedPics but I do not have that one. If I manually make one and populate it with images, they are still ignored if downloading on the fly is enabled.

[ebbit1q]

yes, this is supposed to work, please send us your logs

[matdillen]

Thanks for confirming that I wasn't missing something. I found the problem now, and it was at my end. I missed numerous images when reading the .cod xml files to aggregate card names to throw at scryfall, because I assumed main and side would always be in the same order in the xml. It was these missing images that would keep redownloading, the other ones do get read from local storage if they are in pics/CUSTOM.

So this can be closed. Hope I didn't bother anyone.

[ebbit1q]

I am still confused with your issue, you don't have to redownload images ever, when a picture is downloaded on the fly it is stored indefinitely or until you delete the cache.

[matdillen]

That is the problem. No images are stored after being downloaded on the fly. They are always redownloaded. This can be tested by loading a deck, then reloading it again after a cockatrice reboot and without an internet connection. Example process:

1. I load a deck that contains both cards with images in pics/CUSTOM and cards without images there.
2. PictureLoader downloads them, e.g. for Lotus Petal which is not in CUSTOM:

PictureLoader: [card: "Lotus Petal" set: "P30M"]: Requested "set"property ("muid") for Url template ("https://api.scryfall.com/cards/multiverse/!set:muid!?format=image") is not available
PictureLoader: [card: "Lotus Petal" set: "P30M"]: Requested "set"property ("muid") for Url template ("https://gatherer.wizards.com/Handlers/Image.ashx?multiverseid=!set:muid!&type=card") is not available
...
PictureLoader: [card: "Lotus Petal" set: "P30M"]: Trying to load picture
PictureLoader: [card: "Lotus Petal" set: "P30M"]: No custom picture, trying to download
...
PictureLoader: [card: "Lotus Petal" set: "P30M"]: Trying to fetch picture from url "https://api.scryfall.com/cards/2a7a63b4-2946-4a25-acd4-742fec197ec0?format=image&face=front"
PictureLoader: [card: "Lotus Petal" set: "P30M"]: following redirect to "https://cards.scryfall.io/large/front/2/a/2a7a63b4-2946-4a25-acd4-742fec197ec0.jpg?1703609504"
PictureLoader: [card: "Lotus Petal" set: "P30M"]: Image successfully loaded from cached url "https://cards.scryfall.io/large/front/2/a/2a7a63b4-2946-4a25-acd4-742fec197ec0.jpg?1703609504"

3. Close cockatrice, disconnect internet connection.
4. Reopen the deck in cockatrice.

PictureLoader: [card: "Lotus Petal" set: "P30M"]: Requested "set"property ("muid") for Url template ("https://api.scryfall.com/cards/multiverse/!set:muid!?format=image") is not available
PictureLoader: [card: "Lotus Petal" set: "P30M"]: Requested "set"property ("muid") for Url template ("https://gatherer.wizards.com/Handlers/Image.ashx?multiverseid=!set:muid!&type=card") is not available
...
PictureLoader: [card: "Lotus Petal" set: "P30M"]: Trying to load picture
PictureLoader: [card: "Lotus Petal" set: "P30M"]: No custom picture, trying to download
...
PictureLoader: [card: "Lotus Petal" set: "P30M"]: Trying to fetch picture from url "https://api.scryfall.com/cards/2a7a63b4-2946-4a25-acd4-742fec197ec0?format=image&face=front"
...
PictureLoader: [card: "Lotus Petal" set: "P30M"]: Download failed for url "https://api.scryfall.com/cards/2a7a63b4-2946-4a25-acd4-742fec197ec0?format=image&face=front" ("Host api.scryfall.com not found")
...
PictureLoader: [card: "Lotus Petal" set: "P30M"]: Trying to fetch picture from url "https://gatherer.wizards.com/Handlers/Image.ashx?name=Lotus Petal&type=card"
...
PictureLoader: [card: "Lotus Petal" set: "P30M"]: Download failed for url "https://gatherer.wizards.com/Handlers/Image.ashx?name=Lotus Petal&type=card" ("Host gatherer.wizards.com not found")
PictureLoader: [card: "Lotus Petal" set: "MB1"]: Requested "set"property ("muid") for Url template ("https://api.scryfall.com/cards/multiverse/!set:muid!?format=image") is not available
PictureLoader: [card: "Lotus Petal" set: "MB1"]: Requested "set"property ("muid") for Url template ("https://gatherer.wizards.com/Handlers/Image.ashx?multiverseid=!set:muid!&type=card") is not available

And so on, for all cards without an image in CUSTOM, until the script bails out after too many retries. As far as I can tell, it never looks into cached images. I wouldn't know where they would be stored on my machine. The images in CUSTOM are loaded properly, internet or not. But those only came to be there because I downloaded them from scryfall using a script of my own.

Version info:

Client Version: 2.9.0 (2023-09-14)
Client Operating System: Windows 11 Version 2009
Build Architecture: 64-bit
Qt Version: 6.3.2
System Locale: nl_BE
Install Mode: Standard

I also see this in the logs upon startup:

CardDatabase::loadCardDatabases start
[CockatriceXml4Parser] Trying to parse:  "C:/Users/mdill/AppData/Local/Cockatrice/Cockatrice/cards.xml"
[CockatriceXml4Parser] Unknown item "info" , trying to continue anyway
PictureLoader: cache fail for "_trice_card_back_305278"
[CardDatabase] loadCardDatabase(): Path = "C:/Users/mdill/AppData/Local/Cockatrice/Cockatrice/cards.xml" Status = 0 Cards = 28351 Sets = 733 "1078ms"
[CockatriceXml4Parser] Trying to parse:  "C:/Users/mdill/AppData/Local/Cockatrice/Cockatrice/tokens.xml"
[CockatriceXml4Parser] Unknown item "info" , trying to continue anyway
[CardDatabase] loadCardDatabase(): Path = "C:/Users/mdill/AppData/Local/Cockatrice/Cockatrice/tokens.xml" Status = 0 Cards = 29126 Sets = 733 "24ms"
[CardDatabase] loadCardDatabase(): Path = "C:/Users/mdill/AppData/Local/Cockatrice/Cockatrice/spoiler.xml" Status = 4 Cards = 29126 Sets = 733 "0ms"
CardDatabase::loadCardDatabases success

[ebbit1q]

thank you so much for your logs, they make it clear that there is a regression here, the new url cache (#4756) is not used when there is no network connection.
as a workaround you can still use version 2.8.0 that does not have this regression.

[Elarnon]

This looks like it is https://bugreports.qt.io/browse/QTBUG-40151 — it is not that the cache is not used when offline, it is that redirects are not cached, and scryfall URLs are redirects (as can be seen in the log) :(

At least this should be easier to fix, because we can just remember the resolved (post-redirect) URL associated with the card.

I can reproduce the behavior now, but am fairly surprised — I specifically tested for this while writing #4756 and was fairly certain I had found a workaround by using the ManualRedirectPolicy… I even had api.scryfall.io redirects in cache from that time, but after clearing the cache it seems like I can't get new ones to appear.

Edit: to clarify, the following lines in the log:

PictureLoader: [card: "Lotus Petal" set: "P30M"]: following redirect to "https://cards.scryfall.io/large/front/2/a/2a7a63b4-2946-4a25-acd4-742fec197ec0.jpg?1703609504"
PictureLoader: [card: "Lotus Petal" set: "P30M"]: Image successfully loaded from cached url "https://cards.scryfall.io/large/front/2/a/2a7a63b4-2946-4a25-acd4-742fec197ec0.jpg?1703609504"

should have been:

PictureLoader: [card: "Lotus Petal" set: "P30M"]: following cached redirect to "https://cards.scryfall.io/large/front/2/a/2a7a63b4-2946-4a25-acd4-742fec197ec0.jpg?1703609504"
PictureLoader: [card: "Lotus Petal" set: "P30M"]: Image successfully loaded from cached url "https://cards.scryfall.io/large/front/2/a/2a7a63b4-2946-4a25-acd4-742fec197ec0.jpg?1703609504"

(notice "following cached redirect" instead of "following redirect")

Since the redirect is not cached, when there is no network connectivity, we never reach the second step, even though the image itself is cached on disk.