ReadTimeoutError when using FaceDetailer + CLIPSeg

[Lex-DRL]

I'm reposting the issue from here:
ltdrdata/ComfyUI-Impact-Pack#116

I'm using Impact pack.
Very often (but not always), when I use FaceDetailer node with CLIPSegDetectorProvider - one of them spams ReadTimeoutError errors into console, with text like this:

Loads SAM model: E:\SD\ComfyUI-portable\ComfyUI\models\sams\sam_vit_b_01ec64.pth (device:Prefer GPU)
'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 621b36cd-5b11-4361-bbd3-c1f008d1a9e3)')' thrown while requesting HEAD https://huggingface.co/CIDAS/clipseg-rd64-refined/resolve/main/preprocessor_config.json
'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: 50506107-44dc-4bf4-9fb7-48cf7ad9c640)')' thrown while requesting HEAD https://huggingface.co/CIDAS/clipseg-rd64-refined/resolve/main/tokenizer_config.json
'(ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: c5467a5f-4ec8-4074-a2b4-00765779fe5e)')' thrown while requesting HEAD https://huggingface.co/CIDAS/clipseg-rd64-refined/resolve/main/config.json
# of Detected SEGS: 1
Detailer: segment upscale for ((419, 390)) | crop region (838, 780) x 1.8331748251748252 -> (1536, 1429)

The nodes author claims that they are just wrappers on top of CLIPSeg and the actual issue is here.

The error in question is not from FaceDetailer but rather from CLIPSeg. CLIPSeg has a part where it downloads the model through the network initially, so if there is no internet connection, this can become an issue.

I can't reproduce the issue on demand. It either happens or not, with no obvious cause. Sometimes my whole graph (heavily utilizing FaceDetailer + CLIPSeg) just works like a breeze. Sometimes, when I have a long sequence of such nodes, each of them spams the same set of errors. My best guess is I'm getting antispam-banned at huggingface after a while.

And looks like, the only thing CLIPSeg tries to retrieve is a couple of json files. Why don't just download them once?