Fix download model

[maxjeblick]

Make Download Model functionality robust against various model backbones/tokenizers.

• Fetch tokenizer files from tokenizer.save_pretrained
• Monitor experiment folder and add files that where modified after the backbone/tokenizer have been saved

Fixes #489

[pascal-pfeiffer]

classification_head.pth seems to be missing in the download, too, doesn't it?

[maxjeblick]

Yes, you are completely right, I fixed it

[pascal-pfeiffer]

lgtm!

I think the model_card.md doesn't take into account the option to have multiple classes and will always assume a (1, 1) linear layer, but this is unrelated to this PR (so also wrong about input shape actually)

# settings can be arbitrary here as we overwrite with saved weights
head = torch.nn.Linear(1, 1, bias=False).to("cuda")
head.weight.data = head_weights

inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")

out = model(**inputs).logits

logits = head(out[:,-1])

[maxjeblick]

I think the model_card.md doesn't take into account the option to have multiple classes and will always assume a (1, 1) linear layer, but this is unrelated to this PR (so also wrong about input shape actually)

Yeah, I was also confused, but it is actually ok, head.weight.data will overwrite weights. See also the comment in the md file.