docs: add wise-ft to clip finetuning

CHANGELOG.md

@@ -10,6 +10,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- Support advanced CLIP fine-tuning with WiSE-FT ([#571](https://github.com/jina-ai/finetuner/pull/571))
+
 - Add documentation for callbacks ([#567](https://github.com/jina-ai/finetuner/pull/567))
 
 ### Removed

docs/tasks/text-to-image.md

@@ -163,6 +163,38 @@ In case you set `to_onnx=True` when calling `finetuner.fit` function,
 please use `model = finetuner.get_model('/path/to/YOUR-MODEL.zip', is_onnx=True)`
 ```
 
-Check out [clip-as-service](https://clip-as-service.jina.ai/user-guides/finetuner/?highlight=finetuner#fine-tune-models) to learn how to plug-in a finetuned CLIP model to our CLIP specific service.
+(wise-ft)=
+## Advanced: WiSE-FT 
 
-That's it! If you want to integrate the fine-tuned model into your Jina Flow, please check out {ref}`integrated with the Jina ecosystem <integrate-with-jina>`.
+WiSE-FT, proposed by Mitchell et al. in [Robust fine-tuning of zero-shot models](https://arxiv.org/abs/2109.01903),
+has been proven to be an effective way for fine-tuning models with a strong zero-shot capability,
+such as CLIP.
+As was introduced in the paper:
+
+> Large pre-trained models such as CLIP or ALIGN offer consistent accuracy across a range of data distributions when performing zero-shot inference (i.e., without fine-tuning on a specific dataset). Although existing fine-tuning methods substantially improve accuracy on a given target distribution, they often reduce robustness to distribution shifts. We address this tension by introducing a simple and effective method for improving robustness while fine-tuning: ensembling the weights of the zero-shot and fine-tuned models (WiSE-FT).
+
+Finetuner allows you to apply WiSE-FT easily,
+all you need to do is use the `WiSEFTCallback`.
+Finetuner will trigger the callback when fine-tuning job finished and merge the weights between the pre-trained model and the fine-tuned model:
+
+```diff
+from finetuner.callbakcs import WiSEFTCallback
+
+run = finetuner.fit(
+    model='ViT-B-32#openai',
+    ...,
+    loss='CLIPLoss',
+-   callbacks=[],
++   callbacks=[WiSEFTCallback(alpha=0.5)],
+)
+```
+
+The value you set to `alpha` should be greater equal than 0 and less equal than 1:
+
++ if `alpha` is a float between 0 and 1, we merge the weights between the pre-trained model and the fine-tuned model.
++ if `alpha` is 0, the fine-tuned model is identical to the pre-trained model.
++ if `alpha` is 1, the pre-trained weights will not be utilized.
+
+
+That's it! If you want to integrate the fine-tuned model into your Jina Flow, please check out {ref}`integrated with the Jina ecosystem <integrate-with-jina>`.
+Check out [clip-as-service](https://clip-as-service.jina.ai/user-guides/finetuner/?highlight=finetuner#fine-tune-models) to learn how to plug-in a fine-tuned CLIP model to our CLIP specific service.

docs/walkthrough/using-callbacks.md

@@ -126,3 +126,17 @@ The early stopping callback triggers at the end of training and evaluation batch
 - `minimum_delta`: The minimum amount of improvement that a model can have over the previous best model to be considered worthwhile, zero by default, meaning that the training will not stop early unless the performance starts to decrease
 - `patience`: The number of consecutive rounds without improvement before the training is stopped, two by default.
 - `baseline`: an optional parameter that is used to compare the model's score against instead of the best previous model when checking for improvement. This baseline does not get changed over the course of a run.
+
+
+## WiSEFTCallback
+
+WiSE-FT, proposed by Mitchell et al. in [Robust fine-tuning of zero-shot models](https://arxiv.org/abs/2109.01903),
+has been proven to be an effective way for fine-tuning models with a strong zero-shot capability,
+such as CLIP.
+
+Please refer to {ref}`Apply WiSE-FT <wise-ft>` in the CLIP fine-tuning example.
+
+```{warning}
+It is recommended to use WiSEFTCallback when fine-tuning CLIP.
+We can not ensure it works for other category of models, such as ResNet or Bert.
+```