Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: document multilingual clip #611

Merged
merged 18 commits into from
Nov 24, 2022
Merged

Conversation

LMMilliken
Copy link
Contributor

This pr adds a notebook showing how to use the soon to be added multilingual clip model


  • This PR references an open issue
  • I have added a line about this change to CHANGELOG

@LMMilliken LMMilliken linked an issue Nov 21, 2022 that may be closed by this pull request
Copy link
Member

@bwanglzu bwanglzu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sure in the end all imports are sorted

},
"nbformat": 4,
"nbformat_minor": 5
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3BFEB298-846C-4A3C-A18F-19AE22CEBCA6
we need to login to the team account and push the da to the cloud

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6FED2752-43F4-4361-9B42-A9F175435A4F
which model, can you elaberate?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8E026CAD-060A-441E-9874-A67C6E1FC8ED
always use cuda

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The method of pulling data and the use of 'cpu' were for testing purposes, updated now

Copy link
Member

@bwanglzu bwanglzu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left some comments

@github-actions github-actions bot added size/l and removed size/m labels Nov 22, 2022
@bwanglzu bwanglzu changed the title Docs document multilingual clip docs: document multilingual clip Nov 22, 2022
docs/notebooks/using_mclip.md Outdated Show resolved Hide resolved
docs/notebooks/using_mclip.md Outdated Show resolved Hide resolved
docs/notebooks/using_mclip.md Outdated Show resolved Hide resolved
docs/notebooks/using_mclip.md Outdated Show resolved Hide resolved
docs/notebooks/using_mclip.md Outdated Show resolved Hide resolved
@LMMilliken LMMilliken marked this pull request as ready for review November 24, 2022 15:10
docs/notebooks/using_mclip.md Outdated Show resolved Hide resolved
docs/notebooks/using_mclip.md Outdated Show resolved Hide resolved
docs/notebooks/using_mclip.md Outdated Show resolved Hide resolved
docs/notebooks/using_mclip.md Outdated Show resolved Hide resolved
@github-actions github-actions bot added size/l and removed size/xl labels Nov 24, 2022
Copy link
Member

@guenthermi guenthermi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added two comments

docs/notebooks/using_mclip.md Outdated Show resolved Hide resolved
docs/notebooks/using_mclip.md Outdated Show resolved Hide resolved
docs/notebooks/using_mclip.md Outdated Show resolved Hide resolved
lmilliken and others added 2 commits November 24, 2022 17:59
Co-authored-by: George Mastrapas <32414777+gmastrapas@users.noreply.github.com>
<!-- #endregion -->

## Before and After
We can directly compare the results of our fine-tuned model with an untrained multilingual clip model by displaying the matches each model has for the same query, while the difference between the results of the two models are quite similar for some queries, the examples below clearly show that finetuning increses the quality of the search results:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"while the difference [...] are quite similar" I think here is something wrong in the sentence?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

differences*

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes but also "the difference are similar" makes no sense

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

corrected to differences

Copy link
Member

@gmastrapas gmastrapas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

<!-- #endregion -->

## Before and After
We can directly compare the results of our fine-tuned model with an untrained multilingual clip model by displaying the matches each model has for the same query, while the differences between the results of the two models are quite similar for some queries, the examples below clearly show that finetuning increses the quality of the search results:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
We can directly compare the results of our fine-tuned model with an untrained multilingual clip model by displaying the matches each model has for the same query, while the differences between the results of the two models are quite similar for some queries, the examples below clearly show that finetuning increses the quality of the search results:
We can directly compare the results of our fine-tuned model with an untrained multilingual clip model by displaying the matches each model has for the same query, while the differences between the results of the two models are quite subtile for some queries, the examples below clearly show that finetuning increses the quality of the search results:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its subtle

@LMMilliken LMMilliken force-pushed the docs-document-multilingual-CLIP branch from d61d7f1 to eeacfbf Compare November 24, 2022 17:14
"id": "72867ba9-6a8c-4b14-acbf-487ea0a61836",
"metadata": {},
"source": [
"# Multilingual Text-To-Image search with MultilingualCLIP\n",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"# Multilingual Text-To-Image search with MultilingualCLIP\n",
"# Multilingual Text-to-Image search with MultilingualCLIP\n",

@LMMilliken LMMilliken force-pushed the docs-document-multilingual-CLIP branch from 72b04f6 to 12459b0 Compare November 24, 2022 17:40
@github-actions
Copy link

📝 Docs are deployed on https://ft-docs-document-multilingual-CLIP--jina-docs.netlify.app 🎉

@LMMilliken LMMilliken merged commit d5dcdc7 into main Nov 24, 2022
@LMMilliken LMMilliken deleted the docs-document-multilingual-CLIP branch November 24, 2022 17:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add notebook for multilingual-CLIP
5 participants