Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add before/after comparisons at the end of notebooks #613

Closed
LMMilliken opened this issue Nov 22, 2022 · 5 comments · Fixed by #622
Closed

Add before/after comparisons at the end of notebooks #613

LMMilliken opened this issue Nov 22, 2022 · 5 comments · Fixed by #622
Assignees
@bwanglzu @LMMilliken

Comments

@LMMilliken
Copy link
Contributor

LMMilliken commented Nov 22, 2022 •
edited

Each of our notebooks currently describe the process of finetuning a model, but don't provide any direct comparisons of the search results between the zero-shot and the finetuned models.
A before/after section should be added to each notebook, they should show example queries and the top result(s) returned by both the zero-shot and the pre-trained model
@LMMilliken LMMilliken self-assigned this Nov 22, 2022
@LMMilliken LMMilliken linked a pull request Nov 28, 2022 that will close this issue
docs: add before and after to text-text #622
Merged
2 tasks
@LMMilliken
Copy link
Contributor Author

Currently, the before/after examples in the image-image notebook is not very good, especially compared to a similar example in the documentation of finetuner version 0.4.1. One identified difference between 0.4.1 and current versions is that the preprocessing function now resizes the images, whereas it did not before. Also, the model used to get the before/after examples in the current notebook did not have the correct tuning parameters.

@LMMilliken
Copy link
Contributor Author

currently working on improving finetuning results by not resizing images that are already the correct size. Once that is done I will collect another set of before/after examples, hopefully with better results

@LMMilliken
Copy link
Contributor Author

LMMilliken commented Dec 12, 2022 •
edited

Better before/after comparisons have been added and the performance of the model using the finetuning parameters in the notebook have been validated. Changes made to finetuner-client in https://github.com/jina-ai/finetuner-api/issues/204 are causing the client tests to fail, so other changes in client need to be made before merging

@guenthermi
Copy link
Member

guenthermi commented Dec 16, 2022 •
edited

An example 3D point cloud:
Screenshot from 2022-12-15 10-56-23
Matching Meshes:
matches

results_comparison

@guenthermi
Copy link
Member

clip-example-pt
clip-example-pt

clip-example-ft
clip-example-ft
image-image-triplets-good
image-image-triplets-good
image-image-triplets-bad
image-image-triplets-bad
WandB-mclip
WandB-mclip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bwanglzu bwanglzu

@LMMilliken LMMilliken

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

docs: add before and after to text-text jina-ai/finetuner
3 participants
@guenthermi @bwanglzu @LMMilliken