Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: allow searching for similar images #21

Merged
merged 18 commits into from
Nov 7, 2021
Merged

feat: allow searching for similar images #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
2a085a7
feat: allow searching for similar images
ramayer Sep 29, 2021
deec04f
pycodestyle (squash this commit later)
ramayer Oct 6, 2021
948a6d2
pyright (squash this commit later)
ramayer Oct 6, 2021
a1a86c6
Update rclip/model.py
ramayer Oct 16, 2021
70cde5c
avoid downloading large files
ramayer Oct 16, 2021
2f61636
add rquests to pipfile
ramayer Oct 16, 2021
97a749d
refactored to compute embeddings in groups (squash this commit later)
ramayer Oct 17, 2021
c996308
pyright and pycodestyle (squash this commit later)
ramayer Oct 17, 2021
a4c3afd
Update rclip/model.py
ramayer Oct 24, 2021
68077be
only return a single value from compute_features_for_queries
ramayer Oct 24, 2021
f13455b
moved image loading functions to utils
ramayer Oct 25, 2021
7b92464
moved constants out of functions
ramayer Oct 25, 2021
e638a5c
Update rclip/utils.py
ramayer Oct 31, 2021
85c23c9
renamed function
ramayer Nov 1, 2021
d7c3154
updated Pipfile.lock
yurijmikhalevich Nov 7, 2021
f497f9a
made linter happy
yurijmikhalevich Nov 7, 2021
b0bc2c4
minor refactoring
yurijmikhalevich Nov 7, 2021
39a3cf3
fix pipenv to avoid an issue introduced with 2021.11.5
yurijmikhalevich Nov 7, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
- name: Install Dependencies
run: |
python -m pip install --upgrade pip
pip install pipenv
pip install pipenv==2021.5.29
pipenv run pipenv sync --dev
pipenv run install_pyright
- name: Check Style
Expand Down
1 change: 1 addition & 0 deletions Pipfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ clip = {git = "https://github.com/openai/CLIP.git"}
torch = "==1.9.0+cpu"
torchvision = "==0.10.0+cpu"
pillow = ">=8.3.2"
requests = "~=2.26"
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add it via pipenv install "requests~=2.26" instead of editing the Pipfile?

This will also update Pipfile.lock and will make the tests pass. I can do it if you don't want to.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Struggling to get this to resolve without errors.

PS -- wondering what direction you want to take this project in the future. On my home machines it's now the main way I've been organizing my pictures; and I've hacked on a lot of features that might-or-might-not be out of scope for your vision for this project - including:

  • De-dup --- Embeddings are cool - they can even find duplicate thumbnails of different sizes. I added a script that finds pictures with identical (or near identical) embeddings, and flag them as a duplicate in a column I added to SQLite.
  • Support for explicit captions/tags/picking up words from folders --- for some images I already had them tagged with words that CLIP couldn't figure out. For example, folders of country names I've visited. I added a column for the CLIP embedding of the words I've manually tagged; and optionally also search that column. Without it, a search for 'germany' only returns about 1/10 of my pictures from germany.
  • Face embeddings -- so I can search for a specific person. I'm using https://github.com/serengil/deepface to add a column for faces. Currently I can't input the name of a face - but using the "similar to an existing image" I can find them. Combining this with the "explicit tagging" feature above I'll soon be able to search for people by name (as long as I have tagged them at least once)

No idea if you consider those in or our of scope for the main project.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ramayer, thanks for the update. I'll take a look at it.

About the direction:

  1. I would love to keep the rclip as a small and simple CLI tool, but considering building a more advanced app on top of it.
  2. I like the de-dup idea and think that it can be a part of the rclip CLI tool. But, I am not 100% sure about this one yet, so instead of filing a GHI, I created a Discussion here: Should we and how to add a de-dup functionality to rclip? #23.
  3. This is a must-have feature for a practical photo search tool, but implementing it will fundamentally change how rclip works. Also, if we will implement this in rclip it will start output pictures that don't feature anything German in them (if though they were taken in Germany; for example, a picture of a generic bed made in the hotel). I feel like this feature can be a part of another tool that will incorporate rclip. Or I should change how I think about rclip 😅 Let me take to some time to consider this. I also think that we may consider extracting manual/existing tags from the images themselves.
  4. Face embeddings feature sounds great, and my general attitude to it is more open compared to the one I have towards indexing the folder names.

Thank you for sharing all these. All good ideas that I need to consider.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ramayer, pipenv install "requests~=2.26" worked fine for me. But, you shouldn't forget to execute pipenv shell before doing anything with pipenv. I mention it here, in the Contributing section: https://github.com/yurijmikhalevich/rclip#contributing. Executing pipenv shell sets env variable from .env which is required to install the PyTorch with pipenv from the official PyTorch "repository."


[dev-packages]
pycodestyle = "~=2.7"
Expand Down
Loading