SimVec

Description

In our modern world, we all have lots of photos on our phones, and it can be a real hassle to find the one you're looking for. SimVec is here to make that easy. With SimVec, you can search for photos in two cool ways: you can find similar images by using a picture you already have, or you can type in a description to find pictures that match.

As a user of SimVec, you will be uploading your gallery to the server to be converted to vectors by Machine Learning models and stored in a Vector Database. Once this operation is done, you will be able to upload a brand new picture that you want to find the similar ones in your gallery. Or you will also be able to send a text query to the server that describes the image you are looking for.

SimVec uses a special database called Milvus to make everything fast and smooth. Milvus is a high-performance vector database that is optimized for image similarity search. This means SimVec has a scalability such that it can find similar photos in your collection very quickly, even if you have a billion scale library without a huge decrease in the efficiency, thanks to indexing algorithms that come with Milvus Vector Database.

SimVec is a small and easy-to-use app, with some extra features to make it even better. It's made for people who want a quick and simple way to search through their photo collections and don't have time to wade through thousands of photos to find the one they're looking for.

Simvec Demo Video

Simvec.Demo.mp4

Component Diagram

The description of the overall system architecture above is the following. There shall be a client application, namely Mobile Application, that the user can interact with. This client application shall have a good-looking and user friendly GUI in the front-end part. This client application shall send and receive data with the server. Although there may be several data incoming and outgoing in the client application, the main data that will be sent and received is the image or text.

In the server, there will be a back-end application that will serve for incoming requests. This back-end application shall organize the jobs of other components in the server, and apply any logic related to the project. It can be considered as the orchestration chief of the components in our server. It may have several microservices inside it. There are 3 main components in the server that the back-end application uses: Machine Learning Model, Vector Database and User Gallery. The jobs that they are assigned are organized by the back-end application.

Components and their Interactions:

Frontend Application (User Device - In React-Native)

1. Register:

Allows users to create an account.

2. Login:

Allows users to log in to their account.

3. Image Based Search:

Users can search for images using an image as a query.

4. Text Based Search:

Users can search for images using text as a query.

5. Synchronization:

Ensures data synchronization between the user device and the server.

Java Backend

1. Register, Login:

Manages user authentication and registration.

2. Image Based Search, Text Based Search:

Receives search requests from the frontend and communicates with the Python Vector Database Backend to process these requests.

3. Synchronization:

Manages synchronization tasks.

4. Image transfer:

Facilitates the transfer of images between different components.

Python Vector Database Backend

1. Image Based Search, Text Based Search:

Interfaces with the Java Backend to perform searches using vector databases.

2. Create Embedding:

Converts input images or text into vector embeddings using the MLP Model.

3. Store Embeddings:

Stores these embeddings in the Milvus Vector Database.

4. Create Collection:

Initializes collections in the Milvus Vector Database for storing embeddings.

5. Similarity Search:

Queries the Milvus Vector Database to find similar images based on embeddings.

6. Get Image:

Retrieves images from the storage when needed.

MLP Model

1. Create Embedding:

Generates vector embeddings from input data (images or text).

2. Store Embeddings:

Interacts with the Python Vector Database Backend to store these embeddings in the Milvus Vector Database.

Milvus Vector Database

1. Store Embeddings:

Stores the vector embeddings generated by the MLP Model.

2. Get Similar Images:

Retrieves similar images based on the similarity search queries.

3. Create Collection:

Sets up collections for organizing stored embeddings.

4. Similarity Search:

Conducts searches to find similar embeddings.

Images Storage

1. Get Image:

Provides the actual images when they are requested by other components.

Workflow:

User Interaction:

The user interacts with the frontend application to perform various actions like registering, logging in, and performing image or text-based searches.

Search Requests:

When a user performs a search, the request (image-based or text-based) is sent to the Java Backend.

Processing Search:

• The Java Backend forwards the search request to the Python Vector Database Backend.
• The Python Vector Database Backend uses the MLP Model to create embeddings for the input query.
• These embeddings are then compared against stored embeddings in the Milvus Vector Database to find similar images.

Retrieving Results:

• Once similar images are identified, their references are sent back through the Python Vector Database Backend to the Java Backend.
• The Java Backend then retrieves the actual images from the Images storage and sends them back to the frontend application.

User Receives Results:

The user receives the search results on their device, which includes the images similar to their query.