Skip to content

Tech Stack

Jump to bottom
Soyoung Yoon edited this page Nov 30, 2025 · 5 revisions

Document Revision History

Date Version Details
2025/11/30 v1.0 Initial documentation
2025/11/30 v1.1 Add experimental results

MomenTag Tech Stack

This page documents the AI/ML and search stack used in MomenTag, and explains how the current production implementation evolved from the original PoC design.

The focus is on:

  • How images and text are embedded
  • How search & recommendation are computed
  • How Qdrant and MariaDB work together
  • What changed between the PoC and the production system

1. Production Implementation (Current System)

Code base: backend/

The current implementation is optimized for multi-user, scalable, and maintainable service operation.

1.1. Image Embedding

  • Vector DB: Qdrant
  • Collection: IMAGE_COLLECTION_NAME
  • Vector dimension: 512
  • Distance metric: Cosine similarity
  • Model: clip-ViT-B-32 (SentenceTransformers)
  • Main code: backend/gallery/gpu_tasks.py

Processing flow:

  1. User uploads a photo from the Android client.

  2. A Celery GPU task generates an image embedding.

  3. The resulting vector is upserted into Qdrant’s IMAGE_COLLECTION_NAME.

  4. Payload stored alongside the vector includes:

    • user_id
    • filename
    • photo_path_id
    • created_at
    • lat, lng
    • isTagged (whether any tags are attached)

Indexed payload fields in Qdrant (used for filters/sorting):

  • user_id, filename, photo_path_id, created_at, lat, lng

1.2. Text & Caption Embedding

  • Text embedding model: clip-ViT-B-32-multilingual-v1

  • Caption generation model: BLIP

    • Salesforce/blip-image-captioning-base

  • Caption storage: Photo_Caption table in MariaDB

  • Key modules:

    • Embedding: backend/search/embedding_service.py
    • Captioning: backend/gallery/gpu_tasks.py

Caption pipeline:

  1. For each uploaded image, BLIP generates 5 candidate captions via a Celery GPU task.

  2. The captions are split into tokens using phrase_to_words() (stop words removed).

  3. Each token is stored in Photo_Caption with:

    • token text
    • frequency / weight
    • image linkage

  4. At query time, matching caption tokens are retrieved from the DB to give caption-based bonus in hybrid search.

BLIP generation parameters (production):

  • max_new_tokens: 20
  • num_return_sequences: 5
  • do_sample=True, top_k=50, top_p=0.95

1.3. Representative Vectors (RepVec)

  • Collection: REPVEC_COLLECTION_NAME (Qdrant)
  • Purpose: Store representative vectors per tag for tag recommendation.
  • Main code: backend/gallery/tasks.py → compute_and_store_rep_vectors

Representative vectors are computed as a Celery asynchronous task whenever the composition of a tag changes.

Trigger conditions:

  • Tag has at least 10 photos attached.

Algorithm:

  1. HDBSCAN clustering

    • min_cluster_size = 5

    • min_samples = 3

    • metric = "euclidean"

    • Produces:

      • One or more dense clusters
      • Outliers labeled as -1

  2. Cluster centers

    • For each non-outlier cluster, compute the mean vector as its center.

  3. Final representative set

    • Union of:

      • Cluster centers
      • Outlier vectors

    • All of them are written into REPVEC_COLLECTION_NAME.

  4. Small tags (< 10 photos)

    • For small tags, every image embedding is treated as a representative vector (no clustering).

Execution:

  • Whenever photos are added or removed from a tag, compute_and_store_rep_vectors is enqueued.
  • This task runs on the interactive Celery queue (CPU only).

1.4. Preset Tag Vectors

  • Collection: TAG_PRESET_COLLECTION_NAME
  • Purpose: Provide generic, pre-defined tags for recommendation, such as “Travel”, “Food”, “Friends”, etc.

These preset vectors are not tied to any specific user and act as a fallback for users with limited personal data.

2. Production Search Algorithm

Main modules:

  • Strategy pattern: backend/search/search_strategies.py
  • Hybrid search orchestration: backend/gallery/tasks.py → execute_hybrid_search

The search layer distinguishes between:

  • Tag-only queries: {TagName}
  • Pure natural language queries
  • Hybrid queries: tags + natural language

A strategy pattern is used to switch between three search strategies.

2.1. Strategy 1 — TagOnlySearchStrategy

Condition: Query contains only tag syntax, e.g. {Travel}

Logic:

  1. From MariaDB, retrieve all photos explicitly attached to the given tag(s).

    • Each such photo starts with a base score of 1.0.

  2. For each tag, call Qdrant recommend():

    • positive: all photo IDs belonging to that tag
    • score_threshold: typically 0.2
    • limit: 1000

  3. Combine scores for multiple tags:

    • For each photo, gather scores from each tag.

    • Compute product of tag scores:

      product_score = score₁ × score₂ × ... × scoreₙ

    • To avoid collapsing scores due to repeated multiplication of values < 1, apply a scaling term:

      final_score = product_score × 2^(n-1)

    • If a tag score is missing, minimum value 0.1 is used.

  4. Sort by final_score descending and apply pagination (offset, limit).

2.2. Strategy 2 — SemanticOnlySearchStrategy

Condition: Query contains only natural language, no {} tags.

Logic:

  1. Embed the text query using create_query_embedding().

  2. Call Qdrant search() on IMAGE_COLLECTION_NAME:

    • query_vector: embedded query
    • score_threshold: typically 0.2
    • limit: 1000

  3. Sort by score and paginate results.

2.3. Strategy 3 — HybridSearchStrategy

Condition: Query mixes tags and natural language, e.g. {Travel} beach sunset

Logic consists of three phases:

Phase 1 — Tag Score

  • Same tag-based scoring as in TagOnlySearchStrategy:

    • Photos belonging to tags get base 1.0.

    • Qdrant recommend() is used to expand candidates.

    • Multi-tag scores are multiplied and scaled:

      product_score = score₁ × score₂ × ... × scoreₙ
final_tag_score = product_score × 2^(n-1)

    • Minimum value is clamped at 0.1 to avoid zeros.

Phase 2 — Semantic Score

  • The natural language part of the query is embedded.
  • Qdrant search() returns semantic scores for candidate photos.

Phase 3 — Caption Bonus

  • The NL query is tokenized via phrase_to_words().

  • From the Photo_Caption table, photos whose caption tokens match query words are counted.

  • For each matched word, a 0.1 bonus is added:

    caption_bonus = num_matching_tokens × 0.1

Final Score

final_score = (tag_weight × tag_score) 
            + (semantic_weight × semantic_score) 
            + caption_bonus

Default weights:

  • tag_weight = 1.0
  • semantic_weight = 1.0

Sorted by final_score descending and paginated.

3. Image Recommendation (Tag Album–Based)

Endpoint:

GET /api/gallery/tag/<tag_id>/recommend/

Goal: Suggest additional photos that fit well into a given tag album.

Main code: backend/gallery/tasks.py → recommend_photo_from_tag

Logic:

  1. From DB, fetch all photo IDs currently attached to the tag.

  2. Call Qdrant recommend() on IMAGE_COLLECTION_NAME:

    • positive: all photos of the tag (no RepVec used here)
    • limit: 80
    • Filter by user_id to stay within the same user’s photos.

  3. Filter out all photos already tagged with this tag.

  4. Return the top 40 remaining candidates.

Key idea: Use the entire set of tag photos as positive examples, letting Qdrant infer similar images in the vector space.

4. Tag & Caption Recommendation (Photo-Based)

Endpoint:

GET /api/gallery/tag-recommendations/<photo_id>/

Goal: Recommend suitable tags (and caption-like phrases) for a single photo or a batch of photos.

Main code: backend/gallery/tasks.py

4.1. Single-Photo Recommendation (tag_recommendation)

Steps:

  1. Retrieve the image vector for the photo using Qdrant retrieve() on IMAGE_COLLECTION_NAME.

  2. Preset tag search:

    • Collection: TAG_PRESET_COLLECTION_NAME
    • Query vector: image vector
    • limit = 10
    • score_threshold = 0.0 (wide net)

  3. User tag search:

    • Collection: REPVEC_COLLECTION_NAME
    • Query vector: image vector
    • limit = 10
    • score_threshold = 0.75
    • Filter by user_id so only the user’s tags are considered.

  4. Results are merged and deduplicated:

    • Preset tags:

      { "tag": "<name>", "tag_id": "", "is_preset": true }

    • User tags:

      { "tag": "<name>", "tag_id": "<uuid>", "is_preset": false }

The frontend can present these as a mix of generic and user-specific tag suggestions.

4.2. Batch Tag Recommendation (tag_recommendation_batch)

Use case: Story/Moment generation or any situation that needs tag suggestions for many photos at once.

Optimizations:

  1. Bulk vector fetch:

    • A single Qdrant call retrieves vectors for all target photos.

  2. Parallel recommendation:

    • A ThreadPoolExecutor (up to 10 workers) runs preset & user-tag searches in parallel for each photo.

  3. Bulk DB queries:

    • Tag metadata is resolved from the DB in batch to reduce query overhead.

Recommendation sources:

  1. TAG_PRESET_COLLECTION_NAME

    • Up to 2 preset tags per photo.

  2. REPVEC_COLLECTION_NAME

    • Up to 10 user tags per photo, with score_threshold = 0.75.

Result format:

  • A mapping from photo_id to a list of suggested tags:

    {
  "<photo_id>": [
    { "tag": "...", "tag_id": "", "is_preset": true },
    { "tag": "...", "tag_id": "<uuid>", "is_preset": false }
  ],
  ...
}

Duplicate tags are removed per photo.

5. Experimental Results

This section summarizes the evaluation of the search and recommendation system using a combined dataset of COCO test2017 and 600 personal photos. The experiments measure the effectiveness of tag-based recommendation, hybrid search, and natural-language attribute fusion.

5.1. Experimental Setup

Dataset

  • COCO test2017: ~40,000 images

  • Personal dataset: 500 images (5 sets × 100 images each)

  • Test tags:

    • Escape Room (#escape)
    • Hoodie (#hoodie)
    • Cooking
    • Jeju Trip
    • Screenshot

Methodology

For each tag, 3 random reference images were manually assigned the tag. The system was then evaluated on how accurately it retrieved the remaining personal photos from the combined dataset.

Evaluation focuses on:

  • Image recommendation from tagged examples
  • Hybrid search (tag + natural language)
  • Attribute fusion (color, count, etc.) within NL queries

5.2. Experiment 1 — Tag-Based Image Recommendation Accuracy

This experiment measures how accurately the system recommends the correct personal images based on the 3 tagged reference images, using Qdrant’s recommend() API.

Results

Tag Top 30 Top 60 Top 90 Notes
Escape Room 30/30 (100%) 59/60 (98%) 79/90 (87%) Strong visual distinctiveness
Hoodie 30/30 (100%) 60/60 (100%) 80/90 (88%) Clothing without people reduces precision at Top 90
Screenshot 30/30 (100%) 58/60 (96%) 69/90 (76%) —
Cooking 27/30 (90%) 52/59 (88%) 75/90 (83%) COCO contains many food images → noise
Jeju Trip 20/30 (66%) 30/60 (50%) 40/90 (44%) COCO contains many landscape/ocean images → noise

Key Finding

  • Tags with clear visual identity (Escape Room, Hoodie) achieve extremely high precision.
  • Tags with high visual overlap with the COCO dataset (Cooking, Jeju Trip) suffer from more false positives.

5.3. Experiment 2 — Hybrid Search (Tag + Natural Language)

Escape Room and Hoodie images are rare in COCO, making them strong candidates to test hybrid tag + NL performance.

5.3.1. Effect of Natural Language Specificity

The specificity of the verb in the natural language query significantly influenced retrieval quality.

Target condition: Among 100 hoodie-related images, 35 actually contain people wearing hoodies. These 35 images were treated as the “true target” subset.

Queries Tested

  • Query A (General) — people with #hoodie
  • Query B (Specific) — people wearing #hoodie

Results

Scroll Depth Query A: with (Target / Hoodie / Else) Query B: wearing (Target / Hoodie / Else)
First 15 4 / 4 / 7 11 / 3 / 1
Next 15 1 / 14 / 0 3 / 12 / 0
Next 15 0 / 11 / 4 4 / 9 / 2

(Target : People wearing hoodie / Hoodie : Hoodie without people / Else : No hoodie)

Conclusion

  • Ambiguous prepositions like “with” weaken semantic signals.
  • Specific verbs like “wearing” dramatically improve retrieval of true target images.
  • Natural language precision has a direct and substantial impact on hybrid search quality.

5.3.2. Attribute Fusion: Color & Count

This experiment tests how well the system reflects attribute-level natural language (color, number of people) in combination with tags.

A. Color Attribute ({color} #hoodie)

Color Total Photos Found in First 15 Found in First 30 Notes
Gray 26 12 19 —
Black 36 14 27 —
Many 7 — All 7 within 114 images
Green 4 — All 4 within 63 images

B. Counting Attribute (#escape with {n} people)

True distribution in the dataset:

People Count Ratio
2 32%
3 40%
4 20%
5 8%

However, the search results prioritized the requested count in the NL query more strongly than the natural dataset distribution.

Query (N) First 15 (Matched) Next 15 Next 15
Two 9 7 3
Three 13 5 6
Four 3 3 1
Five 2 3 0

Conclusion

  • The system successfully integrates tag context + NL attributes.
  • Attributes like color and count are accurately reflected even when dataset distribution suggests otherwise.
  • Attribute fusion follows an implicit priority order: NL attribute > tag context > raw visual distribution

5.4. Overall Conclusions

  • Tag-based image recommendation already achieves very high accuracy, especially for visually distinct categories.
  • Hybrid search performance is strongly influenced by the precision of natural language phrasing.
  • The system accurately incorporates attribute-level semantics (color, number of people) on top of tag context.
  • Categories with high overlap with public datasets (Cooking, Jeju Trip) produce more false positives, expected to improve as personal data volume increases.

6. Qdrant API Usage Summary

Feature Qdrant API Purpose
Tag-based expansion recommend() Expand from a set of tagged images
Natural language search search() Semantic similarity search with query vector
Hybrid search recommend() + search() Tag + semantic fusion
Image recommendation recommend() Recommend images for a tag album
Tag recommendation retrieve() + search() Use image vector to search RepVec & preset tags
RepVec management retrieve(), delete(), upsert() CRUD representative vectors

7. PoC Implementation (Historical Design)

The original PoC explored more experimental techniques, especially around personalized fine-tuning and graph-based recommendations. The production system later simplified and hardened these ideas.

7.1. Image Embedding (PoC)

  • Model: openai/clip-vit-large-patch14
  • Vector dimension: 768
  • Fine-tuning: LoRA adapters per user (PEFT)
  • Storage: NumPy arrays (embedding_vectors.npy), plus JSON metadata

LoRA fine-tuning:

  • Target modules: ["q_proj", "v_proj"]

  • Hyperparameters:

    • r = 16
    • lora_alpha = 32

  • Training data:

    • User’s uploaded images
    • BLIP captions

  • Objective:

    • Contrastive learning (image–text matching)

  • Output:

    • Per-user adapters under /user_adapters/{userName}

Flow:

  1. Train user-specific LoRA adapter (~50 epochs).
  2. Use fine-tuned CLIP to generate embeddings.
  3. Store vectors in .npy files.

7.2. Text & Caption Embedding (PoC)

  • Text embedding: Same fine-tuned CLIP model (per user)
  • Captioning: BLIP Salesforce/blip-image-captioning-base
  • Storage: .npy files

BLIP usage:

  • 5 captions per image

  • Optimizations:

    • FP16
    • torch.compile()
    • Batch size 8

Captions were primarily used as training data for LoRA, rather than direct search bonus.

7.3. Representative Vectors (PoC)

  • Storage: Faiss IndexFlatL2
  • Purpose: Image recommendation

Algorithm:

  1. Outlier removal: IsolationForest (contamination=0.05)

  2. Clustering: KMeans (K=10)

  3. Final representative set:

    • Outlier vectors
    • Cluster centers

These representative vectors were then indexed by Faiss to support nearest-neighbor queries.

7.4. Preset Tags (PoC)

  • Not implemented.
  • Tag recommendation relied solely on user-generated tags and RepVec.

7.5. Search Algorithm (PoC)

  1. Query is embedded with fine-tuned CLIP (user-specific LoRA).

  2. Cosine similarity is computed between query vector and stored image vectors using:

    • sklearn.metrics.pairwise.cosine_similarity

  3. Top-K images are returned (e.g., Top-20).

Characteristics:

  • Only natural language search was supported.
  • No tag syntax or hybrid search.
  • No caption bonus; captions were used indirectly via fine-tuning.

7.6. Image Recommendation (PoC)

Two approaches were explored:

7.6.1. RepVec + Faiss

  1. Compute representative vectors for each tag (IsolationForest + KMeans, K=10).

  2. Build a Faiss IndexFlatL2 over these vectors.

  3. For recommendation:

    • Compare candidate image vectors against tag representative vectors.
    • Use L2 distance or cosine similarity (with normalized vectors).

7.6.2. Bipartite Graph + PageRank

  • Graph structure:

    • Nodes:

      • Image nodes (bipartite=0)
      • Word nodes (bipartite=1)

    • Edges:

      • Image–word edges with weights (e.g., caption frequency)

  • Algorithms via NetworkX:

    1. Weighted Rooted PageRank (RWR)

      • Personalized PageRank seeded on input images.

    2. Adamic/Adar index

      • Measures similarity via shared neighbors (words).

  • Final score:

    final_score = 0.6 × normalized_AdamicAdar + 0.4 × normalized_RWR

This graph-based approach was ultimately dropped for production due to complexity and scalability concerns.

7.7. Tag Recommendation (PoC)

  • Representative vectors for each tag were computed using IsolationForest + KMeans (K=3).
  • A Faiss index was built over these tag-level vectors.
  • For a given photo, the closest representative vectors were retrieved; connected tags were recommended (excluding tags already attached to the image).

7.8. Additional Feature (PoC Only): Image De-duplication

  • Model: clip-ViT-B-32 (SentenceTransformers)

  • Approach: Window-based duplicate detection

    • Maintain embeddings of the most recent 5 images.
    • If cosine similarity ≥ 0.85, treat as a duplicate.

  • Optimizations:

    • Compressed input images (JPEG quality ~25)
    • Batch size 32
    • FP16 inference

In production, this responsibility was shifted away from the backend and handled more simply on the client side (e.g., using photo_path_id).

8. Production vs PoC – Key Differences

8.1. Embedding & Storage

Aspect PoC Production
Image model CLIP-ViT-Large (768-dim) CLIP-ViT-B-32 (512-dim)
Fine-tuning LoRA per user Removed
Vector storage NumPy files + JSON Qdrant vector DB
Search engine Faiss (in-memory) Qdrant (distributed, persistent)
Multi-user support Not designed Built-in via user_id filters

Why LoRA and Faiss were dropped:

  • LoRA:

    • Complex training/infrastructure
    • High GPU cost
    • Pretrained models already perform well enough

  • Faiss:

    • In-memory only
    • Limited filtering capabilities
    • No built-in persistence

8.2. Representative Vectors

Aspect PoC Production
Algorithm IsolationForest + KMeans (fixed K) HDBSCAN (automatic cluster count + outlier detection)
Storage Faiss index Qdrant REPVEC_COLLECTION_NAME
Main usage Image recommendation Tag recommendation

8.3. Search

Aspect PoC Production
Query types Natural language only Tag-only, semantic-only, hybrid
Strategy pattern No Yes (3 strategies)
Hybrid scoring N/A Tag score + semantic score + caption bonus
Caption usage Training data for LoRA Direct bonus in hybrid search

8.4. Recommendations

Type PoC Production
Image recommendation RepVec + Faiss / Graph-based PageRank+AA Qdrant recommend() using all tag photos
Tag recommendation RepVec + Faiss Qdrant over RepVec + preset tags
Graph-based methods NetworkX (RWR + Adamic/Adar) Removed (complex & hard to scale)

8.5. Overall Evolution

Carried forward from PoC:

  • Representative vector concept (though algorithm & use changed)
  • BLIP captioning (now used as search signal, not training data)
  • Vector-based semantic search

Simplified or removed:

  • User-specific LoRA fine-tuning
  • Graph-based recommendation with NetworkX
  • Backend duplicate detection with embeddings
  • torch.compile() tricks aimed at local optimization

New in production:

  • Qdrant as a production-grade vector DB
  • Full hybrid search (tags + natural language + caption bonus)
  • Strategy pattern for search behavior
  • Explicit TAG_PRESET_COLLECTION_NAME for generic caption/tag suggestions
  • HDBSCAN for flexible clustering
  • Thread-based parallelization for batch recommendations

9. Summary

The PoC phase allowed aggressive experimentation with fine-tuning, graph algorithms, and in-memory search. The production system then distilled these ideas into a simpler, scalable, and maintainable architecture:

  • Qdrant replaces ad-hoc vector storage and Faiss-only setups.
  • Search quality is driven by a combination of tag signals, semantic similarity, and caption-based bonuses rather than per-user fine-tuning.
  • Representative vectors survive, but focus on tag recommendation, while image recommendation is handled directly via Qdrant’s recommend() over full tag sets.

This evolution keeps the spirit of the PoC—semantic understanding of personal photos—while meeting real-world constraints of reliability, cost, and operational complexity.

Clone this wiki locally