For information about the project solution read Documentation.docx

Logo Similarity Clustering Pipeline

[cite_start]This project is a comprehensive solution designed to automatically group websites by the visual similarity of their company logos[cite: 3]. [cite_start]It tackles the complex task of identifying and clustering logos—a task that is simple for humans but challenging for machines[cite: 12]. The pipeline handles everything from data ingestion and web scraping to deep learning-based feature extraction and clustering.

---

📋 Project Workflow

The project is broken down into three main stages:

1. Data Preparation and Ingestion

[cite_start]The process begins by converting the initial .parquet file containing a list of domains into a more accessible .csv format[cite: 21]. [cite_start]This step ensures the data is easy to visualize and process in the subsequent stages[cite: 21].

2. Multi-Stage Logo Scraping

[cite_start]A robust, three-tiered strategy is used to scrape logos from each domain, ensuring a high success rate (estimated at 98-99%)[cite: 24, 30].

• [cite_start]Tier 1: Static Scraping with BeautifulSoup: The script first attempts to find the logo on a static version of the website by searching for common image tags (e.g., <img> tags where the src or alt attribute contains the word "logo")[cite: 27].

• [cite_start]Tier 2: Dynamic Scraping with Selenium: If the first attempt fails (often due to dynamically loaded content), Selenium is used to load the full webpage in a headless browser and repeat the search[cite: 27].

• Tier 3: Google Custom Search API Fallback: For websites that block scrapers or use unconventional methods to display their logos, the script uses a final fallback. [cite_start]It extracts the company name from the domain and uses the Google Custom Search API to find and download the most relevant image for the query "[company name] logo"[cite: 28, 29].

[cite_start]All scraped logos are saved to a local directory for processing[cite: 31]. [cite_start]The entire scraping process is multi-threaded using ThreadPoolExecutor for improved performance[cite: 32].

3. Feature Extraction and Clustering

Once the logos are collected, they are processed and clustered using a machine learning pipeline:

• [cite_start]Database Storage: Logos are inserted into a MySQL database as binary data for persistent storage and efficient retrieval[cite: 34, 35].

• Feature Extraction: The pre-trained EfficientNetB0 model (via TensorFlow/Keras) is used to analyze each image and convert it into a high-dimensional feature vector. [cite_start]This vector numerically represents the key visual characteristics of the logo[cite: 38, 39].

• [cite_start]Dimensionality Reduction: Scikit-learn's PCA is applied to reduce the dimensionality of the feature vectors, making the clustering process more efficient[cite: 38].

• [cite_start]Clustering: Faiss, a library for efficient similarity search, is used to perform K-Means clustering on the reduced feature vectors, grouping similar logos together[cite: 40].

• [cite_start]Exporting Results: The final clusters are exported to both a .csv file for data analysis and a styled .html file for easy visual inspection of the logo groupings[cite: 40].

---

🛠️ Technology Stack

• Programming Language: Python
• [cite_start]IDE: PyCharm [cite: 20]
• Data Handling: Pandas
• [cite_start]Web Scraping: Selenium, BeautifulSoup, Requests [cite: 25]
• Image Processing: Pillow (PIL)
• [cite_start]Database: MySQL [cite: 34]
• [cite_start]Machine Learning & Deep Learning: TensorFlow, Keras, Scikit-learn [cite: 36]
• [cite_start]Similarity Search/Clustering: Faiss [cite: 36]
• [cite_start]Concurrency: concurrent.futures [cite: 32]
• [cite_start]APIs: Google Custom Search API [cite: 29]

---

🚀 How to Run

1. Clone the Repository:

   git clone [https://github.com/your-username/your-repository-name.git](https://github.com/your-username/your-repository-name.git)
   cd your-repository-name

2. Install Dependencies: Install the necessary Python libraries from requirements.txt.

   pip install -r requirements.txt

3. Setup Database:

   • Ensure you have a MySQL server running.
   • [cite_start]Update the database configuration (db_config) in the main script with your host, user, password, and database name[cite: 41].

4. Configure API Keys:

   • Obtain an API Key and a Search Engine ID from the Google Custom Search API.
   • Create the necessary files (API_KEY and SEARCH_ENGINE_ID) or update the script to load your credentials.

5. Run the Scripts:

   • Execute the web scraping script to gather the logos.
   • Execute the classification script to process the images and generate the final cluster reports.

---

🔮 Future Improvements

[cite_start]While the current solution is robust, scalability could be enhanced by addressing the following[cite: 43]:

• Developing more advanced methods to bypass website scraping restrictions.
• Optimizing the web scraping process to reduce resource consumption.
• Fine-tuning the AI model on a logo-specific dataset to further improve clustering accuracy.