Repository navigation
Starting an AI‐Powered Feature
When I started this feature #327 , I had no idea how many challenges and detours I’d face. I tried different tools, models, and approaches, some worked, and many didn’t. I hope that by sharing what I went through, you’ll save time and avoid making the same mistakes if you ever work on something similar.
The first thing I tried was using existing image classification APIs like Google Vision AI, Amazon Rekognition, and others. The idea was simple: why reinvent the wheel when these services promise high accuracy and are already widely used?
However, as I dug into the pricing, I quickly realized these services were far too expensive for our project, especially given the volume of images we’d need to classify. While they are great for companies with bigger budgets, they weren’t the right fit for a cost-sensitive project like ours.
Next, I decided to experiment with TensorFlow.js and the MobileNet model. This approach seemed promising because TensorFlow.js runs entirely in the browser, and MobileNet is a lightweight, pre-trained model designed for resource-constrained environments. The setup was relatively easy, and I quickly integrated it into my project.
The real test came when I started running classification tasks with images of plumbers, electricians, and other service categories. Unfortunately, while MobileNet is great for general-purpose tasks, it wasn’t specific enough to handle the nuances of these service categories. For example, it couldn’t reliably distinguish between a carpenter’s tools and an electrician’s tools. The results were inconsistent and not accurate enough for a real-world application. It was a step forward but also a dead end for this project.
After realizing that MobileNet’s pre-trained capabilities weren’t enough, I turned to fine-tuning the model with custom data. I collected and labeled datasets for each service category and issues, using Roboflow to help manage the data and train the model. Each dataset contained around 1,000 images, which I hoped would teach the model to recognize the differences between categories like "cleaner" and "carpenter."
The training process took a lot of time and effort, not just because of the sheer number of images but also because fine-tuning requires significant computational resources and experience. In the end, the fine-tuned model performed better than the original MobileNet. However, it still wasn’t good enough. Some categories improved, but others remained problematic. Despite the improvement, I realized I needed a different approach one that didn’t require so much manual work and resources.
At this point, I decided to take a step back and look for pre-trained models that could work out of the box. That’s when I found CLIP and BLIP, two powerful models designed for tasks like image classification and captioning. These models are pre-trained on huge datasets, which makes them highly accurate and versatile.
Testing these models was a game-changer. CLIP was especially effective because it doesn’t rely on traditional classification methods. Instead, it works by comparing image embeddings with text embeddings, making it perfect for categorizing images into well-defined labels like “PLUMBER” or “ELECTRICIAN.” BLIP, on the other hand, was great for generating descriptions of images. For the first time, I felt like I had found tools that fit my needs perfectly.
While CLIP and BLIP worked well locally, I needed a way to integrate them into my app. That’s when I decided to set up a Flask backend. The idea was to host the models on a server so that my app could send images to the backend and receive classification results in return. I used Retrofit in my Android app to make the API calls .
I chose to create a Flask backend to act as a secure and efficient bridge between the app and the AI models (CLIP and BLIP). This approach reduces the load on our app by handling the heavy computational tasks server-side, ensuring the app remains lightweight and responsive. It also allows for centralized logic to manage API requests, pre-process inputs, and post-process outputs, ect giving us more control and flexibility in adjusting the results to our needs. Using a backend also improves security by keeping API keys safe, prevents direct exposure to external services, and ensures better error handling with user-friendly responses. Additionally, this setup is scalable and adaptable, making it easier to expand features or integrate new models in the future without modifying the app itself.
Deploying the backend was a BIG BIG challenge. Initially, I used Heroku, but the free plan’s 512MB memory limit wasn’t enough for these models. To optimize the backend, I added Redis to cache results, which helped reduce resource usage. However, even with Redis, I still ran into performance issues. Hosting the backend turned out to be one of the most time-consuming parts of the project. I’m currently planning to use Docker to improve this setup.
After struggling with backend hosting, I started exploring the Hugging Face Inference API. This platform allows you to use pre-trained models without worrying about hosting them yourself. It seemed like the perfect solution for my use case since it offers a scalable way to run models like CLIP and BLIP.
So far, I’ve been testing the API with sample images and configuring it to handle our service categories. While there have been some hiccups with image data formatting, the results look promising. This approach has the potential to save me from the headaches of maintaining a backend while still providing the accuracy I need and we need.
I wrote this not just to document the process but to help anyone in my position see what worked and what didn’t. I hope this helps you and others in the future to start smarter, avoid the pitfalls I fell into, and build better solutions.