Prototyping: A New Shopping Experience

A Pivot to a New Shopping Experience

After our needfinding within the wine space, we were left a bit stuck, but with a core insight: the concept of "recommendation" is incredibly valuable. As a team, we also liked building around the themes of "recommendation" and "curation," as we found them to be both interesting social and technical problems to tackle.

One of the products we heard about during our chats was Wirecutter. This is a site, maintained by the New York Times, that curates items by hand and makes money by writing articles about these products and using affiliate links to capture the value. At a high-level, this was a super interesting concept: it was a way to stick with the idea of recommendation while also making money only when recommendations made sense. We also felt as though this offering could be made even better: while there is a lot of data on recommendations and reviews out there, it's super hard to sift through all of them. Furthermore, the democratization of large language models (LLMs) gives us the ability to do not just SQL-based search, but also semantic search using embeddings. All of this was to say: we wanted to pivot to a new concept, which was shopping powered by natural language.

Starting with Books

Given a pivot to a more generalized shopping experience, we wanted to ship something that we could validate on quickly. To this end, we decided to take our wine recommendation MVP and adapt it to a new product. We knew we had to start looking at a singular item, and we looked at categories that people searched for often, was bought online (since we could monetize long-term), and had a huge enough audience. To this end, we looked at books. In fact, we found that there were existing sites that tackled similar problems, but they weren't granular enough to handle general natural language.

With this prototype, we wanted to learn a couple of things:

1. Did people like our new value-add? We knew that there were already ways people searched online: is this an improvement, and if so, is it a 10x improvement compared to the alternative?
2. Does our basic recommendation algorithm -- which for the sake of the MVP, was rather simple -- provide quality enough search results for a wide array of queries?
3. What do people still want out of a recommendation engine? As LLM interfaces are still new, they can be tricky to get right, and so better understanding what people's habits and wants were in this context was relevant.

Tech Stack

At a high-level, our team used Retrieval Augmented Generation (RAG), which takes in a user query (i.e., "find me a book that is about fantasy basketball"), turns the query into an embedding, and uses similarity algorithms to find book descriptions and reviews that are similar to the query. The general process is described below:

1. As a data source, we took open-source Amazon review data on Kaggle with associated metadata.
2. Using the title, description, and three reviews, we formatted a descriptive text that was sent to OpenAI's embedding model.
3. We then saved these embeddings into a persistent Chroma DB client. When a user sends a natural language query, we call the same OpenAI embedding model and do a similarity search pulling up the top three most similar results.
4. As for actual serving, we deployed a small EC2 instance with Streamlit listening on port 8501 with permissions to accept incoming HTTP requests from any IP address.
5. After this, we bought a domain to connect to this IP address and were ready to use Prometheus to collect basic telemetry on visits + searches on our product and started sending out our prototype.

You can try our prototype here.

Takeaways and Learnings

After creating our initial MVP for book recommendation, we sent it out to our friends and family which was mostly composed of the Gen-Z/Millenial age group. Quantitatively, we got around 100+ unique page views and 150+ unique searches on the platform. The initial feedback from users that we received was positive, with statements like “[this] reminds me of how when you ask alexa to find a song you hum it” and more general thoughts like “woah this is so cool,” “awesome start,” and “I like it…the Amazon links are dope.” There were also users who noted that they saw strong search results on certain queries, with one person describing how they had read two of the books offered for the description “magic political drama” and asserted their quality.

There were also multiple areas that we were pointed towards to improve on in the offering. While we give solid recommendations on some queries, for others they are off the mark right now.

• For example, multiple people noted how when they asked for books similar to other books like “Lord of the Rings” and “Ranger’s Apprentice” rather than being given similar books they were given the books themselves.
• Furthermore, other people noted how there needs to be a wider variety of books in the database, with one user noting how when they asked for books about Napa Valley they were not given any in the domain, and when another user asked for books about rowing they were not offered any books about rowing.
• In a similar vein, another person asked for books by Ray Bradbury, but was offered the same book three different times as a response. When considering the user interface, people noted how it would be better to give more elaborate examples of how people can use the platform in terms of their descriptions in order to show them how the platform works better.
• People thought it would be beneficial to give more feedback on why certain books were offered, noting how if they ask for a book that is similar to Harry Potter they would like to know what makes the recommended book similar to Harry Potter.
• Another user also stated that they would like to see more than just three recommendations listed at a time, noting have if only the three most likely results are given people might have already have read these books.
• Lastly, a user stated that they thought it would be beneficial to see the cover of the books and be given some reviews of the books that are recommended.

Aside: Survey Data

We also launched a survey to better understand the random online shopper's views on searching. Some insights we gleaned from getting 22 people to anonymously respond to our Prolific survey:

• Most people actually don't have issues when finding things when they shop and search: someone even said it's too easy.
• A lot of people basically just use key words + looking in different places. Some did mention it being like a skill -- i.e., power users crafting boolean searches to find items.
• Biggest painpoint around search -- if any -- was around paid search priority. Some other issues mentioned also include:
  • Narrow down using different filters (price, quality, search) -- i.e., too many search options to choose from
  • Finding the best price for an item
  • Looking across different sites
  • Sort through online reviews
  • Irrelevant choices come up

In general, there seem to be some tailwinds.

Conclusions

While we are still reflecting on our sprint, some initial conclusions are:

• We need to integrate more data, while improving the quality of search recommendations. While this seems obvious, adding a lot data -- from more real-time data sources to API calls and others -- can make for a lot of data, so a way to properly search and recommend this data is required. In the future, we would use more in-depth algorithms like latent feature recommender systems like in CS 246.
• We also want to improve the overall search experience from an HCI perspective. This includes things like listing out recommendations for prompts, making the AI more explainable, and so on.
• Lastly: a big question we want to tackle is whether we want to tackle more recommendation or exploration. It seems as though an interesting use case may be for exploration, but is this path also as monetizable as pure recommendation and search?