Prototype v3: SUPost Battle for Software Fair

Software Fair Prototype

Updating Find and Filter

Retrieval augmented generation lies at the heart of FaF. In this LLM-based technique, a knowledge source outside of the LLM is first utilized to lower the state space that the LLM will eventually have to parse through. Once information is extracted from this knowledge source, it is then passed to the LLM to generate the final output. This approach allows for the LLM to not be overwhelmed while at the same time utilizing the ability of these models to better grasp the nuances of provided data.

For our previous v2 prototype Bookworm, while algorithm design and code implementation was at the forefront of our minds, in order to be able to rapidly iterate and get quick feedback we initially structured our codebase to be heavily centered around the Amazon books dataset relevant to this demo. While the core elements of find and filter still existed, our codebase itself was directly referencing the configuration of this dataset and was not attuned to the B2B software product we now saw as Kenja's main value add. Thus the next step we took was restructuring our code base in order to be able to process any structured csv and run FaF on the data in order to allow companies to easily run this new form of search on their platforms.

Initial Data Processing

The first stage in the new pipeline for our backend revolves around the dataset itself. As noted above, we decided to first structure our approach around structured csv data, as we felt like this formatting is extremely common and pretty ubiquitous. Similar to standard databases, the csv data provided needed to have an identifier (id) to every row in order for us to be able to properly track data in the backend. We also made sure to have functionality where users could provide multiple csv files provided that each row of all the csv files provided contained a column with the relevant identifier. The user lets us know the identifier on the front end when we process our data. Once the all the csv files are provided by the user, we combine all of them in to a single pandas dataframe to make data processing easier at the cost of a larger memory footprint. This increased memory usage is a cost that we were willing to make in order to iterate more quickly, but in the future is something that we would optimize in order to keep a lower memory footprint which would be more crucial when dealing with large-scale datasets.

With the csv data in hand, we then created the vector databases to store the embeddings for each layer of FaF using chromaDB. This step is done before runtime in order to lower the latency of search. We used an open-source embedding model from Nomic AI to create the embeddings. These vector databases were created for all of the stages of FaF before generation from the LLM described below.

Selecting a Distance Function

Within FaF, querying our vector database is a key part of the algorithm. In order to do so, we must utilize a distance function to decide how similar the query embedding and candidate embedding are to one another. The default distance function utilized by our vector database library, chromaDB, is the squared L2 norm. Since this distance function is the default, we used this function for our previous prototype. However, when investigating the technical report for the open-source embedding model from Nomic AI that we utilized to create the embeddings for our vector database, we found that this embedding model was trained using cosine similarity. In turn, we adjusted querying of our vector database to use cosine similarity which should lead to better search results.

Noun Querying

Once this initial data processing stage is completed, we then move on to iteratively funneling down the dataset until only a few candidates remain to be provided to the user as the result of their search query. The first step in this process is noun querying. This step in the pipeline is a direct result of the team's learnings from Breakathon, for we found that the original version of FaF without this stage was not able to handle more traditional keyword based searches very robustly. Thus, we decided to add this initial layer to better select more relevant candidates in the initial stages of the pipeline relating to the user's query to be passed on to the later stages. With noun querying, we first extract out all the nouns in each row of the combined pandas dataframe and create embeddings for each row based on all of the nouns that they individually contain. We do the same for the user's query then run cosine similarity between the user's query embedding and each of these noun embeddings and keep only keep a specified subset of the original dataset which are revealed to be the most similar. This approach is meant to ground potential candidates in their core aspects so that other details that might be contained in details do not overly confuse the algorithm too early in the process. For example, if a user searched "blue sunglasses" but the dataset contained "blue shoes," we do not want the detail of "blue" to cause the shoes to be presented. At the same time, we still want a large enough state space to provide interesting candidates for more exploratory rather than keyword-based searches (think "I am looking for something nice for my lawn"), so we try to strike a balance between the two and utilize the later stages of FaF.

Feature Querying

We next move on to feature querying. For this step of FaF, the user is first asked on the frontend which features (columns of the csv) of their dataset they view as being the most relevant. We made the decision to have the user provide this specification because they would have the most knowledge about their dataset and we thus wanted to utilize this valuable domain knowledge. We also considered iterating on different columns of the csv as features and doing this step more algorithmically (think A/B testing on the customer's website with different features activated), but we decided that for the purposes of this current prototype that this approach would be a little too untenable and out of scope. Once the user has specified the features, we then create different collections (unique databases) for each of the features and run embedding search on each of these collections. In doing so, we extract candidates from each of these feature collections which are then passed on to the next stage of FaF. Note that feature querying is run at the initial stages of the pipeline because each individual feature will have some knowledge of a given identifier but will at the same time not have the same information as other features, so we believe this approach is useful in further narrowing down the state space of candidates but not for squeezing down the candidate list completely.

Combined Feature Querying and Runtime Embeddings

In combined feature querying, a single embedding is created for every candidate identifier remaining in the dataset using the concatenated data from each of the features. This step is the final stage before utilizing a more powerful large language model to select the best candidates from those remaining to present as the final search results. At this stage in the pipeline, the candidates have been honed down to the point where they most likely have a strong correlation to the search result. Still, since the large language model only has so big of a context window adding another layer before this final step which has lots of semantic information for an individual candidate combined into a single embedding can prove helpful in generating more high quality search results. Thus, embedding search is run on the user's search query and the combined feature embeddings for every candidate.

Furthermore, for this layer one approach we have experimented with is to generate its vector database at runtime rather than beforehand. This approach is taken so that the query can be appended to the embedded text before the concatenated features in order to try and bring about better search results. While appending the query has the potential to improve search results in other layers of the pipeline like feature querying, when doing runtime embeddings there is a balance to be struck between potentially better search results and increased latency. In previous layers, not as many candidates have been removed which means that more embeddings will need to be created which means that runtime latency will see an even larger increase. While doing runtime embeddings for the combined feature querying layer does increase latency, since there are less candidates at this point this increase in latency might be worthwhile if search results are significantly better. In order to truly know if the search results are significantly better, however, we would probably have to do A/B testing at scale where runtime embeddings both are and are not used and see what the conversion rates are. This type of testing is not really tenable for this current prototype.

LLM Generation

After finding our initial candidates using our FaF algorithm, we complete the generation part of the RAG pipeline. Each candidate is assigned a string made from appending the searchable features for a product and whatever other relevant information the website provides for the given product. We then take these descriptions and query OpenAI's gpt-3.5-turbo-0125 API to find our final reccomendations. In this stage we provide some basic prompt engineering explaining the job to the model "You are are an expert at looking at different products." We then show the format that we want with variable names for example products to get the recommendations in a regex parsable format. After this we finally add the product descriptions to our query and send. We receive the response and return them to our frontend in its expected format.

Search Refining

An aspect of our product that we had been discussing throughout the quarter was refining searches. More specifically, if a user made a search query on our platform and wanted to elaborate further on what they did or did not like in the results that were presented for example, the user would be able to enter another query. In order to implement this approach, the simplest method that we devised was to simply append the user's previous query and filter out the results that had already been presented presented. If we found conversation would create excessively long queries that confused our embedding models, we could then experiment with summarization models to summarize the list of user generated queries. While the engineering aspect of this approach was not daunting, upon further discussion we realized the user experience aspects that would arise in doing this method. For example, if a user is making a new query and not simply refining an old one, then the previous query should not be appended to the next one. Doing so would almost certainly confuse the model and arise in poor results. One solution to this that we discussed would be introducing a new button on the front end to differentiate between when a user was making a new search and when they were instead refining their most recent search further. While this approach again did not seem difficult from an engineering point to implement, our experience at Breakathon made us weary of going down this path. More specifically, at Breakathon users were having a hard time shifting away from traditional keyword-based search, and so we thought that users would have a hard time in this case being presented with multiple buttons to search for products. This perspective pushed us to the thought that a ChatGPT-esque interface would make sense in terms of the user experience if we were to implement search refining, as there could be multiple different tabs for different searches and users could see their search history and previous results as they shopped. Making this change would propose a pretty massive shift in our front end code base, though, so we ultimately decided to not add search refining to this stage of the product. However, as described above we do have a pretty good idea what the first steps would be to go down this path in the future if we choose to do so.