Prototype v3: SUPost Battle for Software Fair

Software Fair Prototype

Generalizing Find and Filter

For our previous v2 prototype Bookworm, while algorithm design and code implementation was at the forefront of our minds, in order to be able to rapidly iterate and get quick feedback we initially structured our codebase to be heavily centered around the Amazon books dataset relevant to this demo. While the core elements of find and filter still existed, our codebase itself was directly referencing the configuration of this dataset and was not attuned to the B2B software product we now saw as Kenja's main value add. Thus the next step we took was restructuring our code base in order to be able to process any structured csv and run FaF on the data in order to allow companies to easily run this new form of search on their platforms.

Initial Data Processing

The first stage in the new pipeline for our backend revolves around the dataset itself. As noted above, we decided to first structure our approach around structured csv data, as we felt like this formatting is extremely common and pretty ubiquitous. Similar to standard databases, the csv data provided needed to have an identifier (id) to every row in order for us to be able to properly track data in the backend. We also made sure to have functionality where users could provide multiple csv files provided that each row of all the csv files provided contained a column with the relevant identifier. The user lets us know the identifier on the front end when we process our data. Once the all the csv files are provided by the user, we combine all of them in to a single pandas dataframe to make data processing easier at the cost of a larger memory footprint. This increased memory usage is a cost that we were willing to make in order to iterate more quickly, but in the future is something that we would optimize in order to keep a lower memory footprint which would be more crucial when dealing with large-scale datasets.

Selecting a Distance Function

Within FaF, querying our vector database is a key part of the algorithm. In order to do so, we must utilize a distance function to decide how similar the query embedding and candidate embedding are to one another. The default distance function utilized by our vector database library, chromaDB, is the squared L2 norm. Since this distance function is the default, we used this function for our previous prototype. However, when investigating the open-source embedding model from Nomic AI that we utilized to create the embeddings for our vector database, we found that this embedding model was trained using cosine similarity. In turn, we adjusted querying of our vector database to use cosine similarity which should lead to better search results.

Noun Querying

Once this initial data processing stage is completed, we then move on to iteratively funneling down the dataset until only a few candidates remain to be provided to the user as the result of their search query. The first step in this process is noun querying. This step in the pipeline is a direct result of the team's learnings from Breakathon, for we found that the original version of FaF without this stage was not able to handle more traditional keyword based searches very robustly. Thus, we decided to add this initial layer to better select more relevant candidates in the initial stages of the pipeline relating to the user's query to be passed on to the later stages. With noun querying, we first extract out all the nouns in each row of the combined pandas dataframe and create embeddings for each row based on all of the nouns that they individually contain. We do the same for the user's query then run cosine similarity between the user's query embedding and each of these noun embeddings and keep only keep a specified subset of the original dataset which are revealed to be the most similar. This approach is meant to ground potential candidates in their core aspects so that other details that might be contained in details do not overly confuse the algorithm too early in the process. For example, if a user searched "blue sunglasses" but the dataset contained "blue shoes," we do not want the detail of "blue" to cause the shoes to be presented. At the same time, we still want a large enough state space to provide interesting candidates for more exploratory rather than keyword-based searches (think "I am looking for something nice for my lawn"), so we try to strike a balance between the two and utilize the later stages of FaF.