Skip to content

jakehanson/GPT_Web_Scraping

Repository files navigation

Leveraging GPT and Web Scraping to Construct Comprehensive Market Contact Directories

This project uses Large Language Models (LLMs) and web-scraping APIs to create a detailed contact directory to connect buyers with sellers. We demonstrate this with a Florida-based family-owned apiary looking to partner with gourmet grocery stores interested in their products. The resulting contact directory includes store names, descriptions and compatibility scores generated by the LLM, as well as websites, addresses, phone numbers, and emails.

The algorithm starts by iterating through local Google search results using the search query "Gourmet Grocery Stores". The results from this search are then used to query the Google Place API, which provides verification of local businesses and additional information including websites, ratings, reviews, and addresses. This detailed information from the Place API is then processed and cleaned before being fed into an LLM (GPT-4) for compatability analysis using a prompt designed to match the product being sold with the ideal vendors .The prompt is benchmarked against human-evaluated data in order to ensure its effectiveness before being deployed across the entire dataset. The flowchart below illustrates this process.

Directory Structure

  • Prompt Engineering - This directory contains workbooks related to prompt engineering and evaluation using human-scored data.
  • Hyperparameter Tuning - This directory contains workbooks and files related to tuning the parameters of the API calls, including the depth of pagination, the amount of zoom to input to the Google Maps API, and the parameters of the search grid.
  • Contact Directory - This directory contains the final results of the algorithm.
  • img - This directory contains images used in readme

About

Scrape google search results and analyze them with an LLM. Deploy the resulting database as a Flask Application hosted with AWS Elastic Beanstalk.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages