This pull request introduces two main sets of changes: enhancements to the LLM (Large Language Model) records for improved searchability, and a significant cleanup of the prospects table by removing unused columns. #76
Merged
Conversation
Bump version to 2.1.9 and add full-text search support to the llm table: include search_vector in SELECTs, generate it on insert using to_tsvector('english', prompt || ' ' || completion), and add a migration + runner to create the tsvector column and GIN index. Update prospects endpoint to return the llm search_vector and adjust result field mapping. Add several SQL migration files to drop unused columns from the prospects table (cleaned, primary_email_catchall_status, primary_intent_score, primary_intent_topic, qualify_contact, secondary_intent_score, secondary_intent_topic, tertiary_email, tertiary_email_source, tertiary_email_status, tertiary_email_verification_source) and include small runner scripts for removing tertiary_email and its source.
Add four standalone Python migration scripts to remove deprecated columns from the prospects table: do_not_call, email_confidence, home_phone, and work_direct_phone. Each script uses get_db_connection_direct to execute an ALTER TABLE ... DROP COLUMN IF EXISTS statement, commits the change, closes the DB connection, and prints a completion message. The DROP COLUMN IF EXISTS makes the scripts safe to run idempotently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Additionally, the application version is incremented to reflect these updates.
LLM Table Enhancements:
search_vectorcolumn (typetsvector) to thellmtable to support full-text search, along with a GIN index for faster querying. Migration scripts and a Python runner are included. [1] [2]search_vectorfrom thepromptandcompletionfields.search_vectorfield in LLM record retrievals, ensuring it is returned in API responses where appropriate. [1] [2] [3] [4] [5]Prospects Table Cleanup:
prospectstable, including:cleaned,do_not_call,email_confidence,home_phone,primary_email_catchall_status,primary_intent_score,primary_intent_topic,qualify_contact,secondary_intent_score,secondary_intent_topic,tertiary_email,tertiary_email_source,tertiary_email_status,tertiary_email_verification_source, andwork_direct_phone. Each column removal is handled by its own migration script. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15]Other:
2.1.8to2.1.9.