Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

generate_embeddings management command #128

Merged
merged 23 commits into from
Aug 7, 2023

Conversation

alex-nork
Copy link
Contributor

@alex-nork alex-nork commented Aug 4, 2023

The purpose of this PR is to add a management command called generate_embeddings. It allows the user to easily select
the embedding model service, and based on their selection the user is presented with the available embedding models. Once the user provides their username and password, the API key provided for the selected service is used to generating the embeddings.

Aug-04-2023 18-15-02

Copy link
Contributor

@zimventures zimventures left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The approach here feels like it may cause some problems down the road.

Instead of writing the fixture data to an intermediary fixture, then loading that fixture - what about writing the data directly to the database, then using the dumpdata command to export the fixtures?

Notably - the logic where data_to_write is built, is very subject to breakage if the schema changes.

thinking out loud
It's a shame we can't use the existing embedding provider classes to perform the embedding (as opposed to importing the libraries here and doing them). It's not really doable because the providers need a user object...

@alex-nork alex-nork force-pushed the AlexN/generate-embeddings-command branch from 03af06c to c22f71f Compare August 4, 2023 19:52
chirps/embedding/tests.py Fixed Show fixed Hide fixed
Copy link
Contributor

@zimventures zimventures left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FANTASTIC! Really great stuff. Excellent use of the prompts to employ an existing user.

@alex-nork alex-nork merged commit 2a04e81 into main Aug 7, 2023
5 checks passed
@zimventures zimventures deleted the AlexN/generate-embeddings-command branch August 14, 2023 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants