summaries: refactor to use openai api model for inference #59
Conversation
quitrk
force-pushed
the
tavram/openai
branch
from
7c24f74
to
cc85d55
Compare
quitrk
changed the title
quitrk
force-pushed
the
tavram/openai
branch
2 times, most recently
from
9db544d
to
28c7987
Compare
quitrk
force-pushed
the
tavram/openai
branch
from
28c7987
to
9008f9c
Compare
quitrk
force-pushed
the
tavram/openai
branch
2 times, most recently
from
ea480d4
to
256d58a
Compare
quitrk
force-pushed
the
tavram/openai
branch
from
256d58a
to
4cddaf6
Compare
* inference jobso of customers with existing open ai keys will run on open ai servers
quitrk
force-pushed
the
tavram/openai
branch
from
d948388
to
1bcab62
Compare
skynet/auth/openai.py
Outdated
| open_yaml(openai_credentials_file) | ||
| file_watcher = FileWatcher(openai_credentials_file, lambda: open_yaml(openai_credentials_file)) | ||
|
|
||
| file_watcher.start() |
There was a problem hiding this comment.
When are we going to stop this?
There was a problem hiding this comment.
when the process stops. Is there other time when we'd want it to stop?
There was a problem hiding this comment.
Not really, in general I prefer explicit vs implicit. No big deal though.
| log.error(f'Error while polling for file changes: {e}') | ||
| break | ||
|
|
||
| def start(self): |
There was a problem hiding this comment.
I guess there should be a stop here...
| n_ctx=4096, | ||
| n_gpu_layers=llama_n_gpu_layers, | ||
| n_batch=llama_n_batch, | ||
| global llm |
There was a problem hiding this comment.
Is this initialize necessary? Can't we just create it at imprt time?
There was a problem hiding this comment.
if we move it like that, we need to also better isolate the summaries:dispatcher module to make sure it's not importing it indirectly.
There was a problem hiding this comment.
Does the dispatcher import anything from the processor? It kinda shouldn't.
There was a problem hiding this comment.
it imports the jobs module, which in turn has reference to the processor
There was a problem hiding this comment.
Gotcha. No strong opinion then.
|
Pl squash-merge. |
No description provided.