Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: refactor LLMOptions #28

Merged
merged 11 commits into from
May 13, 2024
Merged

refactor: refactor LLMOptions #28

merged 11 commits into from
May 13, 2024

Conversation

micpst
Copy link
Collaborator

@micpst micpst commented May 10, 2024

This PR adds default LLM options for LLM client and enables custom LLM parameterization in collection.ask.

@micpst micpst self-assigned this May 10, 2024
Copy link

Trivy scanning results.

@micpst micpst changed the title reafctor: refactor LLMOptions refactor: refactor LLMOptions May 10, 2024
Copy link

github-actions bot commented May 10, 2024

badge

Code Coverage Summary

Filename                                                  Stmts    Miss  Cover    Missing
------------------------------------------------------  -------  ------  -------  -------------------------------------------------
dbally/_main.py                                              12       0  100.00%
dbally/collection.py                                         92       1  98.91%   262
dbally/assistants/base.py                                    23       0  100.00%
dbally/assistants/openai.py                                  59       2  96.61%   59-76
dbally/audit/event_span.py                                    8       0  100.00%
dbally/audit/event_tracker.py                                34       6  82.35%   35, 48, 59, 69, 86, 92
dbally/audit/event_handlers/base.py                          15       0  100.00%
dbally/audit/event_handlers/cli_event_handler.py             42      27  35.71%   9-12, 38-39, 42-46, 56-58, 70-78, 90-93, 104-107
dbally/audit/event_handlers/langsmith_event_handler.py       25      21  16.00%   6-92
dbally/data_models/audit.py                                  22       0  100.00%
dbally/data_models/execution_result.py                       14       0  100.00%
dbally/embedding_client/base.py                               5       0  100.00%
dbally/embedding_client/openai.py                            19      14  26.32%   20-30, 42-52
dbally/iql/_exceptions.py                                    24       1  95.83%   33
dbally/iql/_processor.py                                     78       7  91.03%   15, 43, 60, 66, 72, 81, 87
dbally/iql/_query.py                                         12       1  91.67%   7
dbally/iql/_type_validators.py                               39       2  94.87%   24, 28
dbally/iql/syntax.py                                         36       9  75.00%   6-9, 27, 36, 60, 63-66
dbally/iql_generator/iql_generator.py                        32       0  100.00%
dbally/iql_generator/iql_prompt_template.py                  12       1  91.67%   38
dbally/llm_client/base.py                                    31       4  87.10%   35-37, 41
dbally/llm_client/openai_client.py                           34      34  0.00%    1-94
dbally/nl_responder/nl_responder.py                          31       5  83.87%   79, 86-95
dbally/nl_responder/nl_responder_prompt_template.py           7       0  100.00%
dbally/nl_responder/query_explainer_prompt_template.py        7       0  100.00%
dbally/nl_responder/token_counters.py                        23      10  56.52%   24-25, 53-63
dbally/prompts/common_validation_utils.py                    15       0  100.00%
dbally/prompts/prompt_builder.py                             21       3  85.71%   7, 27-28
dbally/prompts/prompt_template.py                            28       2  92.86%   27, 35
dbally/similarity/chroma_store.py                            35       0  100.00%
dbally/similarity/detector.py                                73       6  91.78%   21, 200-204
dbally/similarity/faiss_store.py                             36      33  8.33%    5-94
dbally/similarity/fetcher.py                                  5       0  100.00%
dbally/similarity/index.py                                   18       0  100.00%
dbally/similarity/sqlalchemy_base.py                         40      21  47.50%   17, 35-37, 48-50, 59, 68-71, 81-87, 105-108
dbally/similarity/store.py                                    7       0  100.00%
dbally/utils/errors.py                                        2       0  100.00%
dbally/view_selection/base.py                                 7       0  100.00%
dbally/view_selection/llm_view_selector.py                   21       0  100.00%
dbally/view_selection/random_view_selector.py                10      10  0.00%    1-35
dbally/view_selection/view_selector_prompt_template.py       11       2  81.82%   32-33
dbally/views/base.py                                          8       0  100.00%
dbally/views/decorators.py                                    6       0  100.00%
dbally/views/exposed_functions.py                            33       1  96.97%   24
dbally/views/methods_base.py                                 34       2  94.12%   75, 83
dbally/views/pandas_base.py                                  33       1  96.97%   64
dbally/views/sqlalchemy_base.py                              37       7  81.08%   48, 63-65, 83-87
dbally/views/structured.py                                   34       0  100.00%
dbally/views/freeform/text2sql/_autodiscovery.py            113      18  84.07%   107-108, 272, 275, 288-300, 304, 319-321, 333-334
dbally/views/freeform/text2sql/_config.py                    20       0  100.00%
dbally/views/freeform/text2sql/_errors.py                     6       3  50.00%   10-12
dbally/views/freeform/text2sql/_view.py                      48       6  87.50%   84, 88-91, 94
dbally_cli/main.py                                            5       5  0.00%    1-11
dbally_cli/similarity.py                                     21       0  100.00%
TOTAL                                                      1463     265  81.89%

Diff against main

Filename                                         Stmts    Miss  Cover
---------------------------------------------  -------  ------  --------
dbally/collection.py                                 0      -1  +1.08%
dbally/audit/event_span.py                           0      -2  +25.00%
dbally/audit/event_tracker.py                        0      -5  +14.70%
dbally/llm_client/base.py                          +10      -6  +34.72%
dbally/llm_client/openai_client.py                 +13     +13  +100.00%
dbally/view_selection/base.py                       +1       0  +100.00%
dbally/view_selection/random_view_selector.py       +1      +1  +100.00%
dbally/views/base.py                                +1       0  +100.00%
dbally/views/structured.py                           0      -1  +2.94%
TOTAL                                              +26      -1  +0.23%

Results for commit: 6439a71

Minimum allowed coverage is 60%

♻️ This comment has been updated with latest results

@micpst micpst marked this pull request as ready for review May 10, 2024 10:06
Copy link
Contributor

@ds-sebastianchwilczynski ds-sebastianchwilczynski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please write a test (for me can be an integration one that if we execute collection.ask() all LLM Options propagate correctly. + test the or operator

Also the issue with openai dependency is important.

Rest for me is optional, I'll accept the PR if without them

src/dbally/collection.py Show resolved Hide resolved
src/dbally/data_models/llm_options.py Outdated Show resolved Hide resolved
src/dbally/data_models/llm_options.py Outdated Show resolved Hide resolved
src/dbally/data_models/llm_options.py Outdated Show resolved Hide resolved
Copy link
Contributor

@mhordynski mhordynski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@micpst micpst merged commit 3f5c4bf into main May 13, 2024
3 checks passed
@micpst micpst deleted the mp/refactor-llm-options branch May 13, 2024 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants