Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add text generation task type #569

Draft
wants to merge 69 commits into
base: main
Choose a base branch
from
Draft

Add text generation task type #569

wants to merge 69 commits into from

Commits on Apr 19, 2024

  1. copy of Nicks draft for llm guided metric integration tests

    b.nativi committed Apr 19, 2024
    Configuration menu
    Copy the full SHA
    1ad92be View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. updates to llm guided integration test

    b.nativi committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    e5cc8ad View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into add_llm_guided_metrics

    b.nativi committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    73b09d4 View commit details
    Browse the repository at this point in the history

  3. typo in llm guided integration test

    b.nativi committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    f03fba5 View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2024

  1. LLM_EVALUATION task type, evaluate_llm_output(), first draft of test_… 

    …llm_evaluation()
    b.nativi committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    283eb6e View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into add_llm_guided_metrics

    b.nativi committed Apr 23, 2024
    Configuration menu
    Copy the full SHA
    0cc5f11 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2024

  1. Merge branch 'main' into add_llm_guided_metrics

    b.nativi committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    0615a8b View commit details
    Browse the repository at this point in the history

  2. llm_url and llm_api_key added in multiple places in client, api and i… 

    …ntegration tests
    b.nativi committed Apr 24, 2024
    Configuration menu
    Copy the full SHA
    660ee71 View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2024

  1. finalize integration test data llm_evaluation

    b.nativi committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    36a9a17 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into add_llm_guided_metrics

    b.nativi committed Apr 25, 2024
    Configuration menu
    Copy the full SHA
    d7b53b4 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2024

  1. llm_evaluation functional tests

    b.nativi committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    d2f2ebe View commit details
    Browse the repository at this point in the history

  2. all llm evaluation metrics added

    b.nativi committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    c054b47 View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2024

  1. commit with all tests except new tests passing

    b.nativi committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    0ac0f4c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into add_llm_guided_metrics

    b.nativi committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    af90c95 View commit details
    Browse the repository at this point in the history

Commits on May 3, 2024

  1. first draft of coherence working

    b.nativi committed May 3, 2024
    Configuration menu
    Copy the full SHA
    aa9c007 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into add_llm_guided_metrics

    b.nativi committed May 3, 2024
    Configuration menu
    Copy the full SHA
    e1ae78c View commit details
    Browse the repository at this point in the history

Commits on May 6, 2024

  1. Merge branch 'main' into add_llm_guided_metrics

    b.nativi committed May 6, 2024
    Configuration menu
    Copy the full SHA
    b0657d1 View commit details
    Browse the repository at this point in the history

  2. updated metric specifications to metrics_to_return

    b.nativi committed May 6, 2024
    Configuration menu
    Copy the full SHA
    631f3f9 View commit details
    Browse the repository at this point in the history

  3. add OpenAIClient class

    b.nativi committed May 6, 2024
    Configuration menu
    Copy the full SHA
    a761d6c View commit details
    Browse the repository at this point in the history

  4. fix on client side to match with metrics_to_return in EvaluationParam… 

    …eters on api side
    b.nativi committed May 6, 2024
    Configuration menu
    Copy the full SHA
    b041eb1 View commit details
    Browse the repository at this point in the history

  5. fix type error

    b.nativi committed May 6, 2024
    Configuration menu
    Copy the full SHA
    7b3d62e View commit details
    Browse the repository at this point in the history

  6. add mocking for coherence

    b.nativi committed May 6, 2024
    Configuration menu
    Copy the full SHA
    99d880b View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. add test for openai api connection

    b.nativi committed May 8, 2024
    Configuration menu
    Copy the full SHA
    52acfa4 View commit details
    Browse the repository at this point in the history

  2. remove Metric from metric names, improved error handling in coherence

    b.nativi committed May 8, 2024
    Configuration menu
    Copy the full SHA
    36eb7dd View commit details
    Browse the repository at this point in the history

  3. mock openai api connection

    b.nativi committed May 8, 2024
    Configuration menu
    Copy the full SHA
    15c4030 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into add_llm_guided_metrics

    b.nativi committed May 8, 2024
    Configuration menu
    Copy the full SHA
    c9a29cb View commit details
    Browse the repository at this point in the history

  5. use grouper_id = label.key for llm evaluation

    b.nativi committed May 8, 2024
    Configuration menu
    Copy the full SHA
    72aaa7e View commit details
    Browse the repository at this point in the history

  6. change CoherenceMetric from using label_key to using label

    b.nativi committed May 8, 2024
    Configuration menu
    Copy the full SHA
    d31a26c View commit details
    Browse the repository at this point in the history

  7. Merge branch 'main' into add_llm_guided_metrics

    b.nativi committed May 8, 2024
    Configuration menu
    Copy the full SHA
    cbe79ec View commit details
    Browse the repository at this point in the history

Commits on May 10, 2024

  1. Merge branch 'main' into add_llm_guided_metrics

    b.nativi committed May 10, 2024
    Configuration menu
    Copy the full SHA
    ddcb7e3 View commit details
    Browse the repository at this point in the history

Commits on May 15, 2024

  1. text generation examples

    b.nativi committed May 15, 2024
    Configuration menu
    Copy the full SHA
    ffef625 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into add_llm_guided_metrics

    b.nativi committed May 15, 2024
    Configuration menu
    Copy the full SHA
    5a4fee8 View commit details
    Browse the repository at this point in the history

  3. rename misnamed qa folder to text-generation

    b.nativi committed May 15, 2024
    Configuration menu
    Copy the full SHA
    95b25dc View commit details
    Browse the repository at this point in the history

  4. remove .txt from example

    b.nativi committed May 15, 2024
    Configuration menu
    Copy the full SHA
    544b401 View commit details
    Browse the repository at this point in the history

Commits on May 16, 2024

  1. add all three use case examples for text generation

    b.nativi committed May 16, 2024
    Configuration menu
    Copy the full SHA
    8a3a159 View commit details
    Browse the repository at this point in the history

Commits on May 17, 2024

  1. update text generation examples

    b.nativi committed May 17, 2024
    Configuration menu
    Copy the full SHA
    061065c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into add_llm_guided_metrics

    b.nativi committed May 17, 2024
    Configuration menu
    Copy the full SHA
    63d10a8 View commit details
    Browse the repository at this point in the history

  3. rename llm_evaluation to text_generation

    b.nativi committed May 17, 2024
    Configuration menu
    Copy the full SHA
    c7f923a View commit details
    Browse the repository at this point in the history

Commits on May 22, 2024

  1. update datum and annotation to work with text gen, functional tests p… 

    …assing
    b.nativi committed May 22, 2024
    Configuration menu
    Copy the full SHA
    92abc09 View commit details
    Browse the repository at this point in the history

  2. add client side unit tests for Annotation and Datum

    b.nativi committed May 22, 2024
    Configuration menu
    Copy the full SHA
    c927e3f View commit details
    Browse the repository at this point in the history

  3. fix github checks error for Context

    b.nativi committed May 22, 2024
    Configuration menu
    Copy the full SHA
    9224ce7 View commit details
    Browse the repository at this point in the history

  4. attempt number two to fix github checks issue with Context

    b.nativi committed May 22, 2024
    Configuration menu
    Copy the full SHA
    0e701f4 View commit details
    Browse the repository at this point in the history

  5. remove unused create_datums_with_dataset_names() method

    b.nativi committed May 22, 2024
    Configuration menu
    Copy the full SHA
    168867f View commit details
    Browse the repository at this point in the history

  6. rename QAG to Summarization metric

    b.nativi committed May 22, 2024
    Configuration menu
    Copy the full SHA
    092bf7a View commit details
    Browse the repository at this point in the history

Commits on May 23, 2024

  1. add LLMClient parent class

    b.nativi committed May 23, 2024
    Configuration menu
    Copy the full SHA
    59fca69 View commit details
    Browse the repository at this point in the history

  2. add MistralValorClient

    b.nativi committed May 23, 2024
    Configuration menu
    Copy the full SHA
    45118d3 View commit details
    Browse the repository at this point in the history

  3. add mistralai client dependency

    b.nativi committed May 23, 2024
    Configuration menu
    Copy the full SHA
    9ea2b82 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2024

  1. add support for mistral client in valor_api text_generation.py

    b.nativi committed May 24, 2024
    Configuration menu
    Copy the full SHA
    de9911b View commit details
    Browse the repository at this point in the history

  2. fix type error for EvaluationParameters.llm_api_params

    b.nativi committed May 24, 2024
    Configuration menu
    Copy the full SHA
    8fd9ada View commit details
    Browse the repository at this point in the history

Commits on May 29, 2024

  1. temporarily comment out client text gen test until ready

    b.nativi committed May 29, 2024
    Configuration menu
    Copy the full SHA
    db85df5 View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. Merge branch 'main' into add_llm_guided_metrics

    @ntlind
    ntlind committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    c57a464 View commit details
    Browse the repository at this point in the history

  2. fix all tests except test_text_generation.py

    @ntlind
    ntlind committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    ae39f0a View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2024

  1. Merge branch 'main' into add_llm_guided_metrics

    @ntlind
    ntlind committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    b1f8405 View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2024

  1. Merge branch 'main' into add_llm_guided_metrics

    @ntlind
    ntlind committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    191cee2 View commit details
    Browse the repository at this point in the history

  2. fix text generation functional tests

    b.nativi committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    bd4f92a View commit details
    Browse the repository at this point in the history

  3. add metric_params to EvaluationParameters in api and client

    b.nativi committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    70ecee7 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into add_llm_guided_metrics

    b.nativi committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    3616e5e View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2024

  1. Merge branch 'main' into add_llm_guided_metrics

    @ntlind
    ntlind committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    de34b88 View commit details
    Browse the repository at this point in the history

  2. text gen metric names added to client

    b.nativi committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    334442b View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'refs/remotes/origin/add_llm_guided_metr… 

    …ics' into add_llm_guided_metrics
    b.nativi committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    b4f7fb5 View commit details
    Browse the repository at this point in the history

  4. fix Filter issues in text generation

    b.nativi committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    6c11315 View commit details
    Browse the repository at this point in the history

  5. fix filter issue for text generation in coretypes

    b.nativi committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    118dafe View commit details
    Browse the repository at this point in the history

  6. remove label_id from db_mapping of text gen metrics

    b.nativi committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    091df90 View commit details
    Browse the repository at this point in the history

  7. isort pre-commit error in evaluation.py

    b.nativi committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    53d51f4 View commit details
    Browse the repository at this point in the history

  8. text gen integration test passing

    b.nativi committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    dd476c1 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2024

  1. mock client integration test for text generation

    b.nativi committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    9ee7c50 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. cleanup text generation code

    b.nativi committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    dd7188a View commit details
    Browse the repository at this point in the history

  2. add minor PR improvements

    b.nativi committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    34520b3 View commit details
    Browse the repository at this point in the history

  3. Add text comparison metrics to Ben's PR (#632)

    @ntlind
    ntlind committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    de9be2e View commit details
    Browse the repository at this point in the history