Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Evaluator #548

Merged
merged 15 commits into from
Nov 22, 2023
Merged

Evaluator #548

merged 15 commits into from
Nov 22, 2023

Conversation

javipacheco
Copy link
Contributor

@javipacheco javipacheco commented Nov 20, 2023

This PR adds the first approach for an evaluator. The evaluator is a tool that can be used to evaluate the performance of a model on a dataset. In this case, we are using deepeval as evaluation framework for LLMs.

Follow the README to install dependencies and use the Gradlew Task to generate a website with the results like this:

Screenshot 2023-11-22 at 10 22 15

@javipacheco javipacheco marked this pull request as ready for review November 21, 2023 14:23
raulraja
raulraja previously approved these changes Nov 21, 2023
Copy link
Contributor

@raulraja raulraja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @javipacheco looks great!

Montagon
Montagon previously approved these changes Nov 22, 2023
Copy link
Contributor

@Montagon Montagon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just minor comment and a suggestion. Great job, @javipacheco!

evaluator-example/README.md Outdated Show resolved Hide resolved
@tomascayuelas
Copy link
Contributor

tomascayuelas commented Nov 22, 2023

As we have talked I put here the comments:

  • Installing poetry we have it only for macos, I would put directly a link to the poetry website and that's it:
    https://python-poetry.org/docs/#installation

  • Installing pyenv to set the system to 3.10.0, otherwise it will not work because by default in the installation of poetry is brought in some OS 3.12 of python.

  • Consider installing a virtualenv to avoid problems with the versions:
    virtualenv venv --python=python3.10.0.
    source venv/bin/activate.
    this solves all versioning and OS problems...

In the pyproject.toml part:

  • Remove the ragas
  • Remove the README.md or add it, because you get an error when installing the project:
    If you do not want to install the current project use --no-root.
  • check:
    packages = [{include = "xef_evaluator"}]
  • put the files:
    the .py and the .html, .css, .js maybe inside an assets or public folder..

Review the README.md to indicate these steps and you are done.

Nice job @javipacheco 🥳

@javipacheco javipacheco merged commit 95de0d5 into main Nov 22, 2023
6 checks passed
@javipacheco javipacheco deleted the evaluator branch November 22, 2023 11:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants