Skip to content

issues Search Results · repo:aws/fmeval language:Python

Filter by

28 results
 (58 ms)

28 results

inaws/fmeval (press backspace or delete to remove)

aws/fmeval
Cannot load dataset from s3

Hello, When I try to use DataConfig to load my dataset from S3, like so... DataConfig( dataset_name= my-dataset , dataset_uri= s3://my-bucket/some/file.jsonl # rest ...
  • AmrSaber
  • 2
  • Opened 
    on Jan 7
  • #338

aws/fmeval
Problem running package with python 3.13

Hello, I could not run the package using Python 3.13 as I get the following error message on pip install fmeval Collecting markdown (from fmeval) Using cached Markdown-3.7-py3-none-any.whl.metadata ...
  • AmrSaber
  • Opened 
    on Jan 6
  • #337

aws/fmeval
issue with installing fmeval

I am trying to install fmeval, having Sagemaker, python 3.10, ml.m5.2xlarge, but I het the following error: Collecting fmeval Using cached fmeval-1.2.1-py3-none-any.whl.metadata (14 kB) Requirement already ...
  • Erf-dev
  • 1
  • Opened 
    on Dec 10, 2024
  • #336

aws/fmeval
[Feature] Increase model coverage of the BERT Score metric by adding torchmetrics implementation

The current underlying implementation of BERT score supports a limited set of transformer models, and FMEval further truncates this list to microsoft/deberta-xlarge-mnli and roberta-large-mnli. Torchmetrics ...
  • achad4
  • 2
  • Opened 
    on Nov 8, 2024
  • #332

aws/fmeval
[Feature] Lambda model runner

While LLMs will not be hosted directly on lambdas, allowing customers to proxy their eval requests through Lambda invocations would expand the breadth of use cases of FMEval, without needing to maintain ...
  • achad4
  • 1
  • Opened 
    on Nov 8, 2024
  • #331

aws/fmeval
Option to disable BERTScore in QAAccuracy

The addition of BERTScore to the QAAccuracy in version 1.2.0 has massively slowed down evaluation: I ve seen interactive evaluations (with our custom Streamlit solution here) change from ~30 seconds to ...
  • athewsey
  • 3
  • Opened 
    on Oct 28, 2024
  • #330

aws/fmeval
Python 3.11 and 3.12 compatibility?

Are there plans to make this package compatible with Python 3.11 and 3.12?
  • OlivierBinette
  • 4
  • Opened 
    on Aug 12, 2024
  • #322

aws/fmeval
[Bug] Radar plot using plotly not displaying

Running model-comparison.ipynb I realized the radar plot is not displaying. I fixed by adding the following code: import plotly.io as pio pio.renderers.default = notebook
  • jcassiojr
  • Opened 
    on Jul 25, 2024
  • #316

aws/fmeval
ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: "claude-3-sonnet-20240229" is not supported on this API. Please use the Messages API instead.

Hi , Using the code the evaluate summary using below code import json import boto3 import os ##Bedrock clients for model inference bedrock_runtime = boto3.client( bedrock-runtime , region_name= eu-west-3 ...
  • aakash086
  • 4
  • Opened 
    on Jun 3, 2024
  • #286

aws/fmeval
[Feature] Add callback mechanism to evaluation

I m integrating fmeval with experiments tracking solutions (MLflow for now), and the lack of callback mechanisms means that the tracing can only happen after an evaluation is completed. Drawbacks: - ...
  • acere
  • 1
  • Opened 
    on May 29, 2024
  • #284
Issue origami icon

Learn how you can use GitHub Issues to plan and track your work.

Save views for sprints, backlogs, teams, or releases. Rank, sort, and filter issues to suit the occasion. The possibilities are endless.Learn more about GitHub Issues
ProTip! 
Press the
/
key to activate the search input again and adjust your query.
Issue origami icon

Learn how you can use GitHub Issues to plan and track your work.

Save views for sprints, backlogs, teams, or releases. Rank, sort, and filter issues to suit the occasion. The possibilities are endless.Learn more about GitHub Issues
ProTip! 
Press the
/
key to activate the search input again and adjust your query.
Issue search results · GitHub