[AQUA][Evaluate] Added the EvaluationServiceConfig class to manage evaluation service configurations. #940
Conversation
…res to manage evaluation service configurations.
mrDzurb
requested review from
darenr,
mayoor,
VipulMascarenhas,
qiuosier and
ahosler
as code owners
oracle-contributor-agreement
bot
added
the
OCA Verified
|
|
…com/oracle/accelerated-data-science into ODSC-61883/evaluation_supported_metrics
mrDzurb
changed the title
mrDzurb
changed the title
| and evaluation_target not in shape.filter.evaluation_target | ||
| ): | ||
| continue | ||
| results.append(shape) |
There was a problem hiding this comment.
Can this be -
key = evaluation_container or evaluation_target
if key in shape.filter.evaluation_container:
results.append(shape)
There was a problem hiding this comment.
Done. Having one key would not work, however i simplified a logic a bit.
| class InferenceParams(Serializable): | ||
| """Contains inference-related parameters with defaults.""" | ||
|
|
||
| inference_rps: Optional[int] = INFERENCE_RPS |
There was a problem hiding this comment.
Can we move these defaults to the config json? Maybe they are already part of the config json and these initial values get overwritten. Not clear from the code, hence checking.
There was a problem hiding this comment.
Yeah, let me do this. These defaults are placed in the config as well.
|
|
…com/oracle/accelerated-data-science into ODSC-61986/evaluation_supported_shapes
|
| "scikit-learn>=1.0", | ||
| "tabulate>=0.8.9", | ||
| "tqdm>=4.59.0", | ||
| "pydantic>=2.6.3", |
There was a problem hiding this comment.
License needs to be added to THIRD_PARTY_LICENSES.txt?
| def to_json( | ||
| self, | ||
| uri: str = None, | ||
| encoder: callable = json.JSONEncoder, |
There was a problem hiding this comment.
NIT: Seems to me Pydantic suggests using qmodel_dump_json()` with custom serializers.
There was a problem hiding this comment.
Wasn't aware of this, will take a look. Will play with this.
Description
Added the
EvaluationServiceConfigclass along with supporting structures to manage evaluation service configurations. This includes methods to merge evaluation inference parameters, retrieve model parameters based on framework and version, and filter shapes based on provided criteria. The implementation ensures flexibility and scalability for various frameworks and tasks.Dependency
This PR has a dependency on the Introduce Global Configuration for LLM Evaluation
Tests