Skip to content

Conversation

mrDzurb
Copy link
Member

@mrDzurb mrDzurb commented Sep 6, 2024

Description

  • Shifts the management of the supported shapes list from being hardcoded in the ADS SDK to being stored and managed at the global service config level.
  • Reads the evaluation inferencing parameters from the service config.
  • Introduces the inferencing service model sampling parameters. The system parameters that will not be visible for user, but will be applied implicitly for evaluation. The service model params will be extracted based on the container family and version.

@oracle-contributor-agreement oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Sep 6, 2024
DEFAULT_EVALUATION_CONTAINER = "odsc-llm-evaluate"


@cached(cache=TTLCache(maxsize=1, ttl=timedelta(hours=5), timer=datetime.now))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not big fan of caching in our application unless we see that the users are frequently accessing this page. The caching prevents us from urgently addressing a fix if there is a problem. We will have to wait for the cache expiry.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be useful if the containers is not a list but a dictionary and the keys are container name? If it is a list, we have to run for-loop and break. Instead I was wondering if it is better to do a straight lookup.

@mrDzurb mrDzurb merged commit b0ca718 into ODSC-61884/global_evaluation_config Sep 6, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
OCA Verified All contributors have signed the Oracle Contributor Agreement.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants