Skip to content

docs(test): Define requirements for models testing #93

@christian-pinto

Description

@christian-pinto

Description

Similarly to what was done for benchmarks in #3 we need to define the protocol for contributing model tests.

  1. Benchmark Test Case Handling: Define how the system manages test cases, accommodating two distinct types:
    A. Locally provided tests: Model owners can embed their model tests inside the Nexus Package structure
    B. External tests: Model owners have tests defined in their library (or a third party one) and want to re-use them
  2. Test Script Requirements: Define the standard for the script that runs a test.
  • Interface: What is the standard interface the test script must expose? Do we define one (e.g., pytest) or just require the test script return 0 for success or another value for failure?
  • Responsibilities: The script is responsible for sourcing any data wherever required, loading the model and performing all the relevant tests.
  • Ownership: This script will be provided by the model contributor.
  1. Testing models that support serving with vLLM
  • the test must verify the model works as expected with vLLM.
  • Handle vLLM as optional dependency. i.e., should vLLM and non vLLM tests be separated for us to run them conditionally?
  1. Model/Algorithm Contributor Responsibilities: Articulate what contributors must provide when they add a model
  2. Dataset Sourcing & Hosting: Specify requirements for datasets, noting they may be managed internally or be part of an external framework.
  3. Execution Environment: Outline infrastructure requirements, including a mechanism to handle dependencies for external frameworks (e.g., via containerization).

Metadata

Metadata

Labels

documentationImprovements or additions to documentation

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions