Semtest Framework (v0.0)

Semtest Framework

Semtest supports a framework mode that can be executed via the command line, and will automatically crawl and execute test benchmarks within a specified directory.

Capabilities

• Executable via Command Line:

  • Running the following command will trigger the framework mode:

    semtest <directory>

• Directory Crawling:

  • The framework will recursively search the specified directory for test files containing benchmarks.
  • Import and identify all tests marked with the @semtest.benchmark(...) decorator.

• Automatic Execution:

  • All identified benchmarks will be executed automatically.

• Test Result Logging:

  • The framework logs the output of each test execution.
  • Log details include:
    • Result semantic distance: Comparison between test result and the expected semantic output.
    • Semantic distance metrics: Averages, means, or other statistical measures across multiple test executions.
    • Semantic distance type: Cosine similarity, etc...
    • Exceptions thrown: AssertionError, etc...
    • Exception count: 0 ... n