Conversation
- Create tests/test_artifact_preflight.cpp (PF-01..PF-14) with unit
tests for LLMArtifactPreflight, modelBaseDir(), stubModelsEnabled(),
resolveModelPath() and resolveLoraPath()
- Add ${CMAKE_SOURCE_DIR}/benchmarks to tests/CMakeLists.txt include
dirs so test_artifact_preflight.cpp can include the preflight header
- bench_llm_inference_performance.cpp: include benchmark_artifact_preflight.h,
replace all hardcoded /loras/... paths with resolveLoraPath(), add
THEMIS_BENCH_SKIP_IF_ARTIFACT_MISSING guards to every LoRA benchmark
- bench_lora_auto_binding.cpp: same treatment – 12 functions updated
- bench_lora_inline.cpp: same treatment – 5 functions updated
Missing artifact errors now produce actionable SkipWithError messages
pointing to scripts/download_models.sh --stub-only and
docs/BENCHMARK_RUNBOOK.md §LLM/LoRA Model Setup.
Agent-Logs-Url: https://github.com/makr-code/ThemisDB/sessions/c473afc7-d6a4-4be6-8b54-1c300da149c2
Co-authored-by: makr-code <150588092+makr-code@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Copilot Request
feat(benchmarks): standardize LLM/LoRA artifact preflight for benchmarks
Apr 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LLM/LoRA benchmarks were silently producing invalid results (or crashing) because
MultiLoRAManager::loadLoRA()validates file existence but benchmarks passed hardcoded/loras/...paths that never exist. Missing-artifact failures also had no actionable guidance.Changes
New: Unit tests for preflight utilities (
tests/test_artifact_preflight.cpp, PF-01..PF-14)Tests cover
modelBaseDir()env-var priority/fallbacks,stubModelsEnabled()case-insensitive parsing,resolveModelPath()/resolveLoraPath()path resolution, andLLMArtifactPreflight::create()success/failure/LoRA-required modes. Verifies error messages contain setup guidance.tests/CMakeLists.txtAdded
${CMAKE_SOURCE_DIR}/benchmarkstoinclude_directoriesso tests can includebenchmark_artifact_preflight.h.benchmarks/bench_llm_inference_performance.cpp— 9 functionsbenchmarks/bench_lora_auto_binding.cpp— 12 functionsbenchmarks/bench_lora_inline.cpp— 5 functionsAll three files now:
#include "benchmark_artifact_preflight.h"THEMIS_BENCH_SKIP_IF_ARTIFACT_MISSING/loras/....binwithresolveLoraPath()On missing artifact the benchmark emits: "LLM artefact preflight FAILED: LoRA adapter not found. Run
scripts/download_models.sh --stub-onlyor setTHEMIS_MODEL_DIR. Seedocs/BENCHMARK_RUNBOOK.md §LLM/LoRA Model Setup."Type of Change
Breaking Change Checklist
VERSIONandCMakeLists.txtdocs/migration/### Removed/### Changedsection updatedTesting
📚 Research & Knowledge (wenn applicable)
/docs/research/angelegt?/docs/research/implementation_influence/eingetragen?Relevante Quellen:
Checklist
[Unreleased]