test: Intrinsic 'answer_relevance_classifier' as LoRA adapter on base model 'granite-4.0-micro' not found

[planetf1]

uv run pytest -v 2>&1 on macOS (32GB m1) fails with:

==================================== ERRORS ====================================
____ ERROR collecting test/formatters/granite/test_intrinsics_formatters.py ____
test/formatters/granite/test_intrinsics_formatters.py:196: in <module>
    YamlJsonCombo(
E   pydantic_core._pydantic_core.ValidationError: 1 validation error for YamlJsonCombo
E     Value error, Intrinsic 'answer_relevance_classifier' as LoRA adapter on base model 'granite-4.0-micro' not found in ibm-granite/granite-lib-rag-r1.0 repository on Hugging Face Hub. Searched for path answer_relevance_classifier/granite-4.0-micro/lora/io.yaml [type=value_error, input_value={'short_name': 'answer_re...r_relevance_classifier'}, input_type=dict]
E       For further information visit https://errors.pydantic.dev/2.12/v/value_error
------------------------------- Captured s

The cause is a LoRa adapter not being found

This also fails on batch (LSF)

[planetf1]

The R1 RAG repo (granite-lib-rag-r1.0) doesn't have answer_relevance_classifier — it only has 7 intrinsics (answerability, citations, context_relevance, hallucination_detection, query_clarification, query_rewrite, _ollama). The old repo (rag-intrinsics-lib) does have it.

The catalog was switched to R1 in PR #289 (94c35ad, Jan 7). The test was added in PR #609 (8003b44, Mar 12) referencing answer_relevance_classifier on granite-4.0-micro against R1 — but the adapter was never there.

cc @frreiss

[planetf1]

Investigation update

This is now confirmed as a CI-wide blocker affecting all fresh PR runs. Both PR #685 and PR #686 (unrelated fixes) are failing with this same collection error.

Why main looks green: The tip of main (a86fe40, merged 2026-03-17) passes CI, but this appears to be because the GitHub Actions runner is using a cached HuggingFace Hub snapshot from a prior run. PR CI runs start with a cold cache and hit the live Hub, where answer_relevance_classifier/granite-4.0-micro/lora/io.yaml no longer exists.

Code path:

• test_intrinsics_formatters.py defines _YAML_JSON_COMBOS_LIST at module level
• Each YamlJsonCombo(...) instantiation triggers a pydantic @model_validator that calls intrinsics_util.obtain_io_yaml()
• That calls huggingface_hub.snapshot_download() at collection time, before any test runs
• The file path answer_relevance_classifier/granite-4.0-micro/lora/io.yaml is not found in ibm-granite/granite-lib-rag-r1.0

This was working on 2026-03-12 (commit 8003b44 when the test was added passed CI cleanly). The HF Hub content has changed since then, or the granite-4.0-micro adapter for this intrinsic was never published and was previously cached from a different test run.

Structural issue: The pydantic validator hits the network at collection time, which means any missing HF Hub artifact blocks the entire test suite from running — not just the tests for that intrinsic.

[planetf1]

Root cause confirmed — upstream HF Hub change

The failure has a concrete, external cause:

1. Repo renamed: ibm-granite/granite-lib-rag-r1.0 → ibm-granite/granitelib-rag-r1.0 (hyphen removed from granite-lib)

2. Adapters deleted: Commit 43c92f14af ("remove answer relevance (#19)", 2026-03-18) explicitly removed answer_relevance_classifier and answer_relevance_rewriter from the repo. Neither directory is present in the current tree.

This explains everything:

• The test started failing today because the deletion happened today
• The rename means catalog.py's _RAG_REPO = "ibm-granite/granite-lib-rag-r1.0" is now stale (HF Hub redirects but the adapters are gone regardless)
• The test file at collection time calls snapshot_download() to verify the files exist — they don't

What's currently in the repo (top-level dirs): answerability, citations, context_relevance, hallucination_detection, query_clarification, query_rewrite. No answer_relevance_*.

The catalog entries for these two intrinsics, and the tests that reference them, need to either be removed or updated to point to wherever (if anywhere) these adapters have been moved.

[planetf1]

@frreiss Does this reflect work currently in progress? Will this co-incide with our next release (imminent) or do need to make other changes prior that that?

[ajbozarth]

Fixed by #690 (auto-close didn't work)

[frreiss]

@frreiss Does this reflect work currently in progress? Will this co-incide with our next release (imminent) or do need to make other changes prior that that?

Yes, sorry, the answer relevance classifier was pulled from the release, and I forgot to remove the regression tests. Looks like the build is back up now.