fix: issues with tests (alora example, rag intrinsics, mistral tool use, vllm auto-skip)

[jakelorocco]

Misc PR

Type of PR

• Bug Fix
• New Feature
• Documentation
• Other

Description

• Link to Issue: N/A for most of them; bug: broken aLora example after intrinsics refactor #385 for alora 101 example

A few issues with tests that I stumbled on. These were errors in our test code not the mellea code the tests were testing.

• test/backends/test_huggingface_tools.py- we are using a mistral model that requires sentencepiece package installed -> fixed in pyproject.toml

• test/stdlib/components/intrinsic/test_rag.py- changes to the adapters for citations / hallucination detection resulted in slightly different values -> fixed the expected data

• docs/examples/aLora/102_example.py- expected input-> fixed by skipping this example and unskipping the 101_example.py that tests the same functionality.

• test/backends/test_openai_vllm.py -> exceptions raised during vllm setup were causing the test to error out instead of be skipped

Test passes

Testing

• Tests added to the respective file if code was changed
• New code has 100% coverage if code as added
• Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

[github-actions]

The PR description has been updated. Please fill out the template for your PR to be reviewed.

[mergify]

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 Enforce conventional commit

Wonderful, this rule succeeded.

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert|release)(?:\(.+\))?:

[psschwei]

Two of the test_rag tests failed for me:

=================================================================== FAILURES ====================================================================
________________________________________________________________ test_citations _________________________________________________________________

backend = <mellea.backends.huggingface.LocalHFBackend object at 0x351d82480>

    @pytest.mark.qualitative
    def test_citations(backend):
        """Verify that the citations intrinsic functions properly."""
        context, assistant_response, docs = _read_input_json("citations.json")
        expected = _read_output_json("citations.json")

        # First call triggers adapter loading
        result = rag.find_citations(assistant_response, docs, context, backend)
>       assert result == expected
E       assert [{'citation_b...nion. ", ...}] == [{'citation_b...nion. ", ...}]
E
E         At index 0 diff: {'response_begin': 0, 'response_end': 96, 'response_text': 'Murdoch expanded in Australia and New Zealand by acquiring and expanding local newspapers. ', 'citation_doc_id': '0', 'citation_begin'
E
E         ...Full output truncated (2 lines hidden), use '-vv' to show

test/stdlib/components/intrinsic/test_rag.py:130: AssertionError
------------------------------------------------------------- Captured stdout call --------------------------------------------------------------
=== 14:46:06-INFO ======
passing in model options when generating with an adapter; some model options may be overwritten / ignored
------------------------------------------------------------- Captured stderr call --------------------------------------------------------------
Fetching 1 files: 100%|██████████| 1/1 [00:00<00:00, 32263.88it/s]
Fetching 9 files: 100%|██████████| 9/9 [00:00<00:00, 190650.18it/s]
--------------------------------------------------------------- Captured log call ---------------------------------------------------------------
INFO     fancy_logger:huggingface.py:428 passing in model options when generating with an adapter; some model options may be overwritten / ignored
_________________________________________________________ test_hallucination_detection __________________________________________________________

backend = <mellea.backends.huggingface.LocalHFBackend object at 0x351d82480>

    @pytest.mark.qualitative
    def test_hallucination_detection(backend):
        """Verify that the hallucination detection intrinsic functions properly."""
        context, assistant_response, docs = _read_input_json("hallucination_detection.json")
        expected = _read_output_json("hallucination_detection.json")

        # First call triggers adapter loading
        result = rag.flag_hallucinated_content(assistant_response, docs, context, backend)
        # pytest.approx() chokes on lists of records, so we do this complicated dance.
        for r, e in zip(result, expected, strict=True):  # type: ignore
>           assert pytest.approx(r, abs=2e-2) == e
E           AssertionError: assert approx({'resp...he sentence.}) == {'explanation...end': 31, ...}
E
E             comparison failed. Mismatched elements: 1 / 5:
E             Max absolute difference: 5
E             Max relative difference: 0.1388888888888889
E             Index        | Obtained | Expected
E             response_end | 31       | 36 ± 0.02

test/stdlib/components/intrinsic/test_rag.py:164: AssertionError
------------------------------------------------------------- Captured stdout call --------------------------------------------------------------
=== 14:46:15-INFO ======
passing in model options when generating with an adapter; some model options may be overwritten / ignored
------------------------------------------------------------- Captured stderr call --------------------------------------------------------------
Fetching 1 files: 100%|██████████| 1/1 [00:00<00:00, 26886.56it/s]
Fetching 9 files: 100%|██████████| 9/9 [00:00<00:00, 147456.00it/s]
--------------------------------------------------------------- Captured log call ---------------------------------------------------------------
INFO     fancy_logger:huggingface.py:428 passing in model options when generating with an adapter; some model options may be overwritten / ignored

[psschwei]

The other three all passed (or were skipped) successfully

[jakelorocco]

Two of the test_rag tests failed for me:
...

@psschwei, can you please clarify. Did these tests fail when running against this branch and with packages updated?

[psschwei]

@psschwei, can you please clarify. Did these tests fail when running against this branch and with packages updated?

Yes, against this branch with a fresh venv (checkout branch as new worktree and uv sync --all-groups --all-extras in worktree dir)

[psschwei]

though I think your force push came after I checked out, let me retry

[psschwei]

more failures now (though all seem to be related to modules not found after the granite-common merge)

[jakelorocco]

Updated the commit to fix the pyproject packages and tests pass for me locally on mac and on linux with a clean environment.

[psschwei]

tests all pass for me now too