Add local document chat agent using Microsoft Agent Framework + Azure AI Search

[Copilot]

Adds a local document chat agent that utilizes an Azure AI Search index to answer user queries about indexed content, built on the Microsoft Agent Framework.

Changes Made

• Agent implementation (src/agent/agent.py): Created a document chat agent using AzureAIAgent from Semantic Kernel with an AzureAISearchTool configured for vector-semantic-hybrid queries. The agent reads all configuration (endpoint, model deployment, search service name) from config/config.yaml with no CLI arguments required.
• Auto-resolve AI Search connection: The ensure_ai_search_connection_id function lists existing AI Search connections in the AI Foundry project and returns a match by service name. If no connection exists, it auto-creates one via MLClient.connections.create_or_update using AAD authentication.
• Workspace name resolution: The _extract_workspace_name helper extracts the Foundry resource name from the endpoint hostname (e.g., myfoundry from myfoundry.services.ai.azure.com) for use as the workspace_name parameter in MLClient, matching the ARM resource name used by the new (non-hub) Azure AI Foundry.
• Dynamic index name: Uses the shared generate_index_name() utility (same as build_indexer.py) so the index name is consistent across the pipeline.
• Config integration: agent_config in config.yaml uses ${AI_FOUNDRY_PROJECT_URI} and ${AZURE_AI_AGENT_MODEL_DEPLOYMENT_NAME} — no duplicate environment variables.
• Dev container: Added a RUN step in .buildcontainer/Dockerfile to print the azure-search-documents package version at build time.
• Tests (tests/agent/test_agent.py): 12 unit tests covering workspace name extraction, connection resolution/creation, agent creation, conversation handling, and module constants.
• Dependencies: Added azure-ai-projects, azure-ai-agents, semantic-kernel, and azure-ai-ml to root requirements.txt; added src/__init__.py and src/agent/__init__.py for package resolution; added [tool:pytest] pythonpath = . to setup.cfg.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Create an agent that utilizes the index</issue_title>
<issue_description>The project demonstrates how to use AI Search to create an index and indexer including custom skills and data sources.

Now, we need to create an agent that can utilize this index to answer users queries about the indexed content.

The agent should be created using Microsoft Agent Framework.

At this point, we need to create an agent with an ability to run it locally. In later tasks we will add deployment and evaluation capabilities.

The primary idea of the agent is chat with documents that are part of AI Search index.</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Create an agent that utilizes the index #100

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[sbaidachni]

@copilot It will be great to have code that created connection id if it should be created or document about how to make it. Utilize the same Foundry that we already have in config rather than generate a new parameter. Index name should be generated dynamically using the same approach that we are using to create it

[Copilot]

@copilot It will be great to have code that created connection id if it should be created or document about how to make it. Utilize the same Foundry that we already have in config ra...

Addressed in commit 85570b9:

1. Connection ID auto-resolved: Added get_ai_search_connection_id(endpoint, acs_service_name) which calls AIProjectClient.connections.list(connection_type=ConnectionType.AZURE_AI_SEARCH) to find the AI Search connection whose target URL matches acs_service_name from config. Falls back to the default connection if no name match is found, and raises a ValueError with a clear message if no AI Search connections exist in the project.

2. Same Foundry endpoint from config: agent_config.agent_endpoint in config/config.yaml now uses ${AI_FOUNDRY_PROJECT_URI} (already defined in .env.sample). Removed the duplicate AZURE_AI_AGENT_ENDPOINT variable.

3. Index name generated dynamically: main() now calls generate_index_name() (same function used by build_indexer.py) and no longer accepts --index-name or --connection-id CLI arguments. Usage is now:

   python src/agent/agent.py --stage pr --model gpt-4o

[sbaidachni]

@copilot please, follow-up my comments

[sbaidachni]

@copilot any reason to use stage here since our config parts are stage agnostic?

[Copilot]

Removed the --stage argument in commit 8b0f8ac. MLOpsConfig() is now called with no arguments since agent_config and acs_config are not stage-specific.

[sbaidachni]

@copilot we don't need the deployment as a parameter since it's in our config file

[Copilot]

Removed the --model argument in commit 8b0f8ac. The model deployment is now read directly from agent_config["agent_model_deployment"] in config.yaml.

[sbaidachni]

@copilot can we create a connection if it doesn't exist?

[Copilot]

Yes — implemented in commit 8b0f8ac. The function ensure_ai_search_connection_id now:

1. Lists existing CognitiveSearch connections in the AI Foundry project
2. Returns the matching connection ID if one is found whose target URL contains ACS_SERVICE_NAME
3. If no match, calls MLClient.connections.create_or_update(AzureAISearchConnection(...)) using subscription_id and resource_group_name from sub_config — the project name is extracted from the endpoint URL. AAD/managed-identity auth is used (no API key needed).

[Copilot]

@copilot please, follow-up my comments

All three review comments addressed in commit 8b0f8ac:

• Removed --stage (config is stage-agnostic)
• Removed --model (read from config)
• Added auto-creation of AI Search connection in AI Foundry when none exists

[sbaidachni]

@copilot the pr fails with the following error: ./src/agent/agent.py:48:1: D205 1 blank line required between summary line and description
./src/agent/agent.py:48:1: D400 First line should end with a period
1 D205 1 blank line required between summary line and description
1 D400 First line should end with a period
2
Error: Process completed with exit code 1.

[Copilot]

@copilot the pr fails with the following error: ./src/agent/agent.py:48:1: D205 1 blank line required between summary line and description
./src/agent/agent.py:48:1: D400 First line...

Fixed in commit 50945bb. The docstring now places the summary on the opening """ line and ends it with a period, satisfying both D205 and D400.

[sbaidachni]

@copilot Now, I have a problem with unit tests: ============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0
rootdir: /__w/mlops-aisearch-pull/mlops-aisearch-pull
plugins: anyio-4.12.1, mock-3.15.1, cov-7.0.0
collected 1 item / 1 error
==================================== ERRORS ====================================
__________________ ERROR collecting tests/agent/test_agent.py __________________
ImportError while importing test module '/__w/mlops-aisearch-pull/mlops-aisearch-pull/tests/agent/test_agent.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/lib/python3.12/importlib/init.py:90: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tests/agent/test_agent.py:6: in
from azure.ai.projects.models import ConnectionType
E ModuleNotFoundError: No module named 'azure.ai.projects'

• generated xml file: /__w/mlops-aisearch-pull/mlops-aisearch-pull/junit/test-results.xml -
  =========================== short test summary info ============================
  ERROR tests/agent/test_agent.py
  !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
  =============================== 1 error in 0.26s ===============================
  Error: Process completed with exit code 2.

[Copilot]

@copilot Now, I have a problem with unit tests: ============================= test session starts ==============================
platform linux -- Python 3.12.12, pytest-9.0.2, plug...

Fixed in commit 0184368. The root requirements.txt was missing azure-ai-projects, azure-ai-agents, semantic-kernel, and azure-ai-ml, which caused the top-level from azure.ai.projects.models import ConnectionType import in the test file to fail. Added those four packages to requirements.txt so they're available when the CI runs pytest.

[sbaidachni]

@copilot FAILED tests/agent/test_agent.py::TestExtractProjectName::test_extracts_last_path_segment - ModuleNotFoundError: No module named 'src'

[Copilot]

@copilot Look at CI log. There are many warnings from test cases about implementations.

Fixed in commit 8f3cd5a. The 471 warnings all came from azure-ai-ml's marshmallow schema code firing deprecation warnings when the module loaded. Two changes:

1. Lazy imports: moved MLClient/AzureAISearchConnection imports inside ensure_ai_search_connection_id (only used in the connection-creation fallback path), and updated test patches from @patch("src.agent.agent.MLClient") to @patch("azure.ai.ml.MLClient").
2. filterwarnings in setup.cfg: suppresses DeprecationWarning/PendingDeprecationWarning from marshmallow and azure.ai.ml third-party code we don't control.

Result: 12 tests pass, 0 warnings.

[sbaidachni]

@copilot revert 8f3cd5a

[Copilot]

@copilot revert 8f3cd5a

Done — reverted in commit 4675a7c. setup.cfg, src/agent/agent.py, and tests/agent/test_agent.py are all restored to their state before 8f3cd5a.

[sbaidachni]

@copilot when I am looking at Pre-build Dev Container Image and Push to Azure Container Registry Action log I don't see any azure-search-documents package to be installed as a part of container. it should be fixed. add code that will print version of this package at the end of build as well

[Copilot]

@copilot when I am looking at Pre-build Dev Container Image and Push to Azure Container Registry Action log I don't see any azure-search-documents package to be installed as a part o...

Done in commit e91ac6d. azure-search-documents was already in requirements.txt and gets installed into the venv via pip install -r requirements.txt. I've added a RUN step at the end of .buildcontainer/Dockerfile that prints its version:

RUN /home/$USERNAME/llm-env/bin/python -c \
    "import importlib.metadata; print('azure-search-documents version:', importlib.metadata.version('azure-search-documents'))"

This will produce a line like azure-search-documents version: 11.6.0 in the build log, making it easy to confirm the package is present.

[github-actions]

Search Evaluation Results

Metric	Value
AveragePrecision.average_precision	0.0
F1-score@10.f1_score_at_10	0.0
F1-score@3.f1_score_at_3	0.0
F1-score@5.f1_score_at_5	0.0
Precision@10.precision_at_10	0.0
Precision@3.precision_at_3	0.0
Precision@5.precision_at_5	0.0
Recall@10.recall_at_10	0.0
Recall@3.recall_at_3	0.0
Recall@5.recall_at_5	0.0
ReciprocalRank.reciprocal_rank	0.0

[sbaidachni]

@copilot code doesn't create ai searcj connection raising the error: azure.core.exceptions.ResourceNotFoundError: (ResourceNotFound) The Resource 'Microsoft.MachineLearningServices/workspaces/proj-default' under resource group 'mlops-aisearch-pull' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
Code: ResourceNotFound
Message: The Resource 'Microsoft.MachineLearningServices/workspaces/proj-default' under resource group 'mlops-aisearch-pull' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix

[Copilot]

@copilot code doesn't create ai searcj connection raising the error: azure.core.exceptions.ResourceNotFoundError: (ResourceNotFound) The Resource 'Microsoft.MachineLearningServices/works...

Fixed in commit b1f0077.

I updated connection creation to retry ML workspace resolution using both names parsed from the Foundry endpoint:

1. project name (.../api/projects/<project>) and then
2. hub name (<hub>.services.ai.azure.com)

If the project workspace is not found, it now falls back to the hub workspace instead of failing immediately. I also added tests for this fallback path and malformed endpoint parsing, and validated with python -m pytest -q tests/agent/test_agent.py (16 passed).

[sbaidachni]

@copilot revert latest commit. I am not using any Hub based foundry. I am using new foundry

[Copilot]

@copilot revert latest commit. I am not using any Hub based foundry. I am using new foundry

Done in commit 5299a8f.

I reverted the latest hub-based fallback changes and restored src/agent/agent.py and tests/agent/test_agent.py to the pre-b1f0077 state.

Validation: python -m pytest -q tests/agent/test_agent.py → 12 passed.

UI screenshot: N/A (no UI changes in this update).

[sbaidachni]

@copilot Now, fix this issue: azure.core.exceptions.ResourceNotFoundError: (ResourceNotFound) The Resource 'Microsoft.MachineLearningServices/workspaces/proj-default' under resource group 'mlops-aisearch-pull' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
Code: ResourceNotFound
Message: The Resource 'Microsoft.MachineLearningServices/workspaces/proj-default' under resource group 'mlops-aisearch-pull' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix

[Copilot]

@copilot Now, fix this issue: azure.core.exceptions.ResourceNotFoundError: (ResourceNotFound) The Resource 'Microsoft.MachineLearningServices/workspaces/proj-default' under resource gr...

Fixed in commit 89a59f3.

The root cause was that _extract_project_name was pulling the project name from the URL path (proj-default), but in the new AI Foundry the ARM workspace resource is the Foundry resource name from the hostname.

Changed _extract_project_name → _extract_workspace_name, which now parses the hostname prefix (e.g., myfoundry from myfoundry.services.ai.azure.com) and passes that as workspace_name to MLClient. 12 tests pass, lint clean.