Conversation
zhongxuanwang-nv
added
improvement
WalkthroughAdds an ADK-backed test LLM client and registers it as an ADK LLM wrapper, plus two ADK-based integration tests that build and invoke a NAT function using that test LLM and assert deterministic outputs. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
actor Test
participant Builder
participant ADKLLM as ADKTestLLM
participant Chooser as _ResponseChooser
Test->>Builder: build function (ADKFunctionConfig, LLMRef)
Builder->>ADKLLM: instantiate(model="nat_test_llm")
Test->>Builder: invoke function
Builder->>ADKLLM: generate_content_async(prompt, options)
ADKLLM->>Chooser: request next response
Chooser-->>ADKLLM: deterministic text
ADKLLM-->>Builder: LlmResponse(content=text)
Builder-->>Test: return function result
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Suggested labels
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
packages/nvidia_nat_test/src/nat/test/llm.py(1 hunks)packages/nvidia_nat_test/tests/test_test_llm.py(2 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*
⚙️ CodeRabbit configuration file
**/*: # Code Review Instructions
- Ensure the code follows best practices and coding standards. - For Python code, follow
PEP 20 and
PEP 8 for style guidelines.- Check for security vulnerabilities and potential issues. - Python methods should use type hints for all parameters and return values.
Example:def my_function(param1: int, param2: str) -> bool: pass- For Python exception handling, ensure proper stack trace preservation:
- When re-raising exceptions: use bare
raisestatements to maintain the original stack trace,
and uselogger.error()(notlogger.exception()) to avoid duplicate stack trace output.- When catching and logging exceptions without re-raising: always use
logger.exception()
to capture the full stack trace information.Documentation Review Instructions - Verify that documentation and comments are clear and comprehensive. - Verify that the documentation doesn't contain any TODOs, FIXMEs or placeholder text like "lorem ipsum". - Verify that the documentation doesn't contain any offensive or outdated terms. - Verify that documentation and comments are free of spelling mistakes, ensure the documentation doesn't contain any
words listed in the
ci/vale/styles/config/vocabularies/nat/reject.txtfile, words that might appear to be
spelling mistakes but are listed in theci/vale/styles/config/vocabularies/nat/accept.txtfile are OK.Misc. - All code (except .mdc files that contain Cursor rules) should be licensed under the Apache License 2.0,
and should contain an Apache License 2.0 header comment at the top of each file.
- Confirm that copyright years are up-to date whenever a file is changed.
Files:
packages/nvidia_nat_test/src/nat/test/llm.pypackages/nvidia_nat_test/tests/test_test_llm.py
packages/**/*
⚙️ CodeRabbit configuration file
packages/**/*: - This directory contains optional plugin packages for the toolkit, each should contain apyproject.tomlfile. - Thepyproject.tomlfile should declare a dependency onnvidia-nator another package with a name starting
withnvidia-nat-. This dependency should be declared using~=<version>, and the version should be a two
digit version (ex:~=1.0).
- Not all packages contain Python code, if they do they should also contain their own set of tests, in a
tests/directory at the same level as thepyproject.tomlfile.
Files:
packages/nvidia_nat_test/src/nat/test/llm.pypackages/nvidia_nat_test/tests/test_test_llm.py
🪛 Ruff (0.13.3)
packages/nvidia_nat_test/src/nat/test/llm.py
209-209: Unused function argument: builder
(ARG001)
217-218: Avoid specifying long messages outside the exception class
(TRY003)
224-224: Unused method argument: stream
(ARG002)
Signed-off-by: Daniel Wang <daniewang@nvidia.com>
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (1)
packages/nvidia_nat_test/src/nat/test/llm.py (1)
208-235: Add required type hints to all methods.Per coding guidelines, all public APIs must have type hints on parameters and return values. The function and methods are missing return type annotations.
Apply this diff to add the required type hints:
+from typing import AsyncGenerator + @register_llm_client(config_type=TestLLMConfig, wrapper_type=LLMFrameworkEnum.ADK) -async def test_llm_adk(config: TestLLMConfig, builder: Builder): +async def test_llm_adk(config: TestLLMConfig, builder: Builder) -> AsyncGenerator[Any, None]: """LLM client for Google ADK.""" try: from google.adk.models.base_llm import BaseLlm from google.adk.models.llm_response import LlmResponse from google.genai import types except ImportError as exc: raise ImportError("Google ADK is required for using the test_llm with ADK. " "Please install the `nvidia-nat-adk` package. ") from exc chooser = _ResponseChooser(response_seq=config.response_seq, delay_ms=config.delay_ms) class ADKTestLLM(BaseLlm): async def generate_content_async(self, llm_request: Any, - stream: bool = False) -> AsyncGenerator[LlmResponse, None]: + stream: bool = False) -> AsyncGenerator["LlmResponse", None]: self._maybe_append_user_content(llm_request) await chooser.async_sleep() text = chooser.next_response() yield LlmResponse(content=types.Content(role="model", parts=[types.Part.from_text(text=text)])) - def connect(self, *_args: Any, **_kwargs: Any) -> None: + def connect(self, *_args: Any, **_kwargs: Any) -> None: return None yield ADKTestLLM(model="nat_test_llm")Note:
AsyncGeneratoris already imported at line 21. The return type annotation forgenerate_content_asyncshould use a forward reference ("LlmResponse") since it's imported inside the function scope.Based on coding guidelines and past review comments.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
packages/nvidia_nat_test/src/nat/test/llm.py(1 hunks)packages/nvidia_nat_test/tests/test_test_llm.py(2 hunks)
🧰 Additional context used
📓 Path-based instructions (7)
**/*.{py,yaml,yml}
📄 CodeRabbit inference engine (.cursor/rules/nat-test-llm.mdc)
**/*.{py,yaml,yml}: Configure response_seq as a list of strings; values cycle per call, and [] yields an empty string.
Configure delay_ms to inject per-call artificial latency in milliseconds for nat_test_llm.
Files:
packages/nvidia_nat_test/src/nat/test/llm.pypackages/nvidia_nat_test/tests/test_test_llm.py
**/*.py
📄 CodeRabbit inference engine (.cursor/rules/nat-test-llm.mdc)
**/*.py: Programmatic use: create TestLLMConfig(response_seq=[...], delay_ms=...), add with builder.add_llm("", cfg).
When retrieving the test LLM wrapper, use builder.get_llm(name, wrapper_type=LLMFrameworkEnum.) and call the framework’s method (e.g., ainvoke, achat, call).
**/*.py: In code comments/identifiers use NAT abbreviations as specified: nat for API namespace/CLI, nvidia-nat for package name, NAT for env var prefixes; do not use these abbreviations in documentation
Follow PEP 20 and PEP 8; run yapf with column_limit=120; use 4-space indentation; end files with a single trailing newline
Run ruff check --fix as linter (not formatter) using pyproject.toml config; fix warnings unless explicitly ignored
Respect naming: snake_case for functions/variables, PascalCase for classes, UPPER_CASE for constants
Treat pyright warnings as errors during development
Exception handling: use bare raise to re-raise; log with logger.error() when re-raising to avoid duplicate stack traces; use logger.exception() when catching without re-raising
Provide Google-style docstrings for every public module, class, function, and CLI command; first line concise and ending with a period; surround code entities with backticks
Validate and sanitize all user input, especially in web or CLI interfaces
Prefer httpx with SSL verification enabled by default and follow OWASP Top-10 recommendations
Use async/await for I/O-bound work; profile CPU-heavy paths with cProfile or mprof before optimizing; cache expensive computations with functools.lru_cache or external cache; leverage NumPy vectorized operations when beneficial
Files:
packages/nvidia_nat_test/src/nat/test/llm.pypackages/nvidia_nat_test/tests/test_test_llm.py
packages/*/src/**/*.py
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
Importable Python code inside packages must live under packages//src/
Files:
packages/nvidia_nat_test/src/nat/test/llm.py
{src/**/*.py,packages/*/src/**/*.py}
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
All public APIs must have Python 3.11+ type hints on parameters and return values; prefer typing/collections.abc abstractions; use typing.Annotated when useful
Files:
packages/nvidia_nat_test/src/nat/test/llm.py
**/*
⚙️ CodeRabbit configuration file
**/*: # Code Review Instructions
- Ensure the code follows best practices and coding standards. - For Python code, follow
PEP 20 and
PEP 8 for style guidelines.- Check for security vulnerabilities and potential issues. - Python methods should use type hints for all parameters and return values.
Example:def my_function(param1: int, param2: str) -> bool: pass- For Python exception handling, ensure proper stack trace preservation:
- When re-raising exceptions: use bare
raisestatements to maintain the original stack trace,
and uselogger.error()(notlogger.exception()) to avoid duplicate stack trace output.- When catching and logging exceptions without re-raising: always use
logger.exception()
to capture the full stack trace information.Documentation Review Instructions - Verify that documentation and comments are clear and comprehensive. - Verify that the documentation doesn't contain any TODOs, FIXMEs or placeholder text like "lorem ipsum". - Verify that the documentation doesn't contain any offensive or outdated terms. - Verify that documentation and comments are free of spelling mistakes, ensure the documentation doesn't contain any
words listed in the
ci/vale/styles/config/vocabularies/nat/reject.txtfile, words that might appear to be
spelling mistakes but are listed in theci/vale/styles/config/vocabularies/nat/accept.txtfile are OK.Misc. - All code (except .mdc files that contain Cursor rules) should be licensed under the Apache License 2.0,
and should contain an Apache License 2.0 header comment at the top of each file.
- Confirm that copyright years are up-to date whenever a file is changed.
Files:
packages/nvidia_nat_test/src/nat/test/llm.pypackages/nvidia_nat_test/tests/test_test_llm.py
packages/**/*
⚙️ CodeRabbit configuration file
packages/**/*: - This directory contains optional plugin packages for the toolkit, each should contain apyproject.tomlfile. - Thepyproject.tomlfile should declare a dependency onnvidia-nator another package with a name starting
withnvidia-nat-. This dependency should be declared using~=<version>, and the version should be a two
digit version (ex:~=1.0).
- Not all packages contain Python code, if they do they should also contain their own set of tests, in a
tests/directory at the same level as thepyproject.tomlfile.
Files:
packages/nvidia_nat_test/src/nat/test/llm.pypackages/nvidia_nat_test/tests/test_test_llm.py
packages/*/tests/**/*.py
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
If a package contains Python code, include tests in a tests/ directory at the same level as pyproject.toml
Files:
packages/nvidia_nat_test/tests/test_test_llm.py
🧠 Learnings (2)
📓 Common learnings
Learnt from: CR
PR: NVIDIA/NeMo-Agent-Toolkit#0
File: .cursor/rules/nat-test-llm.mdc:0-0
Timestamp: 2025-09-09T20:32:39.016Z
Learning: When stubbing deterministic LLM responses in tests/workflows, use the NAT Test LLM (nat_test_llm).
📚 Learning: 2025-09-09T20:32:39.016Z
Learnt from: CR
PR: NVIDIA/NeMo-Agent-Toolkit#0
File: .cursor/rules/nat-test-llm.mdc:0-0
Timestamp: 2025-09-09T20:32:39.016Z
Learning: Ensure nat.test.llm is importable (install nvidia-nat-test from packages/ or import nat.test.llm once) before use.
Applied to files:
packages/nvidia_nat_test/tests/test_test_llm.py
🧬 Code graph analysis (2)
packages/nvidia_nat_test/src/nat/test/llm.py (2)
src/nat/builder/framework_enum.py (1)
LLMFrameworkEnum(19-25)src/nat/builder/builder.py (1)
Builder(68-290)
packages/nvidia_nat_test/tests/test_test_llm.py (3)
src/nat/data_models/component_ref.py (1)
LLMRef(116-124)packages/nvidia_nat_adk/src/nat/plugins/adk/agent.py (1)
ADKFunctionConfig(30-38)packages/nvidia_nat_test/src/nat/test/llm.py (3)
TestLLMConfig(36-43)ainvoke(87-89)ainvoke(193-195)
🪛 Ruff (0.13.3)
packages/nvidia_nat_test/src/nat/test/llm.py
209-209: Unused function argument: builder
(ARG001)
217-218: Avoid specifying long messages outside the exception class
(TRY003)
226-226: Unused method argument: stream
(ARG002)
🔇 Additional comments (1)
packages/nvidia_nat_test/tests/test_test_llm.py (1)
26-29: LGTM!The new imports are necessary for the ADK integration tests and are used correctly below.
Signed-off-by: Daniel Wang <daniewang@nvidia.com>
|
/merge |
|
@willkill07 Thanks Will! |
2 participants
Description
Closes
This is a small PR that adds the test_llm as a LLM Client for Google ADK, with related tests.
Since in NAT we use ADK's LiteLLM, I did check what public functions/fields were implemented by LiteLLM and overrode them properly. [LiteLLM code source]
By Submitting this PR I confirm:
Summary by CodeRabbit
New Features
Tests