Update evaluate.md to add options to avoid [429] Too Many Requests errors#1056
Merged
rapids-bot[bot] merged 2 commits intoNVIDIA:release/1.3from Oct 21, 2025
Merged
Conversation
Signed-off-by: Yuchen Zhang <yuchenz@nvidia.com>
yczhang-nv
added
doc
WalkthroughAdds guidance in the evaluation docs about handling rate limits (reduce eval concurrency or use a local NVIDIA NIM endpoint) and updates multiple example evaluation configs to set Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant User
participant EvalOrchestrator as Orchestrator
participant LLM
Note over Orchestrator: Config: max_concurrency: 1
User->>Orchestrator: Start evaluation (N tasks)
loop serialized runs (concurrency = 1)
Orchestrator->>LLM: Send task request
LLM-->>Orchestrator: Response
Orchestrator->>User: Record result
end
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Pre-merge checks and finishing touchesImportant Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error)
✅ Passed checks (2 passed)
✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
willkill07
reviewed
Oct 21, 2025
Signed-off-by: Yuchen Zhang <yuchenz@nvidia.com>
willkill07
approved these changes
Oct 21, 2025
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
examples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-tunable-rag-eval.yml (1)
1-29: Remove duplicate Apache License header.The file contains two complete Apache License headers (lines 1-14 and 16-29). Keep only one header at the top of the file.
# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. - -# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License.examples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-sizing-calc.yml (1)
1-29: Remove duplicate Apache License header.The file contains two complete Apache License headers (lines 1-14 and 16-29). Keep only one header at the top of the file.
# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. - -# SPDX-FileCopyrightText: Copyright (c) 2024-2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License.
🧹 Nitpick comments (1)
examples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_upload.yml (1)
61-61: Consider adding a comment explaining the rate-limit mitigation purpose.The
max_concurrency: 1setting addresses the PR's goal of avoiding[429] Too Many Requestserrors. Adding a brief inline comment would help users reading this example understand the rationale.general: + # Set max_concurrency to 1 to serialize evaluation runs and avoid rate limit errors (HTTP 429) max_concurrency: 1
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (10)
examples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-custom-dataset-format.yml(1 hunks)examples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-sizing-calc.yml(1 hunks)examples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-tunable-rag-eval.yml(1 hunks)examples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-with-custom-post-process.yml(1 hunks)examples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config.yml(2 hunks)examples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_llama31.yml(1 hunks)examples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_llama33.yml(1 hunks)examples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_openai.yml(1 hunks)examples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_only_config.yml(1 hunks)examples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_upload.yml(1 hunks)
✅ Files skipped from review due to trivial changes (1)
- examples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-with-custom-post-process.yml
🧰 Additional context used
📓 Path-based instructions (5)
**/*.{yaml,yml}
📄 CodeRabbit inference engine (.cursor/rules/nat-test-llm.mdc)
In workflow/config YAML, set llms.._type: nat_test_llm to stub responses.
Files:
examples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_upload.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_llama33.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_openai.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_only_config.ymlexamples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-tunable-rag-eval.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_llama31.ymlexamples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-custom-dataset-format.ymlexamples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-sizing-calc.yml
**/*.{py,yaml,yml}
📄 CodeRabbit inference engine (.cursor/rules/nat-test-llm.mdc)
**/*.{py,yaml,yml}: Configure response_seq as a list of strings; values cycle per call, and [] yields an empty string.
Configure delay_ms to inject per-call artificial latency in milliseconds for nat_test_llm.
Files:
examples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_upload.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_llama33.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_openai.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_only_config.ymlexamples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-tunable-rag-eval.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_llama31.ymlexamples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-custom-dataset-format.ymlexamples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-sizing-calc.yml
**/configs/**
📄 CodeRabbit inference engine (.cursor/rules/general.mdc)
Configuration files consumed by code must be stored next to that code in a configs/ folder
Files:
examples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_upload.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_llama33.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_openai.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_only_config.ymlexamples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-tunable-rag-eval.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_llama31.ymlexamples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-custom-dataset-format.ymlexamples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-sizing-calc.yml
**/*
⚙️ CodeRabbit configuration file
**/*: # Code Review Instructions
- Ensure the code follows best practices and coding standards. - For Python code, follow
PEP 20 and
PEP 8 for style guidelines.- Check for security vulnerabilities and potential issues. - Python methods should use type hints for all parameters and return values.
Example:def my_function(param1: int, param2: str) -> bool: pass- For Python exception handling, ensure proper stack trace preservation:
- When re-raising exceptions: use bare
raisestatements to maintain the original stack trace,
and uselogger.error()(notlogger.exception()) to avoid duplicate stack trace output.- When catching and logging exceptions without re-raising: always use
logger.exception()
to capture the full stack trace information.Documentation Review Instructions - Verify that documentation and comments are clear and comprehensive. - Verify that the documentation doesn't contain any TODOs, FIXMEs or placeholder text like "lorem ipsum". - Verify that the documentation doesn't contain any offensive or outdated terms. - Verify that documentation and comments are free of spelling mistakes, ensure the documentation doesn't contain any
words listed in the
ci/vale/styles/config/vocabularies/nat/reject.txtfile, words that might appear to be
spelling mistakes but are listed in theci/vale/styles/config/vocabularies/nat/accept.txtfile are OK.Misc. - All code (except .mdc files that contain Cursor rules) should be licensed under the Apache License 2.0,
and should contain an Apache License 2.0 header comment at the top of each file.
- Confirm that copyright years are up-to date whenever a file is changed.
Files:
examples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_upload.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_llama33.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_openai.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_only_config.ymlexamples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-tunable-rag-eval.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_llama31.ymlexamples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-custom-dataset-format.ymlexamples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-sizing-calc.yml
examples/**/*
⚙️ CodeRabbit configuration file
examples/**/*: - This directory contains example code and usage scenarios for the toolkit, at a minimum an example should
contain a README.md or file README.ipynb.
- If an example contains Python code, it should be placed in a subdirectory named
src/and should
contain apyproject.tomlfile. Optionally, it might also contain scripts in ascripts/directory.- If an example contains YAML files, they should be placed in a subdirectory named
configs/. - If an example contains sample data files, they should be placed in a subdirectory nameddata/, and should
be checked into git-lfs.
Files:
examples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_upload.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_llama33.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_openai.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_only_config.ymlexamples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-tunable-rag-eval.ymlexamples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_llama31.ymlexamples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-custom-dataset-format.ymlexamples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-sizing-calc.yml
🔇 Additional comments (9)
examples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-tunable-rag-eval.yml (1)
75-75: Config change looks good.The addition of
max_concurrency: 1undereval.generalcorrectly constrains evaluation concurrency to serialize execution, which helps avoid rate-limiting errors as described in the PR objectives.examples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-custom-dataset-format.yml (1)
57-57: Config change looks good.The addition of
max_concurrency: 1undereval.generalcorrectly constrains evaluation concurrency to serialize execution.examples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_openai.yml (1)
62-62: Config change looks good.The addition of
max_concurrency: 1undereval.generalcorrectly constrains evaluation concurrency.examples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config.yml (1)
55-55: Config change looks good.The addition of
max_concurrency: 1undereval.generalcorrectly constrains evaluation concurrency.examples/evaluation_and_profiling/simple_calculator_eval/src/nat_simple_calculator_eval/configs/config-sizing-calc.yml (1)
66-66: Config change looks good.The addition of
max_concurrency: 1undereval.generalcorrectly constrains evaluation concurrency.examples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_llama31.yml (1)
62-62: Config change looks good.The addition of
max_concurrency: 1undereval.generalcorrectly constrains evaluation concurrency.examples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_only_config.yml (1)
30-30: Config change looks good.The addition of
max_concurrency: 1undereval.generalcorrectly constrains evaluation concurrency.examples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_config_llama33.yml (1)
62-62: Config change looks good.The addition of
max_concurrency: 1undereval.generalcorrectly constrains evaluation concurrency.examples/evaluation_and_profiling/simple_web_query_eval/src/nat_simple_web_query_eval/configs/eval_upload.yml (1)
1-14: License header and copyright years are correct.The Apache 2.0 header is properly in place with up-to-date copyright years (2024-2025).
Member
|
/merge |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes nvbugs-5438561
Closes AIQ-1793
By Submitting this PR I confirm:
Summary by CodeRabbit
Documentation
Chores