[None][Test] Consolidate dis-agg E2E Tests by Shixiaowei02 · Pull Request #16614 · NVIDIA/TensorRT-LLM

Shixiaowei02 · 2026-07-20T09:32:16Z

Description

This pull request removes the remaining UCX-specific disaggregated test configurations and test cases, consolidates and simplifies test parametrizations, and updates test lists and configs to reflect these changes. The main goal is to streamline the test suite by removing redundant or unnecessary UCX backend coverage, as this is now handled elsewhere or is no longer required.

Test configuration and coverage cleanup:

Removed all DeepSeek V3 Lite and Llama 3.1 8B disaggregated tests and configs that specifically targeted the UCX backend, including their YAML config files, test case entries, and test list references. [1] [2] [3] [4] [5] [6] [7] [8]
Updated test configs and test code to use "backend": "DEFAULT" instead of "UCX" for cache_transceiver_config, ensuring consistent backend selection and test coverage. [1] [2] [3] [4] [5] [6]

Test parametrization simplification:

Simplified test_auto_dtype in TestLlama3_1_8BInstruct to only test the two relevant block-reuse settings, removing overlap scheduler parameters which are now considered redundant. [1] [2] [3]
Limited grammar backend testing in guided decoding tests to xgrammar, since backend correctness is now covered by aggregated tests. [1] [2] [3] [4]

Test list and config file maintenance:

Removed references to deleted UCX-specific tests from llm_function_core.txt and updated test config lookups accordingly. [1] [2] [3] [4] [5] [6]

These changes reduce maintenance overhead and make the test suite easier to understand and extend.

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Dev Engineer Review

Removed UCX-specific disaggregated-serving configurations and test cases.
Changed the remaining Llama 3.1 disaggregated configuration to use the DEFAULT transceiver backend.
Simplified automatic-dtype and guided-decoding parameterizations.
Removed the KV-cache-manager-v2 NIXL test.
Removed test_disaggregated_logprobs_serving, which guarded NVBug 5926823. Confirm equivalent coverage or restore and repair the test.
Verify that DEFAULT provides the intended transport coverage.
Investigate the two failed CI runs before merge.

QA Engineer Review

Modified TestLlama3_1_8BInstruct.test_auto_dtype.
Removed the KV-cache-manager-v2 NIXL test.
Reduced guided-decoding coverage to xgrammar.
Removed test_disaggregated_logprobs_serving.
Removed UCX DeepSeek disaggregated-serving tests.
Updated llm_function_core.txt, llm_function_rtx6k.txt, l0_dgx_h100.yml, l0_rtx_pro_6000.yml, l0_dgx_b200.yml, l0_dgx_b300.yml, l0_h100.yml, and waives.txt.
The removed tests are reflected in the manifests.
The NVBug 5926823 scenario has no confirmed replacement coverage.
Two requested CI runs failed in partly tested pipelines.
Verdict: needs follow-up.

coderabbitai · 2026-07-20T09:36:27Z

Walkthrough

The change narrows disaggregated-serving coverage. Auto-dtype tests now use block-reuse cases. Guided-decoding tests retain xgrammar. Selected cache transceiver settings use DEFAULT, and UCX-specific tests and configuration wiring are removed.

Changes

Disaggregated serving test coverage

Layer / File(s)	Summary
Narrow auto-dtype coverage `tests/integration/defs/accuracy/test_disaggregated_serving.py`, `tests/integration/test_lists/qa/llm_function_core.txt`, `tests/integration/test_lists/test-db/l0_dgx_h100.yml`, `tests/integration/test_lists/waives.txt`	Auto-dtype tests now cover `block_reuse` and `no_block_reuse`, use fixed overlap scheduling, and update related test selections and waivers.
Restrict guided-decoding backends `tests/integration/defs/accuracy/test_disaggregated_serving.py`, `tests/integration/test_lists/qa/*`, `tests/integration/test_lists/test-db/l0_rtx_pro_6000.yml`, `tests/integration/test_lists/waives.txt`	Guided-decoding coverage retains `xgrammar` and removes the specified `llguidance` cases.
Update transport and disaggregation coverage `tests/integration/defs/accuracy/test_disaggregated_serving.py`, `tests/integration/defs/disaggregated/test_configs/`, `tests/integration/defs/disaggregated/test_disaggregated.py`, `tests/integration/test_lists/qa/llm_function_core.txt`, `tests/integration/test_lists/test-db/`	Selected cache transceiver settings use `DEFAULT`. The logprobs test uses the standard Llama configuration without `TRTLLM_USE_UCX_KVCACHE`. UCX-specific DeepSeek tests and mappings are removed.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

NVIDIA/TensorRT-LLM#17389: Both PRs remove UCX-based integration and QA test cases and configurations, including disaggregated-serving coverage.

Suggested reviewers: brnguyen2, longlee0622, ruodil

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: consolidating disaggregated serving end-to-end tests.
Description check	✅ Passed	The description explains the cleanup and rationale, but the Test Coverage section is empty and omits coverage for the removed logprobs regression test.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

Shixiaowei02 · 2026-07-20T09:38:11Z

/bot run --stage-list "DGX_H100-2_GPUs-PyTorch-Others-*, DGX_H100-4_GPUs-PyTorch-DeepSeek-1"

tensorrt-cicd · 2026-07-20T09:43:41Z

PR_Github #60342 [ run ] triggered by Bot. Commit: 6fb85e4 Link to invocation

tensorrt-cicd · 2026-07-20T09:58:42Z

PR_Github #60342 [ run ] completed with state FAILURE. Commit: 6fb85e4
/LLM/main/L0_MergeRequest_PR pipeline #48684 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Shixiaowei02 · 2026-07-20T10:52:18Z

/bot run --stage-list "DGX_H100-2_GPUs-PyTorch-Others-*, DGX_H100-4_GPUs-PyTorch-DeepSeek-1"

tensorrt-cicd · 2026-07-20T10:58:17Z

PR_Github #60359 [ run ] triggered by Bot. Commit: f430de6 Link to invocation

tensorrt-cicd · 2026-07-20T11:46:01Z

PR_Github #60359 [ run ] completed with state FAILURE. Commit: f430de6
/LLM/main/L0_MergeRequest_PR pipeline #48701 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Tabrizian · 2026-07-20T15:58:57Z

-@pytest.mark.skip_less_device(4)
-@pytest.mark.parametrize("llama_model_root", ['llama-3.1-8b-instruct'],
-                         indirect=True)
-def test_disaggregated_logprobs_serving(disaggregated_test_root,


Do we test log probs in any other place?

The API-level coverage here is unique: streaming vs. non-streaming logprobs consistency and top_logprobs across the ctx/gen boundary (RCCA for https://nvbugs/5926823). Only the core propagation is covered elsewhere (test_disaggregated_single_gpu.py::test_disaggregated_logprobs).

The test isn't UCX-specific — it just reused the UCX yaml. It also has an open bug (https://nvbugs/6275959, fix in flight: #15221). Suggest migrating it to a DEFAULT/NIXL config and keeping the waive, instead of deleting.

fredricz-20070104 · 2026-07-21T02:15:53Z

One thing worth calling out that isn't mentioned in the PR description: besides the UCX consolidation and parameter trimming, this PR also removes test_disaggregated_logprobs_serving (and its corresponding waive entry). That test is the resident regression guard for a customer bug (NVBug 5926823, disagg + streaming + logprobs).

Per our RCCA / test-escape policy, a customer-bug regression test shouldn't be silently dropped — it should be confirmed by the RCCA owner and the removal should be documented in the description. Could you please either (a) confirm that scenario is now covered elsewhere and note it in the PR description, or (b) keep (fix) the test instead of deleting it? Thanks!

fredricz-20070104

Please resolve my comments before merging.

nv-xtf

LGTM overall — just one question about the backend default behavior (DEFAULT still resolves to UCX in this harness), see inline comment.

…nd config Review follow-up on NVIDIA#16614 (yingguo-trt, fredricz-20070104): the consolidation dropped test_disaggregated_logprobs_serving, which is the resident regression guard for NVBug 5926823 (disagg + streaming + logprobs). Its API-level coverage is unique -- streaming vs. non-streaming logprobs consistency across both the completions and chat APIs, plus chat top_logprobs across the ctx/gen boundary. test_disaggregated_single_gpu.py::test_disaggregated_logprobs only covers the core prefill->decode propagation, so deleting this would be a test escape. The test was never UCX-specific; it only reused the UCX yaml. Restore it on a new DEFAULT-backend config and drop the explicit TRTLLM_USE_UCX_KVCACHE pin, which keeps the de-UCX-specialization goal of this PR intact. UCX_TLS is kept, matching every other non-pinned test in this file. The waive for the open bug (NVBugs 6275959, fix in flight in NVIDIA#15221) is restored with it. Restores the numpy import: np.isclose in this test is its only user. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>

coderabbitai · 2026-08-06T10:15:39Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

…nd config Review follow-up on NVIDIA#16614 (yingguo-trt, fredricz-20070104): the consolidation dropped test_disaggregated_logprobs_serving, which is the resident regression guard for NVBug 5926823 (disagg + streaming + logprobs). Its API-level coverage is unique -- streaming vs. non-streaming logprobs consistency across both the completions and chat APIs, plus chat top_logprobs across the ctx/gen boundary. test_disaggregated_single_gpu.py::test_disaggregated_logprobs only covers the core prefill->decode propagation, so deleting this would be a test escape. The test was never UCX-specific; it only reused the UCX yaml. Restore it on a new DEFAULT-backend config and drop the explicit TRTLLM_USE_UCX_KVCACHE pin, which keeps the de-UCX-specialization goal of this PR intact. UCX_TLS is kept, matching every other non-pinned test in this file. The waive for the open bug (NVBugs 6275959, fix in flight in NVIDIA#15221) is restored with it. Restores the numpy import: np.isclose in this test is its only user. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>

coderabbitai · 2026-08-06T15:56:09Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

Shixiaowei02 · 2026-08-06T15:58:07Z

/bot run --add-multi-gpu-test --disable-fail-fast --extra-stage "DGX_B200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-Post-Merge-2, DGX_B200-4_GPUs-PyTorch-Post-Merge-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-4, RTXPro6000D-PyTorch-Post-Merge-1, RTXPro6000D-4_GPUs-PyTorch-Post-Merge-1, RTXPro6000D-4_GPUs-PyTorch-Post-Merge-2, DGX_B300-4_GPUs-PyTorch-Post-Merge-1, DGX_B300-4_GPUs-PyTorch-Post-Merge-2"

The 2**4 cross-product of ctx/gen overlap-scheduler x ctx/gen block-reuse ran a full MMLU+GSM8K eval per combo. Overlap scheduler is token-invariant (covered by unit tests test_benchmark_disagg.py and test_disagg_gen_transfer_gate.py), so only the block-reuse axis changes which KV blocks are transferred. Keep block-reuse on/off with overlap enabled (production default) and drop the other 14 combos from CI (l0_dgx_h100, llm_function_core). Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…d_decoding Guided decoding runs entirely on the generation worker and the KV-cache handoff is grammar-backend-agnostic, so exercising both xgrammar and llguidance under disaggregated serving only re-validates grammar-engine correctness that the aggregated guided-decoding tests already cover (test_llm_api_pytorch.py has 7 backend-parametrized sites). Keep xgrammar (the default) for the disagg plumbing guard and the DeepSeek guided+MTP interaction; drop the llguidance combos from TestLlama3_1_8BInstruct/TestDeepSeekV3Lite::test_guided_decoding (7 CI entries across l0_rtx_pro_6000, llm_function_core, llm_function_rtx6k). Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…kv_cache_v2 UCX cache-transceiver coverage is model-agnostic plumbing already exercised by the cheap unit/C++ tests (unittest/disaggregated UCX params, cpp/test_multi_gpu test_cache_transceiver ucx_kvcache), so the expensive e2e UCX-specialized integration tests are redundant. Remove test_disaggregated_deepseek_v3_lite_fp8_ucx and test_disaggregated_deepseek_v3_lite_fp8_ucx_tp1_single_gpu, plus their now orphaned config disagg_config_ctxtp2_gentp2_deepseek_v3_lite_ucx.yaml and its config-map entry. Also remove TestLlama3_1_8BInstruct::test_kv_cache_v2_nixl_python: the KVCacheManager-v2 + NIXL + PYTHON transceiver path is model-agnostic and byte-covered by test_cache_transceiver_single_process.py; TestDeepSeekV3Lite keeps the canonical MLA representative. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…ogprobs test Switch the hardcoded backend=UCX in TestDeepSeekV3Lite/TestQwen3_8B test_auto_dtype_with_helix and the TestQwen3_8B chunked-prefill helper to DEFAULT, matching the rest of the accuracy suite (these tests cover helix and chunked prefill, not UCX specifically). Remove test_disaggregated_logprobs_serving and the llama31_8b_ucx config-map entry plus the orphaned yaml it exclusively used. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…nd config Review follow-up on NVIDIA#16614 (yingguo-trt, fredricz-20070104): the consolidation dropped test_disaggregated_logprobs_serving, which is the resident regression guard for NVBug 5926823 (disagg + streaming + logprobs). Its API-level coverage is unique -- streaming vs. non-streaming logprobs consistency across both the completions and chat APIs, plus chat top_logprobs across the ctx/gen boundary. test_disaggregated_single_gpu.py::test_disaggregated_logprobs only covers the core prefill->decode propagation, so deleting this would be a test escape. The test was never UCX-specific; it only reused the UCX yaml. Restore it on a new DEFAULT-backend config and drop the explicit TRTLLM_USE_UCX_KVCACHE pin, which keeps the de-UCX-specialization goal of this PR intact. UCX_TLS is kept, matching every other non-pinned test in this file. The waive for the open bug (NVBugs 6275959, fix in flight in NVIDIA#15221) is restored with it. Restores the numpy import: np.isclose in this test is its only user. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>

test_disaggregated_logprobs_serving is the regression guard for NVBug 5926823 (disagg + streaming + logprobs). The migration to the DEFAULT-backend config kept the test function and its waive, but dropped its entry from qa/llm_function_core.txt, leaving it in no test list at all -- so lifting the 6275959 waive later would not have brought it back. Restore the entry; the waive stays. validate-test-lists was already reporting this as a waive with no active test list. Two more entries that hook rejected, both from this branch's parametrize consolidation rather than from review: - test_guided_decoding[llguidance-mtp_nextn=2]: the backend list is now ["xgrammar"] only, so this ID no longer exists. - test_auto_dtype[False-True-True-True]: the four boolean parametrize decorators became one (ctx,gen) pair with ids block_reuse/no_block_reuse, and the waived combination was the mixed ctx=True/gen=False one, which no longer exists. Both waives point at test IDs that cannot be collected, so they are removed. If 6525893 still reproduces on a surviving test_auto_dtype variant, its waive needs re-pointing rather than deleting. Also record why DEFAULT is used instead of NIXL: launch_disaggregated_llm still sets TRTLLM_USE_UCX_KVCACHE=1 for every backend but NIXL, so dropping the per-test UCX pinning leaves transport coverage unchanged. Signed-off-by: Xiaowei Shi <39303645+Shixiaowei02@users.noreply.github.com>

Shixiaowei02 · 2026-08-07T02:12:31Z

/bot run --add-multi-gpu-test --disable-fail-fast --extra-stage "DGX_B200-4_GPUs-PyTorch-Post-Merge-1, DGX_B200-4_GPUs-PyTorch-Post-Merge-2, DGX_B200-4_GPUs-PyTorch-Post-Merge-3, DGX_B200-4_GPUs-PyTorch-Post-Merge-4, RTXPro6000D-PyTorch-Post-Merge-1, RTXPro6000D-4_GPUs-PyTorch-Post-Merge-1, RTXPro6000D-4_GPUs-PyTorch-Post-Merge-2, DGX_B300-4_GPUs-PyTorch-Post-Merge-1, DGX_B300-4_GPUs-PyTorch-Post-Merge-2"

coderabbitai · 2026-08-07T02:12:32Z

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

tensorrt-cicd · 2026-08-07T02:18:44Z

PR_Github #64455 [ run ] triggered by Bot. Commit: 9732c55 Link to invocation

tensorrt-cicd · 2026-08-07T07:08:06Z

PR_Github #64455 [ run ] completed with state FAILURE. Commit: 9732c55
/LLM/main/L0_MergeRequest_PR pipeline #52330 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

github-actions Bot assigned Shixiaowei02 Jul 20, 2026

Shixiaowei02 marked this pull request as ready for review July 20, 2026 09:33

Shixiaowei02 requested review from a team as code owners July 20, 2026 09:33

Shixiaowei02 requested review from Tabrizian, brnguyen2, chuangz0, fredricz-20070104, joyang-nv, longlee0622, mlefeb01, mzweilz, nv-xtf, reasonsolo, schetlur-nv and yingguo-trt July 20, 2026 09:33

Shixiaowei02 force-pushed the user/xiaoweis/trim-disagg-e2e-accuracy branch from 9ce9bbb to f430de6 Compare July 20, 2026 10:49

Tabrizian approved these changes Jul 20, 2026

View reviewed changes

fredricz-20070104 approved these changes Jul 21, 2026

View reviewed changes

nv-xtf approved these changes Jul 21, 2026

View reviewed changes

Comment thread tests/integration/defs/accuracy/test_disaggregated_serving.py

yingguo-trt approved these changes Jul 28, 2026

View reviewed changes

Shixiaowei02 force-pushed the user/xiaoweis/trim-disagg-e2e-accuracy branch from f430de6 to 973bfa0 Compare August 3, 2026 05:54

Shixiaowei02 requested review from a team as code owners August 3, 2026 05:54

Shixiaowei02 requested review from StanleySun639 and ruodil August 3, 2026 05:54

Shixiaowei02 force-pushed the user/xiaoweis/trim-disagg-e2e-accuracy branch from 973bfa0 to 7a1e5eb Compare August 3, 2026 11:23

StanleySun639 approved these changes Aug 4, 2026

View reviewed changes

Shixiaowei02 force-pushed the user/xiaoweis/trim-disagg-e2e-accuracy branch from 06b81fb to c52c272 Compare August 6, 2026 15:56

Shixiaowei02 and others added 6 commits August 7, 2026 10:12

Shixiaowei02 force-pushed the user/xiaoweis/trim-disagg-e2e-accuracy branch from c52c272 to 9732c55 Compare August 7, 2026 02:12

This was referenced Aug 7, 2026

[TRTLLM-15078][test] Prune non-Llama-3.1-8B Llama tests #17392

Open

[https://nvbugs/6523520][fix] Halve gb200 r1-fp4 128k8k con128 multi_round to fit perf-sanity budget #17397

Merged

coderabbitai Bot mentioned this pull request Aug 7, 2026

[TRTLLM-14814][feat] Kimi K3 serving parsers, chat template, and speculative decoding (suffix automaton + DFlash scaffold) #17327

Open

3 tasks

Uh oh!

Conversation

Shixiaowei02 commented Jul 20, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

GitHub Bot Help

Dev Engineer Review

QA Engineer Review

Uh oh!

coderabbitai Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

Shixiaowei02 commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

Shixiaowei02 commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

tensorrt-cicd commented Jul 20, 2026

Uh oh!

Tabrizian Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

yingguo-trt Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

fredricz-20070104 commented Jul 21, 2026

Uh oh!

fredricz-20070104 left a comment

Choose a reason for hiding this comment

Uh oh!

nv-xtf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot commented Aug 6, 2026

Uh oh!

coderabbitai Bot commented Aug 6, 2026

Uh oh!

Shixiaowei02 commented Aug 6, 2026

Uh oh!

Shixiaowei02 commented Aug 7, 2026

Uh oh!

coderabbitai Bot commented Aug 7, 2026

Uh oh!

tensorrt-cicd commented Aug 7, 2026

Uh oh!

tensorrt-cicd commented Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Shixiaowei02 commented Jul 20, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 20, 2026 •

edited

Loading