chore: batch LoaderV3 program data fetching

[thlorenz]

Summary

Optimize program executable fetching by batching multiple LoaderV3 program data account requests into a single get_multiple_accounts call instead of spawning concurrent tasks for each program. This improves efficiency when cloning multiple programs in a single transaction.

Details

fetch_cloner.rs

Refactored the LoaderV3 program data fetching logic to use a batched approach:

• Collect all program pubkeys and their corresponding program data account pubkeys into a single vector
• Call try_get_multi_until_slots_match once to fetch all accounts together
• Process the returned accounts in pairs (program account + program data account)
• Extract the resolve_account_with_companion logic into a reusable helper method to handle the resolution of program and delegation accounts

The new approach reduces the number of network round trips and task spawning overhead, particularly beneficial when multiple LoaderV3 programs need to be cloned concurrently.

test-cloning

Added a new integration test 08_multi_program_cloning.rs that verifies the batched fetching optimization works correctly:

• Creates a transaction that invokes two LoaderV3 programs (PARALLEL_MINIV3_1 and PARALLEL_MINIV3_2)
• Confirms both programs execute correctly by verifying their logs
• Demonstrates that the batching optimization handles multiple LoaderV3 programs in a single batch

test-chainlink/src/programs.rs

Added program constants for the two parallel programs used in the integration test.

configs/cloning-conf.devnet.toml

Added deployment configuration for the two additional parallel LoaderV3 programs.

Summary by CodeRabbit

• Performance

  • Program data fetching now operates on batches for faster, more efficient loading and fewer remote calls.

• Reliability

  • Centralized account/companion resolution with aggregated results and coordinated cancellation for consistent fetch outcomes.
  • Added an error state to surface companion-resolution failures.

• Tests

  • New integration test validating cloning and executing multiple loader programs in a single transaction.

• Chores

  • Added entries/constants for additional ephemeral loader programs used by integration tests.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[github-actions]

Manual Deploy Available

You can trigger a manual deploy of this PR branch to testnet:

Deploy to Testnet 🚀

Alternative: Comment /deploy on this PR to trigger deployment directly.

⚠️ Note: Manual deploy requires authorization. Only authorized users can trigger deployments.

Comment updated automatically when the PR is synchronized.

[coderabbitai]

📝 Walkthrough

Walkthrough

Replaces per-account remote fetches in fetch_cloner.rs with batched fetching using a collected pubkeys_to_fetch and try_get_multi_until_slots_match, introduces a batch_min_context_slot and batch-level max effective slot, and centralizes account+companion resolution via resolve_account_with_companion (adding a new ChainlinkError variant for missing companions). Updates fetch flow, error aggregation, cancellation logic, and logging to use companion terminology. Adds two LoaderV3 program entries to the devnet cloning TOML, exposes four public test constants for parallel MiniV3 programs, and adds an integration test that clones and executes two LoaderV3 programs in one transaction.

Possibly related issues

• magicblock-labs/magicblock-validator issue 581 — Implements the batched multi-account fetching and centralized companion-account resolution described in that issue.

Possibly related PRs

• magicblock-labs/magicblock-validator PR 578 — Modifies the same chainlink fetcher and companion-resolution/error-handling logic that this PR changes.

Suggested reviewers

• GabrielePicco
• bmuddha

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch thlorenz/parallel-exec-buffer-fetch

---

📜 Recent review details

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 94f12d8 and 011dea7.

📒 Files selected for processing (1)

• magicblock-chainlink/src/chainlink/fetch_cloner.rs (3 hunks)

🧰 Additional context used

🧠 Learnings (5)

📓 Common learnings

Learnt from: Dodecahedr0x
Repo: magicblock-labs/magicblock-validator PR: 639
File: magicblock-chainlink/tests/04_redeleg_other_separate_slots.rs:158-165
Timestamp: 2025-11-18T08:47:39.702Z
Learning: In magicblock-chainlink tests involving compressed accounts, `set_remote_slot()` sets the slot of the `AccountSharedData`, while `compressed_account_shared_with_owner_and_slot()` sets the slot of the delegation record. These are two different fields and both calls are necessary.

📚 Learning: 2025-10-21T14:00:54.642Z

Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 578
File: magicblock-aperture/src/requests/websocket/account_subscribe.rs:18-27
Timestamp: 2025-10-21T14:00:54.642Z
Learning: In magicblock-aperture account_subscribe handler (src/requests/websocket/account_subscribe.rs), the RpcAccountInfoConfig fields data_slice, commitment, and min_context_slot are currently ignored—only encoding is applied. This is tracked as technical debt in issue #579: https://github.com/magicblock-labs/magicblock-validator/issues/579

Applied to files:

• magicblock-chainlink/src/chainlink/fetch_cloner.rs

📚 Learning: 2025-11-19T09:34:37.917Z

Learnt from: thlorenz
Repo: magicblock-labs/magicblock-validator PR: 621
File: test-integration/test-chainlink/tests/ix_remote_account_provider.rs:62-63
Timestamp: 2025-11-19T09:34:37.917Z
Learning: In test-integration/test-chainlink/tests/ix_remote_account_provider.rs and similar test files, the `_fwd_rx` receiver returned by `init_remote_account_provider()` is intentionally kept alive (but unused) to prevent "receiver dropped" errors on the sender side. The pattern `let (remote_account_provider, _fwd_rx) = init_remote_account_provider().await;` should NOT be changed to `let (remote_account_provider, _) = ...` because dropping the receiver would cause send() operations to fail.

Applied to files:

• magicblock-chainlink/src/chainlink/fetch_cloner.rs

📚 Learning: 2025-11-18T08:47:39.702Z

Learnt from: Dodecahedr0x
Repo: magicblock-labs/magicblock-validator PR: 639
File: magicblock-chainlink/tests/04_redeleg_other_separate_slots.rs:158-165
Timestamp: 2025-11-18T08:47:39.702Z
Learning: In magicblock-chainlink tests involving compressed accounts, `set_remote_slot()` sets the slot of the `AccountSharedData`, while `compressed_account_shared_with_owner_and_slot()` sets the slot of the delegation record. These are two different fields and both calls are necessary.

Applied to files:

• magicblock-chainlink/src/chainlink/fetch_cloner.rs

📚 Learning: 2025-11-07T14:20:31.457Z

Learnt from: thlorenz
Repo: magicblock-labs/magicblock-validator PR: 621
File: magicblock-chainlink/src/remote_account_provider/chain_pubsub_actor.rs:457-495
Timestamp: 2025-11-07T14:20:31.457Z
Learning: In magicblock-chainlink/src/remote_account_provider/chain_pubsub_client.rs, the unsubscribe closure returned by PubSubConnection::account_subscribe(...) resolves to () (unit), not a Result. Downstream code should not attempt to inspect an unsubscribe result and can optionally wrap it in a timeout to guard against hangs.

Applied to files:

• magicblock-chainlink/src/chainlink/fetch_cloner.rs

🧬 Code graph analysis (1)

magicblock-chainlink/src/chainlink/fetch_cloner.rs (3)

magicblock-chainlink/src/remote_account_provider/program_account.rs (1)

• get_loaderv3_get_program_data_address (37-41)

magicblock-chainlink/src/remote_account_provider/mod.rs (1)

• default (124-130)

magicblock-chainlink/src/remote_account_provider/remote_account.rs (2)

• slot (203-215)
• account (180-202)

🔇 Additional comments (1)

magicblock-chainlink/src/chainlink/fetch_cloner.rs (1)

1351-1413: resolve_account_with_companion centralization and semantics look correct

This helper cleanly handles all RemoteAccount combinations:

• (NotFound, _) → ResolvedAccountCouldNoLongerBeFound(pubkey) for the primary.
• (Found, NotFound) → return the main account with companion_account: None.
• (Found, Found) → resolve both via resolved_account_shared_data(bank) and distinguish failures via ResolvedCompanionAccountCouldNoLongerBeFound(companion_pubkey) vs ResolvedAccountCouldNoLongerBeFound(pubkey).

The behavior matches the previously described semantics for both delegation-record and LoaderV3 program-data flows, and keeps callers simple. I don't see any functional issues here.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between f67482b and 57cd237.

📒 Files selected for processing (4)

• magicblock-chainlink/src/chainlink/fetch_cloner.rs (2 hunks)
• test-integration/configs/cloning-conf.devnet.toml (1 hunks)
• test-integration/test-chainlink/src/programs.rs (1 hunks)
• test-integration/test-cloning/tests/08_multi_program_cloning.rs (1 hunks)

🧰 Additional context used

🧠 Learnings (4)

📚 Learning: 2025-10-14T09:56:14.047Z

Learnt from: taco-paco
Repo: magicblock-labs/magicblock-validator PR: 564
File: test-integration/programs/flexi-counter/src/processor/call_handler.rs:122-125
Timestamp: 2025-10-14T09:56:14.047Z
Learning: The file test-integration/programs/flexi-counter/src/processor/call_handler.rs contains a test smart contract used for integration testing, not production code.

Applied to files:

• test-integration/test-cloning/tests/08_multi_program_cloning.rs

📚 Learning: 2025-11-07T13:09:52.253Z

Learnt from: bmuddha
Repo: magicblock-labs/magicblock-validator PR: 589
File: test-kit/src/lib.rs:275-0
Timestamp: 2025-11-07T13:09:52.253Z
Learning: In test-kit, the transaction scheduler in ExecutionTestEnv is not expected to shut down during tests. Therefore, using `.unwrap()` in test helper methods like `schedule_transaction` is acceptable and will not cause issues in the test environment.

Applied to files:

• test-integration/test-cloning/tests/08_multi_program_cloning.rs

📚 Learning: 2025-11-19T09:34:37.917Z

Learnt from: thlorenz
Repo: magicblock-labs/magicblock-validator PR: 621
File: test-integration/test-chainlink/tests/ix_remote_account_provider.rs:62-63
Timestamp: 2025-11-19T09:34:37.917Z
Learning: In test-integration/test-chainlink/tests/ix_remote_account_provider.rs and similar test files, the `_fwd_rx` receiver returned by `init_remote_account_provider()` is intentionally kept alive (but unused) to prevent "receiver dropped" errors on the sender side. The pattern `let (remote_account_provider, _fwd_rx) = init_remote_account_provider().await;` should NOT be changed to `let (remote_account_provider, _) = ...` because dropping the receiver would cause send() operations to fail.

Applied to files:

• test-integration/test-cloning/tests/08_multi_program_cloning.rs
• magicblock-chainlink/src/chainlink/fetch_cloner.rs

📚 Learning: 2025-11-07T14:20:31.457Z

Learnt from: thlorenz
Repo: magicblock-labs/magicblock-validator PR: 621
File: magicblock-chainlink/src/remote_account_provider/chain_pubsub_actor.rs:457-495
Timestamp: 2025-11-07T14:20:31.457Z
Learning: In magicblock-chainlink/src/remote_account_provider/chain_pubsub_client.rs, the unsubscribe closure returned by PubSubConnection::account_subscribe(...) resolves to () (unit), not a Result. Downstream code should not attempt to inspect an unsubscribe result and can optionally wrap it in a timeout to guard against hangs.

Applied to files:

• magicblock-chainlink/src/chainlink/fetch_cloner.rs

🧬 Code graph analysis (2)

test-integration/test-cloning/tests/08_multi_program_cloning.rs (1)

magicblock-validator/src/main.rs (1)

• init_logger (13-50)

magicblock-chainlink/src/chainlink/fetch_cloner.rs (2)

magicblock-chainlink/src/remote_account_provider/program_account.rs (1)

• get_loaderv3_get_program_data_address (37-41)

magicblock-chainlink/src/remote_account_provider/remote_account.rs (1)

• account (180-202)

🔇 Additional comments (3)

test-integration/configs/cloning-conf.devnet.toml (1)

47-56: New MiniV3 parallel program entries are consistent with test constants

The two additional [[program]] entries for MiniV321... and MiniV331... both point to the same miniv3 binary and use the expected MiniV4AUTH2.../MiniV4AUTH3... authorities, matching the new PARALLEL_MINIV3_* and *_AUTH constants in test_chainlink::programs. Config looks consistent with the new multi‑program cloning test.

test-integration/test-chainlink/src/programs.rs (1)

53-67: Parallel MiniV3 program constants align with config and tests

The new PARALLEL_MINIV3_1, PARALLEL_MINIV3_2 and corresponding *_AUTH constants are well-documented and match the IDs/auths in cloning-conf.devnet.toml. This gives a clean, typed way for tests (like 08_multi_program_cloning.rs) to reference the additional LoaderV3 programs used for batched cloning scenarios.

test-integration/test-cloning/tests/08_multi_program_cloning.rs (1)

1-73: Integration test correctly exercises two LoaderV3 programs in a single transaction

The test implementation is sound: it sends a single transaction with two MiniSdk::log_msg_instruction calls and asserts both log messages are present in the fetched logs, properly exercising the batched program-data fetching for multiple LoaderV3 programs. The init_logger! macro is correctly imported from test_kit (defined as macro_rules! init_logger in test-kit/src/macros.rs) and used consistently throughout the codebase.

[bmuddha]

LGTM