perf: deduplicate imports as they merge, not once at the end

[adhami3310]

What

merge_imports and merge_parsed_imports concatenated without deduplicating, and collapse_imports deduplicated through a set. This makes the merges dedupe as they go, and makes the collapse preserve first-seen order.

Why it is slow

Component._get_all_imports merges every descendant's imports on the way up the tree. Because the merge only concatenated, a tag shared across a subtree was carried once per node rather than once, so each node's import list grew with the size of its subtree instead of with the number of distinct imports it actually has.

Nothing consumed those duplicates, since collapse_imports dropped them at the end, once per page. But _get_component_hash, which names every auto-memoized component, hashes the node's whole recursive import dict, so the cost landed there instead.

Measured on an app with 80 routes and 42,268 components (7,036 memo wrappers):

_update_deterministic_hash calls	248,572,401
share of compile spent hashing	~65%
one Button: values hashed	2,263,441
one Button: distinct objects behind them	133
that Button's hash	3.5s

That Button has no children. The 2.26M came from 323,296 ImportVar visits over 21 distinct import vars.

Result

reflex compile on that app, run back to back on the same machine:

	before	after
compiler	263s	48s

Reproducibility

collapse_imports used list(set(import_vars)), whose iteration order over ImportVar varies with PYTHONHASHSEED. Two compiles of identical source therefore emitted imports in different orders, and since import order feeds _get_component_hash, every memo name derived from it changed too.

On the same app, two consecutive runs of unmodified reflex rewrote 113 of 157 generated files with different memo names. With this change both runs agree byte for byte.

That is worth having on its own: stable output means a bundler and its downstream caches see unchanged modules when nothing changed.

Deduplicating a library's first batch against itself, added in review below, turned out to be load-bearing for this too. Those repeats varied per process, so they moved the memo hash even once import order was stable. On the app above, the two fixes together take it from 113 differing files to 0, byte-identical between PYTHONHASHSEED=1 and PYTHONHASHSEED=999983.

On the CodSpeed regression

test_get_all_imports is genuinely slower, and the first version of this PR was worse than what is here now. Worth being precise about what that benchmark measures.

It times _get_all_imports() alone. Deduplicating is pure cost inside that call; the payoff is a much smaller dict, and it is collected by whoever consumes the result. On _complicated_page, the result goes from 277 entries to 19 (14.6x), and on _stateful_page from 61 to 10. The benchmark pays for that reduction and never uses it, so it can only show the cost side. The whole-compile benchmarks are unaffected, and the app above spends 65% of its compile hashing exactly what shrank.

I still narrowed it rather than leaving it. A library's first batch has nothing accumulated to collide with, so it is deduplicated against itself and no further, and a batch of one entry skips even that; only from the second batch does a library earn a lookup set. Locally, on _complicated_page over 300 calls: base 0.101s, dedupe-everywhere 0.255s, this version 0.170s.

The batch-of-one guard is doing real work there. Deduplicating every first batch unconditionally costs 36% (0.165s to 0.225s in one run of the same comparison), because the average first batch is 1.4 entries and it allocates a dict a million times for batches that cannot duplicate. The guard gets the same correctness for 9%.

I also tried and dropped two further ideas: inlining the accumulator to avoid a call per library made no measurable difference (0.124s vs 0.125s), and caching ImportVar.__hash__ bought ~10% on the micro but needed a typing workaround to keep the cache out of fields(), which is where _update_deterministic_hash would have picked it up. Neither seemed worth the complexity, but say the word if you would rather have them.

Tests

Four regression tests in tests/units/utils/test_imports.py, each confirmed to fail against the current implementation before the fix:

• test_merge_imports_deduplicates
• test_merge_parsed_imports_deduplicates
• test_merge_imports_keeps_first_seen_order
• test_collapse_imports_is_order_preserving

The existing test_merge_imports compares with set(...) on both sides, so it could not have caught either problem.

Full unit suite: 8007 passed, 18 skipped. pre-commit clean.

One unrelated flake to note: tests/units/test_state.py::test_state_manager_lock_expire and its _contend sibling fail intermittently under full-suite load, on a wall-clock redis lock expiry. Across alternating full-suite runs they failed with this change and also without it, and always pass when test_state.py runs on its own.

Notes

• Both merges still return a defaultdict(list), so no caller sees a changed return type.
• Deduplicating is semantically a no-op, since collapse_imports already removed these duplicates before anything was emitted. This only stops them being built and hashed in the first place.
• There is more headroom here. _get_all_imports is still recomputed from scratch at every node rather than cached, which is a separate quadratic. Left for its own PR, since components are mutated during compile and a cache needs an invalidation story.

[greptile-apps]

Greptile Summary

This PR deduplicates import metadata during merging and preserves deterministic first-seen ordering during collapse.

• Adds ordered normalization and incremental per-library deduplication.
• Adds regression coverage for deduplication and deterministic ordering.
• Adds bug-fix and performance news fragments.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains in the eligible review scope.

No blocking failure remains.

Important Files Changed

Filename	Overview
packages/reflex-base/src/reflex_base/utils/imports.py	Adds deterministic normalization, merge-time deduplication, and order-preserving collapse behavior.
tests/units/utils/test_imports.py	Adds regression tests covering deduplication, first-seen ordering, unordered batches, and sort-key uniqueness.
packages/reflex-base/news/7012.bugfix.md	Documents deterministic generated output across unchanged compilations.
packages/reflex-base/news/7012.performance.md	Documents the compilation performance improvement from reducing duplicate import metadata.

_{Reviews (7): Last reviewed commit: "perf: deduplicate imports as they merge,..." | Re-trigger Greptile}

[codspeed-hq]

Merging this PR will degrade performance by 35.95%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 2 regressed benchmarks
✅ 30 untouched benchmarks
⏩ 8 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	test_get_all_imports[_complicated_page]	2.9 ms	4.6 ms	-37.91%
❌	Simulation	test_get_all_imports[_stateful_page]	542.1 µs	820.5 µs	-33.93%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing khaleel/dedupe-import-merge (845b06e) with main (2145622)}

Footnotes

1. 8 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[adhami3310]

Acting on the CodSpeed report rather than acknowledging it: the regression was real, and the pushed version cuts it roughly in half.

test_get_all_imports times _get_all_imports() on its own. Deduplicating is pure cost inside that call, while the payoff is a smaller result collected by whoever consumes it. Measured on the benchmark's own fixtures, the result goes from 277 entries to 19 on _complicated_page (14.6x) and 61 to 10 on _stateful_page. The benchmark pays for that reduction and never uses it, which is why it can only show one side. On the app in the description, 65% of the compile is spent hashing exactly what shrank.

What changed since the first push: a library reached through only one child cannot have gained a duplicate there, so the first batch is taken as-is and hashes nothing, and only from the second batch onward does a library earn a lookup set. Locally on _complicated_page, 300 calls: base 0.088s, dedupe-everywhere 0.255s, now 0.124s. The end-to-end compile also improved, from 52s to 47s.

Two further ideas measured and dropped: inlining the accumulator made no difference (0.124s vs 0.125s), and caching ImportVar.__hash__ bought ~10% but needed a typing workaround to keep the cached value out of fields(), where _update_deterministic_hash would have folded it into content hashes. Happy to add either if you would prefer the micro closer to parity.

🤖 Addressed by Claude Code

[cubic-dev-ai]

All reported issues were addressed across 3 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 4 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 4 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>