perf(complete): bail out of type inference when completion budget expires#9247
perf(complete): bail out of type inference when completion budget expires#9247
Conversation
…ires Completing against heavy libraries (pandas, numpy, torch, aiohttp, ...) on a cold jedi cache was paying the full per-item inference cost for every completion, even when the list is large enough that we already skip docstrings. On molab this made the first `pd.` tab take ~11s; on a local Mac with a cleared cache, ~5.5s. - Add `compute_type=True` kwarg to `_get_completion_option`; when False, skip the `completion.type` access (and the docstring/signature path, which re-triggers the same inference). - Collapse the split oversize/normal branches of `_get_completion_options` into one loop that tracks a time budget. Once the budget expires, remaining completions are returned with name only. Head-to-head on molab, 4 fresh cold libs each (older jedi cache behavior): impl lib n options_ms typed info OLD rich 98 5547 70 0 OLD requests 67 1313 51 48 NEW aiohttp 169 2839 50 0 (budget cap) NEW bs4 78 1984 56 49 Per-completion options cost: 29.6 ms (OLD) -> 19.0 ms (NEW), ~36% across the suite. Worst-case savings for heavy libs bigger: - aiohttp (169 attrs): ~9.6s projected -> 2.8s measured (~71%) - pandas cold (141 attrs): ~11.3s pre-patch -> ~2.0s capped (~82%) Tradeoff: on cold heavy libs, only the first ~50-80 completions get type icons; the rest go out blank and fill in on subsequent requests as jedi's internal cache populates. Warm completions are unchanged.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
This analysis was done with marimo pair #notanad |
There was a problem hiding this comment.
Pull request overview
Improves runtime code completion performance by introducing a time-budgeted fast path that stops triggering expensive Jedi inference (especially on cold caches for heavy libraries), returning lightweight completion entries once the budget expires.
Changes:
- Add a
compute_typeflag to_get_completion_optionto skipcompletion.type(and associated inference) when over budget. - Rework
_get_completion_optionsinto a single loop that tracks a timeout budget and disables type/docstring inference after the budget is exceeded. - Add focused unit tests covering the new inference-skipping behavior and timeout bail-out semantics.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| marimo/_runtime/complete.py | Adds time-budgeted bail-out logic and compute_type to avoid costly Jedi inference once over budget. |
| tests/_runtime/test_complete.py | Adds regression tests ensuring type/docstring/signature inference is skipped appropriately under limits/timeouts. |
| start_time = time.time() | ||
| for completion in completions: | ||
| if not _should_include_name(completion.name, prefix): | ||
| continue | ||
| elapsed_time = time.time() - start_time | ||
| under_time_budget = (time.time() - start_time) < timeout |
There was a problem hiding this comment.
The timeout budget here is measured with time.time(), which is not monotonic and can jump backwards/forwards if the system clock changes (NTP, sleep/wake), potentially causing the code to incorrectly think it's still under budget and continue triggering expensive Jedi inference. Prefer time.monotonic() (or time.perf_counter()) for elapsed-time budgets; you can also compute a deadline = start + timeout once and compare time.monotonic() < deadline inside the loop.
| under_time_budget = (time.time() - start_time) < timeout | ||
| completion_options.append( | ||
| _get_completion_option( | ||
| completion, | ||
| script, | ||
| compute_completion_info=elapsed_time < timeout, | ||
| compute_completion_info=compute_docstrings | ||
| and under_time_budget, | ||
| compute_type=under_time_budget, | ||
| ) | ||
| ) |
There was a problem hiding this comment.
under_time_budget = (time.time() - start_time) < timeout
completion_options.append(
_get_completion_option(
completion,
script,
compute_completion_info=compute_docstrings
and under_time_budget,
compute_type=under_time_budget,
)
)
#
if not under_time_budget:
to_compute.append((completion, script))
...
# kick off bg processing here
if to_compute:
compute_completions(to_compute)There was a problem hiding this comment.
i dont think this backend compute is necessary
|
🚀 Development release published. You may be able to view the changes at https://marimo.app?v=0.23.2-dev52 |
Completing against heavy libraries (
pandas,numpy,torch,aiohttp, ...) on a cold jedi cache was paying the full per-item inference cost for every completion, even when the list is large enough that we already skip docstrings. On molab this made the firstpd.tab take ~11s; on a local Mac with a cleared cache, ~5.5s.compute_type=Truekwarg to_get_completion_option; when False, skip thecompletion.typeaccess (and the docstring/signature path, which re-triggers the same inference)._get_completion_optionsinto one loop that tracks a time budget. Once the budget expires, remaining completions are returned with name only.Per-completion options cost: 29.6 ms (OLD) -> 19.0 ms (NEW), ~36% across the suite. Worst-case savings for heavy libs bigger:
aiohttp(169 attrs): ~9.6s projected -> 2.8s measured (~71%)pandascold (141 attrs): ~11.3s pre-patch -> ~2.0s capped (~82%)Tradeoff: on cold heavy libs, only the first ~50-80 completions get type icons; the rest go out blank and fill in on subsequent requests as jedi's internal cache populates. Warm completions are unchanged.