Skip to content

fix(ui): load all paginated projects (#1812) - #1819

Closed
rarepops wants to merge 2 commits into
DeusData:mainfrom
rarepops:fix/ui-project-pagination-1812
Closed

fix(ui): load all paginated projects (#1812)#1819
rarepops wants to merge 2 commits into
DeusData:mainfrom
rarepops:fix/ui-project-pagination-1812

Conversation

@rarepops

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes #1812.

  • follows list_projects pagination until has_more is false
  • publishes the complete project list before schema hydration finishes
  • caps schema hydration at four concurrent requests
  • rejects non-advancing pagination metadata instead of looping indefinitely
  • adds regressions for a 51-project result and bounded schema concurrency

Validation

  • cd graph-ui && npm test (41 passed)
  • cd graph-ui && npm run build
  • git diff --check

Checklist

  • Every commit is signed off (git commit -s)
  • Tests pass locally (make -f Makefile.cbm test) (full native suite not run; focused UI suite passed)
  • Lint passes (make -f Makefile.cbm lint-ci) (not run locally)
  • New behavior is covered by a reproduce-first test

@rarepops
rarepops requested a review from DeusData as a code owner August 24, 2026 17:43
@github-actions

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

@rarepops rarepops changed the title fix(ui): load all paginated projects fix(ui): load all paginated projects (#1812) Aug 24, 2026
@rarepops
rarepops force-pushed the fix/ui-project-pagination-1812 branch from bda9282 to f7bccab Compare August 24, 2026 20:00
@rarepops

Copy link
Copy Markdown
Contributor Author

CI note: the fresh run's test-windows-guards failure is outside this PR's two TypeScript files and appears to cascade from guard setup state.

The raw job log shows:

  1. test_non_ascii_path.py first reports SETUP FAIL: ASCII baseline did not index after its Unicode relocation check passed.
  2. The next daemon lifecycle guard then finds an active daemon with a different cache/build and cannot start its isolated daemon.
  3. The daemon stability guard fails for the same active-cache/build mismatch.
  4. Both regular Windows shards pass, as do macOS TSan (the prior unrelated failure), Linux TSan, UI build, lint, DCO, and security checks.

The branch is current with main, mergeable, and changes only:

  • graph-ui/src/hooks/useProjects.ts
  • graph-ui/src/hooks/useProjects.test.tsx

I do not have permission to rerun repository jobs. A maintainer rerun of test-windows-guards should distinguish this guard-state cascade without adding unrelated churn to the PR.

@DeusData DeusData added bug Something isn't working ux/behavior Display bugs, docs, adoption UX priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker. labels Sep 1, 2026
@DeusData

DeusData commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Thank you for the reproduce-first pagination coverage and for bounding schema hydration concurrency while publishing the project list early. The contribution queue is quite full, so the full review may take a little time. We have the PR routed and will come back with grounded feedback as soon as possible.

@DeusData

DeusData commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Approved. The pagination loop is written with the right posture, which is what makes this safe rather than merely correct.

You treat the server's own pagination metadata as untrusted. That is the part I would have asked for:

const pageOffset = Number.isSafeInteger(page.offset) ? page.offset! : offset;
const returned   = Number.isSafeInteger(page.returned) ? page.returned! : pageProjects.length;
const nextOffset = pageOffset + returned;
if (returned <= 0 || nextOffset <= offset) {
  throw new Error("list_projects pagination did not advance");
}

Two distinct ways a while (true) over a paginated API spins forever — a page claiming has_more while returning nothing, and an offset that fails to advance or goes backwards — and both are caught, with non-numeric metadata falling back to locally known values rather than poisoning the arithmetic. A loop that trusted offset and returned would have been shorter and would hang the UI on a server bug instead of surfacing it.

Publishing the complete list before schema hydration finishes is the right ordering: the thing the user asked for appears as soon as it is known, and the enrichment fills in behind it. Capping hydration at four concurrent requests keeps that from turning a 51-project workspace into 51 simultaneous calls.

And the regression is sized to the bug — 51 projects against a 50-item page is exactly the boundary that was broken, rather than a round number that happens to exceed it.

Your CI red is not yours

test / test-windows-guards fails, and it cannot be this change: a graph-ui/src/hooks/*.ts edit does not make a Windows fixture repository index with null counts. The job's own setup could not index a plain ASCII fixture:

SETUP FAIL: ASCII baseline did not index:
  {'nodes': None, 'edges': None, 'definition_nodes': 5}

Everything downstream then skipped on a failed precondition — including test_ui_drive_listing.py's "Is this a UI build?", which is part of the same cluster and fails on non-UI pull requests too, so it is not evidence about your change either.

I have now confirmed this identical cluster, with that identical diagnostic, on five unrelated pull requests spanning several days. It is ours to fix. Please do not chase it.

On the unticked boxes: not running the full native suite or lint-ci for a change confined to graph-ui/ is a reasonable call, and saying so beats ticking them. The focused UI suite and the build are the relevant gates here.

Please rebasemain moved three times yesterday: broken by a duplicate-symbol merge, repaired by #1993, then #1703 landed.

Follow list_projects pagination metadata so the management UI does not stop at 50 projects. Publish the complete list before schema hydration and cap schema lookups at four concurrent requests.

Signed-off-by: Rares Popa <2606875+rarepops@users.noreply.github.com>
@rarepops
rarepops force-pushed the fix/ui-project-pagination-1812 branch from f7bccab to 64ea496 Compare September 2, 2026 16:39
@DeusData

DeusData commented Sep 4, 2026

Copy link
Copy Markdown
Owner

This one needs an honest explanation, because it was approved and was next in the merge order. Since then a direction decision has landed: the graph UI is being reimplemented from the ground up, and the current React app — including the projects hook this fixes — will be replaced wholesale rather than maintained. Merging into a surface that is about to disappear gives the fix a very short life, so we are closing this and #1812 as superseded rather than landing it and deleting it within a release cycle.

What carries forward is the posture rather than the code: treating the server's own pagination metadata as untrusted, bounding the hydration concurrency, and publishing the list early are going in as requirements for the new projects view, with this PR as the reference. Thank you for a reproduce-first fix with more test than change — that is the shape we like — and for the careful CI attribution on the Windows guards leg, which was correct. You are very welcome back.

@DeusData DeusData closed this Sep 4, 2026
DeusData added a commit that referenced this pull request Sep 4, 2026
…istic

lock_registry_absolute_deadline_survives_repeated_wakes has failed on the
test-unix (macos-15-intel) leg of three unrelated pull requests in one
week -- #1342 (08-28), #1811 (09-02) and #1819 (09-03, run 33799475629,
job 100823626965) -- always with the same signature:

  FAIL tests/test_lock_registry.c:1153: ASSERT(tail_queued)
  7648 passed / 1 failed

The registry is not racy. cbm_lock_registry_acquire enqueues the waiter
synchronously under the registry mutex before any wait, so waiter_count
and the attempting count are exact. The defect is in the fixture: it
raced two wall-clock windows against each other, both anchored to a
timestamp the observer thread took before the tail thread had even been
scheduled.

  deadline_start = cbm_now_ms();
  tail.deadline_ms = deadline_start + 200;  /* the tail's acquire deadline */
  queue_deadline   = deadline_start + 100;  /* the observer's budget */

Because both windows start before the tail runs, a loaded runner breaks
the fixture two different ways:

  * the tail is scheduled inside its deadline but after the observer's
    100 ms budget has expired -- the queued state existed and was simply
    no longer being looked at; or
  * the tail is scheduled more than 200 ms late, in which case its
    deadline has already passed when it finally calls acquire, the
    pre-registration deadline check in lock_registry_acquire_internal
    returns BUSY immediately, and the tail never enqueues at all -- the
    asserted state can then never occur, however long the observer waits.

lock_registry is in the parallel wave of run-tests-parallel.sh, not the
serial tail, so on a small CI runner it competes with a full wave of
sanitized suites -- exactly the scheduling delay both paths need.

Widening the observer's budget would only paper over the first path, and
the state it waits for is transient by construction: it exists only
between the tail's enqueue and the tail's own deadline. So the fixture is
rebuilt to observe states that cannot evaporate.

  * The tail anchors its absolute deadline itself, in its own thread,
    immediately before the acquire it bounds. Scheduling delay can no
    longer consume the deadline before the call starts, so the enqueue is
    unconditional, and elapsed is measured from the tail's own anchor --
    it now times the registry instead of timing the scheduler.

  * Enqueue is exposed as a monotonic counter,
    cbm_lock_registry_waiter_enqueue_count_for_test, next to the existing
    test_condition_wait_calls counter it is modelled on. Because the
    count only ever grows, the observer reads it once after the tail has
    returned rather than trying to catch a live queue depth: the polling
    loop, and with it the window, is gone.

  * The fixture's remaining 500 ms and 600 ms budgets become the file's
    LOCK_REGISTRY_TEST_TIMEOUT_MS backstop, and each loop exits on the
    state it waits for instead of on the clock, so the backstop only
    fires when the product is actually broken.

The contract is unchanged: the tail must still return at its absolute
deadline (150 <= elapsed < 350 ms for a 200 ms deadline) despite ~40
unrelated cancel broadcasts, still with BUSY and no lease, with the head
still holding the attempt. The broadcast loop now starts immediately
after the tail is released, so it overlaps the tail's wait at least as
much as it did before.

Verification, all on macOS arm64 with the sanitized runner:

  * The mechanism was reproduced locally by delaying only the tail thread
    after its start gate, with production untouched. A 120 ms delay (the
    state exists, outside the observer's budget) and a 250 ms delay (the
    tail never enqueues) each produced exactly one failure,
    ASSERT(tail_queued) -- the CI signature.
  * After the rebuild the same injections are green at 120, 250, 400 and
    900 ms: an arbitrary scheduling delay no longer decides the verdict.
  * lock_registry 30/30 green plain and 30/30 green under four CPU hogs;
    16/16 inside the saturated 18-job parallel wave.
  * private_file_lock, lock_registry, daemon, project_lock and the
    daemon_* suites: 222 passed, 1 skipped.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working priority/high Needs near-term maintainer attention; high-impact bug, regression, safety issue, or release blocker. ux/behavior Display bugs, docs, adoption UX

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Management UI only displays the first 50 projects and ignores list_projects.has_more

2 participants