feat(ourlogs): Switch needle in haystack to time based

[k-fish]

We were doing a static number of fetches before. Depending on cacheing etc. this can be really fast or slow, but it resulted in you clicking 'continue scanning' a LOT.

This should bring down the number of times you have to click continue. The initial page is 10 seconds to show you the button faster and to correct in case the user accidentally made a huge query. Subsequent presses are longer (20 seconds) since you are assumed to be fine with waiting / have the correct conditions.

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

Bugbot Autofix prepared fixes for all 3 issues found in the latest run.

• ✅ Fixed: Autofetch timer never resets per new query
  • Added useEffect to reset autoFetchStartTime, autoFetchDuration, and autoFetchRequestCount when queryKeyWithInfinite changes.
• ✅ Fixed: Resume duration leaks into later searches
  • The useEffect that triggers on queryKeyWithInfinite changes now resets autoFetchDuration to FLEX_TIME_INITIAL_SEARCH_DURATION_MS for each new query.
• ✅ Fixed: Time-only autofetch can flood API requests
  • Added autoFetchRequestCount state with maxAutoFetchRequests limit (50) to prevent unbounded request loops during fast-response scenarios.

Or push these changes by commenting:

@cursor push 69f20acc5c

Preview (69f20acc5c)

diff --git a/static/app/views/explore/logs/useLogsQuery.tsx b/static/app/views/explore/logs/useLogsQuery.tsx
--- a/static/app/views/explore/logs/useLogsQuery.tsx
+++ b/static/app/views/explore/logs/useLogsQuery.tsx
@@ -694,13 +694,24 @@
   const [autoFetchDuration, setAutoFetchDuration] = useState(
     FLEX_TIME_INITIAL_SEARCH_DURATION_MS
   );
+  const [autoFetchRequestCount, setAutoFetchRequestCount] = useState(0);
+  const maxAutoFetchRequests = 50;
+
+  useEffect(() => {
+    setAutoFetchStartTime(Date.now());
+    setAutoFetchDuration(FLEX_TIME_INITIAL_SEARCH_DURATION_MS);
+    setAutoFetchRequestCount(0);
+  }, [queryKeyWithInfinite]);
+
   const canAutoFetchNextPage =
     !!highFidelity &&
     hasNextPage &&
     nextPageHasData &&
     (lastPageLength === 0 || _data.length < limit);
   const shouldAutoFetchNextPage =
-    canAutoFetchNextPage && Date.now() - autoFetchStartTime < autoFetchDuration;
+    canAutoFetchNextPage &&
+    Date.now() - autoFetchStartTime < autoFetchDuration &&
+    autoFetchRequestCount < maxAutoFetchRequests;
 
   const autoFetchPageCount = useRef(0);
   const prevShouldAutoFetch = useRef(false);
@@ -735,6 +746,7 @@
     }
 
     autoFetchPageCount.current += 1;
+    setAutoFetchRequestCount(prev => prev + 1);
     _fetchNextPage();
   }, [shouldAutoFetchNextPage, isFetchingNextPage, _fetchNextPage, nextPageCursor]);
 
@@ -770,6 +782,7 @@
     resumeAutoFetch: () => {
       setAutoFetchStartTime(Date.now());
       setAutoFetchDuration(FLEX_TIME_RESUME_SEARCH_DURATION_MS);
+      setAutoFetchRequestCount(0);
     },
     dataScanned,
     bytesScanned: totalBytesScanned,

_{This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.}

[cursor]

Autofetch timer never resets per new query

Medium Severity

autoFetchStartTime is initialized once and only refreshed by resumeAutoFetch, so the 10-second autofetch window is measured from hook mount instead of each new search. After the page has been open long enough, new queries in useLogsQuery.tsx stop auto-fetching immediately and require manual Continue Scanning clicks.

Additional Locations (1)

• static/app/views/explore/logs/useLogsQuery.tsx#L769-L773

 

[cursor]

Resume duration leaks into later searches

Low Severity

resumeAutoFetch sets autoFetchDuration to FLEX_TIME_RESUME_SEARCH_DURATION_MS, but that value is never restored for later queries. After one resume, subsequent “initial” searches in useLogsQuery.tsx inherit the longer 20s window, so the shorter first-pass behavior no longer applies.

Additional Locations (1)

• static/app/views/explore/logs/useLogsQuery.tsx#L693-L696

 

[cursor]

Time-only autofetch can flood API requests

Medium Severity

Replacing maxAutoFetches with only a wall-clock check in useLogsQuery.tsx removes the request-count guard. When empty pages return quickly, shouldAutoFetchNextPage stays true and the loop can issue many sequential _fetchNextPage calls within 10–20 seconds, creating significantly higher backend load than before.

Additional Locations (1)

• static/app/views/explore/logs/useLogsQuery.tsx#L736-L738

 

[sentry]

Bug: The auto-fetch timer state (autoFetchStartTime, autoFetchDuration) is not reset when the user changes query parameters, preventing or delaying auto-fetching on subsequent searches.
_{Severity: HIGH}

Suggested Fix

Use a useEffect hook that triggers when the query key (queryKeyWithInfinite) changes. Inside this effect, reset the autoFetchStartTime state to Date.now() and the autoFetchDuration state to the initial value (FLEX_TIME_INITIAL_SEARCH_DURATION_MS). This will ensure every new, distinct query starts with a fresh auto-fetch timer.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: static/app/views/explore/logs/useLogsQuery.tsx#L693

Potential issue: The state variables `autoFetchStartTime` and `autoFetchDuration` are
initialized when the `useInfiniteLogsQuery` hook mounts but are not reset when query
parameters change. If a user waits more than 10 seconds after the page loads and then
modifies their query (e.g., changes a filter or search term), the new query will not
auto-fetch new logs because the condition `Date.now() - autoFetchStartTime <
autoFetchDuration` will be false. Additionally, if a user clicks "Continue Scanning"
(which sets the duration to 20s) and then runs a new query, the initial auto-fetch for
that new query will incorrectly use a 20-second duration instead of the intended 10
seconds.

_{Did we get this right? 👍 / 👎 to inform future reviews.}

[JoshuaKGoldberg]

I'm continuing this in #113219.

[linear-code]

EXP-873 Revisit temporary workaround in PR 111946