fix(perf): virtualize explore sidebar list

[ZeRego]

Closes: PROD-2731

Description:

Implemented virtualized rendering for the explore sidebar to improve performance with large numbers of explores. This replaces the previous static rendering approach with a more efficient virtualized list that only renders items visible in the viewport.

Key improvements:

• Added VirtualizedExploreList component using @tanstack/react-virtual for efficient DOM rendering
• Optimized data processing with single-pass categorization and pre-sorting
• Implemented debounced search for better performance during typing
• Refactored ExploreGroup into a more focused GroupHeader component
• Reduced unnecessary re-renders with proper memoization

This change significantly improves performance and responsiveness when navigating projects with many explores or explore groups.

Now:

CleanShot 2026-01-27 at 13.50.20.mp4 (uploaded via Graphite)

Before:

CleanShot 2026-01-27 at 13.51.47.mp4 (uploaded via Graphite)

How to test this locally with 2000+ explores:

import { type SummaryExplore } from '@lightdash/common';

// Mock data generator for 2000 explores to test search performance
export const generateMockExplores = (count: number = 2000): SummaryExplore[] => {
    const explores: SummaryExplore[] = [];
    
    const tableTypes = ['users', 'orders', 'products', 'transactions', 'events', 'sessions', 'analytics', 'marketing', 'sales', 'support'];
    const prefixes = ['customer', 'vendor', 'internal', 'external', 'staging', 'raw', 'processed', 'aggregated'];
    const suffixes = ['daily', 'weekly', 'monthly', 'historical', 'current', 'snapshot', 'summary', 'detail'];
    
    for (let i = 0; i < count; i++) {
        const tableType = tableTypes[i % tableTypes.length];
        const prefix = prefixes[Math.floor(i / tableTypes.length) % prefixes.length];
        const suffix = suffixes[Math.floor(i / (tableTypes.length * prefixes.length)) % suffixes.length];
        
        const name = `${prefix}_${tableType}_${suffix}_${String(i).padStart(4, '0')}`;
        const label = `${prefix.charAt(0).toUpperCase() + prefix.slice(1)} ${tableType.charAt(0).toUpperCase() + tableType.slice(1)} ${suffix.charAt(0).toUpperCase() + suffix.slice(1)} ${i}`;
        
        explores.push({
            name,
            label,
            tags: [`${prefix}`, `${tableType}`, `table_${i}`],
            // First 500 explores have no group, rest are grouped
            groupLabel: i < 500 ? undefined : `${prefix.charAt(0).toUpperCase() + prefix.slice(1)} Tables`,
            description: `Mock table for testing search performance - ${label}`,
            schemaName: `schema_${Math.floor(i / 100)}`,
            databaseName: `database_${Math.floor(i / 500)}`,
            type: undefined,
            warnings: Math.random() > 0.9 ? [{
                type: 'FIELD_ERROR' as const,
                message: 'Sample warning for testing'
            }] : undefined,
        });
    }
    
    return explores;
};

And modify useExplores hook

import { generateMockExplores } from './mockExploresData';

const getExplores = async (projectUuid: string, filtered?: boolean) => {
    // For testing search performance - use mock data with 2000 tables
    // Set this to true to simulate the performance issue
    const USE_MOCK_DATA = true;
    
    if (USE_MOCK_DATA) {
        // Simulate API delay
        await new Promise(resolve => setTimeout(resolve, 500));
        return generateMockExplores(2000);
    }
    
    return lightdashApi<ApiExploresResults>({
        url: `/projects/${projectUuid}/explores?filtered=${
            filtered ? 'true' : 'false'
        }`,
        method: 'GET',
        body: undefined,
    });
};

[ZeRego]

• fix(perf): virtualize explore sidebar list #19741 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[linear]

PROD-2731 Slow search on tables page

[github-actions]

Your preview environment pr-19741 has been deployed.

Preview environment endpoints are available at:

• https://lightdash-pr-19741.lightdash.okteto.dev
• https://lightdash-preview-pr-19741.lightdash.okteto.dev

[github-actions]

Preview Environment

🌐 URL: https://lightdash-preview-pr-19741.lightdash.okteto.dev

📋 Logs: View in GCP Console

🔧 SSH: ./scripts/okteto-ssh.sh 19741

[rephus]

Tested locally, works great !

[rephus]

Linked issue https://linear.app/lightdash/issue/CENG-2731 doesn't seem to be valid 🤔

[lightdash-bot]

🎉 This PR is included in version 0.2380.1 🎉

The release is available on:

• 0.2380.1
• GitHub release

Your semantic-release bot 📦🚀