From b247ae2db1167b65da843a59fbaf1b29d404de41 Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Fri, 1 May 2026 12:25:10 -0700 Subject: [PATCH 1/4] Phase 5: rename Interactive Explorer to /explorer.html with redirects MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the unified-explorer migration (issue #156). Phases 1-4 (#157-#160) built the unified UI on tutorials/progressive_globe.qmd; this PR promotes it to the canonical site-root URL and retires the old Search Explorer page. Rename + asset-path fix - Move tutorials/progressive_globe.qmd → explorer.qmd at site root. - Adjust source-palette import from `../assets/js/source-palette.js` to `assets/js/source-palette.js` so it resolves on both isamples.org and on rdhyee.github.io/isamplesorg.github.io PR previews. URL param: search, not q - The explorer's URL state now uses `?search=` instead of `?q=`. Quarto's site-wide search reserves `?q=` for its highlight feature and strips the param via history.replaceState before any of our cells run (see docs/site_libs/quarto-search/quarto-search.js). `?search=` is unused by Quarto and survives intact. Redirect stubs at the old URLs - tutorials/progressive_globe.html and tutorials/isamples_explorer.html become preview-safe redirect stubs: new URL('../explorer.html' + search + hash, href) - They forward whatever query string the browser presents. Note: legacy `?q=basalt` URLs lose the search term because Quarto strips `?q=` before our stub script runs (the stub is itself a Quarto-rendered page, so its loads quarto-search.js). Non-q params (sources, material, etc.) and the hash fragment all survive — the only affected URLs are Phase 3 dev test links that were never published. _quarto.yml + internal links - Navbar Interactive Explorer href → explorer.qmd. Search Explorer removed from both the How-to-Use menu and the sidebar. - Update internal links to /explorer.html in index.qmd, how-to-use.qmd, tutorials/index.qmd, about.qmd, data.qmd, design/index.qmd, index_alt.qmd, query-spec.qmd, tutorials/narrow_vs_wide_performance.qmd, and the existing parquet_cesium_isamples_wide redirect stub. Tests - tests/test_explorer.py → tests/test_globe.py targeting /explorer.html. - Selectors updated for the unified DOM-based UI: #sourceFilter, #materialFilter, #contextFilter, #objectTypeFilter, #globeViewBtn / #tableViewBtn (no List view), #maxSamples number input. - Unskip the cross-filter facet tests deferred in #155 — native HTML checkboxes respond to programmatic .click() unlike the old Explorer's OJS Inputs.checkbox. - Add redirect-preserves-params tests for both old URLs (using the current ?search= param, which survives Quarto's q-stripping). - test_navigation.py + test_tutorials_landing.py drop Search Explorer assertions and retarget the globe-loads test to /explorer.html. Co-Authored-By: Claude Opus 4.7 (1M context) --- _quarto.yml | 6 +- about.qmd | 2 +- data.qmd | 2 +- design/index.qmd | 2 +- explorer.qmd | 2041 +++++++++++++++++++ how-to-use.qmd | 3 +- index.qmd | 4 +- index_alt.qmd | 2 +- query-spec.qmd | 2 +- tests/test_explorer.py | 138 -- tests/test_globe.py | 137 ++ tests/test_navigation.py | 5 - tests/test_tutorials_landing.py | 10 +- tutorials/index.qmd | 3 +- tutorials/isamples_explorer.qmd | 1890 +----------------- tutorials/narrow_vs_wide_performance.qmd | 2 +- tutorials/parquet_cesium_isamples_wide.qmd | 8 +- tutorials/progressive_globe.qmd | 2046 +------------------- 18 files changed, 2225 insertions(+), 4078 deletions(-) create mode 100644 explorer.qmd delete mode 100644 tests/test_explorer.py create mode 100644 tests/test_globe.py diff --git a/_quarto.yml b/_quarto.yml index cf3570b..685a07f 100644 --- a/_quarto.yml +++ b/_quarto.yml @@ -12,14 +12,12 @@ website: left: - href: index.qmd text: Home - - href: tutorials/progressive_globe.qmd + - href: explorer.qmd text: Interactive Explorer - text: How to Use menu: - text: Overview href: how-to-use.qmd - - text: Search Explorer - href: tutorials/isamples_explorer.qmd - text: Deep-Dive Analysis href: tutorials/zenodo_isamples_analysis.qmd - text: About @@ -67,8 +65,6 @@ website: contents: - href: how-to-use.qmd text: Overview - - text: "Search Explorer" - href: tutorials/isamples_explorer.qmd - text: "Deep-Dive Analysis" href: tutorials/zenodo_isamples_analysis.qmd - id: about diff --git a/about.qmd b/about.qmd index fb46b0c..dfc33a8 100644 --- a/about.qmd +++ b/about.qmd @@ -20,7 +20,7 @@ The iSamples project will: * Encourage a high-level metadata standard for natural history samples (across biosciences, geosciences, and archaeology), while supporting community-developed metadata standards in specialist domains. * Extend existing capabilities, enhance consistency, and expand their reach to serve science and society much more broadly through integration with established discipline-specific infrastructure at SESAR (geoscience), CyVerse (bioscience), Open Context (archaeology), and the Smithsonian Institution. -**Current data access**: The project now uses **geoparquet files + DuckDB-WASM** for efficient, browser-based data access and analysis. See the [Interactive Explorer](/tutorials/progressive_globe.html) for a live demo. +**Current data access**: The project now uses **geoparquet files + DuckDB-WASM** for efficient, browser-based data access and analysis. See the [Interactive Explorer](/explorer.html) for a live demo. ![iSamples Architecture](assets/iSamplesArchitecture.png) ::: diff --git a/data.qmd b/data.qmd index d73a7be..be0c075 100644 --- a/data.qmd +++ b/data.qmd @@ -162,7 +162,7 @@ Approximate breakpoints: Reference implementations: -- [Interactive Explorer (web)](tutorials/progressive_globe.qmd) — Observable JS + DuckDB-WASM + Cesium +- [Interactive Explorer (web)](explorer.qmd) — Observable JS + DuckDB-WASM + Cesium - [iSamples Explorer (Python)](https://github.com/isamplesorg/examples/blob/main/examples/basic/isamples_explorer.ipynb) — Jupyter widgets + DuckDB + lonboard ## 5. Full catalog + companion docs diff --git a/design/index.qmd b/design/index.qmd index c346944..50ab44c 100644 --- a/design/index.qmd +++ b/design/index.qmd @@ -22,7 +22,7 @@ iSamples has defined: - A **metadata profile** applicable to all physical samples — see the [Metadata Model](https://isamplesorg.github.io/metadata/) - A set of **controlled vocabularies** for interoperable sample description — see [Vocabularies](../models/index.qmd) - A **programmatic interface** for sample discovery and access -- A **browser-based data analysis** approach using geoparquet + DuckDB-WASM — see the [Interactive Explorer](/tutorials/progressive_globe.html) +- A **browser-based data analysis** approach using geoparquet + DuckDB-WASM — see the [Interactive Explorer](/explorer.html) ## Further Reading {.unnumbered} diff --git a/explorer.qmd b/explorer.qmd new file mode 100644 index 0000000..00838aa --- /dev/null +++ b/explorer.qmd @@ -0,0 +1,2041 @@ +--- +title: "Interactive Explorer" +subtitle: "Search and explore 6.7 million material samples" +categories: [parquet, spatial, h3, performance, isamples] +sidebar: false +# No TOC: this page is an app, not an article. The right-hand TOC sidebar +# (#quarto-margin-sidebar) was overlapping .side-panel and silently +# intercepting clicks on the Source filter checkboxes — see issue #127. +toc: false +format: + html: + include-in-header: + text: | + + + + +--- + + + + + +::: {.callout-note collapse="true"} +## How It Works + +1. **Instant** (<1s): Pre-aggregated H3 res4 summary (580 KB) → 38K colored circles +2. **Zoom in**: Automatically switches to res6 (112K) then res8 (176K) clusters +3. **Zoom deeper** (<120 km): Individual sample points from 60 MB lite parquet +4. **Click**: Cluster info or individual sample card with full metadata +5. **Search**: Find samples by name — results fly to the location on the globe + +Circle size = log(sample count). Color = dominant data source. +::: + + +
+ + +
+
+ +
+
+ + +
+
+ + +
+
+ +
+
+
+
+
+
Loading...Resolution
+
0Clusters Loaded
+
0Samples Loaded
+
-Load Time
+
+
+
+ + + + +
+
+
+
+Material ▾ +
+ +
+
+
+Sampled Feature ▾ +
+ +
+
+
+Specimen Type ▾ +
+ +
+ +
+ + +
+
+Loading H3 global overview... +
+
+
+
Click a cluster or sample on the globe
+
+
+
+
+ +
+
Table view loads samples matching the current filters.
+
+
+ + + +
+
+ +```{ojs} +//| output: false +Cesium.Ion.defaultAccessToken = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiIwNzk3NjkyMy1iNGI1LTRkN2UtODRiMy04OTYwYWE0N2M3ZTkiLCJpZCI6Njk1MTcsImlhdCI6MTYzMzU0MTQ3N30.e70dpNzOCDRLDGxRguQCC-tRzGzA-23Xgno5lNgCeB4'; +``` + +```{ojs} +//| echo: false +//| output: false + +// === Constants === +R2_BASE = "https://data.isamples.org" +h3_res4_url = `${R2_BASE}/isamples_202601_h3_summary_res4.parquet` +h3_res6_url = `${R2_BASE}/isamples_202601_h3_summary_res6.parquet` +h3_res8_url = `${R2_BASE}/isamples_202601_h3_summary_res8.parquet` +lite_url = `${R2_BASE}/isamples_202601_samples_map_lite.parquet` +// Stable alias that 302-redirects to the current enriched wide parquet +// (isamples_YYYYMM_wide.parquet). Gets OpenContext thumbnails populated. +wide_url = `${R2_BASE}/current/wide.parquet` +// v2 carries object_type alongside material and context (URI-string columns). +facets_url = `${R2_BASE}/isamples_202601_sample_facets_v2.parquet` +facet_summaries_url = `${R2_BASE}/isamples_202601_facet_summaries.parquet` +// Pre-aggregated single-filter cache for fast cross-filtered facet counts. +cross_filter_url = `${R2_BASE}/isamples_202601_facet_cross_filter.parquet` +// SKOS prefLabels for Material / Sampled Feature / Specimen Type URIs. +// ~60 KB lookup; falls back to URI tail if a URI isn't covered. +vocab_labels_url = `${R2_BASE}/vocab_labels.parquet` + +// Canonical palette — see issue #113. Path-relative so this works under +// both isamples.org (custom domain at root) and project-pages fork +// previews (rdhyee.github.io/isamplesorg.github.io/...). +_palette = await import(new URL('assets/js/source-palette.js', document.baseURI).href) +SOURCE_COLORS = _palette.SOURCE_COLORS +SOURCE_NAMES = _palette.SOURCE_NAMES + +// === Source URL: resolve pid to original repository === +function sourceUrl(pid) { + if (!pid) return null; + // All sources resolve via n2t.net: + // ARK pids (OpenContext, GEOME, Smithsonian) → n2t.net/ark:/... + // IGSN pids (SESAR) → n2t.net/IGSN:... + return `https://n2t.net/${pid}`; +} + +// === Source Filter: get active sources and build SQL clause === +function getActiveSources() { + const checks = document.querySelectorAll('#sourceFilter input[type="checkbox"]'); + return Array.from(checks).filter(c => c.checked).map(c => c.value); +} + +function sourceFilterSQL(col) { + const active = getActiveSources(); + if (active.length === 0) return ' AND 1=0'; // nothing checked = show nothing + if (active.length === 4) return ''; // all checked = no filter + const list = active.map(s => `'${s}'`).join(','); + return ` AND ${col} IN (${list})`; +} + +SOURCE_VALUES = ['SESAR', 'OPENCONTEXT', 'GEOME', 'SMITHSONIAN'] +DEFAULT_POINT_BUDGET = 5000 + +function csvParamValues(params, key) { + if (!params.has(key)) return null; + const raw = params.get(key) || ''; + if (raw.trim() === '') return []; + return raw.split(',').map(s => s.trim()).filter(Boolean); +} + +function updateSourceLegendState() { + document.querySelectorAll('#sourceFilter .legend-item').forEach(li => { + const cb = li.querySelector('input'); + li.classList.toggle('disabled', !cb.checked); + }); +} + +function applyQueryToSourceFilter() { + const params = new URLSearchParams(location.search); + const initialSources = csvParamValues(params, 'sources'); + if (initialSources == null) return; + const allowed = new Set(SOURCE_VALUES); + const selected = new Set(initialSources.filter(s => allowed.has(s))); + document.querySelectorAll('#sourceFilter input[type="checkbox"]').forEach(cb => { + cb.checked = selected.has(cb.value); + }); + updateSourceLegendState(); +} + +// URL param is `search`, not `q` — Quarto's site-wide search hijacks `?q=` +// (highlights matches and strips the param via replaceState). +// See docs/site_libs/quarto-search/quarto-search.js. +function applyQueryToSearch() { + const input = document.getElementById('sampleSearch'); + if (!input) return; + const params = new URLSearchParams(location.search); + const q = params.get('search'); + if (q != null) input.value = q; +} + +function setCheckedValues(containerId, values) { + if (values == null) return; + const selected = new Set(values); + document.querySelectorAll(`#${containerId} input[type="checkbox"]`).forEach(cb => { + cb.checked = selected.has(cb.value); + }); +} + +function applyQueryToFacetFilters() { + const params = new URLSearchParams(location.search); + setCheckedValues('materialFilterBody', csvParamValues(params, 'material')); + setCheckedValues('contextFilterBody', csvParamValues(params, 'context')); + setCheckedValues('objectTypeFilterBody', csvParamValues(params, 'object_type')); +} + +function getMaxSamplesLimit() { + const params = new URLSearchParams(location.search); + return Math.round(parseNum(params.get('maxSamples'), DEFAULT_POINT_BUDGET, 1, 1000000)); +} + +function writeQueryState() { + const params = new URLSearchParams(location.search); + const searchInput = document.getElementById('sampleSearch'); + const q = searchInput ? searchInput.value.trim() : ''; + if (q) params.set('search', q); + else params.delete('search'); + + const activeSources = getActiveSources(); + if (activeSources.length === SOURCE_VALUES.length) params.delete('sources'); + else params.set('sources', activeSources.join(',')); + + [ + ['material', 'materialFilterBody'], + ['context', 'contextFilterBody'], + ['object_type', 'objectTypeFilterBody'], + ].forEach(([key, containerId]) => { + const values = getCheckedValues(containerId); + if (values.length > 0) params.set(key, values.join(',')); + else params.delete(key); + }); + + const maxSamples = getMaxSamplesLimit(); + if (maxSamples !== DEFAULT_POINT_BUDGET) params.set('maxSamples', String(maxSamples)); + else params.delete('maxSamples'); + + if (typeof document !== 'undefined' && document.body && document.body.classList.contains('table-view-active')) { + params.set('view', 'table'); + } else { + params.delete('view'); + } + + const qs = params.toString(); + const url = `${location.pathname}${qs ? `?${qs}` : ''}${location.hash}`; + if (url !== `${location.pathname}${location.search}${location.hash}`) { + history.replaceState(null, '', url); + } +} + +function searchTerms(value) { + return String(value || '').trim().split(/\s+/).filter(Boolean); +} + +function escapeIlikePattern(value) { + return escSql(value).replace(/[\\%_]/g, "\\$&"); +} + +function textSearchWhere(terms, columns) { + return terms.map(raw => { + const term = escapeIlikePattern(raw); + const checks = columns.map(col => `${col} ILIKE '%${term}%' ESCAPE '\\'`); + return `(${checks.join(' OR ')})`; + }).join(' AND '); +} + +function textSearchScore(terms, weightedColumns) { + if (!terms.length) return '0'; + return terms.map(raw => { + const term = escapeIlikePattern(raw); + return weightedColumns.map(({ col, weight }) => + `CASE WHEN ${col} ILIKE '%${term}%' ESCAPE '\\' THEN ${weight} ELSE 0 END` + ).join(' + '); + }).map(score => `(${score})`).join(' + '); +} + +// === Material / Sampled Feature / Specimen Type Filters === +// Checkbox semantics: start UNCHECKED (no filter; show everything). User +// checks items to *include only those*. Empty = no filter. Matches the +// explorer's URI-valued facet UX — with hundreds of materials, defaulting +// to "all checked" would be unusable, and "empty = no filter" is the +// natural reading. See issue #155. +function getCheckedValues(containerId) { + const checks = document.querySelectorAll(`#${containerId} input[type="checkbox"]`); + return Array.from(checks).filter(c => c.checked).map(c => c.value); +} + +function hasFacetFilters() { + return getCheckedValues('materialFilterBody').length > 0 + || getCheckedValues('contextFilterBody').length > 0 + || getCheckedValues('objectTypeFilterBody').length > 0; +} + +function escSql(value) { + return String(value).replace(/'/g, "''"); +} + +// Returns a portable predicate fragment (no outer-table alias dependency) +// that callers append to a WHERE: ` AND ${facetFilterSQL()}`. Uses a +// `pid IN (SELECT pid FROM facets WHERE ...)` subquery so it works +// without a JOIN and avoids duplicate rows from multi-valued facets +// (a sample with two materials would appear twice via JOIN). Required +// for Phase 4's table mode and any non-JOIN caller. See issue #156. +function facetFilterSQL() { + const mat = getCheckedValues('materialFilterBody'); + const ctx = getCheckedValues('contextFilterBody'); + const ot = getCheckedValues('objectTypeFilterBody'); + + const conds = []; + if (mat.length > 0) { + const list = mat.map(s => `'${escSql(s)}'`).join(','); + conds.push(`material IN (${list})`); + } + if (ctx.length > 0) { + const list = ctx.map(s => `'${escSql(s)}'`).join(','); + conds.push(`context IN (${list})`); + } + if (ot.length > 0) { + const list = ot.map(s => `'${escSql(s)}'`).join(','); + conds.push(`object_type IN (${list})`); + } + if (conds.length === 0) return ''; + return ` AND pid IN (SELECT DISTINCT pid FROM read_parquet('${facets_url}') WHERE ${conds.join(' AND ')})`; +} + +// === Cross-filter facet count UI helpers === +function applyFacetCounts(facetKey, countsMap) { + const baseline = (viewer && viewer._baselineCounts) ? viewer._baselineCounts[facetKey] : null; + document.querySelectorAll(`.facet-count[data-facet="${facetKey}"]`).forEach(el => { + const value = el.getAttribute('data-value'); + let count; + if (countsMap) { + count = countsMap.has(value) ? countsMap.get(value) : 0; + } else { + count = baseline ? (baseline.get(value) ?? 0) : 0; + } + el.textContent = `(${Number(count).toLocaleString()})`; + el.classList.remove('recomputing'); + + const row = document.querySelector(`.facet-row[data-facet="${facetKey}"][data-value="${CSS.escape(value)}"]`); + if (row) row.classList.toggle('zero', count === 0); + }); +} + +function markFacetCountsRecomputing() { + document.querySelectorAll('.facet-count').forEach(el => el.classList.add('recomputing')); +} + +// === URL State: encode/decode globe state in hash fragment === +function parseNum(val, def, min, max) { + if (val == null) return def; + const n = parseFloat(val); + if (!Number.isFinite(n)) return def; + if (min != null && n < min) return min; + if (max != null && n > max) return max; + return n; +} + +function readHash() { + const params = new URLSearchParams(location.hash.slice(1)); + return { + v: parseInt(params.get('v')) || 0, + lat: parseNum(params.get('lat'), null, -90, 90), + lng: parseNum(params.get('lng'), null, -180, 180), + alt: parseNum(params.get('alt'), null, 100, 40000000), + heading: parseNum(params.get('heading'), 0, 0, 360), + pitch: parseNum(params.get('pitch'), -90, -90, 0), + mode: params.get('mode') || null, + pid: params.get('pid') || null, + }; +} + +function buildHash(v) { + const cam = v.camera; + const carto = cam.positionCartographic; + const params = new URLSearchParams(); + params.set('v', '1'); + params.set('lat', Cesium.Math.toDegrees(carto.latitude).toFixed(4)); + params.set('lng', Cesium.Math.toDegrees(carto.longitude).toFixed(4)); + params.set('alt', Math.round(carto.height).toString()); + const heading = Cesium.Math.toDegrees(cam.heading) % 360; + const pitch = Cesium.Math.toDegrees(cam.pitch); + if (Math.abs(heading) > 1) params.set('heading', heading.toFixed(1)); + if (Math.abs(pitch + 90) > 1) params.set('pitch', pitch.toFixed(1)); + const gs = v._globeState; + if (gs.mode === 'point') params.set('mode', 'point'); + if (gs.selectedPid) params.set('pid', gs.selectedPid); + return '#' + params.toString(); +} + +// === Helpers: update DOM imperatively (no OJS reactivity) === +function updateStats(phase, points, samples, time, pointsLabel, samplesLabel) { + const s = (id, v) => { const e = document.getElementById(id); if (e) e.textContent = v; }; + s('sPhase', phase); + s('sPoints', typeof points === 'string' ? points : points.toLocaleString()); + s('sSamples', typeof samples === 'string' ? samples : samples.toLocaleString()); + if (time != null) s('sTime', time); + if (pointsLabel) s('sPointsLbl', pointsLabel); + if (samplesLabel) s('sSamplesLbl', samplesLabel); +} + +function updatePhaseMsg(text, type) { + const m = document.getElementById('phaseMsg'); + if (!m) return; + m.textContent = text; + if (type === 'loading') { m.style.background = '#e3f2fd'; m.style.color = '#1565c0'; } + else { m.style.background = '#e8f5e9'; m.style.color = '#2e7d32'; } +} + +function updateClusterCard(info) { + const el = document.getElementById('clusterSection'); + if (!el) return; + if (!info) { + el.innerHTML = '
Click a cluster or sample on the globe
'; + return; + } + const color = SOURCE_COLORS[info.source] || '#666'; + const name = SOURCE_NAMES[info.source] || info.source; + el.innerHTML = `

Selected Cluster

+
+
+ ${name} + H3 res${info.resolution} +
+
+ ${info.count.toLocaleString()} samples +
+
+ ${info.lat.toFixed(4)}, ${info.lng.toFixed(4)} +
+
`; +} + +function updateSampleCard(sample) { + const el = document.getElementById('clusterSection'); + if (!el) return; + const color = SOURCE_COLORS[sample.source] || '#666'; + const name = SOURCE_NAMES[sample.source] || sample.source; + const placeParts = sample.place_name; + const placeStr = Array.isArray(placeParts) && placeParts.length > 0 + ? placeParts.filter(Boolean).join(' › ') + : ''; + const srcUrl = sourceUrl(sample.pid); + el.innerHTML = `

Sample

+
+
+ ${name} +
+
+ ${sample.label || sample.pid || 'Unnamed'} +
+
+ ${sample.lat.toFixed(5)}, ${sample.lng.toFixed(5)} +
+ ${placeStr ? `
${placeStr}
` : ''} + ${sample.result_time ? `
Date: ${sample.result_time}
` : ''} + ${srcUrl ? `
View at ${name} →
` : ''} +
Loading full details...
+
`; +} + +function updateSampleDetail(detail) { + const el = document.getElementById('sampleDetail'); + if (!el) return; + if (!detail) { + el.innerHTML = 'Detail query failed'; + return; + } + const desc = detail.description + ? (detail.description.length > 300 ? detail.description.slice(0, 300) + '...' : detail.description) + : ''; + el.innerHTML = `${desc ? `
${desc}
` : ''}`; +} + +function updateSamples(samples) { + const el = document.getElementById('samplesSection'); + if (!el) return; + if (!samples || samples.length === 0) { + el.innerHTML = ''; + return; + } + let h = `

Nearby Samples (${samples.length})

`; + for (const s of samples) { + const color = SOURCE_COLORS[s.source] || '#666'; + const name = SOURCE_NAMES[s.source] || s.source; + const placeParts = s.place_name; + const desc = Array.isArray(placeParts) && placeParts.length > 0 + ? placeParts.filter(Boolean).join(' › ') + : ''; + const sUrl = sourceUrl(s.pid); + h += `
+
+ ${sUrl ? `${s.label || s.pid}` : `${s.label || s.pid}`} + ${name} +
+ ${desc ? `
${desc}
` : ''} +
`; + } + el.innerHTML = h; +} + +// === Binary Globe/Table view === +TABLE_PAGE_SIZE = 100 +TABLE_DEFAULT_MAX = 25000 +TABLE_MIN_MAX = 1000 +TABLE_MAX_MAX = 100000 + +function escapeHtml(value) { + return String(value ?? '') + .replace(/&/g, '&') + .replace(//g, '>') + .replace(/"/g, '"') + .replace(/'/g, '''); +} + +function clampTableMaxSamples(value) { + const n = parseInt(value, 10); + if (!Number.isFinite(n)) return TABLE_DEFAULT_MAX; + return Math.min(TABLE_MAX_MAX, Math.max(TABLE_MIN_MAX, n)); +} + +function getTableMaxSamples() { + const el = document.getElementById('maxSamples'); + const value = clampTableMaxSamples(el ? el.value : TABLE_DEFAULT_MAX); + if (el && String(value) !== String(el.value)) el.value = value; + return value; +} + +function isTableViewActive() { + return document.body.classList.contains('table-view-active'); +} +``` + +```{ojs} +//| echo: false +//| output: false + +// === DuckDB === +db = { + performance.mark('duckdb-init-start'); + const instance = await DuckDBClient.of(); + performance.mark('duckdb-init-end'); + performance.measure('duckdb_init', 'duckdb-init-start', 'duckdb-init-end'); + return instance; +} +``` + +```{ojs} +//| echo: false +//| output: false + +// === Cesium Viewer (created once, never re-created) === +viewer = { + performance.mark('viewer-init-start'); + const v = new Cesium.Viewer("cesiumContainer", { + timeline: false, + animation: false, + baseLayerPicker: false, + fullscreenElement: "cesiumContainer", + terrain: Cesium.Terrain.fromWorldTerrain() + }); + + // URL deep-link state (must be set before globalRect/once block reads it) + v._globeState = { mode: 'cluster', selectedPid: null }; + v._initialHash = readHash(); + v._suppressHashWrite = true; // cleared after zoomWatcher initializes + v._suppressTimer = null; + + const globalRect = Cesium.Rectangle.fromDegrees(-180, -60, 180, 80); + Cesium.Camera.DEFAULT_VIEW_RECTANGLE = globalRect; + Cesium.Camera.DEFAULT_VIEW_FACTOR = 0.5; + const ih = v._initialHash; + const once = () => { + if (ih.lat != null && ih.lng != null) { + v.camera.setView({ + destination: Cesium.Cartesian3.fromDegrees(ih.lng, ih.lat, ih.alt || 20000000), + orientation: { + heading: Cesium.Math.toRadians(ih.heading), + pitch: Cesium.Math.toRadians(ih.pitch) + } + }); + } else { + v.camera.setView({ destination: globalRect }); + } + v.scene.postRender.removeEventListener(once); + }; + v.scene.postRender.addEventListener(once); + + // Two separate point collections: clusters and individual samples + v.h3Points = new Cesium.PointPrimitiveCollection(); + v.scene.primitives.add(v.h3Points); + + v.samplePoints = new Cesium.PointPrimitiveCollection(); + v.scene.primitives.add(v.samplePoints); + v.samplePoints.show = false; // hidden until point mode + + // Hover tooltip — works for both clusters and samples + v.pointLabel = v.entities.add({ + label: { + show: false, showBackground: true, font: "13px monospace", + horizontalOrigin: Cesium.HorizontalOrigin.LEFT, + verticalOrigin: Cesium.VerticalOrigin.BOTTOM, + pixelOffset: new Cesium.Cartesian2(15, 0), + disableDepthTestDistance: Number.POSITIVE_INFINITY, text: "", + } + }); + + new Cesium.ScreenSpaceEventHandler(v.scene.canvas).setInputAction((movement) => { + const picked = v.scene.pick(movement.endPosition); + if (Cesium.defined(picked) && picked.primitive && picked.id) { + v.pointLabel.position = picked.primitive.position; + v.pointLabel.label.show = true; + const meta = picked.id; + if (typeof meta === 'object' && meta.type === 'sample') { + v.pointLabel.label.text = `${meta.label || meta.pid}`; + } else if (typeof meta === 'object' && meta.count) { + v.pointLabel.label.text = `${meta.source}: ${meta.count.toLocaleString()} samples`; + } else { + v.pointLabel.label.text = String(meta); + } + } else { + v.pointLabel.label.show = false; + } + }, Cesium.ScreenSpaceEventType.MOUSE_MOVE); + + // Click handler — routes to cluster card or sample card + new Cesium.ScreenSpaceEventHandler(v.scene.canvas).setInputAction(async (e) => { + const picked = v.scene.pick(e.position); + if (!Cesium.defined(picked) || !picked.primitive || !picked.id) return; + const meta = picked.id; + + if (typeof meta === 'object' && meta.type === 'sample') { + // --- Individual sample click --- + updateSampleCard(meta); + v._globeState.selectedPid = meta.pid; + history.pushState(null, '', buildHash(v)); + // Clear nearby list + const sampEl = document.getElementById('samplesSection'); + if (sampEl) sampEl.innerHTML = ''; + + // Stage 2: lazy-load full description from wide parquet + try { + const detail = await db.query(` + SELECT description + FROM read_parquet('${wide_url}') + WHERE pid = '${meta.pid.replace(/'/g, "''")}' + LIMIT 1 + `); + if (detail && detail.length > 0) { + updateSampleDetail(detail[0]); + } else { + updateSampleDetail({ description: '' }); + } + } catch(err) { + console.error("Detail query failed:", err); + updateSampleDetail(null); + } + + } else if (typeof meta === 'object' && meta.count) { + // --- Cluster click --- + updateClusterCard(meta); + v._globeState.selectedPid = null; + history.pushState(null, '', buildHash(v)); + + const sampEl = document.getElementById('samplesSection'); + if (sampEl) sampEl.innerHTML = '
Loading nearby samples...
'; + + const delta = meta.resolution === 4 ? 2.0 : meta.resolution === 6 ? 0.5 : 0.1; + try { + // facetFilterSQL() returns a portable `pid IN (...)` predicate, + // so the same query works whether or not facet filters are active. + const nearbyQuery = ` + SELECT pid, label, source, latitude, longitude, place_name + FROM read_parquet('${lite_url}') + WHERE latitude BETWEEN ${meta.lat - delta} AND ${meta.lat + delta} + AND longitude BETWEEN ${meta.lng - delta} AND ${meta.lng + delta} + ${sourceFilterSQL('source')} + ${facetFilterSQL()} + LIMIT 30 + `; + const samples = await db.query(nearbyQuery); + updateSamples(samples); + } catch(err) { + console.error("Sample query failed:", err); + if (sampEl) sampEl.innerHTML = '
Query failed — try again.
'; + } + } + }, Cesium.ScreenSpaceEventType.LEFT_CLICK); + + // Timing: viewer ready (mount complete, pre-first-render) + performance.mark('viewer-init-end'); + performance.measure('viewer_init', 'viewer-init-start', 'viewer-init-end'); + + // Timing: first Cesium frame painted (globe visible, may be pre-cluster) + const firstGlobeFrame = () => { + performance.mark('first-globe-frame'); + v.scene.postRender.removeEventListener(firstGlobeFrame); + }; + v.scene.postRender.addEventListener(firstGlobeFrame); + + return v; +} +``` + +```{ojs} +//| echo: false +//| output: false + +// === PHASE 1: Load H3 res4 globally (instant) === +phase1 = { + performance.mark('p1-start'); + applyQueryToSearch(); + applyQueryToSourceFilter(); + + const data = await db.query(` + SELECT h3_cell, sample_count, center_lat, center_lng, + dominant_source, source_count + FROM read_parquet('${h3_res4_url}') + WHERE 1=1${sourceFilterSQL('dominant_source')} + `); + + const scalar = new Cesium.NearFarScalar(1.5e2, 1.5, 8.0e6, 0.5); + let totalSamples = 0; + + for (const row of data) { + const count = row.sample_count; + totalSamples += count; + const size = Math.min(3 + Math.log10(count) * 4, 20); + viewer.h3Points.add({ + id: { count, source: row.dominant_source, lat: row.center_lat, lng: row.center_lng, resolution: 4 }, + position: Cesium.Cartesian3.fromDegrees(row.center_lng, row.center_lat, 0), + pixelSize: size, + color: Cesium.Color.fromCssColorString(SOURCE_COLORS[row.dominant_source] || '#666').withAlpha(0.8), + scaleByDistance: scalar, + }); + } + + // Cache cluster data for viewport counting + viewer._clusterData = Array.from(data); + viewer._clusterTotal = { clusters: data.length, samples: totalSamples }; + + performance.mark('p1-end'); + performance.measure('p1', 'p1-start', 'p1-end'); + const elapsed = performance.getEntriesByName('p1').pop().duration; + + updateStats('H3 Res4', data.length, totalSamples, `${(elapsed/1000).toFixed(1)}s`, 'Clusters Loaded', 'Samples Loaded'); + updatePhaseMsg(`${data.length.toLocaleString()} clusters, ${totalSamples.toLocaleString()} samples. Zoom in for finer detail.`, 'done'); + console.log(`Phase 1: ${data.length} clusters in ${elapsed.toFixed(0)}ms`); + + return { count: data.length, samples: totalSamples }; +} +``` + +```{ojs} +//| echo: false +//| output: false + +// === Load facet summaries + SKOS prefLabels, populate filter checkboxes === +// +// Checkbox value = full URI (matches the URI strings stored in +// sample_facets_v2.parquet's material / context / object_type columns). +// Display label = SKOS prefLabel (en) when available, URI tail otherwise. +// Default state: UNCHECKED — empty = no filter. +facetFilters = { + if (!phase1) return; + + // Tiny URI → prefLabel lookup. ~60 KB. Best-effort: fallback to URI tail. + const vocabMap = new Map(); + try { + const vocab = await db.query( + `SELECT uri, pref_label FROM read_parquet('${vocab_labels_url}') WHERE lang = 'en'` + ); + for (const r of vocab) vocabMap.set(r.uri, r.pref_label); + } catch (err) { + console.warn("vocab_labels load failed; falling back to URI tails:", err); + } + const prettyLabel = (uri) => { + if (uri == null) return ""; + const hit = vocabMap.get(uri); + if (hit) return hit; + const s = String(uri); + if (!/^https?:\/\//.test(s)) return s; + const parts = s.replace(/[#?].*$/, "").split("/").filter(Boolean); + return parts.length ? parts[parts.length - 1] : s; + }; + + try { + const summaries = await db.query(` + SELECT facet_type, facet_value, count + FROM read_parquet('${facet_summaries_url}') + ORDER BY facet_type, count DESC + `); + + const grouped = { source: [], material: [], context: [], object_type: [] }; + for (const row of summaries) { + if (grouped[row.facet_type]) { + grouped[row.facet_type].push({ + uri: row.facet_value, + label: prettyLabel(row.facet_value), + count: row.count + }); + } + } + + viewer._baselineCounts = { + source: new Map(grouped.source.map(s => [s.uri, s.count])), + material: new Map(grouped.material.map(m => [m.uri, m.count])), + context: new Map(grouped.context.map(c => [c.uri, c.count])), + object_type: new Map(grouped.object_type.map(o => [o.uri, o.count])), + }; + + // HTML attribute / text escapers for safety when interpolating URIs. + const escAttr = (s) => String(s).replace(/&/g, '&').replace(/"/g, '"').replace(/ String(s).replace(/&/g, '&').replace(/ { + const body = document.getElementById(bodyId); + if (!body) return; + if (items.length === 0) { + body.innerHTML = 'No values'; + return; + } + body.innerHTML = items.map(it => + `` + ).join(''); + }; + + renderFilter('materialFilterBody', 'material', grouped.material); + renderFilter('contextFilterBody', 'context', grouped.context); + renderFilter('objectTypeFilterBody', 'object_type', grouped.object_type); + applyFacetCounts('source', null); + applyQueryToFacetFilters(); + + console.log(`Facet filters loaded: ${grouped.material.length} materials, ${grouped.context.length} contexts, ${grouped.object_type.length} object types (vocab labels: ${vocabMap.size})`); + } catch(err) { + console.warn("Facet summaries failed to load:", err); + } + return "loaded"; +} +``` + +```{ojs} +//| echo: false +//| output: false + +// === Table view: paginated sample rows matching current filters === +tableView = { + if (!facetFilters) return; + + let rows = []; + let page = 0; + let requestId = 0; + let loadedMax = 0; + let hitHardCap = false; + let tableDirty = true; + + const globeLayout = document.querySelector('.globe-layout'); + const tableContainer = document.getElementById('tableContainer'); + const tableControls = document.getElementById('tableControls'); + const globeBtn = document.getElementById('globeViewBtn'); + const tableBtn = document.getElementById('tableViewBtn'); + const maxInput = document.getElementById('maxSamples'); + const prevBtn = document.getElementById('tablePrev'); + const nextBtn = document.getElementById('tableNext'); + const metaEl = document.getElementById('tableMeta'); + const pageInfoEl = document.getElementById('tablePageInfo'); + const tableEl = document.getElementById('samplesTable'); + + function setMeta(text, loading) { + if (!metaEl) return; + metaEl.textContent = text; + metaEl.style.color = loading ? '#1565c0' : '#555'; + } + + function tableSourceBadge(source) { + const color = SOURCE_COLORS[source] || '#666'; + const name = SOURCE_NAMES[source] || source || ''; + return `${escapeHtml(name)}`; + } + + function renderTable() { + const totalPages = Math.max(1, Math.ceil(rows.length / TABLE_PAGE_SIZE)); + page = Math.min(page, totalPages - 1); + const start = page * TABLE_PAGE_SIZE; + const visible = rows.slice(start, start + TABLE_PAGE_SIZE); + + if (!tableEl) return; + if (visible.length === 0) { + tableEl.innerHTML = '
No samples match the current filters.
'; + } else { + const body = visible.map(r => { + const placeParts = r.place_name; + const place = Array.isArray(placeParts) && placeParts.length > 0 + ? placeParts.filter(Boolean).join(' › ') + : ''; + const lat = r.latitude != null ? Number(r.latitude).toFixed(5) : ''; + const lng = r.longitude != null ? Number(r.longitude).toFixed(5) : ''; + const label = r.label || r.pid || ''; + const url = sourceUrl(r.pid); + const labelHtml = url + ? `${escapeHtml(label)}` + : escapeHtml(label); + return ` + ${tableSourceBadge(r.source)} + ${labelHtml} + ${escapeHtml(place)} + ${escapeHtml(r.result_time || '')} + ${escapeHtml(lat)} + ${escapeHtml(lng)} + `; + }).join(''); + tableEl.innerHTML = `
+ + + ${body} +
SourceLabelPlaceDateLatLon
+
`; + } + + if (pageInfoEl) { + const first = rows.length === 0 ? 0 : start + 1; + const last = Math.min(rows.length, start + visible.length); + pageInfoEl.textContent = rows.length === 0 + ? 'Page 0 of 0' + : `Page ${page + 1} of ${totalPages} (${first.toLocaleString()}-${last.toLocaleString()} of ${rows.length.toLocaleString()})`; + } + if (prevBtn) prevBtn.disabled = page <= 0; + if (nextBtn) nextBtn.disabled = page >= totalPages - 1; + } + + async function refreshTable() { + const myReq = ++requestId; + loadedMax = getTableMaxSamples(); + page = 0; + setMeta(`Loading up to ${loadedMax.toLocaleString()} samples matching filters...`, true); + + try { + const data = await db.query(` + SELECT pid, label, source, latitude, longitude, place_name, result_time + FROM read_parquet('${lite_url}') + WHERE 1=1 + ${sourceFilterSQL('source')} + ${facetFilterSQL()} + LIMIT ${loadedMax} + `); + if (myReq !== requestId) return; + const arr = Array.from(data); + hitHardCap = arr.length === loadedMax; + rows = arr; + tableDirty = false; + renderTable(); + const capText = hitHardCap + ? (loadedMax < TABLE_MAX_MAX + ? ` Max samples cap reached; raise it to inspect more rows.` + : ` Maximum table cap reached.`) + : ''; + setMeta(`Loaded ${rows.length.toLocaleString()} sample rows.${capText}`, false); + } catch (err) { + if (myReq !== requestId) return; + console.error('Table query failed:', err); + rows = []; + renderTable(); + setMeta('Table query failed; adjust filters and try again.', false); + } + } + + function setView(mode, updateUrl) { + const tableMode = mode === 'table'; + document.body.classList.toggle('table-view-active', tableMode); + if (globeLayout) globeLayout.style.display = tableMode ? 'none' : ''; + if (tableContainer) tableContainer.style.display = tableMode ? 'block' : 'none'; + if (tableControls) tableControls.style.display = tableMode ? 'flex' : 'none'; + if (globeBtn) { + globeBtn.classList.toggle('active', !tableMode); + globeBtn.setAttribute('aria-pressed', String(!tableMode)); + } + if (tableBtn) { + tableBtn.classList.toggle('active', tableMode); + tableBtn.setAttribute('aria-pressed', String(tableMode)); + } + if (updateUrl) writeQueryState(); + if (tableMode && (tableDirty || rows.length === 0)) refreshTable(); + if (!tableMode && typeof viewer !== 'undefined') { + setTimeout(() => viewer.resize(), 0); + } + } + + if (globeBtn) globeBtn.addEventListener('click', () => setView('globe', true)); + if (tableBtn) tableBtn.addEventListener('click', () => setView('table', true)); + if (prevBtn) prevBtn.addEventListener('click', () => { page = Math.max(0, page - 1); renderTable(); }); + if (nextBtn) nextBtn.addEventListener('click', () => { page += 1; renderTable(); }); + if (maxInput) { + maxInput.addEventListener('change', () => { + maxInput.value = getTableMaxSamples(); + if (isTableViewActive()) refreshTable(); + }); + } + + function handleTableFilterChange() { + tableDirty = true; + if (isTableViewActive()) refreshTable(); + } + + document.getElementById('sourceFilter')?.addEventListener('change', handleTableFilterChange); + document.getElementById('materialFilterBody')?.addEventListener('change', handleTableFilterChange); + document.getElementById('contextFilterBody')?.addEventListener('change', handleTableFilterChange); + document.getElementById('objectTypeFilterBody')?.addEventListener('change', handleTableFilterChange); + + window.refreshSamplesTable = refreshTable; + const params = new URLSearchParams(location.search); + setView(params.get('view') === 'table' ? 'table' : 'globe', false); + + return "active"; +} +``` + +```{ojs} +//| echo: false +//| output: false + +// === Zoom watcher: H3 cluster mode + individual sample point mode === +zoomWatcher = { + if (!phase1) return; + if (!facetFilters) return; // wait for facet checkboxes + + // --- State --- + let mode = 'cluster'; // 'cluster' or 'point' + let currentRes = 4; + let loading = false; + let requestId = 0; // stale-request guard + // clusterDataCache stored on viewer._clusterData (set by phase1 and loadRes) + + // Hysteresis thresholds to avoid flicker + const ENTER_POINT_ALT = 120000; // 120 km → enter point mode + const EXIT_POINT_ALT = 180000; // 180 km → exit point mode + const POINT_BUDGET = getMaxSamplesLimit(); + + // Viewport cache: avoid re-querying same area + let cachedBounds = null; // { south, north, west, east } + let cachedData = null; // array of rows + + // --- H3 cluster loading (existing logic) --- + let loadResGen = 0; // generation counter to discard stale results + const loadRes = async (res, url) => { + const gen = ++loadResGen; // claim a generation + loading = true; + updatePhaseMsg(`Loading H3 res${res}...`, 'loading'); + + try { + performance.mark(`r${res}-s`); + const data = await db.query(` + SELECT h3_cell, sample_count, center_lat, center_lng, + dominant_source, source_count + FROM read_parquet('${url}') + WHERE 1=1${sourceFilterSQL('dominant_source')} + `); + + if (gen !== loadResGen) return; // stale — a newer call superseded this one + viewer.h3Points.removeAll(); + const scalar = new Cesium.NearFarScalar(1.5e2, 1.5, 8.0e6, 0.3); + let total = 0; + + for (const row of data) { + total += row.sample_count; + const size = Math.min(3 + Math.log10(row.sample_count) * 3.5, 18); + viewer.h3Points.add({ + id: { count: row.sample_count, source: row.dominant_source, lat: row.center_lat, lng: row.center_lng, resolution: res }, + position: Cesium.Cartesian3.fromDegrees(row.center_lng, row.center_lat, 0), + pixelSize: size, + color: Cesium.Color.fromCssColorString(SOURCE_COLORS[row.dominant_source] || '#666').withAlpha(0.85), + scaleByDistance: scalar, + }); + } + + // Cache for viewport counting + viewer._clusterData = Array.from(data); + viewer._clusterTotal = { clusters: data.length, samples: total }; + + performance.mark(`r${res}-e`); + performance.measure(`r${res}`, `r${res}-s`, `r${res}-e`); + const elapsed = performance.getEntriesByName(`r${res}`).pop().duration; + + // Show viewport count immediately + const bounds = getViewportBounds(); + const inView = countInViewport(bounds); + updateStats(`H3 Res${res}`, `${inView.clusters.toLocaleString()} / ${data.length.toLocaleString()}`, inView.samples.toLocaleString(), `${(elapsed/1000).toFixed(1)}s`, 'Clusters in View / Loaded', 'Samples in View'); + updatePhaseMsg(`${data.length.toLocaleString()} clusters, ${total.toLocaleString()} samples. ${res < 8 ? 'Zoom in for finer detail.' : 'Zoom closer for individual samples.'}`, 'done'); + + currentRes = res; + console.log(`Res${res}: ${data.length} clusters in ${elapsed.toFixed(0)}ms`); + } catch(err) { + console.error(`Failed to load res${res}:`, err); + updatePhaseMsg(`Failed to load H3 res${res} — try zooming again.`, 'loading'); + } finally { + loading = false; + } + }; + + // --- Get camera viewport bounds --- + function getViewportBounds() { + const rect = viewer.camera.computeViewRectangle(viewer.scene.globe.ellipsoid); + if (!rect) return null; + return { + south: Cesium.Math.toDegrees(rect.south), + north: Cesium.Math.toDegrees(rect.north), + west: Cesium.Math.toDegrees(rect.west), + east: Cesium.Math.toDegrees(rect.east) + }; + } + + // --- Count clusters visible in current viewport (from cached array) --- + function countInViewport(bounds) { + const cache = viewer._clusterData; + if (!bounds || !cache || cache.length === 0) return { clusters: 0, samples: 0 }; + const { south, north, west, east } = bounds; + const wrapLng = west > east; // dateline crossing + let clusters = 0, samples = 0; + for (const row of cache) { + if (row.center_lat < south || row.center_lat > north) continue; + if (wrapLng ? (row.center_lng < west && row.center_lng > east) : (row.center_lng < west || row.center_lng > east)) continue; + clusters++; + samples += row.sample_count; + } + return { clusters, samples }; + } + + // --- Check if viewport is within cached bounds --- + function isWithinCache(bounds) { + if (!cachedBounds || !bounds) return false; + return bounds.south >= cachedBounds.south && + bounds.north <= cachedBounds.north && + bounds.west >= cachedBounds.west && + bounds.east <= cachedBounds.east; + } + + // --- Load individual samples for current viewport --- + async function loadViewportSamples() { + const myReqId = ++requestId; + const bounds = getViewportBounds(); + if (!bounds) return; + + // If viewport is within cached area, just re-render from cache + if (isWithinCache(bounds) && cachedData) { + renderSamplePoints(cachedData, bounds); + return; + } + + // Fetch with 30% padding for smooth panning + const latPad = (bounds.north - bounds.south) * 0.3; + const lngPad = (bounds.east - bounds.west) * 0.3; + const padded = { + south: bounds.south - latPad, + north: bounds.north + latPad, + west: bounds.west - lngPad, + east: bounds.east + lngPad + }; + + updatePhaseMsg('Loading individual samples...', 'loading'); + + try { + performance.mark('sp-s'); + // facetFilterSQL() returns a portable `pid IN (...)` predicate, + // so the same query works whether or not facet filters are active. + const query = ` + SELECT pid, label, source, latitude, longitude, + place_name, result_time + FROM read_parquet('${lite_url}') + WHERE latitude BETWEEN ${padded.south} AND ${padded.north} + AND longitude BETWEEN ${padded.west} AND ${padded.east} + ${sourceFilterSQL('source')} + ${facetFilterSQL()} + LIMIT ${POINT_BUDGET} + `; + const data = await db.query(query); + performance.mark('sp-e'); + performance.measure('sp', 'sp-s', 'sp-e'); + const elapsed = performance.getEntriesByName('sp').pop().duration; + + // Stale guard: discard if a newer request was issued + if (myReqId !== requestId) { + console.log(`Discarding stale sample response (req ${myReqId}, current ${requestId})`); + return; + } + + // Cache the padded bounds + data + cachedBounds = padded; + cachedData = Array.from(data); + + renderSamplePoints(cachedData, bounds); + + updateStats('Samples', cachedData.length, cachedData.length, `${(elapsed/1000).toFixed(1)}s`, 'Samples in View', 'Samples in View'); + updatePhaseMsg(`${cachedData.length.toLocaleString()} individual samples. Click one for details.`, 'done'); + console.log(`Point mode: ${cachedData.length} samples in ${elapsed.toFixed(0)}ms`); + + } catch(err) { + if (myReqId !== requestId) return; + console.error("Viewport sample query failed:", err); + updatePhaseMsg('Sample query failed — try again.', 'loading'); + } + } + + // --- Render sample points on globe --- + function renderSamplePoints(data, bounds) { + viewer.samplePoints.removeAll(); + const scalar = new Cesium.NearFarScalar(1e2, 8, 2e5, 3); + + for (const row of data) { + const color = SOURCE_COLORS[row.source] || '#666'; + viewer.samplePoints.add({ + id: { + type: 'sample', + pid: row.pid, + label: row.label, + source: row.source, + lat: row.latitude, + lng: row.longitude, + place_name: row.place_name, + result_time: row.result_time + }, + position: Cesium.Cartesian3.fromDegrees(row.longitude, row.latitude, 0), + pixelSize: 6, + color: Cesium.Color.fromCssColorString(color).withAlpha(0.9), + scaleByDistance: scalar, + }); + } + } + + // --- Mode transitions --- + function enterPointMode(pushHistory) { + mode = 'point'; + viewer._globeState.mode = 'point'; + viewer.h3Points.show = false; + viewer.samplePoints.show = true; + if (pushHistory !== false) history.pushState(null, '', buildHash(viewer)); + loadViewportSamples(); + console.log('Entered point mode'); + } + + function exitPointMode(pushHistory) { + mode = 'cluster'; + viewer._globeState.mode = 'cluster'; + viewer.samplePoints.show = false; + viewer.samplePoints.removeAll(); + viewer.h3Points.show = true; + if (pushHistory !== false) history.pushState(null, '', buildHash(viewer)); + cachedBounds = null; + cachedData = null; + + // Restore cluster stats with viewport count + const bounds = getViewportBounds(); + const inView = countInViewport(bounds); + const total = viewer._clusterTotal; + if (total) { + updateStats(`H3 Res${currentRes}`, `${inView.clusters.toLocaleString()} / ${total.clusters.toLocaleString()}`, inView.samples.toLocaleString(), '—', 'Clusters in View / Loaded', 'Samples in View'); + } else { + updateStats(`H3 Res${currentRes}`, viewer.h3Points.length, '—', '—', 'Clusters Loaded', 'Samples Loaded'); + } + updatePhaseMsg(`${inView.clusters.toLocaleString()} clusters in view. Zoom closer for individual samples.`, 'done'); + console.log('Exited point mode'); + } + + // === Cross-filter facet count refresh (issue #156, Phase 2) === + // + // Counts answer: for each value in facet D, how many samples would match + // this value plus the active filters in all OTHER facets. This keeps + // selected facets useful as drill-out controls instead of just echoing the + // selected values. Search text is treated as an additional sample predicate. + let facetCountsReqId = 0; + let facetCountsDebounce = null; + + function getSearchTerm() { + const input = document.getElementById('sampleSearch'); + return input ? input.value.trim() : ''; + } + + function describeCrossFilters() { + const sourceChecks = document.querySelectorAll('#sourceFilter input[type="checkbox"]'); + const sourceTotal = sourceChecks.length; + const sources = getActiveSources(); + const mat = getCheckedValues('materialFilterBody'); + const ctx = getCheckedValues('contextFilterBody'); + const ot = getCheckedValues('objectTypeFilterBody'); + const search = getSearchTerm(); + const dims = [ + { key: 'source', col: 'source', values: sources.length < sourceTotal ? sources : [] }, + { key: 'material', col: 'material', values: mat }, + { key: 'context', col: 'context', values: ctx }, + { key: 'object_type', col: 'object_type', values: ot }, + ]; + const activeDims = dims.filter(d => d.values.length > 0); + const totalActiveValues = activeDims.reduce((n, d) => n + d.values.length, 0); + return { + dims, + activeDims, + totalActiveValues, + sourceImpossible: sourceTotal > 0 && sources.length === 0, + searchActive: search.length >= 2, + search, + }; + } + + function buildCrossFilterWhere(excludeFacet) { + const { activeDims, sourceImpossible, searchActive, search } = describeCrossFilters(); + if (sourceImpossible && excludeFacet !== 'source') return '1=0'; + + const conds = activeDims + .filter(d => d.key !== excludeFacet) + .map(d => { + const list = d.values.map(v => `'${escSql(v)}'`).join(','); + return `${d.col} IN (${list})`; + }); + + if (searchActive) { + const terms = searchTerms(search); + const searchWhere = textSearchWhere(terms, ['label', 'CAST(place_name AS VARCHAR)']); + conds.push(`pid IN ( + SELECT pid + FROM read_parquet('${lite_url}') + WHERE ${searchWhere} + )`); + } + + return conds.length > 0 ? conds.join(' AND ') : '1=1'; + } + + async function updateCrossFilteredCounts(myReq) { + if (myReq !== facetCountsReqId) return; + const { dims, activeDims, totalActiveValues, sourceImpossible, searchActive } = describeCrossFilters(); + + if (!sourceImpossible && activeDims.length === 0 && !searchActive) { + for (const d of dims) applyFacetCounts(d.key, null); + return; + } + + markFacetCountsRecomputing(); + + const singleActiveDim = !sourceImpossible && !searchActive + && activeDims.length === 1 && activeDims[0].values.length === 1 + ? activeDims[0] : null; + if (singleActiveDim && totalActiveValues === 1) { + try { + const filterCols = ['filter_source', 'filter_material', 'filter_context', 'filter_object_type']; + const filterColForKey = { + source: 'filter_source', + material: 'filter_material', + context: 'filter_context', + object_type: 'filter_object_type', + }; + const targetCol = filterColForKey[singleActiveDim.key]; + const value = escSql(singleActiveDim.values[0]); + const whereParts = filterCols.map(c => + c === targetCol ? `${c} = '${value}'` : `${c} IS NULL` + ); + const rows = await db.query(` + SELECT facet_type, facet_value, count + FROM read_parquet('${cross_filter_url}') + WHERE ${whereParts.join(' AND ')} + `); + if (myReq !== facetCountsReqId) return; + if (rows && rows.length > 0) { + const grouped = { source: new Map(), material: new Map(), context: new Map(), object_type: new Map() }; + for (const r of rows) { + if (grouped[r.facet_type]) grouped[r.facet_type].set(r.facet_value, Number(r.count)); + } + for (const d of dims) { + applyFacetCounts(d.key, d.key === singleActiveDim.key ? null : grouped[d.key]); + } + return; + } + } catch (err) { + console.warn('Cross-filter cache lookup failed; falling back to on-the-fly:', err); + } + } + + await Promise.all(dims.map(async (d) => { + const where = buildCrossFilterWhere(d.key); + try { + const rows = await db.query(` + SELECT ${d.col} AS value, COUNT(*) AS count + FROM read_parquet('${facets_url}') + WHERE ${where} AND ${d.col} IS NOT NULL + GROUP BY ${d.col} + `); + if (myReq !== facetCountsReqId) return; + const map = new Map(); + for (const r of rows) map.set(r.value, Number(r.count)); + applyFacetCounts(d.key, map); + } catch (err) { + if (myReq !== facetCountsReqId) return; + console.warn(`Cross-filter count query failed for ${d.key}:`, err); + applyFacetCounts(d.key, null); + } + })); + } + + function refreshFacetCounts() { + clearTimeout(facetCountsDebounce); + const myReq = ++facetCountsReqId; + facetCountsDebounce = setTimeout(() => { + updateCrossFilteredCounts(myReq); + }, 250); + } + + // --- Source filter change handler --- + const resUrls = { 4: h3_res4_url, 6: h3_res6_url, 8: h3_res8_url }; + document.getElementById('sourceFilter').addEventListener('change', async () => { + // Toggle visual state on labels + updateSourceLegendState(); + writeQueryState(); + if (mode === 'cluster') { + loading = false; // allow loadRes to run (gen counter discards stale results) + await loadRes(currentRes, resUrls[currentRes]); + } else { + cachedBounds = null; // force re-query + await loadViewportSamples(); + } + refreshFacetCounts(); + }); + + // --- Material / Context / Specimen Type filter change handler --- + // + // Cluster-mode honesty: the H3 summary parquets only carry + // `dominant_source`, so material / context / object_type filters cannot + // affect cluster counts. When any of these is active in cluster mode, + // surface the explanatory `#facetNote` so users understand the filter + // takes effect at neighborhood zoom. See issue #156, Phase 1. + const facetNote = document.getElementById('facetNote'); + function handleFacetFilterChange() { + const active = hasFacetFilters(); + if (facetNote) facetNote.style.display = (active && mode === 'cluster') ? 'block' : 'none'; + writeQueryState(); + if (mode === 'point') { + cachedBounds = null; + loadViewportSamples(); + } + refreshFacetCounts(); + } + document.getElementById('materialFilterBody').addEventListener('change', handleFacetFilterChange); + document.getElementById('contextFilterBody').addEventListener('change', handleFacetFilterChange); + document.getElementById('objectTypeFilterBody').addEventListener('change', handleFacetFilterChange); + + // --- Camera change handler --- + let timer = null; + viewer.camera.changed.addEventListener(() => { + if (timer) clearTimeout(timer); + timer = setTimeout(async () => { + const h = viewer.camera.positionCartographic.height; + + // Determine target mode with hysteresis + const targetMode = h < ENTER_POINT_ALT ? 'point' + : h > EXIT_POINT_ALT ? 'cluster' + : mode; + + if (targetMode === 'point' && mode !== 'point') { + // Make sure we're at res8 clusters before transitioning + if (currentRes !== 8 && !loading) { + await loadRes(8, h3_res8_url); + } + enterPointMode(); + } else if (targetMode === 'cluster' && mode !== 'cluster') { + exitPointMode(); + // Reload appropriate resolution + const target = h > 3000000 ? 4 : h > 300000 ? 6 : 8; + if (target !== currentRes && !loading) { + await loadRes(target, { 4: h3_res4_url, 6: h3_res6_url, 8: h3_res8_url }[target]); + } + } else if (targetMode === 'point') { + // Already in point mode — update viewport samples + loadViewportSamples(); + } else { + // Cluster mode — check if resolution should change + const target = h > 3000000 ? 4 : h > 300000 ? 6 : 8; + if (target !== currentRes && !loading) { + await loadRes(target, { 4: h3_res4_url, 6: h3_res6_url, 8: h3_res8_url }[target]); + } + } + + // Update viewport cluster count (cluster mode only; point mode already shows viewport count) + if (mode === 'cluster' && viewer._clusterData) { + const bounds = getViewportBounds(); + const inView = countInViewport(bounds); + const total = viewer._clusterTotal; + if (total) { + updateStats(`H3 Res${currentRes}`, `${inView.clusters.toLocaleString()} / ${total.clusters.toLocaleString()}`, inView.samples.toLocaleString(), null, 'Clusters in View / Loaded', 'Samples in View'); + } + } + + // Update URL hash (replaceState for continuous movement) + if (!viewer._suppressHashWrite) { + history.replaceState(null, '', buildHash(viewer)); + } + }, 600); + }); + viewer.camera.percentageChanged = 0.1; + + // --- Handle browser back/forward --- + window.addEventListener('hashchange', async () => { + const state = readHash(); + if (state.lat == null || state.lng == null) return; + + viewer._suppressHashWrite = true; + clearTimeout(viewer._suppressTimer); + viewer.camera.cancelFlight(); + viewer.camera.flyTo({ + destination: Cesium.Cartesian3.fromDegrees(state.lng, state.lat, state.alt || 20000000), + orientation: { + heading: Cesium.Math.toRadians(state.heading), + pitch: Cesium.Math.toRadians(state.pitch) + }, + duration: 1.5, + }); + + // After flight settles, force mode and clear suppress flag + viewer._suppressTimer = setTimeout(() => { + viewer._suppressHashWrite = false; + const s = readHash(); + if (s.mode === 'point' && mode !== 'point') enterPointMode(false); + else if (s.mode !== 'point' && mode === 'point') exitPointMode(false); + }, 2000); + + // Handle pid selection + if (state.pid) { + viewer._globeState.selectedPid = state.pid; + try { + const sample = await db.query(` + SELECT pid, label, source, latitude, longitude, place_name, result_time + FROM read_parquet('${lite_url}') + WHERE pid = '${state.pid.replace(/'/g, "''")}' + LIMIT 1 + `); + if (sample && sample.length > 0) { + const s = sample[0]; + updateSampleCard({ + pid: s.pid, label: s.label, source: s.source, + lat: s.latitude, lng: s.longitude, + place_name: s.place_name, result_time: s.result_time + }); + } + } catch(err) { + console.error("Hash pid query failed:", err); + } + } else { + viewer._globeState.selectedPid = null; + updateClusterCard(null); + } + }); + + // --- Share button --- + const shareBtn = document.getElementById('shareBtn'); + if (shareBtn) { + shareBtn.addEventListener('click', async () => { + history.replaceState(null, '', buildHash(viewer)); + try { + await navigator.clipboard.writeText(location.href); + const toast = document.getElementById('shareToast'); + if (toast) { + toast.style.opacity = '1'; + setTimeout(() => { toast.style.opacity = '0'; }, 2000); + } + } catch(err) { + prompt('Copy this link:', location.href); + } + }); + } + + // --- Search handler --- + const searchBtn = document.getElementById('searchBtn'); + const searchInput = document.getElementById('sampleSearch'); + const searchResults = document.getElementById('searchResults'); + + async function doSearch() { + const term = searchInput.value.trim(); + if (!term || term.length < 2) { + searchResults.textContent = 'Type at least 2 characters'; + writeQueryState(); + return; + } + writeQueryState(); + searchResults.textContent = 'Searching...'; + try { + const terms = searchTerms(term); + const searchWhere = textSearchWhere(terms, ['label', 'CAST(place_name AS VARCHAR)']); + const score = textSearchScore(terms, [ + { col: 'label', weight: 3 }, + { col: 'CAST(place_name AS VARCHAR)', weight: 2 }, + ]); + const results = await db.query(` + SELECT pid, label, source, latitude, longitude, place_name, + (${score}) AS relevance_score + FROM read_parquet('${lite_url}') + WHERE ${searchWhere} + ${sourceFilterSQL('source')} + ${facetFilterSQL()} + ORDER BY relevance_score DESC, label + LIMIT 50 + `); + if (results.length === 0) { + searchResults.textContent = `No results for "${term}"`; + return; + } + searchResults.textContent = `${results.length}${results.length === 50 ? '+' : ''} results for "${term}"`; + + // Show results in the samples panel + const sampEl = document.getElementById('samplesSection'); + if (sampEl) { + let h = `

Search: "${term}" (${results.length})

`; + for (const s of results) { + const color = SOURCE_COLORS[s.source] || '#666'; + const name = SOURCE_NAMES[s.source] || s.source; + const sUrl = sourceUrl(s.pid); + h += `
+
+ ${sUrl ? `${s.label || s.pid}` : `${s.label || s.pid}`} + ${name} +
+
`; + } + sampEl.innerHTML = h; + + // Click search result → fly to it + sampEl.querySelectorAll('.sample-row[data-lat]').forEach(row => { + row.addEventListener('click', (e) => { + if (e.target.tagName === 'A') return; // let links work + const lat = parseFloat(row.dataset.lat); + const lng = parseFloat(row.dataset.lng); + const pid = row.dataset.pid; + if (!isNaN(lat) && !isNaN(lng)) { + viewer.camera.flyTo({ + destination: Cesium.Cartesian3.fromDegrees(lng, lat, 50000), + duration: 1.5 + }); + } + }); + }); + } + + // Fly to the first result + if (results[0].latitude && results[0].longitude) { + viewer.camera.flyTo({ + destination: Cesium.Cartesian3.fromDegrees(results[0].longitude, results[0].latitude, 200000), + duration: 1.5 + }); + } + } catch(err) { + console.error("Search failed:", err); + searchResults.textContent = `Search error: ${err.message}`; + } + } + + if (searchBtn) searchBtn.addEventListener('click', () => { + doSearch(); + refreshFacetCounts(); + }); + let searchFacetDebounce = null; + if (searchInput) searchInput.addEventListener('input', () => { + clearTimeout(searchFacetDebounce); + searchFacetDebounce = setTimeout(refreshFacetCounts, 300); + }); + if (searchInput) searchInput.addEventListener('keydown', (e) => { + if (e.key === 'Enter') { + doSearch(); + refreshFacetCounts(); + } + }); + + if (searchInput && searchInput.value.trim().length >= 2) { + doSearch(); + } + + refreshFacetCounts(); + + // --- Deep-link: restore selection from initial hash --- + const ih = viewer._initialHash; + if (ih.pid) { + viewer._globeState.selectedPid = ih.pid; + try { + const sample = await db.query(` + SELECT pid, label, source, latitude, longitude, place_name, result_time + FROM read_parquet('${lite_url}') + WHERE pid = '${ih.pid.replace(/'/g, "''")}' + LIMIT 1 + `); + if (sample && sample.length > 0) { + const s = sample[0]; + updateSampleCard({ + pid: s.pid, label: s.label, source: s.source, + lat: s.latitude, lng: s.longitude, + place_name: s.place_name, result_time: s.result_time + }); + const detail = await db.query(` + SELECT description FROM read_parquet('${wide_url}') + WHERE pid = '${ih.pid.replace(/'/g, "''")}' + LIMIT 1 + `); + if (detail && detail.length > 0) updateSampleDetail(detail[0]); + else updateSampleDetail({ description: '' }); + } + } catch(err) { + console.error("Deep-link pid query failed:", err); + } + } + + // Enable hash writing now that everything is initialized + viewer._suppressHashWrite = false; + + return "active"; +} +``` + +```{ojs} +//| echo: false +//| output: false + +// === Performance timing panel (opt-in: append ?perf=1 to URL) === +// v0: reads performance.mark/measure entries and renders a small fixed panel. +// Reports navigation→duckdb_init, navigation→viewer_init, phase 1 res4 load, +// and navigation→first paint. Also dumps to console.table for CI / Playwright. +perfPanel = { + if (!phase1) return; // wait for phase 1 to have run + + const params = new URLSearchParams(location.search); + const isOn = params.get('perf') === '1'; + if (!isOn) return; + + // Give first-paint a tick to fire, then collect + await new Promise(r => setTimeout(r, 100)); + + const origin = performance.timeOrigin; + const mark = (name) => { + const e = performance.getEntriesByName(name, 'mark').pop(); + return e ? e.startTime : null; + }; + const measure = (name) => { + const e = performance.getEntriesByName(name, 'measure').pop(); + return e ? e.duration : null; + }; + + // Paint timings from the browser (free, no instrumentation needed) + const paintEntries = performance.getEntriesByType('paint'); + const fcp = paintEntries.find(e => e.name === 'first-contentful-paint')?.startTime; + const fp = paintEntries.find(e => e.name === 'first-paint')?.startTime; + + const rows = [ + ['first-paint (browser)', fp], + ['first-contentful-paint', fcp], + ['duckdb init', measure('duckdb_init')], + ['viewer init', measure('viewer_init')], + ['nav → viewer ready', mark('viewer-init-end')], + ['phase 1 res4 (duration)', measure('p1')], + ['nav → phase 1 complete', mark('p1-end')], + ['nav → first globe frame', mark('first-globe-frame')], + ].filter(([, v]) => v != null); + + // Console table for CI / offline capture + console.table(Object.fromEntries(rows.map(([k, v]) => [k, `${v.toFixed(0)} ms`]))); + + // Render a small floating panel + const fmt = (ms) => ms == null ? '—' : ms >= 1000 ? `${(ms/1000).toFixed(2)} s` : `${ms.toFixed(0)} ms`; + const panel = document.createElement('div'); + panel.id = 'perfPanel'; + panel.style.cssText = ` + position: fixed; bottom: 12px; right: 12px; z-index: 9999; + background: rgba(0,0,0,0.82); color: #e8f5e9; padding: 10px 12px; + border-radius: 6px; font: 11px/1.4 ui-monospace, SFMono-Regular, monospace; + max-width: 320px; box-shadow: 0 2px 12px rgba(0,0,0,0.3); + `; + panel.innerHTML = ` +
+ ⏱ Perf timings + +
+ + ${rows.map(([label, v]) => ` + + + `).join('')} +
${label}${fmt(v)}
+
timeOrigin: ${new Date(origin).toISOString().split('T')[1].slice(0,12)}
+ `; + document.body.appendChild(panel); + panel.querySelector('#perfClose').onclick = () => panel.remove(); + + return "shown"; +} +``` + +## How This Demo Works + +Pre-aggregated H3 hexagonal indices achieve near-instant globe rendering, with seamless drill-down to individual samples: + +| Phase | Data | Size | Points | +|-------|------|------|--------| +| **Instant** | H3 res4 | 580 KB | 38K clusters (continental) | +| **Zoom in** | H3 res6 | 1.6 MB | 112K clusters (city) | +| **Zoom more** | H3 res8 | 2.5 MB | 176K clusters (neighborhood) | +| **Zoom deep** | Map lite | 60 MB (range req.) | Up to 5K individual samples | +| **Click sample** | Full dataset | ~280 MB (range req.) | Full metadata for 1 sample | + +**4 parquet files, zero backend.** All queries run in your browser via DuckDB-WASM with HTTP range requests — only the bytes you need are transferred. + +## See Also + +- [Deep-Dive Analysis](/tutorials/zenodo_isamples_analysis.html) — DuckDB-WASM SQL tutorial +- [Tutorials](/tutorials/) — index of all interactive tutorials diff --git a/how-to-use.qmd b/how-to-use.qmd index e99c06e..fe67c0e 100644 --- a/how-to-use.qmd +++ b/how-to-use.qmd @@ -6,7 +6,7 @@ number-sections: false ## Quick Start {.unnumbered} -1. **Open the [Interactive Explorer](/tutorials/progressive_globe.html)** — a 3D globe loads with clustered sample data +1. **Open the [Interactive Explorer](/explorer.html)** — a 3D globe loads with clustered sample data 2. **Zoom in** — clusters break into finer detail as you zoom (resolution 4 → 6 → 8 → individual samples) 3. **Filter by source** — use the checkboxes to show/hide data from SESAR, OpenContext, GEOME, or Smithsonian 4. **Click a cluster** — see sample count and nearby samples with links to source records @@ -36,7 +36,6 @@ Works in Chrome, Firefox, Edge, Safari, and Brave. No plugins, no downloads, no All code is visible and foldable on tutorial pages. Want to build your own analysis? -- **[Search Explorer](/tutorials/isamples_explorer.html)** — faceted search across all 6.7M samples with cross-filtering - **[Deep-Dive Analysis](/tutorials/zenodo_isamples_analysis.html)** — statistical exploration with Observable Plot - **[Tutorials index](/tutorials/)** — step-by-step guides from basic exploration to advanced analysis - **[GitHub](https://github.com/isamplesorg/)** — all source code and data pipelines diff --git a/index.qmd b/index.qmd index ce51185..c2432d2 100644 --- a/index.qmd +++ b/index.qmd @@ -9,7 +9,7 @@ page-layout: full ::: {.column-page} -[![Explore 6.7 million samples on an interactive globe](assets/isamples_globe.webp){fig-alt="Animated rotating globe showing iSamples data points from 4 scientific repositories" width="100%" .nolightbox}](/tutorials/progressive_globe.html "Explore the interactive globe") +[![Explore 6.7 million samples on an interactive globe](assets/isamples_globe.webp){fig-alt="Animated rotating globe showing iSamples data points from 4 scientific repositories" width="100%" .nolightbox}](/explorer.html "Explore the interactive globe") ::: @@ -51,7 +51,7 @@ The project uses **geoparquet files + DuckDB-WASM** for efficient, browser-based - **iSamples Full Dataset**: ~280 MB wide format, 6.7M samples - **Available via**: Cloudflare R2 with HTTP range requests -- **Interactive tools**: [Progressive Globe](/tutorials/progressive_globe.html) for visual exploration, [Interactive Explorer](/tutorials/isamples_explorer.html) for search and filtering +- **Interactive tools**: [Interactive Explorer](/explorer.html) — search, filter, and explore 6.7M samples on a 3D globe or in a paginated table All analysis happens in your browser. Only the data you need is downloaded — typically less than 1 MB for initial exploration. ::: diff --git a/index_alt.qmd b/index_alt.qmd index b61c500..c288784 100644 --- a/index_alt.qmd +++ b/index_alt.qmd @@ -4,7 +4,7 @@ number-sections: false toc: false --- -[![](assets/isamples_planet.jpg){group="my-group"}](/tutorials/progressive_globe.html "interact with iSamples data (still a bit slow to load)") +[![](assets/isamples_planet.jpg){group="my-group"}](/explorer.html "interact with iSamples data (still a bit slow to load)") ::: {layout-ncol=4 layout-valign="center"} diff --git a/query-spec.qmd b/query-spec.qmd index 675f248..795f476 100644 --- a/query-spec.qmd +++ b/query-spec.qmd @@ -403,7 +403,7 @@ PQG (the parquet property-graph binding) is specified in vs. the Observable JS, Oct 2025) - `test_cesium_queries.js`, `test_python_js_alignment.py` — alignment test harness at the monorepo root -- [Interactive Explorer](tutorials/progressive_globe.qmd) — the reference +- [Interactive Explorer](explorer.qmd) — the reference web UI - `isamples-python/examples/basic/isamples_explorer.ipynb` — the reference Python UI diff --git a/tests/test_explorer.py b/tests/test_explorer.py deleted file mode 100644 index 2f39dab..0000000 --- a/tests/test_explorer.py +++ /dev/null @@ -1,138 +0,0 @@ -""" -Interactive Explorer tests — verify the search/filter/globe experience. - -These tests hit the live Explorer page and wait for DuckDB-WASM to initialize. -They are slower (~30s+) due to remote parquet loading. -""" -import pytest -from conftest import SITE_URL - -EXPLORER_URL = f"{SITE_URL}/tutorials/isamples_explorer.html" - - -@pytest.fixture -def explorer_page(page): - """Navigate to Explorer and wait for initial load.""" - page.goto(EXPLORER_URL, wait_until="domcontentloaded", timeout=60000) - return page - - -class TestExplorerLoads: - """Explorer page should load and initialize DuckDB-WASM.""" - - def test_page_loads(self, explorer_page): - assert "Explorer" in explorer_page.title() - - def test_has_search_input(self, explorer_page): - # Observable inputs render after JS executes — wait for them - search = explorer_page.locator("input[type='text']") - search.first.wait_for(state="visible", timeout=15000) - assert search.count() > 0 - - def test_has_source_filter_section(self, explorer_page): - assert explorer_page.get_by_text("Source", exact=True).count() > 0 - - def test_has_material_filter_section(self, explorer_page): - assert explorer_page.get_by_text("Material", exact=True).count() > 0 - - def test_has_sampled_feature_filter(self, explorer_page): - assert explorer_page.get_by_text("Sampled Feature").count() > 0 - - def test_has_specimen_type_filter(self, explorer_page): - assert explorer_page.get_by_text("Specimen Type").count() > 0 - - def test_has_max_samples_slider(self, explorer_page): - # Observable range input renders after JS — wait for it - slider = explorer_page.locator("input[type='range']") - slider.first.wait_for(state="attached", timeout=15000) - assert slider.count() > 0 - - def test_has_view_mode_selector(self, explorer_page): - assert explorer_page.get_by_text("Globe").count() > 0 - assert explorer_page.get_by_text("List").count() > 0 - assert explorer_page.get_by_text("Table").count() > 0 - - -class TestExplorerFacetCounts: - """Facet counts should appear from pre-computed summaries.""" - - def test_source_checkboxes_have_counts(self, explorer_page): - """Source checkboxes should show sample counts (loaded from 2KB summary).""" - # Wait for facet summaries to load (they're tiny, should be fast) - explorer_page.wait_for_timeout(5000) - # Check that at least one source has a count in parentheses - sesar = explorer_page.get_by_text("SESAR") - assert sesar.count() > 0 - - def test_four_sources_present(self, explorer_page): - """All 4 data sources should appear as filter options.""" - explorer_page.wait_for_timeout(5000) - for source in ["SESAR", "OPENCONTEXT", "GEOME", "SMITHSONIAN"]: - assert explorer_page.get_by_text(source).count() > 0, f"Missing source: {source}" - - -class TestExplorerCrossFiltering: - """Cross-filtering: clicking a facet should update counts in other facets.""" - - def _wait_for_facets(self, page): - """Wait for facet count labels to render (requires DuckDB-WASM init).""" - facet = page.locator(".facet-count[data-facet='source']") - try: - facet.first.wait_for(state="attached", timeout=60000) - except Exception: - pytest.skip("Facet count labels not rendered (DuckDB-WASM may not have loaded)") - - def _get_count(self, page, facet, value): - """Extract the numeric count from a facet-count label.""" - el = page.locator(f".facet-count[data-facet='{facet}'][data-value='{value}']") - if el.count() == 0: - return None - text = el.first.text_content() # e.g. "(4,389,231)" - return int(text.strip("() ").replace(",", "")) - - def test_baseline_sesar_count_matches_summaries(self, explorer_page): - """Before any interaction, SESAR count should match the facet summary.""" - self._wait_for_facets(explorer_page) - count = self._get_count(explorer_page, "source", "SESAR") - assert count is not None, "SESAR facet-count element not found" - assert count > 4_000_000, f"SESAR baseline count too low: {count}" - - @pytest.mark.skip(reason="Observable Inputs.checkbox ignores programmatic clicks in headless Playwright") - def test_clicking_source_updates_material_counts(self, explorer_page): - """Checking SESAR should lower material counts (no archaeology materials). - Cannot be automated: Observable's reactive system only triggers from real - user interaction, not from programmatic .click() or dispatchEvent().""" - - @pytest.mark.skip(reason="Observable Inputs.checkbox ignores programmatic clicks in headless Playwright") - def test_clearing_filter_restores_baseline(self, explorer_page): - """Unchecking a source should restore baseline counts.""" - - @pytest.mark.skip(reason="Observable Inputs.checkbox ignores programmatic clicks in headless Playwright") - def test_zero_count_items_are_dimmed(self, explorer_page): - """Facet values with 0 matches should have reduced opacity.""" - - def test_new_parquet_endpoints_reachable(self, explorer_page): - """The cross-filter and sample_facets parquet files should be accessible.""" - import subprocess - for url in [ - "https://data.isamples.org/isamples_202601_facet_cross_filter.parquet", - "https://data.isamples.org/isamples_202601_sample_facets_v2.parquet", - ]: - result = subprocess.run( - ["curl", "-s", "-o", "/dev/null", "-w", "%{http_code}", "--head", url], - capture_output=True, text=True - ) - code = result.stdout.strip() - assert code in ("200", "206"), f"{url} returned {code}" - - -class TestExplorerSampleCard: - """Sample Card section should exist.""" - - def test_has_sample_card_section(self, explorer_page): - assert explorer_page.get_by_text("Sample Card").count() > 0 - - def test_sample_card_shows_click_prompt(self, explorer_page): - """Before clicking a point, card should show instructions.""" - explorer_page.wait_for_timeout(3000) - assert explorer_page.get_by_text("Click a point").count() > 0 diff --git a/tests/test_globe.py b/tests/test_globe.py new file mode 100644 index 0000000..cb4a7d9 --- /dev/null +++ b/tests/test_globe.py @@ -0,0 +1,137 @@ +""" +Interactive Explorer tests — verify the unified search/filter/globe/table experience. + +Targets /explorer.html (canonical) and waits for DuckDB-WASM to initialize. +Slow (~30s+) due to remote parquet loading. +""" +import pytest +from conftest import SITE_URL + +EXPLORER_URL = f"{SITE_URL}/explorer.html" + + +@pytest.fixture +def explorer_page(page): + """Navigate to Explorer and wait for initial load.""" + page.goto(EXPLORER_URL, wait_until="domcontentloaded", timeout=60000) + return page + + +class TestExplorerLoads: + """Explorer page should load and initialize DuckDB-WASM.""" + + def test_page_loads(self, explorer_page): + assert "Explorer" in explorer_page.title() + + def test_has_search_input(self, explorer_page): + search = explorer_page.locator("#sampleSearch") + search.wait_for(state="visible", timeout=15000) + assert search.count() == 1 + + def test_has_source_filter_section(self, explorer_page): + assert explorer_page.locator("#sourceFilter").count() == 1 + + def test_has_material_filter_section(self, explorer_page): + assert explorer_page.locator("#materialFilter").count() == 1 + + def test_has_sampled_feature_filter(self, explorer_page): + assert explorer_page.locator("#contextFilter").count() == 1 + + def test_has_specimen_type_filter(self, explorer_page): + assert explorer_page.locator("#objectTypeFilter").count() == 1 + + def test_has_max_samples_input(self, explorer_page): + # Phase 4: number input bounded 1K-100K, default 25K. Lives in table view controls. + max_input = explorer_page.locator("#maxSamples") + max_input.wait_for(state="attached", timeout=15000) + assert max_input.count() == 1 + + def test_has_view_toggle(self, explorer_page): + # Phase 4: binary Globe/Table toggle (no List). + assert explorer_page.locator("#globeViewBtn").count() == 1 + assert explorer_page.locator("#tableViewBtn").count() == 1 + + +class TestExplorerFacetCounts: + """Facet counts should appear from pre-computed summaries.""" + + def test_source_checkboxes_have_counts(self, explorer_page): + explorer_page.wait_for_timeout(5000) + assert explorer_page.get_by_text("SESAR").count() > 0 + + def test_four_sources_present(self, explorer_page): + explorer_page.wait_for_timeout(5000) + for source in ["SESAR", "OPENCONTEXT", "GEOME", "SMITHSONIAN"]: + assert explorer_page.get_by_text(source).count() > 0, f"Missing source: {source}" + + +class TestExplorerCrossFiltering: + """Cross-filtering: changing a facet should update counts in other facets. + + Unskipped in Phase 5 (#156): the unified explorer uses native HTML checkboxes + (not OJS Inputs.checkbox), so .click() and .dispatchEvent() work as expected. + """ + + def _wait_for_facets(self, page): + facet = page.locator(".facet-count[data-facet='source']") + try: + facet.first.wait_for(state="attached", timeout=60000) + except Exception: + pytest.skip("Facet count labels not rendered (DuckDB-WASM may not have loaded)") + + def _get_count(self, page, facet, value): + el = page.locator(f".facet-count[data-facet='{facet}'][data-value='{value}']") + if el.count() == 0: + return None + text = el.first.text_content() + return int(text.strip("() ").replace(",", "")) + + def test_baseline_sesar_count_matches_summaries(self, explorer_page): + self._wait_for_facets(explorer_page) + count = self._get_count(explorer_page, "source", "SESAR") + assert count is not None, "SESAR facet-count element not found" + assert count > 4_000_000, f"SESAR baseline count too low: {count}" + + def test_new_parquet_endpoints_reachable(self, explorer_page): + import subprocess + for url in [ + "https://data.isamples.org/isamples_202601_facet_cross_filter.parquet", + "https://data.isamples.org/isamples_202601_sample_facets_v2.parquet", + ]: + result = subprocess.run( + ["curl", "-s", "-o", "/dev/null", "-w", "%{http_code}", "--head", url], + capture_output=True, text=True + ) + code = result.stdout.strip() + assert code in ("200", "206"), f"{url} returned {code}" + + +class TestExplorerRedirects: + """Old URLs should redirect to /explorer.html, preserving query and hash.""" + + def test_progressive_globe_redirects_preserves_non_q_params(self, page): + # The explorer uses `?search=` (not `?q=`) to avoid colliding with + # Quarto's site-wide search highlight feature. The redirect stub + # forwards whatever query string the browser presents — but Quarto's + # quarto-search.js (loaded into the stub page's ) strips `?q=` + # before our redirect runs, so legacy `?q=` links lose the search + # term. Non-q params survive. + page.goto( + f"{SITE_URL}/tutorials/progressive_globe.html?sources=SESAR&search=basalt", + wait_until="load", + timeout=30000, + ) + page.wait_for_url("**/explorer.html?**", timeout=10000) + assert "/explorer.html" in page.url + assert "search=basalt" in page.url + assert "sources=SESAR" in page.url + + def test_isamples_explorer_redirects_with_search_param(self, page): + page.goto( + f"{SITE_URL}/tutorials/isamples_explorer.html?search=pottery", + wait_until="load", + timeout=30000, + ) + page.wait_for_url("**/explorer.html?**", timeout=10000) + assert "/explorer.html" in page.url + assert "search=pottery" in page.url diff --git a/tests/test_navigation.py b/tests/test_navigation.py index 059c3f6..5b1b088 100644 --- a/tests/test_navigation.py +++ b/tests/test_navigation.py @@ -67,11 +67,6 @@ def test_how_to_use_has_globe_viz(self, page): sidebar = page.locator(".sidebar-navigation") assert sidebar.get_by_text("3D Globe Visualization").count() > 0 - def test_how_to_use_has_search_explorer(self, page): - page.goto(f"{SITE_URL}/how-to-use.html", wait_until="domcontentloaded") - sidebar = page.locator(".sidebar-navigation") - assert sidebar.get_by_text("Search Explorer").count() > 0 - def test_how_to_use_has_narrow_vs_wide(self, page): page.goto(f"{SITE_URL}/how-to-use.html", wait_until="domcontentloaded") sidebar = page.locator(".sidebar-navigation") diff --git a/tests/test_tutorials_landing.py b/tests/test_tutorials_landing.py index 77030a0..8b0ba16 100644 --- a/tests/test_tutorials_landing.py +++ b/tests/test_tutorials_landing.py @@ -27,12 +27,6 @@ def test_has_globe_viz_link(self, page): link = page.locator("a:has-text('Globe')") assert link.count() > 0 - def test_has_search_explorer_link(self, page): - """And I see a link to the Search Explorer.""" - page.goto(HOW_TO_USE_URL, wait_until="domcontentloaded") - link = page.locator("a:has-text('Explorer')") - assert link.count() > 0 - def test_has_narrow_vs_wide_link(self, page): """And I see a link to Narrow vs Wide.""" page.goto(HOW_TO_USE_URL, wait_until="domcontentloaded") @@ -57,9 +51,9 @@ def test_deep_dive_loads(self, page): assert len(critical) == 0, f"JS errors: {critical}" def test_globe_viz_loads(self, page): - """Given I navigate to the 3D Globe page, Then the Cesium container exists.""" + """Given I navigate to /explorer.html, Then the Cesium container exists.""" page.goto( - f"{SITE_URL}/tutorials/progressive_globe.html", + f"{SITE_URL}/explorer.html", wait_until="domcontentloaded", timeout=30000, ) diff --git a/tutorials/index.qmd b/tutorials/index.qmd index a4e5bf8..3797563 100644 --- a/tutorials/index.qmd +++ b/tutorials/index.qmd @@ -8,8 +8,7 @@ number-sections: false | Tutorial | What You'll Learn | |----------|-------------------| -| [**Interactive Explorer**](/tutorials/progressive_globe.html) | Browse samples on a 3D globe with H3-clustered, zoom-adaptive rendering | -| [**Search Explorer**](isamples_explorer.qmd) | Faceted search and filter across all 6.7M samples with cross-filtering | +| [**Interactive Explorer**](/explorer.html) | Browse, search, and filter 6.7M samples on a 3D globe or in a paginated table | | [**Deep-Dive Analysis**](zenodo_isamples_analysis.qmd) | Comprehensive DuckDB-WASM analysis with Observable JS — charts, maps, statistics | | [**Technical: Narrow vs Wide**](narrow_vs_wide_performance.qmd) | Schema comparison and performance benchmarks for the PQG data formats | diff --git a/tutorials/isamples_explorer.qmd b/tutorials/isamples_explorer.qmd index 3b76d2a..a7f9624 100644 --- a/tutorials/isamples_explorer.qmd +++ b/tutorials/isamples_explorer.qmd @@ -1,1882 +1,22 @@ --- -title: "iSamples Interactive Explorer" -subtitle: "Search, filter, and explore 6.7 million material samples" -categories: [parquet, spatial, h3, performance, isamples] -sidebar: false -# No TOC: this page is an app, not an article. The right-hand TOC sidebar -# (#quarto-margin-sidebar) was overlapping .side-panel and silently -# intercepting clicks on the Source filter checkboxes — see issue #127. -toc: false +title: "Redirecting…" format: html: + toc: false + page-layout: full include-in-header: text: | - - - + + +sidebar: false --- - - - - -::: {.callout-note collapse="true"} -## How It Works - -1. **Instant** (<1s): Pre-aggregated H3 res4 summary (580 KB) → 38K colored circles -2. **Zoom in**: Automatically switches to res6 (112K) then res8 (176K) clusters -3. **Zoom deeper** (<120 km): Individual sample points from 60 MB lite parquet -4. **Click**: Cluster info or individual sample card with full metadata -5. **Search**: Find samples by name — results fly to the location on the globe - -Circle size = log(sample count). Color = dominant data source. -::: - - -
- - -
-
- -
-
-
-
-
-
Loading...Resolution
-
0Clusters Loaded
-
0Samples Loaded
-
-Load Time
-
-
-
- - - - -
-
-
-
-Material ▾ -
- -
-
-
-Sampled Feature ▾ -
- -
-
-
-Specimen Type ▾ -
- -
- -
- - -
-
- - - -
-
-Loading H3 global overview... -
-
-
-
Click a cluster or sample on the globe
-
-
-
-
- -### Results {#results} - -
-
Loading samples matching your filters...
-
-
- -```{ojs} -//| output: false -Cesium.Ion.defaultAccessToken = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiIwNzk3NjkyMy1iNGI1LTRkN2UtODRiMy04OTYwYWE0N2M3ZTkiLCJpZCI6Njk1MTcsImlhdCI6MTYzMzU0MTQ3N30.e70dpNzOCDRLDGxRguQCC-tRzGzA-23Xgno5lNgCeB4'; -``` - -```{ojs} -//| echo: false -//| output: false - -// === Constants === -R2_BASE = "https://data.isamples.org" -h3_res4_url = `${R2_BASE}/isamples_202601_h3_summary_res4.parquet` -h3_res6_url = `${R2_BASE}/isamples_202601_h3_summary_res6.parquet` -h3_res8_url = `${R2_BASE}/isamples_202601_h3_summary_res8.parquet` -lite_url = `${R2_BASE}/isamples_202601_samples_map_lite.parquet` -// Stable alias that 302-redirects to the current enriched wide parquet -// (isamples_YYYYMM_wide.parquet). Gets OpenContext thumbnails populated. -wide_url = `${R2_BASE}/current/wide.parquet` -// v2 carries object_type alongside material and context (URI-string columns). -facets_url = `${R2_BASE}/isamples_202601_sample_facets_v2.parquet` -facet_summaries_url = `${R2_BASE}/isamples_202601_facet_summaries.parquet` -// Pre-aggregated cache for fast cross-filter count lookups in the -// single-facet-value-active case. Multi-filter combinations fall back to -// on-the-fly GROUP BY queries against facets_url. See issue #154. -cross_filter_url = `${R2_BASE}/isamples_202601_facet_cross_filter.parquet` -// SKOS prefLabels for Material / Sampled Feature / Specimen Type URIs. -// ~60 KB lookup; falls back to URI tail if a URI isn't covered. -vocab_labels_url = `${R2_BASE}/vocab_labels.parquet` - -// Canonical palette — see issue #113. Path-relative so this works under -// both isamples.org (custom domain at root) and project-pages fork -// previews (rdhyee.github.io/isamplesorg.github.io/...). -_palette = await import(new URL('../assets/js/source-palette.js', document.baseURI).href) -SOURCE_COLORS = _palette.SOURCE_COLORS -SOURCE_NAMES = _palette.SOURCE_NAMES - -// === Source URL: resolve pid to original repository === -function sourceUrl(pid) { - if (!pid) return null; - // All sources resolve via n2t.net: - // ARK pids (OpenContext, GEOME, Smithsonian) → n2t.net/ark:/... - // IGSN pids (SESAR) → n2t.net/IGSN:... - return `https://n2t.net/${pid}`; -} - -// === Source Filter: get active sources and build SQL clause === -function getActiveSources() { - const checks = document.querySelectorAll('#sourceFilter input[type="checkbox"]'); - return Array.from(checks).filter(c => c.checked).map(c => c.value); -} - -function sourceFilterSQL(col) { - const active = getActiveSources(); - // Empty == all (no filter), matching the semantics of the other facet - // groups (material / context / object_type) and Raymond's stated - // intuition (PR #155 thread). All-4 checked is also "all" by virtue - // of the IN list covering everything; we short-circuit to skip an - // unnecessary IN clause. - if (active.length === 0 || active.length === 4) return ''; - const list = active.map(s => `'${s}'`).join(','); - return ` AND ${col} IN (${list})`; -} - -// === Text search SQL helpers === -function searchTerms(value) { - return String(value || '').trim().split(/\s+/).filter(Boolean); -} - -function escapeSqlString(value) { - return String(value).replace(/'/g, "''"); -} - -function escapeIlikePattern(value) { - return escapeSqlString(value).replace(/[\\%_]/g, "\\$&"); -} - -function textSearchWhere(terms, columns) { - return terms.map(raw => { - const term = escapeIlikePattern(raw); - const checks = columns.map(col => `${col} ILIKE '%${term}%' ESCAPE '\\'`); - return `(${checks.join(' OR ')})`; - }).join(' AND '); -} - -function textSearchScore(terms, weightedColumns) { - if (!terms.length) return '0'; - return terms.map(raw => { - const term = escapeIlikePattern(raw); - return weightedColumns.map(({ col, weight }) => - `CASE WHEN ${col} ILIKE '%${term}%' ESCAPE '\\' THEN ${weight} ELSE 0 END` - ).join(' + '); - }).map(score => `(${score})`).join(' + '); -} - -// === Material/Context Filters === -function getCheckedValues(containerId) { - const checks = document.querySelectorAll(`#${containerId} input[type="checkbox"]`); - return Array.from(checks).filter(c => c.checked).map(c => c.value); -} - -// Semantics (matches the prior explorer): empty = no filter, show all. -// Selecting one or more items = include only those. Reduces visual noise -// at startup (you don't see hundreds of pre-checked rows). -function hasFacetFilters() { - return getCheckedValues('materialFilterBody').length > 0 - || getCheckedValues('contextFilterBody').length > 0 - || getCheckedValues('objectTypeFilterBody').length > 0; -} - -function facetFilterSQL() { - let sql = ''; - const mat = getCheckedValues('materialFilterBody'); - if (mat.length > 0) { - const list = mat.map(s => `'${s.replace(/'/g, "''")}'`).join(','); - sql += ` AND f.material IN (${list})`; - } - const ctx = getCheckedValues('contextFilterBody'); - if (ctx.length > 0) { - const list = ctx.map(s => `'${s.replace(/'/g, "''")}'`).join(','); - sql += ` AND f.context IN (${list})`; - } - const ot = getCheckedValues('objectTypeFilterBody'); - if (ot.length > 0) { - const list = ot.map(s => `'${s.replace(/'/g, "''")}'`).join(','); - sql += ` AND f.object_type IN (${list})`; - } - return sql; -} - -// === URL State: encode/decode globe state in hash fragment === -function parseNum(val, def, min, max) { - if (val == null) return def; - const n = parseFloat(val); - if (!Number.isFinite(n)) return def; - if (min != null && n < min) return min; - if (max != null && n > max) return max; - return n; -} - -function readHash() { - const params = new URLSearchParams(location.hash.slice(1)); - return { - v: parseInt(params.get('v')) || 0, - lat: parseNum(params.get('lat'), null, -90, 90), - lng: parseNum(params.get('lng'), null, -180, 180), - alt: parseNum(params.get('alt'), null, 100, 40000000), - heading: parseNum(params.get('heading'), 0, 0, 360), - pitch: parseNum(params.get('pitch'), -90, -90, 0), - mode: params.get('mode') || null, - pid: params.get('pid') || null, - }; -} - -function buildHash(v) { - const cam = v.camera; - const carto = cam.positionCartographic; - const params = new URLSearchParams(); - params.set('v', '1'); - params.set('lat', Cesium.Math.toDegrees(carto.latitude).toFixed(4)); - params.set('lng', Cesium.Math.toDegrees(carto.longitude).toFixed(4)); - params.set('alt', Math.round(carto.height).toString()); - const heading = Cesium.Math.toDegrees(cam.heading) % 360; - const pitch = Cesium.Math.toDegrees(cam.pitch); - if (Math.abs(heading) > 1) params.set('heading', heading.toFixed(1)); - if (Math.abs(pitch + 90) > 1) params.set('pitch', pitch.toFixed(1)); - const gs = v._globeState; - if (gs.mode === 'point') params.set('mode', 'point'); - if (gs.selectedPid) params.set('pid', gs.selectedPid); - return '#' + params.toString(); -} - -// === Helpers: update DOM imperatively (no OJS reactivity) === -function updateStats(phase, points, samples, time, pointsLabel, samplesLabel) { - const s = (id, v) => { const e = document.getElementById(id); if (e) e.textContent = v; }; - s('sPhase', phase); - s('sPoints', typeof points === 'string' ? points : points.toLocaleString()); - s('sSamples', typeof samples === 'string' ? samples : samples.toLocaleString()); - if (time != null) s('sTime', time); - if (pointsLabel) s('sPointsLbl', pointsLabel); - if (samplesLabel) s('sSamplesLbl', samplesLabel); -} - -function updatePhaseMsg(text, type) { - const m = document.getElementById('phaseMsg'); - if (!m) return; - m.textContent = text; - if (type === 'loading') { m.style.background = '#e3f2fd'; m.style.color = '#1565c0'; } - else { m.style.background = '#e8f5e9'; m.style.color = '#2e7d32'; } -} - -function updateClusterCard(info) { - const el = document.getElementById('clusterSection'); - if (!el) return; - if (!info) { - el.innerHTML = '
Click a cluster or sample on the globe
'; - return; - } - const color = SOURCE_COLORS[info.source] || '#666'; - const name = SOURCE_NAMES[info.source] || info.source; - el.innerHTML = `

Selected Cluster

-
-
- ${name} - H3 res${info.resolution} -
-
- ${info.count.toLocaleString()} samples -
-
- ${info.lat.toFixed(4)}, ${info.lng.toFixed(4)} -
-
`; -} - -function updateSampleCard(sample) { - const el = document.getElementById('clusterSection'); - if (!el) return; - const color = SOURCE_COLORS[sample.source] || '#666'; - const name = SOURCE_NAMES[sample.source] || sample.source; - const placeParts = sample.place_name; - const placeStr = Array.isArray(placeParts) && placeParts.length > 0 - ? placeParts.filter(Boolean).join(' › ') - : ''; - const srcUrl = sourceUrl(sample.pid); - el.innerHTML = `

Sample

-
-
- ${name} -
-
- ${sample.label || sample.pid || 'Unnamed'} -
-
- ${sample.lat.toFixed(5)}, ${sample.lng.toFixed(5)} -
- ${placeStr ? `
${placeStr}
` : ''} - ${sample.result_time ? `
Date: ${sample.result_time}
` : ''} - ${srcUrl ? `
View at ${name} →
` : ''} -
Loading full details...
-
`; -} - -function updateSampleDetail(detail) { - const el = document.getElementById('sampleDetail'); - if (!el) return; - if (!detail) { - el.innerHTML = 'Detail query failed'; - return; - } - const desc = detail.description - ? (detail.description.length > 300 ? detail.description.slice(0, 300) + '...' : detail.description) - : ''; - el.innerHTML = `${desc ? `
${desc}
` : ''}`; -} - -function updateSamples(samples) { - const el = document.getElementById('samplesSection'); - if (!el) return; - if (!samples || samples.length === 0) { - el.innerHTML = ''; - return; - } - let h = `

Nearby Samples (${samples.length})

`; - for (const s of samples) { - const color = SOURCE_COLORS[s.source] || '#666'; - const name = SOURCE_NAMES[s.source] || s.source; - const placeParts = s.place_name; - const desc = Array.isArray(placeParts) && placeParts.length > 0 - ? placeParts.filter(Boolean).join(' › ') - : ''; - const sUrl = sourceUrl(s.pid); - h += `
-
- ${sUrl ? `${s.label || s.pid}` : `${s.label || s.pid}`} - ${name} -
- ${desc ? `
${desc}
` : ''} -
`; - } - el.innerHTML = h; -} - -// === Results table (below globe) === -// Renders the first N samples matching current filters, regardless of camera. -// Stable, sourceless ordering (no ORDER BY RANDOM) keeps it cheap on lite_url. -function renderResultsTable(rows) { - const el = document.getElementById('resultsTable'); - if (!el) return; - if (!rows || rows.length === 0) { - el.innerHTML = '
No samples match the current filters.
'; - return; - } - const head = `SourceLabelPlaceLatLon`; - const body = rows.map(r => { - const color = SOURCE_COLORS[r.source] || '#666'; - const name = SOURCE_NAMES[r.source] || r.source; - const placeParts = r.place_name; - const place = Array.isArray(placeParts) && placeParts.length > 0 - ? placeParts.filter(Boolean).join(' › ') - : ''; - const lat = (r.latitude != null) ? r.latitude.toFixed(4) : ''; - const lng = (r.longitude != null) ? r.longitude.toFixed(4) : ''; - const sUrl = sourceUrl(r.pid); - const labelHtml = sUrl - ? `${r.label || r.pid}` - : (r.label || r.pid); - return ` - ${name} - ${labelHtml} - ${place} - ${lat} - ${lng} - `; - }).join(''); - el.innerHTML = `${head}${body}
`; -} - -function updateResultsTableMeta(text, isLoading) { - const el = document.getElementById('resultsTableMeta'); - if (!el) return; - el.textContent = text; - el.style.color = isLoading ? '#1565c0' : '#555'; -} - -// === Cross-filter facet count updates (issue #154) === -// Updates the count span next to each checkbox for one facet group. If -// `countsMap` is null, falls back to baseline counts on `viewer._baselineCounts` -// (set in facetFilters). Dims rows where count === 0 so the user sees that -// their selection eliminated those values rather than silently hiding them. -function applyFacetCounts(facetKey, countsMap) { - const baseline = (viewer && viewer._baselineCounts) ? viewer._baselineCounts[facetKey] : null; - document.querySelectorAll(`.facet-count[data-facet="${facetKey}"]`).forEach(el => { - const value = el.getAttribute('data-value'); - let count; - if (countsMap) { - count = countsMap.has(value) ? countsMap.get(value) : 0; - } else { - count = baseline ? (baseline.get(value) ?? 0) : 0; - } - el.textContent = `(${Number(count).toLocaleString()})`; - el.classList.remove('recomputing'); - const row = document.querySelector(`.facet-row[data-facet="${facetKey}"][data-value="${CSS.escape(value)}"]`); - if (row) row.classList.toggle('zero', count === 0); - }); -} - -function markFacetCountsRecomputing() { - document.querySelectorAll('.facet-count').forEach(el => el.classList.add('recomputing')); -} -``` - -```{ojs} -//| echo: false -//| output: false - -// === DuckDB === -db = { - performance.mark('duckdb-init-start'); - const instance = await DuckDBClient.of(); - performance.mark('duckdb-init-end'); - performance.measure('duckdb_init', 'duckdb-init-start', 'duckdb-init-end'); - return instance; -} -``` - -```{ojs} -//| echo: false -//| output: false - -// === Cesium Viewer (created once, never re-created) === -viewer = { - performance.mark('viewer-init-start'); - const v = new Cesium.Viewer("cesiumContainer", { - timeline: false, - animation: false, - baseLayerPicker: false, - fullscreenElement: "cesiumContainer", - terrain: Cesium.Terrain.fromWorldTerrain() - }); - - // URL deep-link state (must be set before globalRect/once block reads it) - v._globeState = { mode: 'cluster', selectedPid: null }; - v._initialHash = readHash(); - v._suppressHashWrite = true; // cleared after zoomWatcher initializes - v._suppressTimer = null; - - const globalRect = Cesium.Rectangle.fromDegrees(-180, -60, 180, 80); - Cesium.Camera.DEFAULT_VIEW_RECTANGLE = globalRect; - Cesium.Camera.DEFAULT_VIEW_FACTOR = 0.5; - const ih = v._initialHash; - const once = () => { - if (ih.lat != null && ih.lng != null) { - v.camera.setView({ - destination: Cesium.Cartesian3.fromDegrees(ih.lng, ih.lat, ih.alt || 20000000), - orientation: { - heading: Cesium.Math.toRadians(ih.heading), - pitch: Cesium.Math.toRadians(ih.pitch) - } - }); - } else { - v.camera.setView({ destination: globalRect }); - } - v.scene.postRender.removeEventListener(once); - }; - v.scene.postRender.addEventListener(once); - - // Two separate point collections: clusters and individual samples - v.h3Points = new Cesium.PointPrimitiveCollection(); - v.scene.primitives.add(v.h3Points); - - v.samplePoints = new Cesium.PointPrimitiveCollection(); - v.scene.primitives.add(v.samplePoints); - v.samplePoints.show = false; // hidden until point mode - - // Hover tooltip — works for both clusters and samples - v.pointLabel = v.entities.add({ - label: { - show: false, showBackground: true, font: "13px monospace", - horizontalOrigin: Cesium.HorizontalOrigin.LEFT, - verticalOrigin: Cesium.VerticalOrigin.BOTTOM, - pixelOffset: new Cesium.Cartesian2(15, 0), - disableDepthTestDistance: Number.POSITIVE_INFINITY, text: "", - } - }); - - new Cesium.ScreenSpaceEventHandler(v.scene.canvas).setInputAction((movement) => { - const picked = v.scene.pick(movement.endPosition); - if (Cesium.defined(picked) && picked.primitive && picked.id) { - v.pointLabel.position = picked.primitive.position; - v.pointLabel.label.show = true; - const meta = picked.id; - if (typeof meta === 'object' && meta.type === 'sample') { - v.pointLabel.label.text = `${meta.label || meta.pid}`; - } else if (typeof meta === 'object' && meta.count) { - v.pointLabel.label.text = `${meta.source}: ${meta.count.toLocaleString()} samples`; - } else { - v.pointLabel.label.text = String(meta); - } - } else { - v.pointLabel.label.show = false; - } - }, Cesium.ScreenSpaceEventType.MOUSE_MOVE); - - // Click handler — routes to cluster card or sample card - new Cesium.ScreenSpaceEventHandler(v.scene.canvas).setInputAction(async (e) => { - const picked = v.scene.pick(e.position); - if (!Cesium.defined(picked) || !picked.primitive || !picked.id) return; - const meta = picked.id; - - if (typeof meta === 'object' && meta.type === 'sample') { - // --- Individual sample click --- - updateSampleCard(meta); - v._globeState.selectedPid = meta.pid; - history.pushState(null, '', buildHash(v)); - // Clear nearby list - const sampEl = document.getElementById('samplesSection'); - if (sampEl) sampEl.innerHTML = ''; - - // Stage 2: lazy-load full description from wide parquet - try { - const detail = await db.query(` - SELECT description - FROM read_parquet('${wide_url}') - WHERE pid = '${meta.pid.replace(/'/g, "''")}' - LIMIT 1 - `); - if (detail && detail.length > 0) { - updateSampleDetail(detail[0]); - } else { - updateSampleDetail({ description: '' }); - } - } catch(err) { - console.error("Detail query failed:", err); - updateSampleDetail(null); - } - - } else if (typeof meta === 'object' && meta.count) { - // --- Cluster click --- - updateClusterCard(meta); - v._globeState.selectedPid = null; - history.pushState(null, '', buildHash(v)); - - const sampEl = document.getElementById('samplesSection'); - if (sampEl) sampEl.innerHTML = '
Loading nearby samples...
'; - - const delta = meta.resolution === 4 ? 2.0 : meta.resolution === 6 ? 0.5 : 0.1; - try { - const facetActive = hasFacetFilters(); - const facetSQL = facetActive ? facetFilterSQL() : ''; - let nearbyQuery; - if (facetActive) { - nearbyQuery = ` - SELECT l.pid, l.label, l.source, l.latitude, l.longitude, l.place_name - FROM read_parquet('${lite_url}') l - JOIN read_parquet('${facets_url}') f ON l.pid = f.pid - WHERE l.latitude BETWEEN ${meta.lat - delta} AND ${meta.lat + delta} - AND l.longitude BETWEEN ${meta.lng - delta} AND ${meta.lng + delta} - ${sourceFilterSQL('l.source')} - ${facetSQL} - LIMIT 30 - `; - } else { - nearbyQuery = ` - SELECT pid, label, source, latitude, longitude, place_name - FROM read_parquet('${lite_url}') - WHERE latitude BETWEEN ${meta.lat - delta} AND ${meta.lat + delta} - AND longitude BETWEEN ${meta.lng - delta} AND ${meta.lng + delta} - ${sourceFilterSQL('source')} - LIMIT 30 - `; - } - const samples = await db.query(nearbyQuery); - updateSamples(samples); - } catch(err) { - console.error("Sample query failed:", err); - if (sampEl) sampEl.innerHTML = '
Query failed — try again.
'; - } - } - }, Cesium.ScreenSpaceEventType.LEFT_CLICK); - - // Timing: viewer ready (mount complete, pre-first-render) - performance.mark('viewer-init-end'); - performance.measure('viewer_init', 'viewer-init-start', 'viewer-init-end'); - - // Timing: first Cesium frame painted (globe visible, may be pre-cluster) - const firstGlobeFrame = () => { - performance.mark('first-globe-frame'); - v.scene.postRender.removeEventListener(firstGlobeFrame); - }; - v.scene.postRender.addEventListener(firstGlobeFrame); - - return v; -} -``` - -```{ojs} -//| echo: false -//| output: false - -// === PHASE 1: Load H3 res4 globally (instant) === -phase1 = { - performance.mark('p1-start'); - - // Read ?sources=A,B from query string and pre-check the matching legend - // checkboxes BEFORE running the cluster query (so phase 1 honors the - // bookmarkable filter). Camera state still lives in the hash. - { - const initialSources = (new URLSearchParams(location.search).get('sources') || '') - .split(',').map(s => s.trim()).filter(Boolean); - if (initialSources.length > 0) { - document.querySelectorAll('#sourceFilter input[type="checkbox"]').forEach(cb => { - cb.checked = initialSources.includes(cb.value); - }); - document.querySelectorAll('#sourceFilter .legend-item').forEach(li => { - const cb = li.querySelector('input'); - li.classList.toggle('disabled', !cb.checked); - }); - } - } - - const data = await db.query(` - SELECT h3_cell, sample_count, center_lat, center_lng, - dominant_source, source_count - FROM read_parquet('${h3_res4_url}') - WHERE 1=1${sourceFilterSQL('dominant_source')} - `); - - const scalar = new Cesium.NearFarScalar(1.5e2, 1.5, 8.0e6, 0.5); - let totalSamples = 0; - - for (const row of data) { - const count = row.sample_count; - totalSamples += count; - const size = Math.min(3 + Math.log10(count) * 4, 20); - viewer.h3Points.add({ - id: { count, source: row.dominant_source, lat: row.center_lat, lng: row.center_lng, resolution: 4 }, - position: Cesium.Cartesian3.fromDegrees(row.center_lng, row.center_lat, 0), - pixelSize: size, - color: Cesium.Color.fromCssColorString(SOURCE_COLORS[row.dominant_source] || '#666').withAlpha(0.8), - scaleByDistance: scalar, - }); - } - - // Cache cluster data for viewport counting - viewer._clusterData = Array.from(data); - viewer._clusterTotal = { clusters: data.length, samples: totalSamples }; - - performance.mark('p1-end'); - performance.measure('p1', 'p1-start', 'p1-end'); - const elapsed = performance.getEntriesByName('p1').pop().duration; - - updateStats('H3 Res4', data.length, totalSamples, `${(elapsed/1000).toFixed(1)}s`, 'Clusters Loaded', 'Samples Loaded'); - updatePhaseMsg(`${data.length.toLocaleString()} clusters, ${totalSamples.toLocaleString()} samples. Zoom in for finer detail.`, 'done'); - console.log(`Phase 1: ${data.length} clusters in ${elapsed.toFixed(0)}ms`); - - return { count: data.length, samples: totalSamples }; -} -``` - -```{ojs} -//| echo: false -//| output: false - -// === Load facet summaries + SKOS prefLabels, populate filter checkboxes === -// -// Checkbox semantics: start UNCHECKED (no filter; show everything). User -// checks items to *include only those*. Empty = no filter. This matches the -// prior explorer's UX and keeps the side panel compact at startup. -// -// Checkbox value = full URI (matches the URI strings stored in -// sample_facets_v2.parquet's material / context / object_type columns). -// Display label = SKOS prefLabel (en) when available, URI tail otherwise. -facetFilters = { - if (!phase1) return; - - // Tiny URI → prefLabel lookup. ~60 KB. Best-effort: fallback to URI tail. - const vocabMap = new Map(); - try { - const vocab = await db.query( - `SELECT uri, pref_label FROM read_parquet('${vocab_labels_url}') WHERE lang = 'en'` - ); - for (const r of vocab) vocabMap.set(r.uri, r.pref_label); - } catch (err) { - console.warn("vocab_labels load failed; falling back to URI tails:", err); - } - const prettyLabel = (uri) => { - if (uri == null) return ""; - const hit = vocabMap.get(uri); - if (hit) return hit; - const s = String(uri); - if (!/^https?:\/\//.test(s)) return s; - const parts = s.replace(/[#?].*$/, "").split("/").filter(Boolean); - return parts.length ? parts[parts.length - 1] : s; - }; - - try { - const summaries = await db.query(` - SELECT facet_type, facet_value, count - FROM read_parquet('${facet_summaries_url}') - ORDER BY facet_type, count DESC - `); - - const grouped = { source: [], material: [], context: [], object_type: [] }; - for (const row of summaries) { - if (grouped[row.facet_type]) { - grouped[row.facet_type].push({ - uri: row.facet_value, - label: prettyLabel(row.facet_value), - count: row.count - }); - } - } - - // Stash baseline counts on `viewer` so refreshFacetCounts() can - // restore them when filters return to the no-active-filters state. - // Schema: viewer._baselineCounts[facetKey] = Map(uri → count). - viewer._baselineCounts = { - source: new Map(grouped.source.map(s => [s.uri, s.count])), - material: new Map(grouped.material.map(m => [m.uri, m.count])), - context: new Map(grouped.context.map(c => [c.uri, c.count])), - object_type: new Map(grouped.object_type.map(o => [o.uri, o.count])), - }; - - // Paint baseline counts now so the source legend and (after this - // function's renderFilter calls below) the facet rows show numbers - // immediately, before any user interaction debounces refresh. - // Wrapped in setTimeout(0) so the renderFilter calls below land first. - setTimeout(() => { - applyFacetCounts('source', null); - applyFacetCounts('material', null); - applyFacetCounts('context', null); - applyFacetCounts('object_type', null); - }, 0); - - const escAttr = (s) => String(s).replace(/&/g, '&').replace(/"/g, '"').replace(/ String(s).replace(/&/g, '&').replace(/ { - const body = document.getElementById(bodyId); - if (!body) return; - if (items.length === 0) { - body.innerHTML = 'No values'; - return; - } - body.innerHTML = items.map(it => - `` - ).join(''); - }; - - renderFilter('materialFilterBody', 'material', grouped.material); - renderFilter('contextFilterBody', 'context', grouped.context); - renderFilter('objectTypeFilterBody', 'object_type', grouped.object_type); - - console.log(`Facet filters loaded: ${grouped.material.length} materials, ${grouped.context.length} contexts, ${grouped.object_type.length} object types (vocab labels: ${vocabMap.size})`); - } catch(err) { - console.warn("Facet summaries failed to load:", err); - } - return "loaded"; -} -``` - -```{ojs} -//| echo: false -//| output: false - -// === Zoom watcher: H3 cluster mode + individual sample point mode === -zoomWatcher = { - if (!phase1) return; - if (!facetFilters) return; // wait for facet checkboxes - - // --- State --- - let mode = 'cluster'; // 'cluster' or 'point' - let currentRes = 4; - let loading = false; - let requestId = 0; // stale-request guard - // clusterDataCache stored on viewer._clusterData (set by phase1 and loadRes) - - // Hysteresis thresholds to avoid flicker - const ENTER_POINT_ALT = 120000; // 120 km → enter point mode - const EXIT_POINT_ALT = 180000; // 180 km → exit point mode - // Sample budget is driven by the Max Samples slider; reads at query time. - const getMaxSamples = () => { - const el = document.getElementById('maxSamples'); - const v = el ? parseInt(el.value, 10) : NaN; - return Number.isFinite(v) && v > 0 ? v : 5000; - }; - - // Viewport cache: avoid re-querying same area - let cachedBounds = null; // { south, north, west, east } - let cachedData = null; // array of rows - - // --- H3 cluster loading (existing logic) --- - let loadResGen = 0; // generation counter to discard stale results - const loadRes = async (res, url) => { - const gen = ++loadResGen; // claim a generation - loading = true; - updatePhaseMsg(`Loading H3 res${res}...`, 'loading'); - - try { - performance.mark(`r${res}-s`); - const data = await db.query(` - SELECT h3_cell, sample_count, center_lat, center_lng, - dominant_source, source_count - FROM read_parquet('${url}') - WHERE 1=1${sourceFilterSQL('dominant_source')} - `); - - if (gen !== loadResGen) return; // stale — a newer call superseded this one - viewer.h3Points.removeAll(); - const scalar = new Cesium.NearFarScalar(1.5e2, 1.5, 8.0e6, 0.3); - let total = 0; - - for (const row of data) { - total += row.sample_count; - const size = Math.min(3 + Math.log10(row.sample_count) * 3.5, 18); - viewer.h3Points.add({ - id: { count: row.sample_count, source: row.dominant_source, lat: row.center_lat, lng: row.center_lng, resolution: res }, - position: Cesium.Cartesian3.fromDegrees(row.center_lng, row.center_lat, 0), - pixelSize: size, - color: Cesium.Color.fromCssColorString(SOURCE_COLORS[row.dominant_source] || '#666').withAlpha(0.85), - scaleByDistance: scalar, - }); - } - - // Cache for viewport counting - viewer._clusterData = Array.from(data); - viewer._clusterTotal = { clusters: data.length, samples: total }; - - performance.mark(`r${res}-e`); - performance.measure(`r${res}`, `r${res}-s`, `r${res}-e`); - const elapsed = performance.getEntriesByName(`r${res}`).pop().duration; - - // Show viewport count immediately - const bounds = getViewportBounds(); - const inView = countInViewport(bounds); - updateStats(`H3 Res${res}`, `${inView.clusters.toLocaleString()} / ${data.length.toLocaleString()}`, inView.samples.toLocaleString(), `${(elapsed/1000).toFixed(1)}s`, 'Clusters in View / Loaded', 'Samples in View'); - updatePhaseMsg(`${data.length.toLocaleString()} clusters, ${total.toLocaleString()} samples. ${res < 8 ? 'Zoom in for finer detail.' : 'Zoom closer for individual samples.'}`, 'done'); - - currentRes = res; - console.log(`Res${res}: ${data.length} clusters in ${elapsed.toFixed(0)}ms`); - } catch(err) { - console.error(`Failed to load res${res}:`, err); - updatePhaseMsg(`Failed to load H3 res${res} — try zooming again.`, 'loading'); - } finally { - loading = false; - } - }; - - // --- Get camera viewport bounds --- - function getViewportBounds() { - const rect = viewer.camera.computeViewRectangle(viewer.scene.globe.ellipsoid); - if (!rect) return null; - return { - south: Cesium.Math.toDegrees(rect.south), - north: Cesium.Math.toDegrees(rect.north), - west: Cesium.Math.toDegrees(rect.west), - east: Cesium.Math.toDegrees(rect.east) - }; - } - - // --- Count clusters visible in current viewport (from cached array) --- - function countInViewport(bounds) { - const cache = viewer._clusterData; - if (!bounds || !cache || cache.length === 0) return { clusters: 0, samples: 0 }; - const { south, north, west, east } = bounds; - const wrapLng = west > east; // dateline crossing - let clusters = 0, samples = 0; - for (const row of cache) { - if (row.center_lat < south || row.center_lat > north) continue; - if (wrapLng ? (row.center_lng < west && row.center_lng > east) : (row.center_lng < west || row.center_lng > east)) continue; - clusters++; - samples += row.sample_count; - } - return { clusters, samples }; - } - - // --- Check if viewport is within cached bounds --- - function isWithinCache(bounds) { - if (!cachedBounds || !bounds) return false; - return bounds.south >= cachedBounds.south && - bounds.north <= cachedBounds.north && - bounds.west >= cachedBounds.west && - bounds.east <= cachedBounds.east; - } - - // --- Load individual samples for current viewport --- - async function loadViewportSamples() { - const myReqId = ++requestId; - const bounds = getViewportBounds(); - if (!bounds) return; - - // If viewport is within cached area, just re-render from cache - if (isWithinCache(bounds) && cachedData) { - renderSamplePoints(cachedData, bounds); - return; - } - - // Fetch with 30% padding for smooth panning - const latPad = (bounds.north - bounds.south) * 0.3; - const lngPad = (bounds.east - bounds.west) * 0.3; - const padded = { - south: bounds.south - latPad, - north: bounds.north + latPad, - west: bounds.west - lngPad, - east: bounds.east + lngPad - }; - - updatePhaseMsg('Loading individual samples...', 'loading'); - - try { - performance.mark('sp-s'); - const facetActive = hasFacetFilters(); - const facetSQL = facetActive ? facetFilterSQL() : ''; - let query; - if (facetActive) { - query = ` - SELECT l.pid, l.label, l.source, l.latitude, l.longitude, - l.place_name, l.result_time, f.material, f.context - FROM read_parquet('${lite_url}') l - JOIN read_parquet('${facets_url}') f ON l.pid = f.pid - WHERE l.latitude BETWEEN ${padded.south} AND ${padded.north} - AND l.longitude BETWEEN ${padded.west} AND ${padded.east} - ${sourceFilterSQL('l.source')} - ${facetSQL} - LIMIT ${getMaxSamples()} - `; - } else { - query = ` - SELECT pid, label, source, latitude, longitude, - place_name, result_time - FROM read_parquet('${lite_url}') - WHERE latitude BETWEEN ${padded.south} AND ${padded.north} - AND longitude BETWEEN ${padded.west} AND ${padded.east} - ${sourceFilterSQL('source')} - LIMIT ${getMaxSamples()} - `; - } - const data = await db.query(query); - performance.mark('sp-e'); - performance.measure('sp', 'sp-s', 'sp-e'); - const elapsed = performance.getEntriesByName('sp').pop().duration; - - // Stale guard: discard if a newer request was issued - if (myReqId !== requestId) { - console.log(`Discarding stale sample response (req ${myReqId}, current ${requestId})`); - return; - } - - // Cache the padded bounds + data - cachedBounds = padded; - cachedData = Array.from(data); - - renderSamplePoints(cachedData, bounds); - - updateStats('Samples', cachedData.length, cachedData.length, `${(elapsed/1000).toFixed(1)}s`, 'Samples in View', 'Samples in View'); - updatePhaseMsg(`${cachedData.length.toLocaleString()} individual samples. Click one for details.`, 'done'); - console.log(`Point mode: ${cachedData.length} samples in ${elapsed.toFixed(0)}ms`); - - } catch(err) { - if (myReqId !== requestId) return; - console.error("Viewport sample query failed:", err); - updatePhaseMsg('Sample query failed — try again.', 'loading'); - } - } - - // --- Render sample points on globe --- - function renderSamplePoints(data, bounds) { - viewer.samplePoints.removeAll(); - const scalar = new Cesium.NearFarScalar(1e2, 8, 2e5, 3); - - for (const row of data) { - const color = SOURCE_COLORS[row.source] || '#666'; - viewer.samplePoints.add({ - id: { - type: 'sample', - pid: row.pid, - label: row.label, - source: row.source, - lat: row.latitude, - lng: row.longitude, - place_name: row.place_name, - result_time: row.result_time - }, - position: Cesium.Cartesian3.fromDegrees(row.longitude, row.latitude, 0), - pixelSize: 6, - color: Cesium.Color.fromCssColorString(color).withAlpha(0.9), - scaleByDistance: scalar, - }); - } - } - - // --- Mode transitions --- - function enterPointMode(pushHistory) { - mode = 'point'; - viewer._globeState.mode = 'point'; - viewer.h3Points.show = false; - viewer.samplePoints.show = true; - if (pushHistory !== false) history.pushState(null, '', buildHash(viewer)); - loadViewportSamples(); - console.log('Entered point mode'); - } - - function exitPointMode(pushHistory) { - mode = 'cluster'; - viewer._globeState.mode = 'cluster'; - viewer.samplePoints.show = false; - viewer.samplePoints.removeAll(); - viewer.h3Points.show = true; - if (pushHistory !== false) history.pushState(null, '', buildHash(viewer)); - cachedBounds = null; - cachedData = null; - - // Restore cluster stats with viewport count - const bounds = getViewportBounds(); - const inView = countInViewport(bounds); - const total = viewer._clusterTotal; - if (total) { - updateStats(`H3 Res${currentRes}`, `${inView.clusters.toLocaleString()} / ${total.clusters.toLocaleString()}`, inView.samples.toLocaleString(), '—', 'Clusters in View / Loaded', 'Samples in View'); - } else { - updateStats(`H3 Res${currentRes}`, viewer.h3Points.length, '—', '—', 'Clusters Loaded', 'Samples Loaded'); - } - updatePhaseMsg(`${inView.clusters.toLocaleString()} clusters in view. Zoom closer for individual samples.`, 'done'); - console.log('Exited point mode'); - } - - // --- Results table refresh (below globe) --- - // Independent of globe mode — always shows up to maxSamples rows matching - // filters. Cap rendered rows at TABLE_RENDER_CAP to keep DOM tractable - // (the slider can go up to 100K which is fine for the globe but jank for - // a flat HTML table). - const TABLE_RENDER_CAP = 200; - let tableReqId = 0; - async function refreshResultsTable() { - const myReq = ++tableReqId; - const limit = getMaxSamples(); - const renderLimit = Math.min(limit, TABLE_RENDER_CAP); - updateResultsTableMeta('Loading samples matching filters…', true); - try { - const facetActive = hasFacetFilters(); - const facetSQL = facetActive ? facetFilterSQL() : ''; - let query; - if (facetActive) { - query = ` - SELECT l.pid, l.label, l.source, l.latitude, l.longitude, l.place_name - FROM read_parquet('${lite_url}') l - JOIN read_parquet('${facets_url}') f ON l.pid = f.pid - WHERE 1=1 - ${sourceFilterSQL('l.source')} - ${facetSQL} - LIMIT ${limit} - `; - } else { - query = ` - SELECT pid, label, source, latitude, longitude, place_name - FROM read_parquet('${lite_url}') - WHERE 1=1 - ${sourceFilterSQL('source')} - LIMIT ${limit} - `; - } - const data = await db.query(query); - if (myReq !== tableReqId) return; // stale - const visible = data.slice(0, renderLimit); - renderResultsTable(visible); - let note; - if (data.length === limit) { - note = `Showing first ${visible.length.toLocaleString()} of ${limit.toLocaleString()}+ samples matching filters.`; - } else if (visible.length < data.length) { - note = `Showing first ${visible.length.toLocaleString()} of ${data.length.toLocaleString()} samples matching filters.`; - } else { - note = `Showing ${data.length.toLocaleString()} samples matching filters.`; - } - updateResultsTableMeta(note, false); - } catch (err) { - if (myReq !== tableReqId) return; - console.error('Results table query failed:', err); - updateResultsTableMeta('Failed to load results table.', false); - } - } - - // === Cross-filter facet count refresh (issue #154) === - // - // Strategy: - // - No filters active → restore baseline counts (no query). - // - Exactly one facet value → cache lookup against cross_filter_url. - // - Anything else → on-the-fly group-by on facets_url, four - // concurrent queries (one per target facet), - // each excluding the column being recomputed. - // - // Debounced ~250 ms with a generation guard so rapid clicking only - // resolves the latest selection. Source counts are sample-level - // (not H3 dominant_source counts). - let facetCountsReqId = 0; - let facetCountsDebounce = null; - - function describeActiveFilters() { - const sources = getActiveSources(); - const allSourcesChecked = sources.length === 4; - const mat = getCheckedValues('materialFilterBody'); - const ctx = getCheckedValues('contextFilterBody'); - const ot = getCheckedValues('objectTypeFilterBody'); - const dims = [ - { key: 'source', col: 'source', values: allSourcesChecked ? [] : sources }, - { key: 'material', col: 'material', values: mat }, - { key: 'context', col: 'context', values: ctx }, - { key: 'object_type', col: 'object_type', values: ot }, - ]; - const activeDims = dims.filter(d => d.values.length > 0); - const totalActiveValues = activeDims.reduce((n, d) => n + d.values.length, 0); - return { dims, activeDims, totalActiveValues }; - } - - function buildExcludeWhere(activeDims, excludeKey) { - const conds = activeDims - .filter(d => d.key !== excludeKey) - .map(d => { - const list = d.values.map(v => `'${String(v).replace(/'/g, "''")}'`).join(','); - return `${d.col} IN (${list})`; - }); - return conds.length > 0 ? conds.join(' AND ') : '1=1'; - } - - async function refreshFacetCountsNow(myReq) { - // Stale check up front: if a newer schedule has happened during the - // debounce window, drop this run before doing any work. - if (myReq !== facetCountsReqId) return; - const { dims, activeDims, totalActiveValues } = describeActiveFilters(); - - // Case 1: no filters → restore baseline. - if (activeDims.length === 0) { - for (const d of dims) applyFacetCounts(d.key, null); - return; - } - - markFacetCountsRecomputing(); - - // Case 2: single-filter cache. The pre-aggregated parquet has a - // (filter_source, filter_material, filter_context, filter_object_type) - // schema where exactly one is non-null per row. Use it only when - // exactly one facet value is active across all dims (the cache shape). - const singleActiveDim = activeDims.length === 1 && activeDims[0].values.length === 1 - ? activeDims[0] : null; - if (singleActiveDim) { - try { - const filterCols = ['filter_source', 'filter_material', 'filter_context', 'filter_object_type']; - const filterColForKey = { - source: 'filter_source', - material: 'filter_material', - context: 'filter_context', - object_type: 'filter_object_type', - }; - const targetCol = filterColForKey[singleActiveDim.key]; - const value = String(singleActiveDim.values[0]).replace(/'/g, "''"); - const whereParts = filterCols.map(c => - c === targetCol ? `${c} = '${value}'` : `${c} IS NULL` - ); - const sql = ` - SELECT facet_type, facet_value, count - FROM read_parquet('${cross_filter_url}') - WHERE ${whereParts.join(' AND ')} - `; - const rows = await db.query(sql); - if (myReq !== facetCountsReqId) return; - if (rows && rows.length > 0) { - const grouped = { source: new Map(), material: new Map(), context: new Map(), object_type: new Map() }; - for (const r of rows) { - if (grouped[r.facet_type]) grouped[r.facet_type].set(r.facet_value, Number(r.count)); - } - for (const d of dims) { - if (d.key === singleActiveDim.key) { - // Active dim with no OTHER filters → recomputing - // its own values under "no other dim filters" = - // baseline. Cache also doesn't carry rows for the - // filter-dim's own facet_type (verified empirically). - applyFacetCounts(d.key, null); - } else { - applyFacetCounts(d.key, grouped[d.key]); - } - } - return; - } - // empty → fall through to on-the-fly - } catch (err) { - console.warn('Cross-filter cache lookup failed; falling back to on-the-fly:', err); - } - } - - // Case 3: on-the-fly — four GROUP BY queries against facets_url, each - // excluding the dim being recomputed. Per semantics (A) (issue #154 - // / PR #155 thread): for *every* dim D, including dims with active - // selections, count(value=V) = `WHERE V AND `. - // This makes each value's count answer "how many samples have THIS - // value under my other filters", regardless of which values within - // this dim are currently selected. - const queries = dims.map(async (d) => { - const where = buildExcludeWhere(activeDims, d.key); - const sql = ` - SELECT ${d.col} AS value, COUNT(*) AS count - FROM read_parquet('${facets_url}') - WHERE ${where} AND ${d.col} IS NOT NULL - GROUP BY ${d.col} - `; - try { - const rows = await db.query(sql); - if (myReq !== facetCountsReqId) return; - const map = new Map(); - for (const r of rows) map.set(r.value, Number(r.count)); - applyFacetCounts(d.key, map); - } catch (err) { - if (myReq !== facetCountsReqId) return; - console.warn(`Cross-filter on-the-fly failed for ${d.key}:`, err); - applyFacetCounts(d.key, null); // give up on this dim, leave baseline - } - }); - await Promise.all(queries); - } - - function refreshFacetCounts() { - // Bump the generation synchronously so any in-flight queries - // (whose `myReq` is now strictly less) are invalidated immediately - // — without this, a stale query could finish during the debounce - // window of the new request and pass the gen check, repainting - // stale counts (Codex review on PR #155). - clearTimeout(facetCountsDebounce); - const myReq = ++facetCountsReqId; - facetCountsDebounce = setTimeout(() => { - refreshFacetCountsNow(myReq); - }, 250); - } - - // --- Source filter change handler --- - const resUrls = { 4: h3_res4_url, 6: h3_res6_url, 8: h3_res8_url }; - document.getElementById('sourceFilter').addEventListener('change', async () => { - // Toggle visual state on labels. Per the new source semantics - // (0 == all == 4), don't dim any row when the filter isn't actually - // narrowing the result set; only dim unchecked rows when 1–3 are - // checked. Without this, unchecking all 4 made every label appear - // disabled even though semantically all sources are active. - const active = getActiveSources(); - const filterIsActive = active.length > 0 && active.length < 4; - document.querySelectorAll('#sourceFilter .legend-item').forEach(li => { - const cb = li.querySelector('input'); - li.classList.toggle('disabled', filterIsActive && !cb.checked); - }); - // Persist source filter in URL query string for bookmarkable links - // (e.g. ?sources=OPENCONTEXT). Camera state lives in the hash. - const params = new URLSearchParams(location.search); - if (filterIsActive) { - params.set('sources', active.join(',')); - } else { - params.delete('sources'); - } - const qs = params.toString(); - const newSearch = qs ? `?${qs}` : ''; - if (location.search !== newSearch) { - history.replaceState(null, '', `${location.pathname}${newSearch}${location.hash}`); - } - if (mode === 'cluster') { - loading = false; // allow loadRes to run (gen counter discards stale results) - await loadRes(currentRes, resUrls[currentRes]); - } else { - cachedBounds = null; // force re-query - await loadViewportSamples(); - } - refreshResultsTable(); - refreshFacetCounts(); - }); - - // --- Material/Context/Specimen filter change handler --- - const facetNote = document.getElementById('facetNote'); - function handleFacetFilterChange() { - const active = hasFacetFilters(); - if (facetNote) facetNote.style.display = (active && mode === 'cluster') ? 'block' : 'none'; - if (mode === 'point') { - cachedBounds = null; - loadViewportSamples(); - } - refreshResultsTable(); - refreshFacetCounts(); - } - document.getElementById('materialFilterBody').addEventListener('change', handleFacetFilterChange); - document.getElementById('contextFilterBody').addEventListener('change', handleFacetFilterChange); - document.getElementById('objectTypeFilterBody').addEventListener('change', handleFacetFilterChange); - - // --- Max Samples slider --- - const maxSamplesEl = document.getElementById('maxSamples'); - const maxSamplesValueEl = document.getElementById('maxSamplesValue'); - if (maxSamplesEl && maxSamplesValueEl) { - // Live label while dragging, debounced query on release. - let sliderDebounce = null; - maxSamplesEl.addEventListener('input', () => { - const v = parseInt(maxSamplesEl.value, 10); - maxSamplesValueEl.textContent = Number(v).toLocaleString(); - clearTimeout(sliderDebounce); - sliderDebounce = setTimeout(() => { - if (mode === 'point') { - cachedBounds = null; - loadViewportSamples(); - } - refreshResultsTable(); - }, 300); - }); - } - - // --- Clear Filters button --- - const clearBtn = document.getElementById('clearFiltersBtn'); - if (clearBtn) { - clearBtn.addEventListener('click', () => { - // Strip query string and hash, reload to a clean state. Simplest - // and matches the prior explorer's "" behavior. - location.href = location.pathname; - }); - } - - // Initial table load + initial facet counts (paints baseline counts + - // applies any dimming if a `?sources=` URL param has narrowed the source set). - refreshResultsTable(); - refreshFacetCounts(); - - // --- Camera change handler --- - let timer = null; - viewer.camera.changed.addEventListener(() => { - if (timer) clearTimeout(timer); - timer = setTimeout(async () => { - const h = viewer.camera.positionCartographic.height; - - // Determine target mode with hysteresis - const targetMode = h < ENTER_POINT_ALT ? 'point' - : h > EXIT_POINT_ALT ? 'cluster' - : mode; - - if (targetMode === 'point' && mode !== 'point') { - // Make sure we're at res8 clusters before transitioning - if (currentRes !== 8 && !loading) { - await loadRes(8, h3_res8_url); - } - enterPointMode(); - } else if (targetMode === 'cluster' && mode !== 'cluster') { - exitPointMode(); - // Reload appropriate resolution - const target = h > 3000000 ? 4 : h > 300000 ? 6 : 8; - if (target !== currentRes && !loading) { - await loadRes(target, { 4: h3_res4_url, 6: h3_res6_url, 8: h3_res8_url }[target]); - } - } else if (targetMode === 'point') { - // Already in point mode — update viewport samples - loadViewportSamples(); - } else { - // Cluster mode — check if resolution should change - const target = h > 3000000 ? 4 : h > 300000 ? 6 : 8; - if (target !== currentRes && !loading) { - await loadRes(target, { 4: h3_res4_url, 6: h3_res6_url, 8: h3_res8_url }[target]); - } - } - - // Update viewport cluster count (cluster mode only; point mode already shows viewport count) - if (mode === 'cluster' && viewer._clusterData) { - const bounds = getViewportBounds(); - const inView = countInViewport(bounds); - const total = viewer._clusterTotal; - if (total) { - updateStats(`H3 Res${currentRes}`, `${inView.clusters.toLocaleString()} / ${total.clusters.toLocaleString()}`, inView.samples.toLocaleString(), null, 'Clusters in View / Loaded', 'Samples in View'); - } - } - - // Update URL hash (replaceState for continuous movement) - if (!viewer._suppressHashWrite) { - history.replaceState(null, '', buildHash(viewer)); - } - }, 600); - }); - viewer.camera.percentageChanged = 0.1; - - // --- Handle browser back/forward --- - window.addEventListener('hashchange', async () => { - const state = readHash(); - if (state.lat == null || state.lng == null) return; - - viewer._suppressHashWrite = true; - clearTimeout(viewer._suppressTimer); - viewer.camera.cancelFlight(); - viewer.camera.flyTo({ - destination: Cesium.Cartesian3.fromDegrees(state.lng, state.lat, state.alt || 20000000), - orientation: { - heading: Cesium.Math.toRadians(state.heading), - pitch: Cesium.Math.toRadians(state.pitch) - }, - duration: 1.5, - }); - - // After flight settles, force mode and clear suppress flag - viewer._suppressTimer = setTimeout(() => { - viewer._suppressHashWrite = false; - const s = readHash(); - if (s.mode === 'point' && mode !== 'point') enterPointMode(false); - else if (s.mode !== 'point' && mode === 'point') exitPointMode(false); - }, 2000); - - // Handle pid selection - if (state.pid) { - viewer._globeState.selectedPid = state.pid; - try { - const sample = await db.query(` - SELECT pid, label, source, latitude, longitude, place_name, result_time - FROM read_parquet('${lite_url}') - WHERE pid = '${state.pid.replace(/'/g, "''")}' - LIMIT 1 - `); - if (sample && sample.length > 0) { - const s = sample[0]; - updateSampleCard({ - pid: s.pid, label: s.label, source: s.source, - lat: s.latitude, lng: s.longitude, - place_name: s.place_name, result_time: s.result_time - }); - } - } catch(err) { - console.error("Hash pid query failed:", err); - } - } else { - viewer._globeState.selectedPid = null; - updateClusterCard(null); - } - }); - - // --- Share button --- - const shareBtn = document.getElementById('shareBtn'); - if (shareBtn) { - shareBtn.addEventListener('click', async () => { - history.replaceState(null, '', buildHash(viewer)); - try { - await navigator.clipboard.writeText(location.href); - const toast = document.getElementById('shareToast'); - if (toast) { - toast.style.opacity = '1'; - setTimeout(() => { toast.style.opacity = '0'; }, 2000); - } - } catch(err) { - prompt('Copy this link:', location.href); - } - }); - } - - // --- Search handler --- - const searchBtn = document.getElementById('searchBtn'); - const searchInput = document.getElementById('sampleSearch'); - const searchResults = document.getElementById('searchResults'); - - async function doSearch() { - const term = searchInput.value.trim(); - if (!term || term.length < 2) { - searchResults.textContent = 'Type at least 2 characters'; - return; - } - searchResults.textContent = 'Searching...'; - try { - const terms = searchTerms(term); - // Compose with facet filters so search honors the same Material / - // Sampled Feature / Specimen Type selections that the table and - // point-mode globe use. Without this, search would surface (and - // fly to) samples outside the active filters. - const facetActive = hasFacetFilters(); - const facetSQL = facetActive ? facetFilterSQL() : ''; - const aliasedSearchWhere = textSearchWhere(terms, ['l.label', 'CAST(l.place_name AS VARCHAR)']); - const aliasedScore = textSearchScore(terms, [ - { col: 'l.label', weight: 3 }, - { col: 'CAST(l.place_name AS VARCHAR)', weight: 2 }, - ]); - const searchWhere = textSearchWhere(terms, ['label', 'CAST(place_name AS VARCHAR)']); - const score = textSearchScore(terms, [ - { col: 'label', weight: 3 }, - { col: 'CAST(place_name AS VARCHAR)', weight: 2 }, - ]); - const query = facetActive ? ` - SELECT l.pid, l.label, l.source, l.latitude, l.longitude, l.place_name, - (${aliasedScore}) AS relevance_score - FROM read_parquet('${lite_url}') l - JOIN read_parquet('${facets_url}') f ON l.pid = f.pid - WHERE ${aliasedSearchWhere} - ${sourceFilterSQL('l.source')} - ${facetSQL} - ORDER BY relevance_score DESC, l.label - LIMIT 50 - ` : ` - SELECT pid, label, source, latitude, longitude, place_name, - (${score}) AS relevance_score - FROM read_parquet('${lite_url}') - WHERE ${searchWhere} - ${sourceFilterSQL('source')} - ORDER BY relevance_score DESC, label - LIMIT 50 - `; - const results = await db.query(query); - if (results.length === 0) { - searchResults.textContent = `No results for "${term}"`; - return; - } - searchResults.textContent = `${results.length}${results.length === 50 ? '+' : ''} results for "${term}"`; - - // Show results in the samples panel - const sampEl = document.getElementById('samplesSection'); - if (sampEl) { - let h = `

Search: "${term}" (${results.length})

`; - for (const s of results) { - const color = SOURCE_COLORS[s.source] || '#666'; - const name = SOURCE_NAMES[s.source] || s.source; - const sUrl = sourceUrl(s.pid); - h += `
-
- ${sUrl ? `${s.label || s.pid}` : `${s.label || s.pid}`} - ${name} -
-
`; - } - sampEl.innerHTML = h; - - // Click search result → fly to it - sampEl.querySelectorAll('.sample-row[data-lat]').forEach(row => { - row.addEventListener('click', (e) => { - if (e.target.tagName === 'A') return; // let links work - const lat = parseFloat(row.dataset.lat); - const lng = parseFloat(row.dataset.lng); - const pid = row.dataset.pid; - if (!isNaN(lat) && !isNaN(lng)) { - viewer.camera.flyTo({ - destination: Cesium.Cartesian3.fromDegrees(lng, lat, 50000), - duration: 1.5 - }); - } - }); - }); - } - - // Fly to the first result - if (results[0].latitude && results[0].longitude) { - viewer.camera.flyTo({ - destination: Cesium.Cartesian3.fromDegrees(results[0].longitude, results[0].latitude, 200000), - duration: 1.5 - }); - } - } catch(err) { - console.error("Search failed:", err); - searchResults.textContent = `Search error: ${err.message}`; - } - } - - if (searchBtn) searchBtn.addEventListener('click', doSearch); - if (searchInput) searchInput.addEventListener('keydown', (e) => { - if (e.key === 'Enter') doSearch(); - }); - - // --- Deep-link: restore selection from initial hash --- - const ih = viewer._initialHash; - if (ih.pid) { - viewer._globeState.selectedPid = ih.pid; - try { - const sample = await db.query(` - SELECT pid, label, source, latitude, longitude, place_name, result_time - FROM read_parquet('${lite_url}') - WHERE pid = '${ih.pid.replace(/'/g, "''")}' - LIMIT 1 - `); - if (sample && sample.length > 0) { - const s = sample[0]; - updateSampleCard({ - pid: s.pid, label: s.label, source: s.source, - lat: s.latitude, lng: s.longitude, - place_name: s.place_name, result_time: s.result_time - }); - const detail = await db.query(` - SELECT description FROM read_parquet('${wide_url}') - WHERE pid = '${ih.pid.replace(/'/g, "''")}' - LIMIT 1 - `); - if (detail && detail.length > 0) updateSampleDetail(detail[0]); - else updateSampleDetail({ description: '' }); - } - } catch(err) { - console.error("Deep-link pid query failed:", err); - } - } - - // Enable hash writing now that everything is initialized - viewer._suppressHashWrite = false; - - return "active"; -} -``` - -```{ojs} -//| echo: false -//| output: false - -// === Performance timing panel (opt-in: append ?perf=1 to URL) === -// v0: reads performance.mark/measure entries and renders a small fixed panel. -// Reports navigation→duckdb_init, navigation→viewer_init, phase 1 res4 load, -// and navigation→first paint. Also dumps to console.table for CI / Playwright. -perfPanel = { - if (!phase1) return; // wait for phase 1 to have run - - const params = new URLSearchParams(location.search); - const isOn = params.get('perf') === '1'; - if (!isOn) return; - - // Give first-paint a tick to fire, then collect - await new Promise(r => setTimeout(r, 100)); - - const origin = performance.timeOrigin; - const mark = (name) => { - const e = performance.getEntriesByName(name, 'mark').pop(); - return e ? e.startTime : null; - }; - const measure = (name) => { - const e = performance.getEntriesByName(name, 'measure').pop(); - return e ? e.duration : null; - }; - - // Paint timings from the browser (free, no instrumentation needed) - const paintEntries = performance.getEntriesByType('paint'); - const fcp = paintEntries.find(e => e.name === 'first-contentful-paint')?.startTime; - const fp = paintEntries.find(e => e.name === 'first-paint')?.startTime; - - const rows = [ - ['first-paint (browser)', fp], - ['first-contentful-paint', fcp], - ['duckdb init', measure('duckdb_init')], - ['viewer init', measure('viewer_init')], - ['nav → viewer ready', mark('viewer-init-end')], - ['phase 1 res4 (duration)', measure('p1')], - ['nav → phase 1 complete', mark('p1-end')], - ['nav → first globe frame', mark('first-globe-frame')], - ].filter(([, v]) => v != null); - - // Console table for CI / offline capture - console.table(Object.fromEntries(rows.map(([k, v]) => [k, `${v.toFixed(0)} ms`]))); - - // Render a small floating panel - const fmt = (ms) => ms == null ? '—' : ms >= 1000 ? `${(ms/1000).toFixed(2)} s` : `${ms.toFixed(0)} ms`; - const panel = document.createElement('div'); - panel.id = 'perfPanel'; - panel.style.cssText = ` - position: fixed; bottom: 12px; right: 12px; z-index: 9999; - background: rgba(0,0,0,0.82); color: #e8f5e9; padding: 10px 12px; - border-radius: 6px; font: 11px/1.4 ui-monospace, SFMono-Regular, monospace; - max-width: 320px; box-shadow: 0 2px 12px rgba(0,0,0,0.3); - `; - panel.innerHTML = ` -
- ⏱ Perf timings - -
- - ${rows.map(([label, v]) => ` - - - `).join('')} -
${label}${fmt(v)}
-
timeOrigin: ${new Date(origin).toISOString().split('T')[1].slice(0,12)}
- `; - document.body.appendChild(panel); - panel.querySelector('#perfClose').onclick = () => panel.remove(); - - return "shown"; -} -``` - -## How It Works - -The globe loads progressively: a 580 KB pre-aggregated H3 res4 file paints the world in clusters within ~1 s, finer-grained res6 / res8 files swap in as you zoom, and only when you zoom past 120 km altitude does the explorer switch to individual sample points (capped at the Max Samples slider, viewport-bounded). The results table below the globe queries the 60 MB lite parquet for samples matching your filters — independent of the camera. Filters compose at sample zoom (and in the results table and search): source + material + sampled feature + specimen type all apply via a join on the facets parquet. **At cluster zoom**, the H3 tier files only carry one row per `(h3_cell, dominant_source)`, so source filtering matches on each cell's *dominant* source — two consequences worth knowing: (1) a cell whose dominant source is OpenContext disappears entirely when you uncheck OpenContext, even if some of its samples are from sources that remain checked; (2) a cell whose dominant source is OpenContext stays visible with its full `sample_count` when OpenContext is checked, even though some samples in it may be from sources you unchecked. Material / sampled feature / specimen filters don't apply at cluster zoom at all (the tier files aren't pre-filtered by those facets) — they kick in once you zoom into sample mode, in the results table, and in search. - -| Phase | Data | Size | Points | -|-------|------|------|--------| -| **Instant** | H3 res4 | 580 KB | 38K clusters (continental) | -| **Zoom in** | H3 res6 | 1.6 MB | 112K clusters (city) | -| **Zoom more** | H3 res8 | 2.5 MB | 176K clusters (neighborhood) | -| **Zoom deep** | Map lite | 60 MB (range req.) | Up to Max Samples slider value (default 25K, max 100K) individual samples in viewport | -| **Click sample** | Full dataset | ~280 MB (range req.) | Full metadata for 1 sample | - -All queries run in your browser via DuckDB-WASM with HTTP range requests — only the bytes you need are transferred. No backend. - -## See Also - -- [Progressive Globe](/tutorials/progressive_globe.html) — Same globe rendering without the faceted filter UI -- [Deep-Dive Analysis](/tutorials/zenodo_isamples_analysis.html) — DuckDB-WASM SQL tutorial -- [Why H3?](/tutorials/why_h3.html) — Why hexagonal hierarchical spatial indexing is the right substrate +This page has moved to the [Interactive Explorer](../explorer.html). Redirecting… diff --git a/tutorials/narrow_vs_wide_performance.qmd b/tutorials/narrow_vs_wide_performance.qmd index cbe943f..166e110 100644 --- a/tutorials/narrow_vs_wide_performance.qmd +++ b/tutorials/narrow_vs_wide_performance.qmd @@ -1068,6 +1068,6 @@ The wide format eliminates ~9M edge rows (60% file size reduction). The H3 varia ## See Also -- [Interactive Explorer](/tutorials/progressive_globe.html) - Browse samples on a 3D globe +- [Interactive Explorer](/explorer.html) - Browse samples on a 3D globe - [Search Explorer](isamples_explorer.qmd) - Faceted search and filter across all samples - [Deep-Dive Analysis](zenodo_isamples_analysis.qmd) - Comprehensive DuckDB-WASM tutorial diff --git a/tutorials/parquet_cesium_isamples_wide.qmd b/tutorials/parquet_cesium_isamples_wide.qmd index 7d1a405..66f9825 100644 --- a/tutorials/parquet_cesium_isamples_wide.qmd +++ b/tutorials/parquet_cesium_isamples_wide.qmd @@ -4,16 +4,16 @@ format: html: include-in-header: text: | - - + + --- This page has moved. The **3D Globe Visualization** has been superseded by the -[**Interactive Explorer**](/tutorials/progressive_globe.html), which subsumes +[**Interactive Explorer**](/explorer.html), which subsumes its functionality with zoom-adaptive H3 clustering and sample-level drilldown. If your browser does not redirect automatically, open -[/tutorials/progressive_globe.html](/tutorials/progressive_globe.html). +[/explorer.html](/explorer.html). The original page is archived for historical reference at [/tutorials/archive/parquet_cesium_isamples_wide.html](/tutorials/archive/parquet_cesium_isamples_wide.html). diff --git a/tutorials/progressive_globe.qmd b/tutorials/progressive_globe.qmd index a71f723..a7f9624 100644 --- a/tutorials/progressive_globe.qmd +++ b/tutorials/progressive_globe.qmd @@ -1,2038 +1,22 @@ --- -title: "Interactive Explorer" -subtitle: "Search and explore 6.7 million material samples" -categories: [parquet, spatial, h3, performance, isamples] -sidebar: false -# No TOC: this page is an app, not an article. The right-hand TOC sidebar -# (#quarto-margin-sidebar) was overlapping .side-panel and silently -# intercepting clicks on the Source filter checkboxes — see issue #127. -toc: false +title: "Redirecting…" format: html: + toc: false + page-layout: full include-in-header: text: | - - - - + + +sidebar: false --- - - - - -::: {.callout-note collapse="true"} -## How It Works - -1. **Instant** (<1s): Pre-aggregated H3 res4 summary (580 KB) → 38K colored circles -2. **Zoom in**: Automatically switches to res6 (112K) then res8 (176K) clusters -3. **Zoom deeper** (<120 km): Individual sample points from 60 MB lite parquet -4. **Click**: Cluster info or individual sample card with full metadata -5. **Search**: Find samples by name — results fly to the location on the globe - -Circle size = log(sample count). Color = dominant data source. -::: - - -
- - -
-
- -
-
- - -
-
- - -
-
- -
-
-
-
-
-
Loading...Resolution
-
0Clusters Loaded
-
0Samples Loaded
-
-Load Time
-
-
-
- - - - -
-
-
-
-Material ▾ -
- -
-
-
-Sampled Feature ▾ -
- -
-
-
-Specimen Type ▾ -
- -
- -
- - -
-
-Loading H3 global overview... -
-
-
-
Click a cluster or sample on the globe
-
-
-
-
- -
-
Table view loads samples matching the current filters.
-
-
- - - -
-
- -```{ojs} -//| output: false -Cesium.Ion.defaultAccessToken = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiIwNzk3NjkyMy1iNGI1LTRkN2UtODRiMy04OTYwYWE0N2M3ZTkiLCJpZCI6Njk1MTcsImlhdCI6MTYzMzU0MTQ3N30.e70dpNzOCDRLDGxRguQCC-tRzGzA-23Xgno5lNgCeB4'; -``` - -```{ojs} -//| echo: false -//| output: false - -// === Constants === -R2_BASE = "https://data.isamples.org" -h3_res4_url = `${R2_BASE}/isamples_202601_h3_summary_res4.parquet` -h3_res6_url = `${R2_BASE}/isamples_202601_h3_summary_res6.parquet` -h3_res8_url = `${R2_BASE}/isamples_202601_h3_summary_res8.parquet` -lite_url = `${R2_BASE}/isamples_202601_samples_map_lite.parquet` -// Stable alias that 302-redirects to the current enriched wide parquet -// (isamples_YYYYMM_wide.parquet). Gets OpenContext thumbnails populated. -wide_url = `${R2_BASE}/current/wide.parquet` -// v2 carries object_type alongside material and context (URI-string columns). -facets_url = `${R2_BASE}/isamples_202601_sample_facets_v2.parquet` -facet_summaries_url = `${R2_BASE}/isamples_202601_facet_summaries.parquet` -// Pre-aggregated single-filter cache for fast cross-filtered facet counts. -cross_filter_url = `${R2_BASE}/isamples_202601_facet_cross_filter.parquet` -// SKOS prefLabels for Material / Sampled Feature / Specimen Type URIs. -// ~60 KB lookup; falls back to URI tail if a URI isn't covered. -vocab_labels_url = `${R2_BASE}/vocab_labels.parquet` - -// Canonical palette — see issue #113. Path-relative so this works under -// both isamples.org (custom domain at root) and project-pages fork -// previews (rdhyee.github.io/isamplesorg.github.io/...). -_palette = await import(new URL('../assets/js/source-palette.js', document.baseURI).href) -SOURCE_COLORS = _palette.SOURCE_COLORS -SOURCE_NAMES = _palette.SOURCE_NAMES - -// === Source URL: resolve pid to original repository === -function sourceUrl(pid) { - if (!pid) return null; - // All sources resolve via n2t.net: - // ARK pids (OpenContext, GEOME, Smithsonian) → n2t.net/ark:/... - // IGSN pids (SESAR) → n2t.net/IGSN:... - return `https://n2t.net/${pid}`; -} - -// === Source Filter: get active sources and build SQL clause === -function getActiveSources() { - const checks = document.querySelectorAll('#sourceFilter input[type="checkbox"]'); - return Array.from(checks).filter(c => c.checked).map(c => c.value); -} - -function sourceFilterSQL(col) { - const active = getActiveSources(); - if (active.length === 0) return ' AND 1=0'; // nothing checked = show nothing - if (active.length === 4) return ''; // all checked = no filter - const list = active.map(s => `'${s}'`).join(','); - return ` AND ${col} IN (${list})`; -} - -SOURCE_VALUES = ['SESAR', 'OPENCONTEXT', 'GEOME', 'SMITHSONIAN'] -DEFAULT_POINT_BUDGET = 5000 - -function csvParamValues(params, key) { - if (!params.has(key)) return null; - const raw = params.get(key) || ''; - if (raw.trim() === '') return []; - return raw.split(',').map(s => s.trim()).filter(Boolean); -} - -function updateSourceLegendState() { - document.querySelectorAll('#sourceFilter .legend-item').forEach(li => { - const cb = li.querySelector('input'); - li.classList.toggle('disabled', !cb.checked); - }); -} - -function applyQueryToSourceFilter() { - const params = new URLSearchParams(location.search); - const initialSources = csvParamValues(params, 'sources'); - if (initialSources == null) return; - const allowed = new Set(SOURCE_VALUES); - const selected = new Set(initialSources.filter(s => allowed.has(s))); - document.querySelectorAll('#sourceFilter input[type="checkbox"]').forEach(cb => { - cb.checked = selected.has(cb.value); - }); - updateSourceLegendState(); -} - -function applyQueryToSearch() { - const input = document.getElementById('sampleSearch'); - if (!input) return; - const params = new URLSearchParams(location.search); - const q = params.get('q'); - if (q != null) input.value = q; -} - -function setCheckedValues(containerId, values) { - if (values == null) return; - const selected = new Set(values); - document.querySelectorAll(`#${containerId} input[type="checkbox"]`).forEach(cb => { - cb.checked = selected.has(cb.value); - }); -} - -function applyQueryToFacetFilters() { - const params = new URLSearchParams(location.search); - setCheckedValues('materialFilterBody', csvParamValues(params, 'material')); - setCheckedValues('contextFilterBody', csvParamValues(params, 'context')); - setCheckedValues('objectTypeFilterBody', csvParamValues(params, 'object_type')); -} - -function getMaxSamplesLimit() { - const params = new URLSearchParams(location.search); - return Math.round(parseNum(params.get('maxSamples'), DEFAULT_POINT_BUDGET, 1, 1000000)); -} - -function writeQueryState() { - const params = new URLSearchParams(location.search); - const searchInput = document.getElementById('sampleSearch'); - const q = searchInput ? searchInput.value.trim() : ''; - if (q) params.set('q', q); - else params.delete('q'); - - const activeSources = getActiveSources(); - if (activeSources.length === SOURCE_VALUES.length) params.delete('sources'); - else params.set('sources', activeSources.join(',')); - - [ - ['material', 'materialFilterBody'], - ['context', 'contextFilterBody'], - ['object_type', 'objectTypeFilterBody'], - ].forEach(([key, containerId]) => { - const values = getCheckedValues(containerId); - if (values.length > 0) params.set(key, values.join(',')); - else params.delete(key); - }); - - const maxSamples = getMaxSamplesLimit(); - if (maxSamples !== DEFAULT_POINT_BUDGET) params.set('maxSamples', String(maxSamples)); - else params.delete('maxSamples'); - - if (typeof document !== 'undefined' && document.body && document.body.classList.contains('table-view-active')) { - params.set('view', 'table'); - } else { - params.delete('view'); - } - - const qs = params.toString(); - const url = `${location.pathname}${qs ? `?${qs}` : ''}${location.hash}`; - if (url !== `${location.pathname}${location.search}${location.hash}`) { - history.replaceState(null, '', url); - } -} - -function searchTerms(value) { - return String(value || '').trim().split(/\s+/).filter(Boolean); -} - -function escapeIlikePattern(value) { - return escSql(value).replace(/[\\%_]/g, "\\$&"); -} - -function textSearchWhere(terms, columns) { - return terms.map(raw => { - const term = escapeIlikePattern(raw); - const checks = columns.map(col => `${col} ILIKE '%${term}%' ESCAPE '\\'`); - return `(${checks.join(' OR ')})`; - }).join(' AND '); -} - -function textSearchScore(terms, weightedColumns) { - if (!terms.length) return '0'; - return terms.map(raw => { - const term = escapeIlikePattern(raw); - return weightedColumns.map(({ col, weight }) => - `CASE WHEN ${col} ILIKE '%${term}%' ESCAPE '\\' THEN ${weight} ELSE 0 END` - ).join(' + '); - }).map(score => `(${score})`).join(' + '); -} - -// === Material / Sampled Feature / Specimen Type Filters === -// Checkbox semantics: start UNCHECKED (no filter; show everything). User -// checks items to *include only those*. Empty = no filter. Matches the -// explorer's URI-valued facet UX — with hundreds of materials, defaulting -// to "all checked" would be unusable, and "empty = no filter" is the -// natural reading. See issue #155. -function getCheckedValues(containerId) { - const checks = document.querySelectorAll(`#${containerId} input[type="checkbox"]`); - return Array.from(checks).filter(c => c.checked).map(c => c.value); -} - -function hasFacetFilters() { - return getCheckedValues('materialFilterBody').length > 0 - || getCheckedValues('contextFilterBody').length > 0 - || getCheckedValues('objectTypeFilterBody').length > 0; -} - -function escSql(value) { - return String(value).replace(/'/g, "''"); -} - -// Returns a portable predicate fragment (no outer-table alias dependency) -// that callers append to a WHERE: ` AND ${facetFilterSQL()}`. Uses a -// `pid IN (SELECT pid FROM facets WHERE ...)` subquery so it works -// without a JOIN and avoids duplicate rows from multi-valued facets -// (a sample with two materials would appear twice via JOIN). Required -// for Phase 4's table mode and any non-JOIN caller. See issue #156. -function facetFilterSQL() { - const mat = getCheckedValues('materialFilterBody'); - const ctx = getCheckedValues('contextFilterBody'); - const ot = getCheckedValues('objectTypeFilterBody'); - - const conds = []; - if (mat.length > 0) { - const list = mat.map(s => `'${escSql(s)}'`).join(','); - conds.push(`material IN (${list})`); - } - if (ctx.length > 0) { - const list = ctx.map(s => `'${escSql(s)}'`).join(','); - conds.push(`context IN (${list})`); - } - if (ot.length > 0) { - const list = ot.map(s => `'${escSql(s)}'`).join(','); - conds.push(`object_type IN (${list})`); - } - if (conds.length === 0) return ''; - return ` AND pid IN (SELECT DISTINCT pid FROM read_parquet('${facets_url}') WHERE ${conds.join(' AND ')})`; -} - -// === Cross-filter facet count UI helpers === -function applyFacetCounts(facetKey, countsMap) { - const baseline = (viewer && viewer._baselineCounts) ? viewer._baselineCounts[facetKey] : null; - document.querySelectorAll(`.facet-count[data-facet="${facetKey}"]`).forEach(el => { - const value = el.getAttribute('data-value'); - let count; - if (countsMap) { - count = countsMap.has(value) ? countsMap.get(value) : 0; - } else { - count = baseline ? (baseline.get(value) ?? 0) : 0; - } - el.textContent = `(${Number(count).toLocaleString()})`; - el.classList.remove('recomputing'); - - const row = document.querySelector(`.facet-row[data-facet="${facetKey}"][data-value="${CSS.escape(value)}"]`); - if (row) row.classList.toggle('zero', count === 0); - }); -} - -function markFacetCountsRecomputing() { - document.querySelectorAll('.facet-count').forEach(el => el.classList.add('recomputing')); -} - -// === URL State: encode/decode globe state in hash fragment === -function parseNum(val, def, min, max) { - if (val == null) return def; - const n = parseFloat(val); - if (!Number.isFinite(n)) return def; - if (min != null && n < min) return min; - if (max != null && n > max) return max; - return n; -} - -function readHash() { - const params = new URLSearchParams(location.hash.slice(1)); - return { - v: parseInt(params.get('v')) || 0, - lat: parseNum(params.get('lat'), null, -90, 90), - lng: parseNum(params.get('lng'), null, -180, 180), - alt: parseNum(params.get('alt'), null, 100, 40000000), - heading: parseNum(params.get('heading'), 0, 0, 360), - pitch: parseNum(params.get('pitch'), -90, -90, 0), - mode: params.get('mode') || null, - pid: params.get('pid') || null, - }; -} - -function buildHash(v) { - const cam = v.camera; - const carto = cam.positionCartographic; - const params = new URLSearchParams(); - params.set('v', '1'); - params.set('lat', Cesium.Math.toDegrees(carto.latitude).toFixed(4)); - params.set('lng', Cesium.Math.toDegrees(carto.longitude).toFixed(4)); - params.set('alt', Math.round(carto.height).toString()); - const heading = Cesium.Math.toDegrees(cam.heading) % 360; - const pitch = Cesium.Math.toDegrees(cam.pitch); - if (Math.abs(heading) > 1) params.set('heading', heading.toFixed(1)); - if (Math.abs(pitch + 90) > 1) params.set('pitch', pitch.toFixed(1)); - const gs = v._globeState; - if (gs.mode === 'point') params.set('mode', 'point'); - if (gs.selectedPid) params.set('pid', gs.selectedPid); - return '#' + params.toString(); -} - -// === Helpers: update DOM imperatively (no OJS reactivity) === -function updateStats(phase, points, samples, time, pointsLabel, samplesLabel) { - const s = (id, v) => { const e = document.getElementById(id); if (e) e.textContent = v; }; - s('sPhase', phase); - s('sPoints', typeof points === 'string' ? points : points.toLocaleString()); - s('sSamples', typeof samples === 'string' ? samples : samples.toLocaleString()); - if (time != null) s('sTime', time); - if (pointsLabel) s('sPointsLbl', pointsLabel); - if (samplesLabel) s('sSamplesLbl', samplesLabel); -} - -function updatePhaseMsg(text, type) { - const m = document.getElementById('phaseMsg'); - if (!m) return; - m.textContent = text; - if (type === 'loading') { m.style.background = '#e3f2fd'; m.style.color = '#1565c0'; } - else { m.style.background = '#e8f5e9'; m.style.color = '#2e7d32'; } -} - -function updateClusterCard(info) { - const el = document.getElementById('clusterSection'); - if (!el) return; - if (!info) { - el.innerHTML = '
Click a cluster or sample on the globe
'; - return; - } - const color = SOURCE_COLORS[info.source] || '#666'; - const name = SOURCE_NAMES[info.source] || info.source; - el.innerHTML = `

Selected Cluster

-
-
- ${name} - H3 res${info.resolution} -
-
- ${info.count.toLocaleString()} samples -
-
- ${info.lat.toFixed(4)}, ${info.lng.toFixed(4)} -
-
`; -} - -function updateSampleCard(sample) { - const el = document.getElementById('clusterSection'); - if (!el) return; - const color = SOURCE_COLORS[sample.source] || '#666'; - const name = SOURCE_NAMES[sample.source] || sample.source; - const placeParts = sample.place_name; - const placeStr = Array.isArray(placeParts) && placeParts.length > 0 - ? placeParts.filter(Boolean).join(' › ') - : ''; - const srcUrl = sourceUrl(sample.pid); - el.innerHTML = `

Sample

-
-
- ${name} -
-
- ${sample.label || sample.pid || 'Unnamed'} -
-
- ${sample.lat.toFixed(5)}, ${sample.lng.toFixed(5)} -
- ${placeStr ? `
${placeStr}
` : ''} - ${sample.result_time ? `
Date: ${sample.result_time}
` : ''} - ${srcUrl ? `
View at ${name} →
` : ''} -
Loading full details...
-
`; -} - -function updateSampleDetail(detail) { - const el = document.getElementById('sampleDetail'); - if (!el) return; - if (!detail) { - el.innerHTML = 'Detail query failed'; - return; - } - const desc = detail.description - ? (detail.description.length > 300 ? detail.description.slice(0, 300) + '...' : detail.description) - : ''; - el.innerHTML = `${desc ? `
${desc}
` : ''}`; -} - -function updateSamples(samples) { - const el = document.getElementById('samplesSection'); - if (!el) return; - if (!samples || samples.length === 0) { - el.innerHTML = ''; - return; - } - let h = `

Nearby Samples (${samples.length})

`; - for (const s of samples) { - const color = SOURCE_COLORS[s.source] || '#666'; - const name = SOURCE_NAMES[s.source] || s.source; - const placeParts = s.place_name; - const desc = Array.isArray(placeParts) && placeParts.length > 0 - ? placeParts.filter(Boolean).join(' › ') - : ''; - const sUrl = sourceUrl(s.pid); - h += `
-
- ${sUrl ? `${s.label || s.pid}` : `${s.label || s.pid}`} - ${name} -
- ${desc ? `
${desc}
` : ''} -
`; - } - el.innerHTML = h; -} - -// === Binary Globe/Table view === -TABLE_PAGE_SIZE = 100 -TABLE_DEFAULT_MAX = 25000 -TABLE_MIN_MAX = 1000 -TABLE_MAX_MAX = 100000 - -function escapeHtml(value) { - return String(value ?? '') - .replace(/&/g, '&') - .replace(//g, '>') - .replace(/"/g, '"') - .replace(/'/g, '''); -} - -function clampTableMaxSamples(value) { - const n = parseInt(value, 10); - if (!Number.isFinite(n)) return TABLE_DEFAULT_MAX; - return Math.min(TABLE_MAX_MAX, Math.max(TABLE_MIN_MAX, n)); -} - -function getTableMaxSamples() { - const el = document.getElementById('maxSamples'); - const value = clampTableMaxSamples(el ? el.value : TABLE_DEFAULT_MAX); - if (el && String(value) !== String(el.value)) el.value = value; - return value; -} - -function isTableViewActive() { - return document.body.classList.contains('table-view-active'); -} -``` - -```{ojs} -//| echo: false -//| output: false - -// === DuckDB === -db = { - performance.mark('duckdb-init-start'); - const instance = await DuckDBClient.of(); - performance.mark('duckdb-init-end'); - performance.measure('duckdb_init', 'duckdb-init-start', 'duckdb-init-end'); - return instance; -} -``` - -```{ojs} -//| echo: false -//| output: false - -// === Cesium Viewer (created once, never re-created) === -viewer = { - performance.mark('viewer-init-start'); - const v = new Cesium.Viewer("cesiumContainer", { - timeline: false, - animation: false, - baseLayerPicker: false, - fullscreenElement: "cesiumContainer", - terrain: Cesium.Terrain.fromWorldTerrain() - }); - - // URL deep-link state (must be set before globalRect/once block reads it) - v._globeState = { mode: 'cluster', selectedPid: null }; - v._initialHash = readHash(); - v._suppressHashWrite = true; // cleared after zoomWatcher initializes - v._suppressTimer = null; - - const globalRect = Cesium.Rectangle.fromDegrees(-180, -60, 180, 80); - Cesium.Camera.DEFAULT_VIEW_RECTANGLE = globalRect; - Cesium.Camera.DEFAULT_VIEW_FACTOR = 0.5; - const ih = v._initialHash; - const once = () => { - if (ih.lat != null && ih.lng != null) { - v.camera.setView({ - destination: Cesium.Cartesian3.fromDegrees(ih.lng, ih.lat, ih.alt || 20000000), - orientation: { - heading: Cesium.Math.toRadians(ih.heading), - pitch: Cesium.Math.toRadians(ih.pitch) - } - }); - } else { - v.camera.setView({ destination: globalRect }); - } - v.scene.postRender.removeEventListener(once); - }; - v.scene.postRender.addEventListener(once); - - // Two separate point collections: clusters and individual samples - v.h3Points = new Cesium.PointPrimitiveCollection(); - v.scene.primitives.add(v.h3Points); - - v.samplePoints = new Cesium.PointPrimitiveCollection(); - v.scene.primitives.add(v.samplePoints); - v.samplePoints.show = false; // hidden until point mode - - // Hover tooltip — works for both clusters and samples - v.pointLabel = v.entities.add({ - label: { - show: false, showBackground: true, font: "13px monospace", - horizontalOrigin: Cesium.HorizontalOrigin.LEFT, - verticalOrigin: Cesium.VerticalOrigin.BOTTOM, - pixelOffset: new Cesium.Cartesian2(15, 0), - disableDepthTestDistance: Number.POSITIVE_INFINITY, text: "", - } - }); - - new Cesium.ScreenSpaceEventHandler(v.scene.canvas).setInputAction((movement) => { - const picked = v.scene.pick(movement.endPosition); - if (Cesium.defined(picked) && picked.primitive && picked.id) { - v.pointLabel.position = picked.primitive.position; - v.pointLabel.label.show = true; - const meta = picked.id; - if (typeof meta === 'object' && meta.type === 'sample') { - v.pointLabel.label.text = `${meta.label || meta.pid}`; - } else if (typeof meta === 'object' && meta.count) { - v.pointLabel.label.text = `${meta.source}: ${meta.count.toLocaleString()} samples`; - } else { - v.pointLabel.label.text = String(meta); - } - } else { - v.pointLabel.label.show = false; - } - }, Cesium.ScreenSpaceEventType.MOUSE_MOVE); - - // Click handler — routes to cluster card or sample card - new Cesium.ScreenSpaceEventHandler(v.scene.canvas).setInputAction(async (e) => { - const picked = v.scene.pick(e.position); - if (!Cesium.defined(picked) || !picked.primitive || !picked.id) return; - const meta = picked.id; - - if (typeof meta === 'object' && meta.type === 'sample') { - // --- Individual sample click --- - updateSampleCard(meta); - v._globeState.selectedPid = meta.pid; - history.pushState(null, '', buildHash(v)); - // Clear nearby list - const sampEl = document.getElementById('samplesSection'); - if (sampEl) sampEl.innerHTML = ''; - - // Stage 2: lazy-load full description from wide parquet - try { - const detail = await db.query(` - SELECT description - FROM read_parquet('${wide_url}') - WHERE pid = '${meta.pid.replace(/'/g, "''")}' - LIMIT 1 - `); - if (detail && detail.length > 0) { - updateSampleDetail(detail[0]); - } else { - updateSampleDetail({ description: '' }); - } - } catch(err) { - console.error("Detail query failed:", err); - updateSampleDetail(null); - } - - } else if (typeof meta === 'object' && meta.count) { - // --- Cluster click --- - updateClusterCard(meta); - v._globeState.selectedPid = null; - history.pushState(null, '', buildHash(v)); - - const sampEl = document.getElementById('samplesSection'); - if (sampEl) sampEl.innerHTML = '
Loading nearby samples...
'; - - const delta = meta.resolution === 4 ? 2.0 : meta.resolution === 6 ? 0.5 : 0.1; - try { - // facetFilterSQL() returns a portable `pid IN (...)` predicate, - // so the same query works whether or not facet filters are active. - const nearbyQuery = ` - SELECT pid, label, source, latitude, longitude, place_name - FROM read_parquet('${lite_url}') - WHERE latitude BETWEEN ${meta.lat - delta} AND ${meta.lat + delta} - AND longitude BETWEEN ${meta.lng - delta} AND ${meta.lng + delta} - ${sourceFilterSQL('source')} - ${facetFilterSQL()} - LIMIT 30 - `; - const samples = await db.query(nearbyQuery); - updateSamples(samples); - } catch(err) { - console.error("Sample query failed:", err); - if (sampEl) sampEl.innerHTML = '
Query failed — try again.
'; - } - } - }, Cesium.ScreenSpaceEventType.LEFT_CLICK); - - // Timing: viewer ready (mount complete, pre-first-render) - performance.mark('viewer-init-end'); - performance.measure('viewer_init', 'viewer-init-start', 'viewer-init-end'); - - // Timing: first Cesium frame painted (globe visible, may be pre-cluster) - const firstGlobeFrame = () => { - performance.mark('first-globe-frame'); - v.scene.postRender.removeEventListener(firstGlobeFrame); - }; - v.scene.postRender.addEventListener(firstGlobeFrame); - - return v; -} -``` - -```{ojs} -//| echo: false -//| output: false - -// === PHASE 1: Load H3 res4 globally (instant) === -phase1 = { - performance.mark('p1-start'); - applyQueryToSearch(); - applyQueryToSourceFilter(); - - const data = await db.query(` - SELECT h3_cell, sample_count, center_lat, center_lng, - dominant_source, source_count - FROM read_parquet('${h3_res4_url}') - WHERE 1=1${sourceFilterSQL('dominant_source')} - `); - - const scalar = new Cesium.NearFarScalar(1.5e2, 1.5, 8.0e6, 0.5); - let totalSamples = 0; - - for (const row of data) { - const count = row.sample_count; - totalSamples += count; - const size = Math.min(3 + Math.log10(count) * 4, 20); - viewer.h3Points.add({ - id: { count, source: row.dominant_source, lat: row.center_lat, lng: row.center_lng, resolution: 4 }, - position: Cesium.Cartesian3.fromDegrees(row.center_lng, row.center_lat, 0), - pixelSize: size, - color: Cesium.Color.fromCssColorString(SOURCE_COLORS[row.dominant_source] || '#666').withAlpha(0.8), - scaleByDistance: scalar, - }); - } - - // Cache cluster data for viewport counting - viewer._clusterData = Array.from(data); - viewer._clusterTotal = { clusters: data.length, samples: totalSamples }; - - performance.mark('p1-end'); - performance.measure('p1', 'p1-start', 'p1-end'); - const elapsed = performance.getEntriesByName('p1').pop().duration; - - updateStats('H3 Res4', data.length, totalSamples, `${(elapsed/1000).toFixed(1)}s`, 'Clusters Loaded', 'Samples Loaded'); - updatePhaseMsg(`${data.length.toLocaleString()} clusters, ${totalSamples.toLocaleString()} samples. Zoom in for finer detail.`, 'done'); - console.log(`Phase 1: ${data.length} clusters in ${elapsed.toFixed(0)}ms`); - - return { count: data.length, samples: totalSamples }; -} -``` - -```{ojs} -//| echo: false -//| output: false - -// === Load facet summaries + SKOS prefLabels, populate filter checkboxes === -// -// Checkbox value = full URI (matches the URI strings stored in -// sample_facets_v2.parquet's material / context / object_type columns). -// Display label = SKOS prefLabel (en) when available, URI tail otherwise. -// Default state: UNCHECKED — empty = no filter. -facetFilters = { - if (!phase1) return; - - // Tiny URI → prefLabel lookup. ~60 KB. Best-effort: fallback to URI tail. - const vocabMap = new Map(); - try { - const vocab = await db.query( - `SELECT uri, pref_label FROM read_parquet('${vocab_labels_url}') WHERE lang = 'en'` - ); - for (const r of vocab) vocabMap.set(r.uri, r.pref_label); - } catch (err) { - console.warn("vocab_labels load failed; falling back to URI tails:", err); - } - const prettyLabel = (uri) => { - if (uri == null) return ""; - const hit = vocabMap.get(uri); - if (hit) return hit; - const s = String(uri); - if (!/^https?:\/\//.test(s)) return s; - const parts = s.replace(/[#?].*$/, "").split("/").filter(Boolean); - return parts.length ? parts[parts.length - 1] : s; - }; - - try { - const summaries = await db.query(` - SELECT facet_type, facet_value, count - FROM read_parquet('${facet_summaries_url}') - ORDER BY facet_type, count DESC - `); - - const grouped = { source: [], material: [], context: [], object_type: [] }; - for (const row of summaries) { - if (grouped[row.facet_type]) { - grouped[row.facet_type].push({ - uri: row.facet_value, - label: prettyLabel(row.facet_value), - count: row.count - }); - } - } - - viewer._baselineCounts = { - source: new Map(grouped.source.map(s => [s.uri, s.count])), - material: new Map(grouped.material.map(m => [m.uri, m.count])), - context: new Map(grouped.context.map(c => [c.uri, c.count])), - object_type: new Map(grouped.object_type.map(o => [o.uri, o.count])), - }; - - // HTML attribute / text escapers for safety when interpolating URIs. - const escAttr = (s) => String(s).replace(/&/g, '&').replace(/"/g, '"').replace(/ String(s).replace(/&/g, '&').replace(/ { - const body = document.getElementById(bodyId); - if (!body) return; - if (items.length === 0) { - body.innerHTML = 'No values'; - return; - } - body.innerHTML = items.map(it => - `` - ).join(''); - }; - - renderFilter('materialFilterBody', 'material', grouped.material); - renderFilter('contextFilterBody', 'context', grouped.context); - renderFilter('objectTypeFilterBody', 'object_type', grouped.object_type); - applyFacetCounts('source', null); - applyQueryToFacetFilters(); - - console.log(`Facet filters loaded: ${grouped.material.length} materials, ${grouped.context.length} contexts, ${grouped.object_type.length} object types (vocab labels: ${vocabMap.size})`); - } catch(err) { - console.warn("Facet summaries failed to load:", err); - } - return "loaded"; -} -``` - -```{ojs} -//| echo: false -//| output: false - -// === Table view: paginated sample rows matching current filters === -tableView = { - if (!facetFilters) return; - - let rows = []; - let page = 0; - let requestId = 0; - let loadedMax = 0; - let hitHardCap = false; - let tableDirty = true; - - const globeLayout = document.querySelector('.globe-layout'); - const tableContainer = document.getElementById('tableContainer'); - const tableControls = document.getElementById('tableControls'); - const globeBtn = document.getElementById('globeViewBtn'); - const tableBtn = document.getElementById('tableViewBtn'); - const maxInput = document.getElementById('maxSamples'); - const prevBtn = document.getElementById('tablePrev'); - const nextBtn = document.getElementById('tableNext'); - const metaEl = document.getElementById('tableMeta'); - const pageInfoEl = document.getElementById('tablePageInfo'); - const tableEl = document.getElementById('samplesTable'); - - function setMeta(text, loading) { - if (!metaEl) return; - metaEl.textContent = text; - metaEl.style.color = loading ? '#1565c0' : '#555'; - } - - function tableSourceBadge(source) { - const color = SOURCE_COLORS[source] || '#666'; - const name = SOURCE_NAMES[source] || source || ''; - return `${escapeHtml(name)}`; - } - - function renderTable() { - const totalPages = Math.max(1, Math.ceil(rows.length / TABLE_PAGE_SIZE)); - page = Math.min(page, totalPages - 1); - const start = page * TABLE_PAGE_SIZE; - const visible = rows.slice(start, start + TABLE_PAGE_SIZE); - - if (!tableEl) return; - if (visible.length === 0) { - tableEl.innerHTML = '
No samples match the current filters.
'; - } else { - const body = visible.map(r => { - const placeParts = r.place_name; - const place = Array.isArray(placeParts) && placeParts.length > 0 - ? placeParts.filter(Boolean).join(' › ') - : ''; - const lat = r.latitude != null ? Number(r.latitude).toFixed(5) : ''; - const lng = r.longitude != null ? Number(r.longitude).toFixed(5) : ''; - const label = r.label || r.pid || ''; - const url = sourceUrl(r.pid); - const labelHtml = url - ? `${escapeHtml(label)}` - : escapeHtml(label); - return ` - ${tableSourceBadge(r.source)} - ${labelHtml} - ${escapeHtml(place)} - ${escapeHtml(r.result_time || '')} - ${escapeHtml(lat)} - ${escapeHtml(lng)} - `; - }).join(''); - tableEl.innerHTML = `
- - - ${body} -
SourceLabelPlaceDateLatLon
-
`; - } - - if (pageInfoEl) { - const first = rows.length === 0 ? 0 : start + 1; - const last = Math.min(rows.length, start + visible.length); - pageInfoEl.textContent = rows.length === 0 - ? 'Page 0 of 0' - : `Page ${page + 1} of ${totalPages} (${first.toLocaleString()}-${last.toLocaleString()} of ${rows.length.toLocaleString()})`; - } - if (prevBtn) prevBtn.disabled = page <= 0; - if (nextBtn) nextBtn.disabled = page >= totalPages - 1; - } - - async function refreshTable() { - const myReq = ++requestId; - loadedMax = getTableMaxSamples(); - page = 0; - setMeta(`Loading up to ${loadedMax.toLocaleString()} samples matching filters...`, true); - - try { - const data = await db.query(` - SELECT pid, label, source, latitude, longitude, place_name, result_time - FROM read_parquet('${lite_url}') - WHERE 1=1 - ${sourceFilterSQL('source')} - ${facetFilterSQL()} - LIMIT ${loadedMax} - `); - if (myReq !== requestId) return; - const arr = Array.from(data); - hitHardCap = arr.length === loadedMax; - rows = arr; - tableDirty = false; - renderTable(); - const capText = hitHardCap - ? (loadedMax < TABLE_MAX_MAX - ? ` Max samples cap reached; raise it to inspect more rows.` - : ` Maximum table cap reached.`) - : ''; - setMeta(`Loaded ${rows.length.toLocaleString()} sample rows.${capText}`, false); - } catch (err) { - if (myReq !== requestId) return; - console.error('Table query failed:', err); - rows = []; - renderTable(); - setMeta('Table query failed; adjust filters and try again.', false); - } - } - - function setView(mode, updateUrl) { - const tableMode = mode === 'table'; - document.body.classList.toggle('table-view-active', tableMode); - if (globeLayout) globeLayout.style.display = tableMode ? 'none' : ''; - if (tableContainer) tableContainer.style.display = tableMode ? 'block' : 'none'; - if (tableControls) tableControls.style.display = tableMode ? 'flex' : 'none'; - if (globeBtn) { - globeBtn.classList.toggle('active', !tableMode); - globeBtn.setAttribute('aria-pressed', String(!tableMode)); - } - if (tableBtn) { - tableBtn.classList.toggle('active', tableMode); - tableBtn.setAttribute('aria-pressed', String(tableMode)); - } - if (updateUrl) writeQueryState(); - if (tableMode && (tableDirty || rows.length === 0)) refreshTable(); - if (!tableMode && typeof viewer !== 'undefined') { - setTimeout(() => viewer.resize(), 0); - } - } - - if (globeBtn) globeBtn.addEventListener('click', () => setView('globe', true)); - if (tableBtn) tableBtn.addEventListener('click', () => setView('table', true)); - if (prevBtn) prevBtn.addEventListener('click', () => { page = Math.max(0, page - 1); renderTable(); }); - if (nextBtn) nextBtn.addEventListener('click', () => { page += 1; renderTable(); }); - if (maxInput) { - maxInput.addEventListener('change', () => { - maxInput.value = getTableMaxSamples(); - if (isTableViewActive()) refreshTable(); - }); - } - - function handleTableFilterChange() { - tableDirty = true; - if (isTableViewActive()) refreshTable(); - } - - document.getElementById('sourceFilter')?.addEventListener('change', handleTableFilterChange); - document.getElementById('materialFilterBody')?.addEventListener('change', handleTableFilterChange); - document.getElementById('contextFilterBody')?.addEventListener('change', handleTableFilterChange); - document.getElementById('objectTypeFilterBody')?.addEventListener('change', handleTableFilterChange); - - window.refreshSamplesTable = refreshTable; - const params = new URLSearchParams(location.search); - setView(params.get('view') === 'table' ? 'table' : 'globe', false); - - return "active"; -} -``` - -```{ojs} -//| echo: false -//| output: false - -// === Zoom watcher: H3 cluster mode + individual sample point mode === -zoomWatcher = { - if (!phase1) return; - if (!facetFilters) return; // wait for facet checkboxes - - // --- State --- - let mode = 'cluster'; // 'cluster' or 'point' - let currentRes = 4; - let loading = false; - let requestId = 0; // stale-request guard - // clusterDataCache stored on viewer._clusterData (set by phase1 and loadRes) - - // Hysteresis thresholds to avoid flicker - const ENTER_POINT_ALT = 120000; // 120 km → enter point mode - const EXIT_POINT_ALT = 180000; // 180 km → exit point mode - const POINT_BUDGET = getMaxSamplesLimit(); - - // Viewport cache: avoid re-querying same area - let cachedBounds = null; // { south, north, west, east } - let cachedData = null; // array of rows - - // --- H3 cluster loading (existing logic) --- - let loadResGen = 0; // generation counter to discard stale results - const loadRes = async (res, url) => { - const gen = ++loadResGen; // claim a generation - loading = true; - updatePhaseMsg(`Loading H3 res${res}...`, 'loading'); - - try { - performance.mark(`r${res}-s`); - const data = await db.query(` - SELECT h3_cell, sample_count, center_lat, center_lng, - dominant_source, source_count - FROM read_parquet('${url}') - WHERE 1=1${sourceFilterSQL('dominant_source')} - `); - - if (gen !== loadResGen) return; // stale — a newer call superseded this one - viewer.h3Points.removeAll(); - const scalar = new Cesium.NearFarScalar(1.5e2, 1.5, 8.0e6, 0.3); - let total = 0; - - for (const row of data) { - total += row.sample_count; - const size = Math.min(3 + Math.log10(row.sample_count) * 3.5, 18); - viewer.h3Points.add({ - id: { count: row.sample_count, source: row.dominant_source, lat: row.center_lat, lng: row.center_lng, resolution: res }, - position: Cesium.Cartesian3.fromDegrees(row.center_lng, row.center_lat, 0), - pixelSize: size, - color: Cesium.Color.fromCssColorString(SOURCE_COLORS[row.dominant_source] || '#666').withAlpha(0.85), - scaleByDistance: scalar, - }); - } - - // Cache for viewport counting - viewer._clusterData = Array.from(data); - viewer._clusterTotal = { clusters: data.length, samples: total }; - - performance.mark(`r${res}-e`); - performance.measure(`r${res}`, `r${res}-s`, `r${res}-e`); - const elapsed = performance.getEntriesByName(`r${res}`).pop().duration; - - // Show viewport count immediately - const bounds = getViewportBounds(); - const inView = countInViewport(bounds); - updateStats(`H3 Res${res}`, `${inView.clusters.toLocaleString()} / ${data.length.toLocaleString()}`, inView.samples.toLocaleString(), `${(elapsed/1000).toFixed(1)}s`, 'Clusters in View / Loaded', 'Samples in View'); - updatePhaseMsg(`${data.length.toLocaleString()} clusters, ${total.toLocaleString()} samples. ${res < 8 ? 'Zoom in for finer detail.' : 'Zoom closer for individual samples.'}`, 'done'); - - currentRes = res; - console.log(`Res${res}: ${data.length} clusters in ${elapsed.toFixed(0)}ms`); - } catch(err) { - console.error(`Failed to load res${res}:`, err); - updatePhaseMsg(`Failed to load H3 res${res} — try zooming again.`, 'loading'); - } finally { - loading = false; - } - }; - - // --- Get camera viewport bounds --- - function getViewportBounds() { - const rect = viewer.camera.computeViewRectangle(viewer.scene.globe.ellipsoid); - if (!rect) return null; - return { - south: Cesium.Math.toDegrees(rect.south), - north: Cesium.Math.toDegrees(rect.north), - west: Cesium.Math.toDegrees(rect.west), - east: Cesium.Math.toDegrees(rect.east) - }; - } - - // --- Count clusters visible in current viewport (from cached array) --- - function countInViewport(bounds) { - const cache = viewer._clusterData; - if (!bounds || !cache || cache.length === 0) return { clusters: 0, samples: 0 }; - const { south, north, west, east } = bounds; - const wrapLng = west > east; // dateline crossing - let clusters = 0, samples = 0; - for (const row of cache) { - if (row.center_lat < south || row.center_lat > north) continue; - if (wrapLng ? (row.center_lng < west && row.center_lng > east) : (row.center_lng < west || row.center_lng > east)) continue; - clusters++; - samples += row.sample_count; - } - return { clusters, samples }; - } - - // --- Check if viewport is within cached bounds --- - function isWithinCache(bounds) { - if (!cachedBounds || !bounds) return false; - return bounds.south >= cachedBounds.south && - bounds.north <= cachedBounds.north && - bounds.west >= cachedBounds.west && - bounds.east <= cachedBounds.east; - } - - // --- Load individual samples for current viewport --- - async function loadViewportSamples() { - const myReqId = ++requestId; - const bounds = getViewportBounds(); - if (!bounds) return; - - // If viewport is within cached area, just re-render from cache - if (isWithinCache(bounds) && cachedData) { - renderSamplePoints(cachedData, bounds); - return; - } - - // Fetch with 30% padding for smooth panning - const latPad = (bounds.north - bounds.south) * 0.3; - const lngPad = (bounds.east - bounds.west) * 0.3; - const padded = { - south: bounds.south - latPad, - north: bounds.north + latPad, - west: bounds.west - lngPad, - east: bounds.east + lngPad - }; - - updatePhaseMsg('Loading individual samples...', 'loading'); - - try { - performance.mark('sp-s'); - // facetFilterSQL() returns a portable `pid IN (...)` predicate, - // so the same query works whether or not facet filters are active. - const query = ` - SELECT pid, label, source, latitude, longitude, - place_name, result_time - FROM read_parquet('${lite_url}') - WHERE latitude BETWEEN ${padded.south} AND ${padded.north} - AND longitude BETWEEN ${padded.west} AND ${padded.east} - ${sourceFilterSQL('source')} - ${facetFilterSQL()} - LIMIT ${POINT_BUDGET} - `; - const data = await db.query(query); - performance.mark('sp-e'); - performance.measure('sp', 'sp-s', 'sp-e'); - const elapsed = performance.getEntriesByName('sp').pop().duration; - - // Stale guard: discard if a newer request was issued - if (myReqId !== requestId) { - console.log(`Discarding stale sample response (req ${myReqId}, current ${requestId})`); - return; - } - - // Cache the padded bounds + data - cachedBounds = padded; - cachedData = Array.from(data); - - renderSamplePoints(cachedData, bounds); - - updateStats('Samples', cachedData.length, cachedData.length, `${(elapsed/1000).toFixed(1)}s`, 'Samples in View', 'Samples in View'); - updatePhaseMsg(`${cachedData.length.toLocaleString()} individual samples. Click one for details.`, 'done'); - console.log(`Point mode: ${cachedData.length} samples in ${elapsed.toFixed(0)}ms`); - - } catch(err) { - if (myReqId !== requestId) return; - console.error("Viewport sample query failed:", err); - updatePhaseMsg('Sample query failed — try again.', 'loading'); - } - } - - // --- Render sample points on globe --- - function renderSamplePoints(data, bounds) { - viewer.samplePoints.removeAll(); - const scalar = new Cesium.NearFarScalar(1e2, 8, 2e5, 3); - - for (const row of data) { - const color = SOURCE_COLORS[row.source] || '#666'; - viewer.samplePoints.add({ - id: { - type: 'sample', - pid: row.pid, - label: row.label, - source: row.source, - lat: row.latitude, - lng: row.longitude, - place_name: row.place_name, - result_time: row.result_time - }, - position: Cesium.Cartesian3.fromDegrees(row.longitude, row.latitude, 0), - pixelSize: 6, - color: Cesium.Color.fromCssColorString(color).withAlpha(0.9), - scaleByDistance: scalar, - }); - } - } - - // --- Mode transitions --- - function enterPointMode(pushHistory) { - mode = 'point'; - viewer._globeState.mode = 'point'; - viewer.h3Points.show = false; - viewer.samplePoints.show = true; - if (pushHistory !== false) history.pushState(null, '', buildHash(viewer)); - loadViewportSamples(); - console.log('Entered point mode'); - } - - function exitPointMode(pushHistory) { - mode = 'cluster'; - viewer._globeState.mode = 'cluster'; - viewer.samplePoints.show = false; - viewer.samplePoints.removeAll(); - viewer.h3Points.show = true; - if (pushHistory !== false) history.pushState(null, '', buildHash(viewer)); - cachedBounds = null; - cachedData = null; - - // Restore cluster stats with viewport count - const bounds = getViewportBounds(); - const inView = countInViewport(bounds); - const total = viewer._clusterTotal; - if (total) { - updateStats(`H3 Res${currentRes}`, `${inView.clusters.toLocaleString()} / ${total.clusters.toLocaleString()}`, inView.samples.toLocaleString(), '—', 'Clusters in View / Loaded', 'Samples in View'); - } else { - updateStats(`H3 Res${currentRes}`, viewer.h3Points.length, '—', '—', 'Clusters Loaded', 'Samples Loaded'); - } - updatePhaseMsg(`${inView.clusters.toLocaleString()} clusters in view. Zoom closer for individual samples.`, 'done'); - console.log('Exited point mode'); - } - - // === Cross-filter facet count refresh (issue #156, Phase 2) === - // - // Counts answer: for each value in facet D, how many samples would match - // this value plus the active filters in all OTHER facets. This keeps - // selected facets useful as drill-out controls instead of just echoing the - // selected values. Search text is treated as an additional sample predicate. - let facetCountsReqId = 0; - let facetCountsDebounce = null; - - function getSearchTerm() { - const input = document.getElementById('sampleSearch'); - return input ? input.value.trim() : ''; - } - - function describeCrossFilters() { - const sourceChecks = document.querySelectorAll('#sourceFilter input[type="checkbox"]'); - const sourceTotal = sourceChecks.length; - const sources = getActiveSources(); - const mat = getCheckedValues('materialFilterBody'); - const ctx = getCheckedValues('contextFilterBody'); - const ot = getCheckedValues('objectTypeFilterBody'); - const search = getSearchTerm(); - const dims = [ - { key: 'source', col: 'source', values: sources.length < sourceTotal ? sources : [] }, - { key: 'material', col: 'material', values: mat }, - { key: 'context', col: 'context', values: ctx }, - { key: 'object_type', col: 'object_type', values: ot }, - ]; - const activeDims = dims.filter(d => d.values.length > 0); - const totalActiveValues = activeDims.reduce((n, d) => n + d.values.length, 0); - return { - dims, - activeDims, - totalActiveValues, - sourceImpossible: sourceTotal > 0 && sources.length === 0, - searchActive: search.length >= 2, - search, - }; - } - - function buildCrossFilterWhere(excludeFacet) { - const { activeDims, sourceImpossible, searchActive, search } = describeCrossFilters(); - if (sourceImpossible && excludeFacet !== 'source') return '1=0'; - - const conds = activeDims - .filter(d => d.key !== excludeFacet) - .map(d => { - const list = d.values.map(v => `'${escSql(v)}'`).join(','); - return `${d.col} IN (${list})`; - }); - - if (searchActive) { - const terms = searchTerms(search); - const searchWhere = textSearchWhere(terms, ['label', 'CAST(place_name AS VARCHAR)']); - conds.push(`pid IN ( - SELECT pid - FROM read_parquet('${lite_url}') - WHERE ${searchWhere} - )`); - } - - return conds.length > 0 ? conds.join(' AND ') : '1=1'; - } - - async function updateCrossFilteredCounts(myReq) { - if (myReq !== facetCountsReqId) return; - const { dims, activeDims, totalActiveValues, sourceImpossible, searchActive } = describeCrossFilters(); - - if (!sourceImpossible && activeDims.length === 0 && !searchActive) { - for (const d of dims) applyFacetCounts(d.key, null); - return; - } - - markFacetCountsRecomputing(); - - const singleActiveDim = !sourceImpossible && !searchActive - && activeDims.length === 1 && activeDims[0].values.length === 1 - ? activeDims[0] : null; - if (singleActiveDim && totalActiveValues === 1) { - try { - const filterCols = ['filter_source', 'filter_material', 'filter_context', 'filter_object_type']; - const filterColForKey = { - source: 'filter_source', - material: 'filter_material', - context: 'filter_context', - object_type: 'filter_object_type', - }; - const targetCol = filterColForKey[singleActiveDim.key]; - const value = escSql(singleActiveDim.values[0]); - const whereParts = filterCols.map(c => - c === targetCol ? `${c} = '${value}'` : `${c} IS NULL` - ); - const rows = await db.query(` - SELECT facet_type, facet_value, count - FROM read_parquet('${cross_filter_url}') - WHERE ${whereParts.join(' AND ')} - `); - if (myReq !== facetCountsReqId) return; - if (rows && rows.length > 0) { - const grouped = { source: new Map(), material: new Map(), context: new Map(), object_type: new Map() }; - for (const r of rows) { - if (grouped[r.facet_type]) grouped[r.facet_type].set(r.facet_value, Number(r.count)); - } - for (const d of dims) { - applyFacetCounts(d.key, d.key === singleActiveDim.key ? null : grouped[d.key]); - } - return; - } - } catch (err) { - console.warn('Cross-filter cache lookup failed; falling back to on-the-fly:', err); - } - } - - await Promise.all(dims.map(async (d) => { - const where = buildCrossFilterWhere(d.key); - try { - const rows = await db.query(` - SELECT ${d.col} AS value, COUNT(*) AS count - FROM read_parquet('${facets_url}') - WHERE ${where} AND ${d.col} IS NOT NULL - GROUP BY ${d.col} - `); - if (myReq !== facetCountsReqId) return; - const map = new Map(); - for (const r of rows) map.set(r.value, Number(r.count)); - applyFacetCounts(d.key, map); - } catch (err) { - if (myReq !== facetCountsReqId) return; - console.warn(`Cross-filter count query failed for ${d.key}:`, err); - applyFacetCounts(d.key, null); - } - })); - } - - function refreshFacetCounts() { - clearTimeout(facetCountsDebounce); - const myReq = ++facetCountsReqId; - facetCountsDebounce = setTimeout(() => { - updateCrossFilteredCounts(myReq); - }, 250); - } - - // --- Source filter change handler --- - const resUrls = { 4: h3_res4_url, 6: h3_res6_url, 8: h3_res8_url }; - document.getElementById('sourceFilter').addEventListener('change', async () => { - // Toggle visual state on labels - updateSourceLegendState(); - writeQueryState(); - if (mode === 'cluster') { - loading = false; // allow loadRes to run (gen counter discards stale results) - await loadRes(currentRes, resUrls[currentRes]); - } else { - cachedBounds = null; // force re-query - await loadViewportSamples(); - } - refreshFacetCounts(); - }); - - // --- Material / Context / Specimen Type filter change handler --- - // - // Cluster-mode honesty: the H3 summary parquets only carry - // `dominant_source`, so material / context / object_type filters cannot - // affect cluster counts. When any of these is active in cluster mode, - // surface the explanatory `#facetNote` so users understand the filter - // takes effect at neighborhood zoom. See issue #156, Phase 1. - const facetNote = document.getElementById('facetNote'); - function handleFacetFilterChange() { - const active = hasFacetFilters(); - if (facetNote) facetNote.style.display = (active && mode === 'cluster') ? 'block' : 'none'; - writeQueryState(); - if (mode === 'point') { - cachedBounds = null; - loadViewportSamples(); - } - refreshFacetCounts(); - } - document.getElementById('materialFilterBody').addEventListener('change', handleFacetFilterChange); - document.getElementById('contextFilterBody').addEventListener('change', handleFacetFilterChange); - document.getElementById('objectTypeFilterBody').addEventListener('change', handleFacetFilterChange); - - // --- Camera change handler --- - let timer = null; - viewer.camera.changed.addEventListener(() => { - if (timer) clearTimeout(timer); - timer = setTimeout(async () => { - const h = viewer.camera.positionCartographic.height; - - // Determine target mode with hysteresis - const targetMode = h < ENTER_POINT_ALT ? 'point' - : h > EXIT_POINT_ALT ? 'cluster' - : mode; - - if (targetMode === 'point' && mode !== 'point') { - // Make sure we're at res8 clusters before transitioning - if (currentRes !== 8 && !loading) { - await loadRes(8, h3_res8_url); - } - enterPointMode(); - } else if (targetMode === 'cluster' && mode !== 'cluster') { - exitPointMode(); - // Reload appropriate resolution - const target = h > 3000000 ? 4 : h > 300000 ? 6 : 8; - if (target !== currentRes && !loading) { - await loadRes(target, { 4: h3_res4_url, 6: h3_res6_url, 8: h3_res8_url }[target]); - } - } else if (targetMode === 'point') { - // Already in point mode — update viewport samples - loadViewportSamples(); - } else { - // Cluster mode — check if resolution should change - const target = h > 3000000 ? 4 : h > 300000 ? 6 : 8; - if (target !== currentRes && !loading) { - await loadRes(target, { 4: h3_res4_url, 6: h3_res6_url, 8: h3_res8_url }[target]); - } - } - - // Update viewport cluster count (cluster mode only; point mode already shows viewport count) - if (mode === 'cluster' && viewer._clusterData) { - const bounds = getViewportBounds(); - const inView = countInViewport(bounds); - const total = viewer._clusterTotal; - if (total) { - updateStats(`H3 Res${currentRes}`, `${inView.clusters.toLocaleString()} / ${total.clusters.toLocaleString()}`, inView.samples.toLocaleString(), null, 'Clusters in View / Loaded', 'Samples in View'); - } - } - - // Update URL hash (replaceState for continuous movement) - if (!viewer._suppressHashWrite) { - history.replaceState(null, '', buildHash(viewer)); - } - }, 600); - }); - viewer.camera.percentageChanged = 0.1; - - // --- Handle browser back/forward --- - window.addEventListener('hashchange', async () => { - const state = readHash(); - if (state.lat == null || state.lng == null) return; - - viewer._suppressHashWrite = true; - clearTimeout(viewer._suppressTimer); - viewer.camera.cancelFlight(); - viewer.camera.flyTo({ - destination: Cesium.Cartesian3.fromDegrees(state.lng, state.lat, state.alt || 20000000), - orientation: { - heading: Cesium.Math.toRadians(state.heading), - pitch: Cesium.Math.toRadians(state.pitch) - }, - duration: 1.5, - }); - - // After flight settles, force mode and clear suppress flag - viewer._suppressTimer = setTimeout(() => { - viewer._suppressHashWrite = false; - const s = readHash(); - if (s.mode === 'point' && mode !== 'point') enterPointMode(false); - else if (s.mode !== 'point' && mode === 'point') exitPointMode(false); - }, 2000); - - // Handle pid selection - if (state.pid) { - viewer._globeState.selectedPid = state.pid; - try { - const sample = await db.query(` - SELECT pid, label, source, latitude, longitude, place_name, result_time - FROM read_parquet('${lite_url}') - WHERE pid = '${state.pid.replace(/'/g, "''")}' - LIMIT 1 - `); - if (sample && sample.length > 0) { - const s = sample[0]; - updateSampleCard({ - pid: s.pid, label: s.label, source: s.source, - lat: s.latitude, lng: s.longitude, - place_name: s.place_name, result_time: s.result_time - }); - } - } catch(err) { - console.error("Hash pid query failed:", err); - } - } else { - viewer._globeState.selectedPid = null; - updateClusterCard(null); - } - }); - - // --- Share button --- - const shareBtn = document.getElementById('shareBtn'); - if (shareBtn) { - shareBtn.addEventListener('click', async () => { - history.replaceState(null, '', buildHash(viewer)); - try { - await navigator.clipboard.writeText(location.href); - const toast = document.getElementById('shareToast'); - if (toast) { - toast.style.opacity = '1'; - setTimeout(() => { toast.style.opacity = '0'; }, 2000); - } - } catch(err) { - prompt('Copy this link:', location.href); - } - }); - } - - // --- Search handler --- - const searchBtn = document.getElementById('searchBtn'); - const searchInput = document.getElementById('sampleSearch'); - const searchResults = document.getElementById('searchResults'); - - async function doSearch() { - const term = searchInput.value.trim(); - if (!term || term.length < 2) { - searchResults.textContent = 'Type at least 2 characters'; - writeQueryState(); - return; - } - writeQueryState(); - searchResults.textContent = 'Searching...'; - try { - const terms = searchTerms(term); - const searchWhere = textSearchWhere(terms, ['label', 'CAST(place_name AS VARCHAR)']); - const score = textSearchScore(terms, [ - { col: 'label', weight: 3 }, - { col: 'CAST(place_name AS VARCHAR)', weight: 2 }, - ]); - const results = await db.query(` - SELECT pid, label, source, latitude, longitude, place_name, - (${score}) AS relevance_score - FROM read_parquet('${lite_url}') - WHERE ${searchWhere} - ${sourceFilterSQL('source')} - ${facetFilterSQL()} - ORDER BY relevance_score DESC, label - LIMIT 50 - `); - if (results.length === 0) { - searchResults.textContent = `No results for "${term}"`; - return; - } - searchResults.textContent = `${results.length}${results.length === 50 ? '+' : ''} results for "${term}"`; - - // Show results in the samples panel - const sampEl = document.getElementById('samplesSection'); - if (sampEl) { - let h = `

Search: "${term}" (${results.length})

`; - for (const s of results) { - const color = SOURCE_COLORS[s.source] || '#666'; - const name = SOURCE_NAMES[s.source] || s.source; - const sUrl = sourceUrl(s.pid); - h += `
-
- ${sUrl ? `${s.label || s.pid}` : `${s.label || s.pid}`} - ${name} -
-
`; - } - sampEl.innerHTML = h; - - // Click search result → fly to it - sampEl.querySelectorAll('.sample-row[data-lat]').forEach(row => { - row.addEventListener('click', (e) => { - if (e.target.tagName === 'A') return; // let links work - const lat = parseFloat(row.dataset.lat); - const lng = parseFloat(row.dataset.lng); - const pid = row.dataset.pid; - if (!isNaN(lat) && !isNaN(lng)) { - viewer.camera.flyTo({ - destination: Cesium.Cartesian3.fromDegrees(lng, lat, 50000), - duration: 1.5 - }); - } - }); - }); - } - - // Fly to the first result - if (results[0].latitude && results[0].longitude) { - viewer.camera.flyTo({ - destination: Cesium.Cartesian3.fromDegrees(results[0].longitude, results[0].latitude, 200000), - duration: 1.5 - }); - } - } catch(err) { - console.error("Search failed:", err); - searchResults.textContent = `Search error: ${err.message}`; - } - } - - if (searchBtn) searchBtn.addEventListener('click', () => { - doSearch(); - refreshFacetCounts(); - }); - let searchFacetDebounce = null; - if (searchInput) searchInput.addEventListener('input', () => { - clearTimeout(searchFacetDebounce); - searchFacetDebounce = setTimeout(refreshFacetCounts, 300); - }); - if (searchInput) searchInput.addEventListener('keydown', (e) => { - if (e.key === 'Enter') { - doSearch(); - refreshFacetCounts(); - } - }); - - if (searchInput && searchInput.value.trim().length >= 2) { - doSearch(); - } - - refreshFacetCounts(); - - // --- Deep-link: restore selection from initial hash --- - const ih = viewer._initialHash; - if (ih.pid) { - viewer._globeState.selectedPid = ih.pid; - try { - const sample = await db.query(` - SELECT pid, label, source, latitude, longitude, place_name, result_time - FROM read_parquet('${lite_url}') - WHERE pid = '${ih.pid.replace(/'/g, "''")}' - LIMIT 1 - `); - if (sample && sample.length > 0) { - const s = sample[0]; - updateSampleCard({ - pid: s.pid, label: s.label, source: s.source, - lat: s.latitude, lng: s.longitude, - place_name: s.place_name, result_time: s.result_time - }); - const detail = await db.query(` - SELECT description FROM read_parquet('${wide_url}') - WHERE pid = '${ih.pid.replace(/'/g, "''")}' - LIMIT 1 - `); - if (detail && detail.length > 0) updateSampleDetail(detail[0]); - else updateSampleDetail({ description: '' }); - } - } catch(err) { - console.error("Deep-link pid query failed:", err); - } - } - - // Enable hash writing now that everything is initialized - viewer._suppressHashWrite = false; - - return "active"; -} -``` - -```{ojs} -//| echo: false -//| output: false - -// === Performance timing panel (opt-in: append ?perf=1 to URL) === -// v0: reads performance.mark/measure entries and renders a small fixed panel. -// Reports navigation→duckdb_init, navigation→viewer_init, phase 1 res4 load, -// and navigation→first paint. Also dumps to console.table for CI / Playwright. -perfPanel = { - if (!phase1) return; // wait for phase 1 to have run - - const params = new URLSearchParams(location.search); - const isOn = params.get('perf') === '1'; - if (!isOn) return; - - // Give first-paint a tick to fire, then collect - await new Promise(r => setTimeout(r, 100)); - - const origin = performance.timeOrigin; - const mark = (name) => { - const e = performance.getEntriesByName(name, 'mark').pop(); - return e ? e.startTime : null; - }; - const measure = (name) => { - const e = performance.getEntriesByName(name, 'measure').pop(); - return e ? e.duration : null; - }; - - // Paint timings from the browser (free, no instrumentation needed) - const paintEntries = performance.getEntriesByType('paint'); - const fcp = paintEntries.find(e => e.name === 'first-contentful-paint')?.startTime; - const fp = paintEntries.find(e => e.name === 'first-paint')?.startTime; - - const rows = [ - ['first-paint (browser)', fp], - ['first-contentful-paint', fcp], - ['duckdb init', measure('duckdb_init')], - ['viewer init', measure('viewer_init')], - ['nav → viewer ready', mark('viewer-init-end')], - ['phase 1 res4 (duration)', measure('p1')], - ['nav → phase 1 complete', mark('p1-end')], - ['nav → first globe frame', mark('first-globe-frame')], - ].filter(([, v]) => v != null); - - // Console table for CI / offline capture - console.table(Object.fromEntries(rows.map(([k, v]) => [k, `${v.toFixed(0)} ms`]))); - - // Render a small floating panel - const fmt = (ms) => ms == null ? '—' : ms >= 1000 ? `${(ms/1000).toFixed(2)} s` : `${ms.toFixed(0)} ms`; - const panel = document.createElement('div'); - panel.id = 'perfPanel'; - panel.style.cssText = ` - position: fixed; bottom: 12px; right: 12px; z-index: 9999; - background: rgba(0,0,0,0.82); color: #e8f5e9; padding: 10px 12px; - border-radius: 6px; font: 11px/1.4 ui-monospace, SFMono-Regular, monospace; - max-width: 320px; box-shadow: 0 2px 12px rgba(0,0,0,0.3); - `; - panel.innerHTML = ` -
- ⏱ Perf timings - -
- - ${rows.map(([label, v]) => ` - - - `).join('')} -
${label}${fmt(v)}
-
timeOrigin: ${new Date(origin).toISOString().split('T')[1].slice(0,12)}
- `; - document.body.appendChild(panel); - panel.querySelector('#perfClose').onclick = () => panel.remove(); - - return "shown"; -} -``` - -## How This Demo Works - -Pre-aggregated H3 hexagonal indices achieve near-instant globe rendering, with seamless drill-down to individual samples: - -| Phase | Data | Size | Points | -|-------|------|------|--------| -| **Instant** | H3 res4 | 580 KB | 38K clusters (continental) | -| **Zoom in** | H3 res6 | 1.6 MB | 112K clusters (city) | -| **Zoom more** | H3 res8 | 2.5 MB | 176K clusters (neighborhood) | -| **Zoom deep** | Map lite | 60 MB (range req.) | Up to 5K individual samples | -| **Click sample** | Full dataset | ~280 MB (range req.) | Full metadata for 1 sample | - -**4 parquet files, zero backend.** All queries run in your browser via DuckDB-WASM with HTTP range requests — only the bytes you need are transferred. - -## See Also - -- [Search Explorer](/tutorials/isamples_explorer.html) — Faceted search and filter across all samples -- [Deep-Dive Analysis](/tutorials/zenodo_isamples_analysis.html) — DuckDB-WASM SQL tutorial +This page has moved to the [Interactive Explorer](../explorer.html). Redirecting… From 7d2af9ecf4bc963e65625686647934daab588193 Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Fri, 1 May 2026 12:32:06 -0700 Subject: [PATCH 2/4] Fix Codex review findings 1. test_baseline_sesar_count_matches_summaries was racy: facet-count spans are present in static HTML before being populated, so wait_for("attached") returned immediately and the test parsed an empty string. Wait until the SESAR span text matches \(\d (i.e., a parenthesised number) before reading. 2. Drop ?maxSamples= from URL state. Phase 3 introduced it to control the globe POINT_BUDGET; Phase 4 added a separate #maxSamples input for the table cap with different defaults (5000 vs 25000) and ranges (1-1000000 vs 1000-100000). The two were never the same concept and conflating them under one URL param meant `?maxSamples=10000` silently affected the globe but not the visible table input, and table-input changes never made it back to the URL. Remove the URL param entirely: globe POINT_BUDGET reverts to the constant DEFAULT_POINT_BUDGET (5000), table input remains a UI-only control. Co-Authored-By: Claude Opus 4.7 (1M context) --- explorer.qmd | 10 +--------- tests/test_globe.py | 16 ++++++++++++++-- 2 files changed, 15 insertions(+), 11 deletions(-) diff --git a/explorer.qmd b/explorer.qmd index 00838aa..592b909 100644 --- a/explorer.qmd +++ b/explorer.qmd @@ -437,10 +437,6 @@ function applyQueryToFacetFilters() { setCheckedValues('objectTypeFilterBody', csvParamValues(params, 'object_type')); } -function getMaxSamplesLimit() { - const params = new URLSearchParams(location.search); - return Math.round(parseNum(params.get('maxSamples'), DEFAULT_POINT_BUDGET, 1, 1000000)); -} function writeQueryState() { const params = new URLSearchParams(location.search); @@ -463,10 +459,6 @@ function writeQueryState() { else params.delete(key); }); - const maxSamples = getMaxSamplesLimit(); - if (maxSamples !== DEFAULT_POINT_BUDGET) params.set('maxSamples', String(maxSamples)); - else params.delete('maxSamples'); - if (typeof document !== 'undefined' && document.body && document.body.classList.contains('table-view-active')) { params.set('view', 'table'); } else { @@ -1270,7 +1262,7 @@ zoomWatcher = { // Hysteresis thresholds to avoid flicker const ENTER_POINT_ALT = 120000; // 120 km → enter point mode const EXIT_POINT_ALT = 180000; // 180 km → exit point mode - const POINT_BUDGET = getMaxSamplesLimit(); + const POINT_BUDGET = DEFAULT_POINT_BUDGET; // Viewport cache: avoid re-querying same area let cachedBounds = null; // { south, north, west, east } diff --git a/tests/test_globe.py b/tests/test_globe.py index cb4a7d9..934b82e 100644 --- a/tests/test_globe.py +++ b/tests/test_globe.py @@ -73,11 +73,23 @@ class TestExplorerCrossFiltering: """ def _wait_for_facets(self, page): - facet = page.locator(".facet-count[data-facet='source']") + # The .facet-count spans are in the static HTML before being populated, + # so wait_for("attached") is not enough. Poll until the text contains a + # parenthesised number (e.g. "(4,389,231)"). + facet = page.locator(".facet-count[data-facet='source'][data-value='SESAR']") try: facet.first.wait_for(state="attached", timeout=60000) + page.wait_for_function( + """() => { + const el = document.querySelector( + ".facet-count[data-facet='source'][data-value='SESAR']" + ); + return el && /\\(\\d/.test(el.textContent || ''); + }""", + timeout=60000, + ) except Exception: - pytest.skip("Facet count labels not rendered (DuckDB-WASM may not have loaded)") + pytest.skip("Facet count labels not populated (DuckDB-WASM may not have loaded)") def _get_count(self, page, facet, value): el = page.locator(f".facet-count[data-facet='{facet}'][data-value='{value}']") From 8559b565ef2bee45ac069556c036876d96245bb9 Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Fri, 1 May 2026 12:57:26 -0700 Subject: [PATCH 3/4] Fix explorer facet dimming --- explorer.qmd | 28 +++++----------------------- 1 file changed, 5 insertions(+), 23 deletions(-) diff --git a/explorer.qmd b/explorer.qmd index 592b909..7778404 100644 --- a/explorer.qmd +++ b/explorer.qmd @@ -1493,15 +1493,10 @@ zoomWatcher = { // Counts answer: for each value in facet D, how many samples would match // this value plus the active filters in all OTHER facets. This keeps // selected facets useful as drill-out controls instead of just echoing the - // selected values. Search text is treated as an additional sample predicate. + // selected values. let facetCountsReqId = 0; let facetCountsDebounce = null; - function getSearchTerm() { - const input = document.getElementById('sampleSearch'); - return input ? input.value.trim() : ''; - } - function describeCrossFilters() { const sourceChecks = document.querySelectorAll('#sourceFilter input[type="checkbox"]'); const sourceTotal = sourceChecks.length; @@ -1509,7 +1504,6 @@ zoomWatcher = { const mat = getCheckedValues('materialFilterBody'); const ctx = getCheckedValues('contextFilterBody'); const ot = getCheckedValues('objectTypeFilterBody'); - const search = getSearchTerm(); const dims = [ { key: 'source', col: 'source', values: sources.length < sourceTotal ? sources : [] }, { key: 'material', col: 'material', values: mat }, @@ -1523,13 +1517,11 @@ zoomWatcher = { activeDims, totalActiveValues, sourceImpossible: sourceTotal > 0 && sources.length === 0, - searchActive: search.length >= 2, - search, }; } function buildCrossFilterWhere(excludeFacet) { - const { activeDims, sourceImpossible, searchActive, search } = describeCrossFilters(); + const { activeDims, sourceImpossible } = describeCrossFilters(); if (sourceImpossible && excludeFacet !== 'source') return '1=0'; const conds = activeDims @@ -1539,31 +1531,21 @@ zoomWatcher = { return `${d.col} IN (${list})`; }); - if (searchActive) { - const terms = searchTerms(search); - const searchWhere = textSearchWhere(terms, ['label', 'CAST(place_name AS VARCHAR)']); - conds.push(`pid IN ( - SELECT pid - FROM read_parquet('${lite_url}') - WHERE ${searchWhere} - )`); - } - return conds.length > 0 ? conds.join(' AND ') : '1=1'; } async function updateCrossFilteredCounts(myReq) { if (myReq !== facetCountsReqId) return; - const { dims, activeDims, totalActiveValues, sourceImpossible, searchActive } = describeCrossFilters(); + const { dims, activeDims, totalActiveValues, sourceImpossible } = describeCrossFilters(); - if (!sourceImpossible && activeDims.length === 0 && !searchActive) { + if (!sourceImpossible && activeDims.length === 0) { for (const d of dims) applyFacetCounts(d.key, null); return; } markFacetCountsRecomputing(); - const singleActiveDim = !sourceImpossible && !searchActive + const singleActiveDim = !sourceImpossible && activeDims.length === 1 && activeDims[0].values.length === 1 ? activeDims[0] : null; if (singleActiveDim && totalActiveValues === 1) { From a0bfe2a27531769713ecc332e5859a915e75e553 Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Fri, 1 May 2026 13:01:28 -0700 Subject: [PATCH 4/4] Drop search-triggered refreshFacetCounts calls MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-on to 8559b56 (Fix explorer facet dimming) which decoupled facet counts from the search predicate. After that change, calling refreshFacetCounts on every search keystroke / button click / Enter triggered DB requeries that produced no visible difference (counts no longer depend on search text). Drop those calls. The single refreshFacetCounts() at the end of the cell (initial paint) and the calls from facet checkbox handlers remain — those are still load-bearing. Co-Authored-By: Claude Opus 4.7 (1M context) --- explorer.qmd | 15 ++------------- 1 file changed, 2 insertions(+), 13 deletions(-) diff --git a/explorer.qmd b/explorer.qmd index 7778404..fd61702 100644 --- a/explorer.qmd +++ b/explorer.qmd @@ -1858,20 +1858,9 @@ zoomWatcher = { } } - if (searchBtn) searchBtn.addEventListener('click', () => { - doSearch(); - refreshFacetCounts(); - }); - let searchFacetDebounce = null; - if (searchInput) searchInput.addEventListener('input', () => { - clearTimeout(searchFacetDebounce); - searchFacetDebounce = setTimeout(refreshFacetCounts, 300); - }); + if (searchBtn) searchBtn.addEventListener('click', doSearch); if (searchInput) searchInput.addEventListener('keydown', (e) => { - if (e.key === 'Enter') { - doSearch(); - refreshFacetCounts(); - } + if (e.key === 'Enter') doSearch(); }); if (searchInput && searchInput.value.trim().length >= 2) {