Skip to content
3 changes: 2 additions & 1 deletion CANONICAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,8 @@ Exactly what production `explorer.qmd` loads, all under `https://data.isamples.o

The ~9-file facet family looks baroque but is load-bearing: it is the price of
fast multi-filter counts with no server. See `EXPLORER_QUERIES.md` for how each
is queried and `DATA_PROVENANCE.md` for how each is built.
is queried and `DATA_PROVENANCE.md` for how the derived files are built (that
doc covers the pre-#290/#304 substrate; see its coverage caveat).

## 2. Superseded versions (still served; do not use)

Expand Down
26 changes: 25 additions & 1 deletion DATA_PROVENANCE.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,23 @@
# iSamples Explorer — Data Provenance

How every parquet file the explorer uses is generated, from root to publish.
How the explorer's derived parquet files are generated, from root to publish.
**Not exhaustive as of 2026-08-05 — see the coverage caveat below.**
*Reviewed 2026-06-02 (CC, via codebase audit). Complements `SERIALIZATIONS.md` (format/schema reference); this file is the end-to-end build chain + the automation gaps.*

> ⚠️ **Coverage caveat (2026-08-05).** The DAG below documents the **seven-file
> derived substrate as of the 2026-06-02 review**. It does *not* cover the whole
> live `202608` family. Known omissions:
>
> - `sample_facet_masks`, `facet_node_bits`, `sample_facet_index`,
> `sample_facet_index_meta` (the bitmask count path, #299/#304/#305/#313)
> - `sample_facet_membership`
> - `facet_tree_summaries`, `facet_tree_cross_filter` (the tree facet path, #290)
> - the sharded search index `isamples_202608_search_index_v1/` (#171)
>
> The *build chain and the automation gaps* it describes are still accurate for
> the files it does cover; treat it as incomplete rather than wrong. Authoritative
> current inventory: `isamples_202608_release_manifest.json` / `CANONICAL.md`.

> **Load-bearing constraint:** the **root export cannot be regenerated.** It was produced from the iSamples Central Solr API (`central.isample.xyz`), **offline since Aug 2025**. The Zenodo-archived export is a **frozen root**. Any *new* data (e.g. concept URIs, thumbnails) therefore must come from a **per-source supplementary file merged into the base by `pid`** — the "sidecar" pattern (see Stage 3) — not from re-exporting.

## Pipeline DAG
Expand Down Expand Up @@ -70,6 +85,15 @@ Eric Kansa maintains OpenContext PQG **independently** on GCS (`storage.googleap

## Documentation / automation gaps (remaining)

> ⚠️ **Snapshot note (2026-08-05).** The version-skew bullets below were written
> when the deployed derived files were `202601` and the wide was `202604`. That is
> no longer the live state: the Explorer now serves the **`202608`** family
> (`sample_facets_v4`, `samples_map_lite_v3`, `wide` at 300,303,095 B). The
> *reproducibility* gap the bullets describe is still real and still unresolved —
> only the version numbers in them are historical. Authoritative current inventory:
> [`isamples_202608_release_manifest.json`](https://data.isamples.org/isamples_202608_release_manifest.json),
> human twin `CANONICAL.md`.

- ⚠️ **The deployed `202601` derived files are NOT reproducible** from any available wide. A rebuild yields **528,983** root-material rows (pre-#271); the deployed `sample_facets_v2` has **346,768** — so the live files came from a different/unrecorded Stage-4 process, *and* the data has since rolled (wide is now `202604`). Treat a fresh `build_frontend_derived.py` run as the new source of truth, not as a bit-for-bit reproduction of the deployed files.
- **Version skew:** the deployed derived files are `202601` while the wide they should derive from is `202604` (the popup reads `202604`). Rebuilding from `202604` resolves it (tracked in the pipeline epic).
- **No R2 upload automation** — file upload to bucket `isamples-ry` + `current/manifest.json` update are manual `wrangler`/dashboard steps.
Expand Down
30 changes: 15 additions & 15 deletions EXPLORER_QUERIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,17 +23,17 @@ server. You can open any of these URLs directly, or point DuckDB at them
one place, `explorer.qmd` around **line 800-864**, e.g.:

```js
lite_url = `${R2_BASE}/isamples_202608_samples_map_lite_v2.parquet` // map points + table
lite_url = `${R2_BASE}/isamples_202608_samples_map_lite_v3.parquet` // map points + table
wide_url = `${R2_BASE}/isamples_202608_wide.parquet` // full sample detail
facets_url = `${R2_BASE}/isamples_202608_sample_facets_v3.parquet` // material/context/object_type + search text
facets_url = `${R2_BASE}/isamples_202608_sample_facets_v4.parquet` // material/context/object_type + search text
h3_res4_url = `${R2_BASE}/isamples_202608_h3_summary_res4.parquet` // pre-counted globe dots (world zoom)
```

| File | Plain-English role | Roughly how big |
|---|---|---|
| `..._wide.parquet` | Full detail for every sample (one row each) — everything else is derived from this | ~280 MB |
| `..._samples_map_lite_v2.parquet` | Slim version with just what the map/table need: coords, label, place, date | ~50-60 MB |
| `..._sample_facets_v3.parquet` | One row per sample: material/context(sampled feature)/object_type as plain URIs, plus a search-text blob | ~60 MB |
| `..._wide.parquet` | Full detail for every sample (one row each) — everything else is derived from this | ~300 MB |
| `..._samples_map_lite_v3.parquet` | Slim version with just what the map/table need: coords, label, place, date | ~63 MB |
| `..._sample_facets_v4.parquet` | One row per sample: material/context(sampled feature)/object_type as plain URIs, plus a search-text blob | ~69 MB |
| `..._h3_summary_res{4,6,8}.parquet` | Pre-counted dots for the globe at 3 zoom tiers (continent / region / neighborhood), so zooming out never counts 6M rows live | tiny–few MB |
| `..._facet_summaries.parquet`, `..._facet_cross_filter.parquet`, `..._facet_tree_*.parquet` | Pre-computed facet-checkbox counts at various levels of "how many filters are active" — the whole point of these is to avoid a live COUNT over millions of rows | KB–tens of MB |
| `..._sample_facet_masks.parquet`, `..._sample_facet_index.parquet` | Bitmask tricks so 2+ facet filters at once are still fast (see `SERIALIZATIONS.md` §4.12 if you want the gory detail) | ~10 MB each |
Expand Down Expand Up @@ -62,7 +62,7 @@ pre-counted hexagon summaries instead.

As you zoom past a threshold, the Explorer swaps to res6, then res8 H3
tiles (same idea, finer hexagons), and eventually to individual points from
`samples_map_lite_v2.parquet` once there are few enough in view to draw
`samples_map_lite_v3.parquet` once there are few enough in view to draw
directly.

### ...click a facet checkbox (Material / Sampled Feature / Object Type / Source)
Expand Down Expand Up @@ -98,15 +98,15 @@ not tens of MB. *(`buildSearchFilterSubstrate()` + `assets/js/search_substrate.j
index contract in `SEARCH_INDEX_V1.md`.)*

**Fallback path (`?fts=off`, and automatically for identifier queries):** the
original `ILIKE`-style scan against `sample_facets_v3.parquet`'s description
original `ILIKE`-style scan against `sample_facets_v4.parquet`'s description
column:

```sql
SELECT pid, label, source, place_name FROM read_parquet('sample_facets_v3.parquet')
SELECT pid, label, source, place_name FROM read_parquet('sample_facets_v4.parquet')
WHERE description ILIKE '%pottery%'
```
*(`buildSearchFilter()` in `explorer.qmd`.)* This path downloads much more
data on first search (the scan touches most of the ~60 MB file) but handles
data on first search (the scan touches most of the ~69 MB file) but handles
one thing the index cannot: **pasted identifiers** (ARK / IGSN / DOI — e.g.
`ark:/28722/k2000hz7r`), which get exact-matched against the `pid` column.
Identifier-looking queries are routed here automatically; you never need the
Expand All @@ -119,20 +119,20 @@ mode.

### ...view the Samples table

The table pages through `samples_map_lite_v2.parquet` (coords/label/place/
date) and, as of #311, joins in `sample_facets_v3.parquet` for
The table pages through `samples_map_lite_v3.parquet` (coords/label/place/
date) and, as of #311, joins in `sample_facets_v4.parquet` for
material/object type/sampled feature — one query per page (default page
size), not the whole result set:

```sql
WITH page AS (
SELECT pid, label, source, latitude, longitude, place_name, result_time
FROM read_parquet('samples_map_lite_v2.parquet')
FROM read_parquet('samples_map_lite_v3.parquet')
WHERE <your active filters>
ORDER BY pid LIMIT 50 OFFSET 0
)
SELECT page.*, f.material, f.context, f.object_type
FROM page LEFT JOIN read_parquet('sample_facets_v3.parquet') AS f ON f.pid = page.pid
FROM page LEFT JOIN read_parquet('sample_facets_v4.parquet') AS f ON f.pid = page.pid
```
*(`loadPage()`, `explorer.qmd` ~line 2755-2775.)* "Download CSV" (#312) runs
the same shape without the `LIMIT`/`OFFSET` (capped at 50,000 rows so an
Expand All @@ -151,7 +151,7 @@ LEFT JOIN read_parquet('vocab_labels.parquet') mat_lbl ON mat_lbl.uri = mat.pid
WHERE s.pid = '<clicked pid>'
```
This is the one query that reads from `wide.parquet` on click (everything
above deliberately avoids touching the 280 MB wide file until you actually
above deliberately avoids touching the 300 MB wide file until you actually
need full detail on one sample).

## Try it yourself
Expand All @@ -170,7 +170,7 @@ GROUP BY n ORDER BY 2 DESC;
-- the default path since 2026-07-17 probes the sharded search index instead
-- (JS, not a single SQL statement — see SEARCH_INDEX_V1.md)
SELECT pid, label, source
FROM read_parquet('https://data.isamples.org/isamples_202608_sample_facets_v3.parquet')
FROM read_parquet('https://data.isamples.org/isamples_202608_sample_facets_v4.parquet')
WHERE description ILIKE '%pottery%'
LIMIT 20;
```
Expand Down
8 changes: 8 additions & 0 deletions EXPLORER_STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -328,6 +328,14 @@ compatible with all of them.

### Light-path addendum: two-button scope selection ([#178](https://github.com/isamplesorg/isamplesorg.github.io/issues/178), 2026-05-08)

> ⚠️ **Historical (2026-05-08).** This addendum records the decision as taken and
> the SQL as it stood then — including the `sample_facets_v2` / `samples_map_lite`
> filenames, which are two generations stale (now `_v4` / `_v3`), and the in-browser
> ILIKE scan, which is no longer the default search (the sharded BM25 substrate
> shipped as default 2026-07-17; ILIKE survives as `?fts=off`). The two-button scope
> *decision* still holds; treat the filenames and search mechanics here as a record
> of the time, not as current reference.

Hana's mockup ([Figma 213:394](https://www.figma.com/design/Nqkuqh3Z4aqVh0nmwUAgKg/iSamples-Wireframe-1.0?node-id=213-394))
proposed a two-button search UI: "Search Selected Areas" (viewport-scoped)
and "Search Entire World" (full-corpus). Implemented as a Light extension
Expand Down
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,15 @@ title: isamples.github.io
subtitle: README for the isamples.github.io source
---

> ⚠️ **Snapshot note (2026-08-05).** The **`202601`** file examples below pin a
> stable, versioned snapshot. Those objects still exist and their byte counts are
> correct, but they are **not what the Interactive Explorer serves today** — the
> live app reads the **`202608`** family, including `sample_facets_v4`,
> `samples_map_lite_v3`, `wide`, and the sharded search index. (Other generations
> may also appear here, such as the `current/` alias or the `202512` narrow file.)
> Use `202601` for a stable citable snapshot; for what the Explorer actually
> loads, see `isamples_202608_release_manifest.json` (human twin: `CANONICAL.md`).

# isamplesorg.github.io

This repository provides the source for [SMR fork isamplesorg.github.io](https://smrgeoinfo.github.io/isamplesorg.github.io/).
Expand Down
9 changes: 9 additions & 0 deletions SERIALIZATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,15 @@ toc: true
categories: [data, architecture, parquet]
---

> ⚠️ **Snapshot note (2026-08-05).** The **`202601`** file examples below pin a
> stable, versioned snapshot. Those objects still exist and their byte counts are
> correct, but they are **not what the Interactive Explorer serves today** — the
> live app reads the **`202608`** family, including `sample_facets_v4`,
> `samples_map_lite_v3`, `wide`, and the sharded search index. (Other generations
> may also appear here, such as the `current/` alias or the `202512` narrow file.)
> Use `202601` for a stable citable snapshot; for what the Explorer actually
> loads, see `isamples_202608_release_manifest.json` (human twin: `CANONICAL.md`).

## 1. Purpose and scope

iSamples has roughly a dozen parquet files in circulation at any given
Expand Down
16 changes: 16 additions & 0 deletions data.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,22 @@ toc: true
categories: [data, parquet, download]
---

::: {.callout-important}
## Snapshot note (2026-08-05)

The **`202601`** file examples on this page pin a stable, versioned snapshot.
Those objects still exist and their byte counts are correct, but they are **not
what the Interactive Explorer serves today** — the live app reads the **`202608`**
family, including `sample_facets_v4`, `samples_map_lite_v3`, `wide`, and the
sharded search index. (This page may also reference other generations, such as
the `current/` alias or the `202512` narrow file.)

Use `202601` when you want a stable, citable reference. For "what the Explorer is
actually loading right now", the authoritative inventory is
[`isamples_202608_release_manifest.json`](https://data.isamples.org/isamples_202608_release_manifest.json)
(human twin: `CANONICAL.md`), which the Explorer cross-checks at boot.
:::

::: {.callout-tip}
**Quick start**: every file on this page is queryable directly from a URL
— no bulk download needed. DuckDB's `httpfs` extension fetches only the
Expand Down
Loading
Loading