🏛️ Vantage — The Vantage Chronicle

A server-rendered Next.js app that turns real New York City locations into immersive historical scenes you can explore on desktop, phone, or in Google Cardboard. Open the map, choose a landmark, step into a 360° AI-aged panorama of that place in a specific year — with hotspots, voice clips, and editorial period notes anchored to the world around you.

🗞️ "See New York through time."

The current build ships with 12 scenes spanning Manhattan, Brooklyn, the Bronx, Queens, Staten Island, and New York harbor — anchored on years between 1873 and 1917. Each panorama is a real modern Mapillary 360° photo of the actual location, edited by Google Gemini 2.5 Flash Image (Nano Banana) to look period-correct using web-grounded historical research. Character voice clips are synthesised at runtime via a server-side ElevenLabs proxy with disk caching, so each unique line costs once.

🗺️ System Architecture

flowchart TB
    USER["👤 User<br/><i>Desktop / Phone / Cardboard</i>"]

    subgraph FE["🖥️ Frontend — Next.js App Router"]
        LANDING["📄 Landing Page<br/><code>/</code>"]
        MAP["🗺️ Map Page<br/><code>/map</code>"]
        VIEWER["🥽 VR Viewer<br/><code>/viewer/[locId]/[eraId]</code>"]
        EDITOR["✏️ Hotspot Editor<br/><code>/editor/[locId]/[eraId]</code>"]
    end

    subgraph API["⚡ API Routes — Server-side"]
        TTS["🔊 TTS Proxy<br/><code>/api/tts/[sceneId]/[hotspotId]</code>"]
        TTS_INTRO["🎤 Intro TTS<br/><code>/api/tts/intro/[locId]/[eraId]</code>"]
        CACHE["💾 Disk Cache<br/><code>cache/tts/{sha256}.mp3</code>"]
    end

    subgraph BUILD["🔧 Build-time Pipeline"]
        B1["📸 scrape-panorama"]
        B2["📚 gather-period-research"]
        B3["🎨 age-panorama"]
        B4["🎯 generate-hotspots-vision"]
        B5["📝 generate-summaries"]
        B6["🖼️ bake-infopanels"]
    end

    subgraph EXT["☁️ External APIs"]
        MAPILLARY["🗺️ Mapillary API<br/><i>360° Street Imagery</i>"]
        GEMINI["✨ Google Gemini 2.5 Flash<br/><i>Image Gen + Search + Vision</i>"]
        ELEVEN["🎙️ ElevenLabs API<br/><i>Text-to-Speech</i>"]
    end

    %% User → Frontend
    USER -->|"Browse"| LANDING
    USER -->|"Explore map"| MAP
    USER -->|"Enter scene"| VIEWER
    VIEWER -->|"Gaze hotspot"| TTS
    VIEWER -->|"Landmark intro"| TTS_INTRO

    %% Frontend → API
    TTS -->|"Cache miss?"| ELEVEN
    TTS_INTRO -->|"Cache miss?"| ELEVEN
    TTS -->|"Cache hit"| CACHE
    TTS_INTRO -->|"Cache hit"| CACHE
    ELEVEN -->|"Write MP3"| CACHE

    %% Build pipeline → External APIs
    B1 -->|"Download 360° photos"| MAPILLARY
    B2 -->|"Search-grounded research"| GEMINI
    B3 -->|"Image-to-image aging"| GEMINI
    B4 -->|"Vision + bbox detection"| GEMINI
    B5 -->|"Summarize"| GEMINI

    %% Build outputs feed the frontend
    B1 -.->|"public/panoramas-modern/"| B3
    B2 -.->|"data/period-research.json"| B3
    B3 -.->|"public/panoramas/"| VIEWER
    B4 -.->|"data/scenes/"| VIEWER
    B5 -.->|"data/location-summaries.json"| MAP
    B6 -.->|"public/infopanels/"| VIEWER

    %% Styling
    classDef user fill:#f4efe6,stroke:#8a2a1f,color:#333,stroke-width:2px
    classDef frontend fill:#fff3e0,stroke:#e65100,color:#333,stroke-width:2px
    classDef api fill:#e8f5e9,stroke:#2e7d32,color:#333,stroke-width:2px
    classDef build fill:#e3f2fd,stroke:#1565c0,color:#333,stroke-width:2px
    classDef external fill:#f3e5f5,stroke:#7b1fa2,color:#333,stroke-width:2px

    class USER user
    class LANDING,MAP,VIEWER,EDITOR frontend
    class TTS,TTS_INTRO,CACHE api
    class B1,B2,B3,B4,B5,B6 build
    class MAPILLARY,GEMINI,ELEVEN external

🎬 The 12 Scenes

#	📍 Location	🏙️ Borough	📅 Year	⚓ Anchor
I	Times Square	Manhattan	1904	IRT subway opens; Longacre→Times rename
II	Brooklyn Bridge	Brooklyn	1883	May 24 opening day
III	Liberty Island	NY Harbor	1907	Peak immigration year across the bay
IV	Central Park	Manhattan	1873	Bethesda Terrace unveiled
V	Lower East Side	Manhattan	1911	Triangle Shirtwaist fire; Hester Street market peak
VI	Coney Island	Brooklyn	1904	Luna Park's million-bulb opening
VII	Prospect Park	Brooklyn	1873	Olmsted/Vaux's Brooklyn masterpiece
VIII	DUMBO	Brooklyn	1900	Empire Stores warehouses, working waterfront
IX	Grand Concourse	Bronx	1909	Risse's Champs-Élysées-modeled boulevard opens
X	South Bronx Hub	Bronx	1909	Third Ave El + commercial peak
XI	Jackson Heights	Queens	1917	First cooperative garden apartments rise
XII	St. George Ferry Terminal	Staten Island	1908	New terminal opens after the 1905 fire

⚙️ How It Works

build-time pipeline (run once)                 runtime (Next.js server)
──────────────────────────────                 ─────────────────────────
1. 📸 scrape-panorama       Mapillary 360 →   GET  /              📄 landing
   → public/panoramas-modern/                  GET  /map           🗺️ NYC map + accordion sidebar
                                               GET  /viewer/[loc]/[era]
2. 📚 gather-period-research  Gemini Search →                      🥽 A-Frame 360 + hotspots
                              ~200 words/scene                     (gaze 1.5s → InfoPanel)

3. 🎨 age-panorama           Nano Banana →    GET  /api/tts/[scene]/[hotspot]
                              (modern + research)                  🔊 ElevenLabs proxy + disk cache
   → public/panoramas/                            (only runtime API call)

4. 🎯 generate-hotspots-vision Gemini vision +
                              bbox detection →
   → data/scenes/

5. 📝 generate-summaries     Wikipedia + Gemini
                              Flash-Lite →
   → data/location-summaries.json

6. 🖼️ bake-infopanels        @napi-rs/canvas →
   → public/infopanels/             (PNG textures used inside Cardboard mode where
                                     HTML/CSS doesn't reach into the WebGL scene)

The only runtime API call is /api/tts/[sceneId]/[hotspotId] — it proxies ElevenLabs, hashes the (text + voiceId) pair with SHA-256, and caches the resulting MP3 on disk under cache/tts/{sha}.mp3. Each unique line burns the API once for the lifetime of the cache.

🛠️ Tech Stack

Category	Technology
⚡ Framework	Next.js 15 App Router + TypeScript, server-rendered (with API routes — not a static export)
🎨 Styling	Tailwind for layout primitives + hand-written editorial CSS in `app/globals.css`
🌊 Motion	`motion` library + CSS keyframes + `IntersectionObserver` for scroll reveals
🗺️ Map	`react-leaflet` with CartoDB `light_nolabels` tiles, run through an SVG `feColorMatrix` + `feComponentTransfer` color-lookup filter that maps tile luminance through a 6-stop gradient (ink → 4 oxblood intermediates → paper) so the map reads in the site palette instead of fighting it
🥽 VR / 360 viewer	A-Frame (loaded via `next/script`), with built-in stereoscopic Cardboard mode and magic-window fallback
🖼️ Build-time canvas	`@napi-rs/canvas` for InfoPanel PNG bakes
🔤 Fonts	Playfair Display + EB Garamond + Inter Tight + JetBrains Mono (self-hosted via `@fontsource`)

📰 Design System — The Vantage Chronicle

Cream paper background (#f4efe6), oxblood accent (#8a2a1f), generous serif typography, hairline rules, animated rule-line draws, letter-by-letter title reveals, drop caps, kicker labels, paper grain overlay. Thematically the UI should feel like opening the morning paper to read about Times Square in 1904 — not like a generic Tailwind site.

CSS tokens live in app/globals.css. Editorial primitives live in components/ui/.

🚀 Setup

📋 Prerequisites

🟢 Node.js 18+
🗺️ A Mapillary access token (free): https://www.mapillary.com/dashboard/developers
✨ A Google Gemini API key with billing enabled (image generation and search grounding are paid features; total spend per full re-build is under $1): https://aistudio.google.com/app/apikey
🎙️ An ElevenLabs API key (free tier is fine — runtime cache means each unique line burns the budget once): https://elevenlabs.io/app/settings/api-keys

📦 Install + First Run

# install dependencies
npm install

# create env file (do not commit)
cp .env.local.example .env.local
# then edit .env.local and fill in:
#   MAPILLARY_ACCESS_TOKEN=...
#   GEMINI_API_KEY=...
#   ELEVENLABS_API_KEY=...

# run the build-time pipeline (in order)
npm run scrape-panorama          # 📸 Mapillary → public/panoramas-modern/
npm run gather-period-research   # 📚 Gemini search → data/period-research.json
npm run age-panorama             # 🎨 Nano Banana → public/panoramas/
npm run generate-hotspots-vision # 🎯 Gemini vision → data/scenes/
npm run generate-summaries       # 📝 Wikipedia + Gemini → data/location-summaries.json
npm run bake-infopanels          # 🖼️ canvas → public/infopanels/

# start the dev server
npm run dev

Open http://localhost:3000. 🎉

📱 Phone + Cardboard Testing (HTTPS Required)

iOS Safari requires HTTPS for DeviceOrientationEvent.requestPermission(). Plain LAN HTTP from npm run dev won't trigger the gyroscope on iPhone. Use Cloudflare Tunnel:

# from the project directory, with the dev server running on port 3000
cloudflared tunnel --url http://localhost:3000 --protocol http2

The --protocol http2 flag is important — without it cloudflared defaults to QUIC over UDP/443, which many networks block. Cloudflared prints a https://*.trycloudflare.com URL — open that on your phone, navigate to a viewer route, tap A-Frame's VR icon, grant motion permission, drop in headset. 🥽

🔧 Build-time Pipeline Scripts

All scripts are idempotent — safe to re-run. Most accept --force to regenerate ignoring cached state.

Script	📖 What it does	💰 Cost (full 12-scene run)
`npm run scrape-panorama`	Calls Mapillary Graph API for each location, picks the highest-resolution 360° photo, downloads to `public/panoramas-modern/{locId}.jpg`. Records contributor attribution in `data/panorama-sources.json`.	Free
`npm run gather-period-research`	Calls Gemini 2.5 Flash with Google Search grounding to produce ~200 words of web-sourced visual context per (loc, era) — buildings present that year, signage by name, vehicles, dress, surfaces, lighting. Cited URLs preserved in the JSON.	~$0.42
`npm run age-panorama`	Sends each modern 360 + the locked aging prompt + research blob to Nano Banana (`gemini-2.5-flash-image`); writes `public/panoramas/{locId}__{eraId}.jpg`. Append `-- --force` to overwrite.	~$0.47
`npm run generate-hotspots-vision`	Sends each aged JPG to Gemini 2.5 Flash with vision; uses bbox detection (`box_2d` in `[ymin, xmin, ymax, xmax]/[0, 1000]`) to spatially place 3–5 period-grounded hotspots per scene. Computes box centers and converts to (yaw, pitch) via the equirectangular projection.	~$0.05
`npm run generate-summaries`	Wikipedia REST + Gemini Flash-Lite → ~50-word period blurbs that show in the sidebar accordion.	Negligible
`npm run bake-infopanels`	Renders each hotspot's editorial paper-card to a PNG via `@napi-rs/canvas`. The PNGs are loaded as `<a-image>` planes inside Cardboard mode, where HTML/CSS can't reach into the WebGL scene.	Free

Total cost for a full re-build of all 12 scenes: under $1 💵

For the older Wikidata-based hotspot generator (kept as a fallback), run npm run generate-hotspots.

🗺️ Mapillary Coverage Gaps

If a lat/lon search returns nothing (common in dense central NYC and outer-borough residential areas), the scrape script tries progressively wider bboxes (150m → 300m → 500m → 800m → 1200m). If still empty, you have two escape hatches:

🔍 Find a Mapillary 360 image manually at https://www.mapillary.com/, copy its image ID from the URL, set MAPILLARY_FALLBACK_<LOCID_UPPER>=<id> in .env.local, and re-run.
📂 Drop a hand-sourced equirectangular JPG into public/panoramas-modern/{locId}.jpg. The aging step picks it up regardless of source.

The current build uses fallback IDs for LIBERTY_ISLAND, DUMBO, SOUTH_BRONX_HUB, and JACKSON_HEIGHTS.

➕ Adding a New Scene

📝 Add an entry to data/locations.json with id, name, lat, lon, numeral, blurb, and one or more eras (each with id, year, label, labelLong, and an event anchor sentence).
🗺️ If the lat/lon has no Mapillary 360 coverage, set the fallback env var (above).
📅 Add an entry to ERA_NOTES in scripts/age-panorama.ts for any new year (vehicles, dress, signage cues for the period).
🔄 Run the pipeline scripts in order.

A hidden /editor/[locId]/[eraId] route exists for manual hotspot fine-tuning if you want pixel-perfect placement; it's gated behind NODE_ENV === 'development'. 🛠️

🎯 Hotspot Positioning + Gaze Hit Detection

Hotspots are placed by Gemini's vision bounding-box detection — the model returns box_2d in [ymin, xmin, ymax, xmax]/[0, 1000] (the format Gemini was trained on for grounding) and the script computes the box center, then maps to (yaw, pitch) via:

yaw   = (x - 0.5) × 360°    // x in [0, 1] from the equirect
pitch = -(y - 0.5) × 180°

The viewer's visible gaze marker is a thin oxblood ring (cosmetic only, ~0.04m thick annulus). The actual raycaster hit target is an invisible 0.45m circle at each hotspot — gives ~10× the catch area so the gaze reticle reliably triggers the hotspot when the user holds their gaze in the rough vicinity. 👁️

🔊 How Runtime Audio Works

The browser plays MP3s via the <audio> element. The bytes come from /api/tts/{sceneId}/{hotspotId} — a Next.js server route that:

🔎 Looks up the hotspot's voice.text + voice.voiceId in the scene JSON
🔐 Hashes that pair (SHA-256) and checks cache/tts/{hash}.mp3
💨 On miss: calls ElevenLabs, writes the bytes to disk, returns them
⚡ On hit: streams the cached file

Each unique line burns its character cost once for the lifetime of the cache. Twelve scenes × ~120 chars/line ≈ 1500 chars total, well under the 10k chars/month free tier. 🎶

☁️ Vercel Deploy (Optional)

npm run dev on a laptop is the demo target. If you want a public share link:

vercel

⚠️ Caveat: Vercel functions are stateless, so the cache/tts/ directory does not survive cold starts in production. Either commit the prewarmed MP3s into the repo (remove cache/ from .gitignore and git add cache/tts/*.mp3), or move the cache to Vercel Blob storage (free tier).

📁 Project Structure

app/                          — ⚡ Next.js routes (page.tsx, viewer/, editor/, api/tts/)
components/                   — 🧩 UI components (LandingClient, MapPageClient, VRScene, InfoPanel, etc.)
components/ui/                — ✨ Editorial design primitives (Rule, SplitText, Kicker, Button, PaperGrain)
data/locations.json           — 📍 12 NYC locations
data/scenes/                  — 🎯 Per-scene hotspot JSON (generated)
data/period-research.json     — 📚 Web-grounded visual research per (loc, era) (generated)
data/location-summaries.json  — 📝 Sidebar period blurbs (generated)
data/panorama-sources.json    — 🗺️ Mapillary contributor attribution (generated)
scripts/                      — 🔧 Build-time pipelines
public/panoramas-modern/      — 📸 Modern Mapillary 360s (input to aging step)
public/panoramas/             — 🎨 Aged 360 JPGs (final output)
public/infopanels/            — 🖼️ Baked editorial paper-card PNGs (generated)
public/fonts/raw/             — 🔤 TTF files for the canvas bake (auto-downloaded)
public/ambient/               — 🎵 Looping ambient MP3s (drop in CC0 audio from freesound.org)
cache/tts/                    — 💾 Runtime ElevenLabs cache (gitignored)
lib/                          — 📦 Server-side helpers (scenes loader, ElevenLabs client, types)

🚧 Hard Constraints (Intentional)

💰 $0 to ship the demo locally beyond the one-time build pipeline (~$1 total). Runtime spend is bounded by the number of unique voice lines, not the number of users, because of the disk cache.
⚠️ A-Frame + Next.js SSR breaks on first import. All A-Frame mounts must use dynamic({ ssr: false }).
🖼️ Image-to-image aging is bounded by the modern Mapillary frame's geometry. Nano Banana can re-skin facades, swap signage, change dress and vehicles, and remove modern displays — but it cannot add buildings that weren't in the source photo, can't move the camera, and can't recreate literal historical geometry.

🙏 Attribution

✨ Period imagery aged via Google Gemini 2.5 Flash Image (Nano Banana).
🎙️ Voice clips synthesised with ElevenLabs.
📚 Period research grounded in citations returned by Google Search via Gemini's grounding tool.

Name		Name	Last commit message	Last commit date
Latest commit History 10 Commits
app		app
components		components
data		data
lib		lib
public		public
scripts		scripts
.env.local.example		.env.local.example
.gitignore		.gitignore
README.md		README.md
aframe.d.ts		aframe.d.ts
next.config.mjs		next.config.mjs
package-lock.json		package-lock.json
package.json		package.json
postcss.config.mjs		postcss.config.mjs
tailwind.config.ts		tailwind.config.ts
tsconfig.json		tsconfig.json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

🏛️ Vantage — The Vantage Chronicle

🗺️ System Architecture

🎬 The 12 Scenes

⚙️ How It Works

🛠️ Tech Stack

📰 Design System — The Vantage Chronicle

🚀 Setup

📋 Prerequisites

📦 Install + First Run

📱 Phone + Cardboard Testing (HTTPS Required)

🔧 Build-time Pipeline Scripts

🗺️ Mapillary Coverage Gaps

➕ Adding a New Scene

🎯 Hotspot Positioning + Gaze Hit Detection

🔊 How Runtime Audio Works

☁️ Vercel Deploy (Optional)

📁 Project Structure

🚧 Hard Constraints (Intentional)

🙏 Attribution

About

Uh oh!

Releases

Packages

Uh oh!

Contributors

Uh oh!

Languages

Folders and files

Latest commit

History

Repository files navigation

🏛️ Vantage — The Vantage Chronicle

🗺️ System Architecture

🎬 The 12 Scenes

⚙️ How It Works

🛠️ Tech Stack

📰 Design System — The Vantage Chronicle

🚀 Setup

📋 Prerequisites

📦 Install + First Run

📱 Phone + Cardboard Testing (HTTPS Required)

🔧 Build-time Pipeline Scripts

🗺️ Mapillary Coverage Gaps

➕ Adding a New Scene

🎯 Hotspot Positioning + Gaze Hit Detection

🔊 How Runtime Audio Works

☁️ Vercel Deploy (Optional)

📁 Project Structure

🚧 Hard Constraints (Intentional)

🙏 Attribution

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages