Search places by address and any other OSM tag

[escapedcat]

Rebuilds the undocumented GET /v4/search into an omnisearch over areas and places.
Searching hamburg now returns the Hamburg area and the places whose addr:city is
Hamburg. Localized name:* tags are searched for free.

This is the server-side version of the client-side search btcmap.org ran until 2025-07-31,
which matched Object.values(tags) — every OSM tag value, never a key. It was removed in
01127138 during the v4 Places migration and never replaced; the API has only ever matched
$.tags.name.

Per @bubelov's steer in chat: /v4/places/search stays scoped to places and is not
touched; the omnisearch lives on the global endpoint.

What changed

• Places match every OSM tag value via json_each — never a key, so q=addr matches
  nothing. All query words must match, each possibly in a different tag, so q=hamburg cafe
  means a place with addr:city=Hamburg and cuisine=cafe. (The old client-side version
  was an any-word OR.)
• Ranking: exact name > name prefix > name substring > any other tag. Areas precede places at
  equal rank. Optional lat/lon break remaining ties by proximity — this matters, because
  a query like hamburg leaves thousands of places tied at the lowest rank.
• Ranking, LIMIT and OFFSET moved into SQL. The handler previously loaded every match
  into a Vec<Element> before slicing 20 rows — untenable once q=yes matches every
  payment:*=yes place. Each side fetches its own top offset+limit; the handler merges.
• Areas match name and alias only. Their tags hold the full geo_json polygon, so matching
  tag values there would substring-match coordinate digits.
• Area rows now carry bbox, so a client can fly the map to a city. The world-rectangle
  default is reported as absent rather than inviting a zoom-to-planet.
• Documented at docs/rest/v4/search.md. It was previously undocumented.

Compatibility

• GET /v4/places/search is unchanged.
• JSON-RPC search is unchanged. The new matching lives in new DB functions, so
  btcmap-admin's area picker is unaffected. git diff touches zero lines in src/rpc/search.rs
  and adds/removes zero lines mentioning select_by_search_query.
• The type discriminator for a place changes from "element" to "place", matching v4
  naming. Safe: no client calls this endpoint.

Defects fixed in passing

• LIKE wildcards were never escaped, so q=% matched every row.
• LOWER(x) LIKE UPPER(?) only ever worked because SQLite's LIKE is already
  ASCII-case-insensitive; the two cancel each other's intent.
• SearchedPlace::from does it.lat.unwrap() and panics on an element with NULL
  coordinates. The new query excludes them. (The same latent panic remains on
  /v4/places/search — out of scope here.)
• Area search deserialised every matching geo_json polygon into a serde_json::Map.

The first commit reformats src/service/wallet.rs. It is unrelated, but rustfmt on current
stable reformats xpub_balance and cargo fmt -- --check fails on master without it.

Known limitations

• q=cafe does not match Café. SQLite's LIKE folds ASCII case only and there is no ICU
  collation; the old client-side toLowerCase() had the identical limitation. Fixing it means
  FTS5 with unicode61 remove_diacritics=2.
• Still a leading-wildcard full table scan, as before. All-value matching multiplies the
  per-row cost by the tag count. FTS5 over a precomputed search document is the fix if it bites.
• Matching every tag value is deliberate — it is the behaviour that ran for three years. It
  does mean q=yes matches every payment:*=yes place and q=2025 every check_date.

Verification

cargo fmt --check, cargo clippy -- -D warnings and cargo test all pass (537 tests).
31 new tests. Also exercised against a running server with seeded data: q=hamburg returns
the area then the places; q=hamburg cafe narrows to one; q=addr and q=%%% return
nothing; q=ハンブルク matches a name:ja tag.

🤖 Generated with Claude Code

[dadofsambonzuki]

Worth using this rather than rolling our own?

https://www.sqlite.org/fts5.html

[escapedcat]

Worth using this rather than rolling our own? https://www.sqlite.org/fts5.html

(Claude here, drafted for @escapedcat — he asked me to dig into this and reply.)

Good instinct, and you're right that there's a performance problem — but measuring it at production scale (28,802 places, benchmarked with 30k) turned up a twist worth knowing before we reach for FTS5.

The slowness isn't the text matching — it's the JSON. We re-parse every merchant's tag blob on every query via json_each. Release build, 30k rows:

approach	q=hamburg
master's old name-only LIKE	28 ms
this PR (json_each over all tag values)	85 ms
precompute the searchable text into a column, plain LIKE	7 ms
FTS5 trigram index	2 ms

So this PR is ~3× slower than master, and /v4/search runs the predicate twice (select + count). Fair catch. But a precomputed search_doc column gets us to 7 ms — 12× faster than this PR, 4× faster than master — with no virtual table, no new query language, and zero behaviour change. That's where most of the win is.

FTS5's default tokenizer would also be a regression. It indexes whole words, so it can't match inside one:

normal FTS index   MATCH 'buck'   → 0   (can't find "Starbucks" by its middle)
normal FTS index   MATCH 'star*'  → 1   (prefixes are fine)
what we do today   LIKE '%buck%'  → 1

Matching an infix is exactly what the old client-side search did (value.includes(word)) and what @bubelov asked to bring back. Only the trigram tokenizer preserves it — and it costs ~3.6× the text size in index, needs a write-path sync, and can't handle search terms under 3 characters (so q="cafe am markt" would silently drop the am).

Where FTS5 genuinely wins: it's the only way to make cafe match Café (via trigram + MATCH, not LIKE) — the one limitation I'd flagged as unfixable. That's a real future capability.

Proposal: land the search_doc column now (kills the regression, no behaviour change), and layer trigram FTS5 on the same column later when we want diacritic-folding and sub-ms misses. Doing FTS5 first would mean paying storage + a sync path for a 5 ms gain while inheriting the 3-char-per-word limit.

Numbers are in-memory SQLite; prod is on-disk WAL so absolutes will shift, but the ratios should hold. Happy to add the column to this PR or as a follow-up — @escapedcat's call.

[Copilot]

Pull request overview

This PR rebuilds GET /v4/search into a single “omnisearch” endpoint that searches both areas and places, expanding place matching from tags.name to any OSM tag value and adding deterministic, SQL-driven ranking/pagination (with optional proximity tie-breaking). It also documents the endpoint in the v4 REST docs.

Changes:

• Added shared search utilities (escape_like, split_words) with tests to safely build LIKE patterns and cap query word count.
• Implemented new DB search/count queries for areas and elements (tag-value search) with ranking and row limiting done in SQL.
• Reworked the /v4/search REST handler to merge ranked area+place results, enforce input constraints, and publish docs.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/service/wallet.rs	rustfmt-only change to keep formatting consistent with current stable toolchain.
src/service/search.rs	Introduces LIKE escaping + query word splitting helpers (with tests) for new search SQL.
src/service/overpass.rs	Adds a test-only helper to create mock elements with multiple tags.
src/service/mod.rs	Exposes the new service::search module.
src/rest/v4/search.rs	Replaces prior search behavior with merged ranked area+place omnisearch, SQL-backed pagination, and new response shape.
src/db/main/element/queries.rs	Adds async wrappers for new tag-value search and count queries + re-exports RankedElement.
src/db/main/element/blocking_queries.rs	Implements ranked tag-value search/count SQL with wildcard escaping and tie-breaking.
src/db/main/area/queries.rs	Adds async wrappers for new area search/count queries + re-exports RankedArea.
src/db/main/area/blocking_queries.rs	Implements ranked area search/count SQL (name/alias only) and bbox reporting behavior.
docs/rest/v4/search.md	Documents the new /v4/search omnisearch behavior, parameters, ordering, and response types.
docs/rest/v4/README.md	Adds the Search endpoint to the v4 documentation index.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[escapedcat]

Addressed in 2d5d38d — clamped the conversion so total_count/pagination.total saturate instead of wrapping past u32::MAX (unreachable in practice, since total is bounded by the table size, but cheap to make obviously correct). — drafted with Claude.