REST converters: page by id windows - #232
Open
ivorbosloper wants to merge 2 commits into
Open
Conversation
m-mohr
reviewed
Sep 11, 2026
This was referenced Sep 11, 2026
ivorbosloper
force-pushed
the
split/rest-mixin
branch
from
September 12, 2026 12:51
090f4d7 to
400191f
Compare
Paging by `sortBy` + `attribute > last_id` asks the server to sort the whole layer for every page. On Cantabria's joined SIGPAC layers that costs about 100 seconds per request — days for one edition — where a range filter on the same indexed key answers in one or two. Galicia is 15.8 million recintos per edition, in pages of 1000. Paging by half-open id windows removes the sort: the key is unique, so a window of `page_size` ids cannot overflow a page, and gaps only produce empty windows, which are skipped. Pages cached by the old scheme are keyed by the previous page's last id; on a dense layer those coincide exactly with a window, so one is reused when its own ids prove it covers the window completely. Three more things this turned up, each invisible in the output: - **Layer ids repeat across services.** Every SIXPAC_<year> MapServer has its Recintos layer at id 2, so a cache keyed on the layer alone served one year's pages for another. The service is part of the key now. - **An error response was cached as if it were data.** Esri answers a failed query with 200 and a JSON error body; that body was written to the cache file and every later run read it back. So was a download that broke off halfway. Neither survives now. - **Joined layers qualify every field with its table name**, so "OBJECTID" matches nothing and the paging filter silently returns everything. The real key field is discovered from a one-row probe before paging starts. `rest_format`, the fixture-reading branch and the way `rest_params["where"]` is combined with the paging filter are unchanged; the test that pins them is updated for the new clause, which now reads `OBJECTID>0 AND OBJECTID<=1000 AND (<the converter's own filter>)`. tests/test_converter_rest.py serves a small fake service of five features in pages of two, which is enough to hold every one of these: the window arithmetic, the cache naming, the two guards, the qualified key, and the reuse of a legacy page only when its ids cover the window. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG
ivorbosloper
force-pushed
the
split/rest-mixin
branch
from
September 12, 2026 15:11
400191f to
07aafe9
Compare
ivorbosloper
changed the title
ivorbosloper
force-pushed
the
split/rest-mixin
branch
2 times, most recently
from
September 12, 2026 16:14
bfa7242 to
6fa8a3b
Compare
Co-authored-by: Matthias Mohr <m.mohr@moregeo.it>
ivorbosloper
force-pushed
the
split/rest-mixin
branch
from
September 12, 2026 16:20
6fa8a3b to
9271232
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Paging by
sortBy+attribute > last_idsorts the whole layer per page: ~100 s per request on Cantabria's joined layers, and Galicia is 15.8M recintos per edition. Half-open id windows remove the sort, and pages cached by the old scheme are reused where their ids prove they cover a window.Three bugs found on the way, none visible in the output:
SIXPAC_<year>has Recintos at id 2, so one year's cache served anotherRECINTOS.OBJECTID, so the paging filter matched nothing and returned everythingNot a copy: main's
rest_formatandwhere-clause work is kept. Newtests/test_converter_rest.py, mixin coverage 19% → 94%.🤖 Generated with Claude Code
https://claude.ai/code/session_01DVx9uQV2QPM8ecPAY3ZjXG