Repository navigation
Replies: 3 comments
|
Production numbers for the field-selection half of this, in case they are useful. Same conclusion from a different angle: on a content site the body is most of the database, and every list read carries it. Measured today on a live EmDash site (D1, Cloudflare Workers, 0.38.0):
The part I had not expected is what it does to how you write a page. A I have put an implementation of the field-selection half behind this in #3204, as a draft: a It is a draft on purpose and I am not asking for it to be merged ahead of a decision here. If the shape is wrong, or field selection should wait for the relations design so the two share an API, I would rather know that than have the PR reviewed. One design question that came out of building it, which belongs in this thread more than in the PR. A projected entry is a partial entry, so anything deriving a value from an unselected column sees nothing rather than falling back: an excerpt computed from |
|
Design PR #4724 proposes |
|
Design PR #4849 proposes |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Summary
A listing page may only need an entry's title, URL and publication date. The collection API prepares the full content and relationship data even when the page never reads most of it.
In a synthetic listing fixture on Cloudflare Workers with D1, a page using
getEmDashCollectionconsumed substantially more CPU than a page selecting only the four columns it rendered. Both produced the same HTML for 500 published entries. This benchmark's baseline already includes the Archives improvements in emdash-cms/emdash PR 3068.I would like to discuss optional top-level field selection and optional relationship loading on
getEmDashCollection, preserving the current default behaviour. The existinggetPublishedDatesquery in that PR handles one specific need. A general selection API would let other callers express which content fields and relationships they need.What the collection path prepares
In the tested uncached D1 path, the loader selects every content column for the matching records, together with folded byline and taxonomy data. The database binding materialises those rows; the loader maps them into entries, converts dates and processes JSON-looking content strings. Successfully parsed JSON values pass through local-media normalisation. Invalid JSON strings remain unchanged, and native object values do not take that normalisation path.
The query layer then prepares bylines and terms, primes related request-cache entries and passes the collection through a JSON-safe snapshot/revive format, even without an object-cache backend. Request-cache hits can skip the load, and preview/edit reads have separate handling.
These steps support the current full-result contract. The listing fixture, however, reads only id, slug, title and publication date. Reducing the result after it reaches the page would leave the database result conversion and entry preparation work in place. Field selection needs to reach the loader's SQL, and relationship selection needs to control both the folded aggregates and subsequent hydration.
Live listing measurements
Two runs used a synthetic D1 database with 500 published posts over 24 months, 8 bylines and 2 taxonomies. Both use the same core source at
bdafeeefrom emdash-cms/emdash PR 3068, with core version 0.37.0, Astro 7.0.0, adapter 14.0.0, no object cache or plugins,session: "primary-first"and default placement. The runs were approximately six hours apart on 12 September 2026.Both columns below use the same PR-based baseline. They are separate executions of the listing experiment, not upstream main versus the PR. Run B also included two additional optimisation candidates, discussed below; this table shows only its baseline arm.
Values are minimum / median / maximum of eligible per-isolate CPU medians, using at least 19 warm measurements per route. CPU comes from Workers Tail for the complete request, including deferred invocation work. Each run included its own controls in an interleaved schedule.
The EmDash rows cover 16 eligible isolates in Run A and 15 in Run B's baseline arm. Bare Worker controls cover 4 isolates in each run; Astro controls cover 2 and 4. The direct-D1 EmDash HTML route still passes through EmDash middleware and layout prefetch. JSON Accept suppresses HTML layout prefetch; the fixture still renders HTML. Corresponding page bodies were byte-identical for the same row count, which checks the fixture output rather than every returned API value.
The direct query returns fewer columns and performs no relationship hydration. This comparison motivates a smaller-result API; it does not isolate the collection function's CPU or measure the savings of the proposed API. General projection has not been implemented or benchmarked.
Identical bundles also produced different CPU distributions between isolates and runs. The physical cause remains unresolved. The useful comparison here is between data-loading routes within each run, with all eligible isolates retained, rather than treating either run's absolute CPU as a fixed cost of the code.
Earlier comparisons and smaller changes
There was also a separate live comparison against the pre-PR upstream base. On 11 September, base
b750ddfversus the initial PR atdb122778measured Archives post CPU medians of 79 and 49 ms. Home, minimal HTML and RSS medians stayed at 12, 8 and 4 ms. This was an Archives post-page experiment, separate from the listing measurements above.A later comparison of the initial PR against
bdafeee, which adds the Archives date projection, measured post medians of 47 and 28 ms. Home rose from 12 to 18 ms and RSS from 4 to 5 ms. A subsequent diagnostic serving the identical previous-PR bundle from both Workers also produced different CPU values. The control increases remain unexplained; the separate runs cannot be combined into a single 79-to-49-to-28 progression. These observations support investigating narrower reads while keeping the scope of each result explicit.Run B then compared its baseline with two additional changes: replacing snapshot/revive with shallow copies when no object-cache backend is configured, and preserving unchanged branches during local-media normalisation. Four copies per arm ran in one interleaved schedule. The 500-row collection route with JSON Accept had medians of isolate medians of 49 ms for the baseline, 54.25 ms for the first change and 47.5 ms for both. Their relative performance reversed between passes, and control routes moved too. Neither candidate established a consistent live CPU improvement; they remain separate from the general projection proposal.
Local workerd profiles pointed to row conversion, entry mapping, snapshots, relationship preparation and garbage collection. Those samples identify work to investigate; they are not percentages of live CPU that projection is guaranteed to remove. Cold-start module initialisation is a separate issue from these warm collection reads.
Proposal
Only
selectandwithbelow are proposed additions; their names are provisional. The other options already exist.The intended contracts and open design questions are:
selectandwithretains today's content fields and relationship loading. Relationship options default to the existing behaviour.EXISTSconditions required to find the correct entries, even when their relationship data is not returned. Sorting by an unselected field must also remain valid.Per-isolate results, method details and the separate experiment summaries are available as supporting material.
Would optional field selection and selective relationship loading be a good fit for
getEmDashCollection? In particular, how should selected results interact with preview/edit annotations and subsequent relationship reads?All reactions