Scaling improvements to allow very large collection pushes and queries

README.md

@@ -215,20 +215,25 @@ The protocol and the platform are documented together:
 
 ### Key API endpoints
 
-All pushes use the negotiate protocol — a three-step flow similar to git's pack negotiation:
-
-| Endpoint                                         | Purpose                                                     |
-| ------------------------------------------------ | ----------------------------------------------------------- |
-| `POST .../versions/negotiate`                    | Start a push session (server returns which hashes it needs) |
-| `POST .../versions/negotiate/:sessionId/records` | Send only the needed records (NDJSON)                       |
-| `POST .../versions/negotiate/:sessionId/commit`  | Validate, hash, and create the immutable version            |
-| `GET .../versions/:semver/manifest`              | Version manifest (add `?since=` for delta)                  |
-| `GET .../versions/:semver/records`               | Paginated records                                           |
-| `GET .../versions/:semver/diff?from=...`         | Diff between two versions                                   |
-| `POST /api/records/batch`                        | Fetch records by hash (JSONL stream)                        |
-| `GET /api/records/:hash/provenance`              | Find all collections containing a record                    |
-| `POST .../fork`                                  | Fork a collection (copies manifest, not data)               |
-| `GET /api/schemas`                               | Search schemas across all collections                       |
+All pushes use the negotiate protocol — a three-step flow similar to git's pack negotiation. Two of
+those steps have a chunked form for collections that don't fit in a single request: the manifest can
+upload in pieces, and the commit can run in the background. A chunked, asynchronous push produces
+the same version hash as the simple one.
+
+| Endpoint                                          | Purpose                                                                                                     |
+| ------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
+| `POST .../versions/negotiate`                     | Start a push session (server returns which hashes it needs)                                                 |
+| `POST .../versions/negotiate/:sessionId/manifest` | Upload the manifest in NDJSON chunks, when it is too large to send in one body                              |
+| `POST .../versions/negotiate/:sessionId/records`  | Send only the needed records (NDJSON)                                                                       |
+| `POST .../versions/negotiate/:sessionId/commit`   | Validate, hash, and create the immutable version. `?async=true` returns 202 and finalizes in the background |
+| `GET .../versions/negotiate/:sessionId`           | Session status, and the result or error of an async commit                                                  |
+| `GET .../versions/:semver/manifest`               | Version manifest (add `?since=` for delta; both keyset-paginated)                                           |
+| `GET .../versions/:semver/records`                | Paginated records                                                                                           |
+| `GET .../versions/:semver/diff?from=...`          | Diff between two versions                                                                                   |
+| `POST /api/records/batch`                         | Fetch records by hash (JSONL stream)                                                                        |
+| `GET /api/records/:hash/provenance`               | Find all collections containing a record                                                                    |
+| `POST .../fork`                                   | Fork a collection (copies manifest, not data)                                                               |
+| `GET /api/schemas`                                | Search schemas across all collections                                                                       |
 
 ## Privacy
 

docker-compose.local.yml

@@ -11,13 +11,22 @@ services:
       POSTGRES_USER: underlay
       POSTGRES_PASSWORD: underlay
       POSTGRES_DB: underlay
+    # Sized for bulk ingest as well as everyday dev. A multi-million-record push
+    # sorts the whole record set twice to fold the version digests, and builds
+    # indexes over it — at 8MB work_mem that spills to disk in many small merge
+    # passes. max_connections is low, so the worst case here is bounded.
     command: >
       postgres
-      -c shared_buffers=256MB
-      -c effective_cache_size=512MB
-      -c work_mem=8MB
-      -c maintenance_work_mem=64MB
+      -c shared_buffers=1GB
+      -c effective_cache_size=3GB
+      -c work_mem=64MB
+      -c maintenance_work_mem=512MB
+      -c max_wal_size=4GB
+      -c checkpoint_completion_target=0.9
       -c max_connections=50
+    # Matches the production stack: Docker's 64 MB /dev/shm default is too small
+    # for Postgres parallel query workers once tables get large.
+    shm_size: 1gb
     ports:
       - '5433:5432'
     volumes:

docker-compose.yml

@@ -8,7 +8,20 @@ services:
       - .env
     environment:
       NODE_ENV: production
-      NODE_OPTIONS: '--max-old-space-size=448'
+      # Push commits accumulate per-record structures in memory, so the heap has
+      # to scale with the largest collection pushed, not with steady-state
+      # traffic. 448 MB could not commit much past ~100k records.
+      #
+      # Measured: a 500k-record push costs ~900 MB of heap above baseline
+      # (~1.8 KB/record), so 2 GB carries a ~1M-record push with headroom.
+      # Raise APP_HEAP_MB (and APP_MEMORY_LIMIT with it) for a one-off larger
+      # ingest rather than making it the standing default — dev and prod share
+      # one 16 GB box, so the defaults here are paid four times over.
+      #
+      # Do not read this as "3.11M records will fit if we go high enough": at
+      # that size the push path needs the streaming commit, not a bigger number.
+      # See planning/local/demos/arxiv-ingest-measurements-v1.md.
+      NODE_OPTIONS: '--max-old-space-size=${APP_HEAP_MB:-2048}'
       PORT: ${PORT:-3000}
       UNDERLAY_MODE: ${UNDERLAY_MODE:-origin}
       UNDERLAY_UPSTREAM: ${UNDERLAY_UPSTREAM:-}
@@ -20,7 +33,15 @@ services:
       replicas: ${APP_REPLICAS:-2}
       resources:
         limits:
-          memory: 640m
+          # Must exceed APP_HEAP_MB — the V8 heap is only part of RSS.
+          #
+          # Sizing on the shared 16 GB box: dev + prod × 2 replicas = 4 app
+          # containers. They idle at a few hundred MB each and only one climbs
+          # at a time (a push is one request on one replica of one stack), so
+          # the realistic peak is 3× idle + 1× limit, not 4× limit. Reservations
+          # stay low deliberately — they are what Swarm schedules against, and
+          # over-reserving would strand memory the other stack needs.
+          memory: ${APP_MEMORY_LIMIT:-2560m}
           cpus: '1.0'
         reservations:
           memory: 384m
@@ -109,6 +130,13 @@ services:
       -c random_page_cost=1.1
     volumes:
       - pgdata:/var/lib/postgresql/data
+    # Postgres allocates dynamic shared memory in /dev/shm for parallel query
+    # workers. Docker's 64 MB default is enough for small scans and not enough
+    # for a parallel aggregate over a few million rows, which fails with
+    # "could not resize shared memory segment ... No space left on device" —
+    # observed on a 500k-record commit.
+    tmpfs:
+      - /dev/shm:size=1g
     ports:
       - '${DB_PORT:-5432}:5432'
     networks: