Recipe： Per tier sparse GSI markers

Recipe: Per-tier sparse GSI markers

The SQL-equivalent question this answers: same as Pattern 1 — "all states globally", "all facilities globally". But here we want per-tier control over projection, storage cost, or eventually-sharded partitioning. One GSI per tier, not one GSI spanning every tier.

Pattern: one sparse GSI per tier you care about, each keyed on a tier-specific marker attribute that's present only on records of that tier. Smaller per-tier indexes, finer control over what each index projects, and a sharding axis per tier when leaf-tier scale outgrows Pattern 1.

When Pattern 1 isn't enough

Pattern 1 (typeField + single GSI) is the simplest cross-partition tier-listing recipe: one GSI, every record indexed, partition per tier label. It works until one of these trips:

1. Leaf-tier scale. Every vehicle sits on kind = 'vehicle' → one GSI partition holds every vehicle in the fleet. At millions of records you blow through the 3000 RCU / 1000 WCU partition ceiling.
2. Per-tier projection differs. You want 'all' projection on the ~50 states (tiny, convenient for reads) and 'keys-only' on vehicles (cheap, hydrate only when needed). Pattern 1's single GSI forces one projection for everyone.
3. Per-tier write costs differ. Every base-table write updates the GSI. When vehicles get 1000× more writes than states, the GSI partition for kind = 'vehicle' bears the whole write load.

Per-tier GSIs split those concerns: each tier pays for its own GSI, which is sized (and projected) for that tier's workload.

The pattern

One sparse GSI per tier, each keyed on a marker attribute present only on records of that tier:

import {Adapter} from 'dynamodb-toolkit';

const adapter = new Adapter({
  client: docClient,
  table: 'rentals',

  technicalPrefix: '_',
  keyFields: ['state', 'facility', 'vehicle'],
  structuralKey: '_sk',
  typeLabels: ['state', 'facility', 'vehicle'],

  indices: {
    // Sparse GSI for states. Only state records have `_stateMarker`, so
    // only state records are in the index. ~50 entries; 'all' projection
    // because the set is small.
    'all-states': {
      type: 'gsi',
      pk: '_stateMarker',
      sk: 'state',                           // sort by state code
      projection: 'all'
    },
    // Sparse GSI for facilities. Thousands of records; keys-only keeps
    // index storage small, second hop hydrates full items only when asked.
    'all-facilities': {
      type: 'gsi',
      pk: '_facilityMarker',
      sk: '_sk',                             // structural key as secondary sort
      projection: 'keys-only'
    }
  },

  hooks: {
    prepare: (item, isPatch) => {
      if (isPatch) return item;
      const tier =
        item.vehicle !== undefined ? 'vehicle' : item.facility !== undefined ? 'facility' : 'state';
      if (tier === 'state') return {...item, _stateMarker: 'Y'};
      if (tier === 'facility') return {...item, _facilityMarker: 'Y'};
      // Vehicles — no marker. Not in any of these GSIs.
      return item;
    }
  }
});

Each marker attribute is a constant ('Y') — the value doesn't matter, only that the attribute is present (sparse by absence). The marker name is what differentiates the per-tier indexes.

Query shape

// All state records, sorted by state code:
const states = await client.send(
  new QueryCommand({
    TableName: 'rentals',
    IndexName: 'all-states',
    KeyConditionExpression: '#pk = :pk',
    ExpressionAttributeNames: {'#pk': '_stateMarker'},
    ExpressionAttributeValues: {':pk': 'Y'}
  })
);

// All facilities globally, keys-only → hydrate via BatchGet:
const facilityKeys = await client.send(
  new QueryCommand({
    TableName: 'rentals',
    IndexName: 'all-facilities',
    KeyConditionExpression: '#pk = :pk',
    ExpressionAttributeNames: {'#pk': '_facilityMarker'},
    ExpressionAttributeValues: {':pk': 'Y'}
  })
);
const facilities = await adapter.getByKeys(
  facilityKeys.Items.map(k => ({state: k.state, facility: k.facility}))
);

Cost & scale

Compared to Pattern 1 (single typeField GSI):

	Pattern 1 (single GSI)	Pattern 2 (per-tier GSIs)
Number of GSIs	1	N (one per tier you care about)
Write amplification	1× base-table writes	1× per GSI the record matches (sparse)
Partition-per-tier hot-spot	Every tier shares one GSI	Each tier has its own GSI
Per-tier projection control	No	Yes
Storage	Full replication × 1	Sparse: only each tier's records replicated

Write amplification is where Pattern 2 pays off for leaf tiers: if a vehicle record carries no marker, it's in zero of these GSIs → zero GSI write cost. Compare with Pattern 1, where every write updates by-kind regardless.

Partition hot-spot is not automatically solved. A tier with ~50 states (all sharing _stateMarker = 'Y') fits fine on one partition. A tier with millions of vehicles all sharing _vehicleMarker = 'Y' still hot-partitions. For that, shard the marker value.

Sharded-marker variant — the scale escape hatch

When a tier outgrows a single partition, replace the constant marker value with a hash bucket:

hooks: {
  prepare: (item, isPatch) => {
    if (isPatch) return item;
    if (item.vehicle !== undefined) {
      // Hash the VIN into one of 16 shards. Same VIN always lands on the
      // same shard (stable).
      const shard = hashToShard(item.vehicle, 16);
      return {...item, _vehicleMarker: `Y#${shard}`};
    }
    return item;
  }
}

// Then the "all vehicles" query becomes 16 Queries run in parallel:
const pages = await Promise.all(
  Array.from({length: 16}, (_, s) =>
    client.send(new QueryCommand({
      TableName: 'rentals',
      IndexName: 'all-vehicles',
      KeyConditionExpression: '#pk = :pk',
      ExpressionAttributeNames: {'#pk': '_vehicleMarker'},
      ExpressionAttributeValues: {':pk': `Y#${s}`}
    }))
  )
);
const allVehicles = pages.flatMap(p => p.Items);

Scatter-gather: fan out N Queries, merge results. Trades round-trip count for per-partition throughput. Pick N based on your peak read rate divided by the 3000 RCU ceiling — e.g. 50k RCU peak → at least 17 shards.

When this pattern fits

• Tier cardinalities / write rates differ enough that one shared GSI is wasteful.
• You want per-tier projection control — 'all' for small tiers, 'keys-only' for big ones.
• You're already near Pattern 1's hot-partition ceiling on at least one tier.
• Scaling beyond 3000 RCU / 1000 WCU per tier is on the horizon → the sharded-marker variant slots in without a schema migration (add the bucket suffix on the next full prepare change).

When it doesn't fit

• Tier listings are always within a partition — use Pattern 3 (LSI within-partition) for strictly cheaper storage.
• Tiers have comparable cardinality + write rates → Pattern 1 is simpler and one GSI is enough.
• You haven't hit scale yet → Pattern 1 is the no-overthinking default. Premature per-tier optimisation costs developer time + index storage without a real scale problem.

Related

• Pattern 1 — single GSI on auto-populated typeField — the no-sharding default.
• Pattern 3 — within-partition tier filtering via sparse LSI — cheaper when you don't need cross-partition reach.
• AWS docs: Partitions and data distribution — where the 3000 RCU / 1000 WCU ceiling comes from.