fix: orderable fractional indexing case-sensitivity issue with PostgreSQL (#14867)

## fix: orderable fractional indexing case-sensitivity issue with
PostgreSQL

### Problem

When using the `orderable` feature with PostgreSQL, reordering documents
to the "first" position generates keys that break subsequent reordering
operations.

#### Scenario

1. Create two orderable documents → keys are `a0`, `a1`
2. Drag `a1` before `a0` (to make it first)
3. The system generates key `Zz` for the moved document
4. **Result**: Ordering is completely broken - cannot reorder any
documents anymore

#### Root Cause

The [fractional indexing
algorithm](https://observablehq.com/@dgreensp/implementing-fractional-indexing)
uses:
- **Uppercase `A-Z`** for "smaller" integer keys
- **Lowercase `a-z`** for "larger" integer keys

This relies on **ASCII ordering** where `'Z'` (code 90) < `'a'` (code
97), so `Zz < a0`.

**However**, PostgreSQL's default collation (`en_US.UTF-8`) uses
**case-insensitive comparison**, treating `'Z'` as `'z'`. This means:
- ASCII: `Zz < a0` ✓
- PostgreSQL: `Zz` treated as `zz`, so `zz > a0` ✗

This mismatch causes:
1. Database queries return incorrect adjacent documents
2. The `generateKeyBetween` function receives arguments in wrong order
3. Subsequent reorder attempts either error out or create more broken
keys

### Solution

Modified the fractional indexing algorithm to use only characters that
sort consistently across **all** database collations:

| Range | Old Encoding | New Encoding |
|-------|--------------|--------------|
| Small keys | `A-Z` (uppercase) | `0-9` (digits) |
| Large keys | `a-z` (lowercase) | `a-z` (lowercase, unchanged) |

**Key insight**: Digits (`0-9`) **always** sort before letters in both
ASCII ordering and case-insensitive collations.

#### Before (broken)
```
decrementInteger('a0') → 'Zz'
'Zz' < 'a0' in ASCII ✓
'Zz' > 'a0' in PostgreSQL (case-insensitive) ✗
```

#### After (fixed)
```
decrementInteger('a0') → '9z'
'9z' < 'a0' in ASCII ✓
'9z' < 'a0' in PostgreSQL ✓
'9z' < 'a0' in MongoDB ✓
'9z' < 'a0' in SQLite ✓
```

### Changes

**`packages/payload/src/config/orderable/fractional-indexing.js`**
   - Changed integer part encoding to use digits for "small" range
   - Maintains backward compatibility with existing `a-z` keys
- Legacy `A-Z` keys are still parsed (for backward compatibility) but
won't be generated

### Backward Compatibility

- ✅ Existing keys starting with `a-z` continue to work correctly
- ⚠️ Existing keys starting with `A-Z` (uppercase) will be parsed but
may sort incorrectly in case-insensitive databases. Users with such keys
should run a migration to regenerate them.

### Testing

Verified the algorithm produces correct ordering:

```javascript
generateKeyBetween(null, null)     // → 'a0'
generateKeyBetween(null, 'a0')     // → '9z' (previously 'Zz')
generateKeyBetween(null, '9z')     // → '9y'

// Sorting works correctly
['9z', 'a0', 'a1'].sort()          // → ['9z', 'a0', 'a1'] ✓
```

### Related

- Fractional Indexing algorithm:
https://observablehq.com/@dgreensp/implementing-fractional-indexing
- Original library: https://github.com/rocicorp/fractional-indexing

---------

Co-authored-by: Paul Popus <paul@payloadcms.com>

Author: vladyslavprosolupov

packages/payload/src/config/orderable/fractional-indexing.js

@@ -1,12 +1,32 @@
 // @ts-no-check
 
 /**
- * THIS FILE IS COPIED FROM:
+ * THIS FILE IS BASED ON:
  * https://github.com/rocicorp/fractional-indexing/blob/main/src/index.js
  *
- * I AM NOT INSTALLING THAT LIBRARY BECAUSE JEST COMPLAINS ABOUT THE ESM MODULE AND THE TESTS FAIL.
- * DO NOT MODIFY IT
- * ALSO, I'M DISABLING TS WITH `@ts-no-check` BECAUSE THEY DON'T USE STRICT NULL CHECKS IN THAT REPOSITORY
+ * MODIFIED FOR PAYLOAD CMS:
+ * - Changed the integer part encoding to use only digits for "small" keys and
+ *   only lowercase letters for "large" keys, ensuring consistent ordering
+ *   across databases with different collations.
+ *
+ * - Original algorithm used A-Z (uppercase) for "smaller" integers and a-z (lowercase)
+ *   for "larger" integers, relying on ASCII ordering where 'Z' < 'a'.
+ *
+ * - Some databases (e.g., PostgreSQL with default collation) use case-insensitive
+ *   comparison, treating 'Z' as 'z', which breaks the ordering.
+ *
+ * - New encoding:
+ *   - Uses digits '0'-'9' for "small" integers (10 values, lengths 11 down to 2)
+ *   - Uses lowercase 'a'-'z' for "large" integers (26 values, lengths 2 up to 27)
+ *   - Digits ALWAYS sort before letters in both ASCII and case-insensitive orderings.
+ *
+ * - Ordering: '0...' < '1...' < ... < '9..' < 'a.' < 'b..' < ... < 'z...'
+ *
+ * BACKWARD COMPATIBILITY:
+ * - Existing keys starting with lowercase 'a'-'z' remain valid and work correctly.
+ * - Keys starting with uppercase 'A'-'Z' (from the old algorithm) will still be
+ *   parsed for backward compatibility, but they may sort incorrectly in
+ *   case-insensitive databases. Consider running a migration to convert them.
  */
 
 // License: CC0 (no rights reserved).
@@ -80,14 +100,25 @@ function validateInteger(int) {
 }
 
 /**
+ * Returns the length of the integer part based on the head character.
+ *
+ * New encoding (case-insensitive safe):
+ * - SMALL range (digits): '0' = 11 chars, '1' = 10 chars, ..., '9' = 2 chars
+ * - LARGE range (lowercase): 'a' = 2 chars, 'b' = 3 chars, ..., 'z' = 27 chars
+ *
+ * Legacy encoding (for backward compatibility with existing keys):
+ * - 'A'-'Z' uppercase: 'A' = 27 chars, 'B' = 26 chars, ..., 'Z' = 2 chars
+ *
  * @param {string} head
  * @return {number}
  */
-
 function getIntegerLength(head) {
-  if (head >= 'a' && head <= 'z') {
+  if (head >= '0' && head <= '9') {
+    return 11 - (head.charCodeAt(0) - '0'.charCodeAt(0))
+  } else if (head >= 'a' && head <= 'z') {
     return head.charCodeAt(0) - 'a'.charCodeAt(0) + 2
   } else if (head >= 'A' && head <= 'Z') {
+    // Legacy encoding
     return 'Z'.charCodeAt(0) - head.charCodeAt(0) + 2
   } else {
     throw new Error('invalid order key head: ' + head)
@@ -107,13 +138,23 @@ function getIntegerPart(key) {
   return key.slice(0, integerPartLength)
 }
 
+/**
+ * Smallest possible key (for validation)
+ * '0' + 10 zeros = smallest valid key in new format
+ */
+const SMALLEST_KEY = '0' + BASE_36_DIGITS[0].repeat(10)
+
 /**
  * @param {string} key
  * @param {string} digits
  * @return {void}
  */
 
 function validateOrderKey(key, digits) {
+  if (key === SMALLEST_KEY) {
+    throw new Error('invalid order key: ' + key)
+  }
+  // Legacy check for old format
   if (key === 'A' + digits[0].repeat(26)) {
     throw new Error('invalid order key: ' + key)
   }
@@ -147,17 +188,30 @@ function incrementInteger(x, digits) {
     }
   }
   if (carry) {
+    if (head === '9') {
+      return 'a' + digits[0]
+    }
+    // Handle legacy uppercase transition
     if (head === 'Z') {
       return 'a' + digits[0]
     }
     if (head === 'z') {
       return null
     }
-    const h = String.fromCharCode(head.charCodeAt(0) + 1)
-    if (h > 'a') {
+
+    let h
+    if (head >= '0' && head <= '8') {
+      h = String.fromCharCode(head.charCodeAt(0) + 1)
+      digs.pop()
+    } else if (head >= 'a' && head <= 'y') {
+      h = String.fromCharCode(head.charCodeAt(0) + 1)
       digs.push(digits[0])
-    } else {
+    } else if (head >= 'A' && head <= 'Y') {
+      // Legacy uppercase
+      h = String.fromCharCode(head.charCodeAt(0) + 1)
       digs.pop()
+    } else {
+      throw new Error('invalid head: ' + head)
     }
     return h + digs.join('')
   } else {
@@ -187,16 +241,28 @@ function decrementInteger(x, digits) {
   }
   if (borrow) {
     if (head === 'a') {
-      return 'Z' + digits.slice(-1)
+      return '9' + digits.slice(-1)
     }
-    if (head === 'A') {
+    if (head === '0') {
       return null
     }
-    const h = String.fromCharCode(head.charCodeAt(0) - 1)
-    if (h < 'Z') {
+
+    let h
+    if (head >= '1' && head <= '9') {
+      h = String.fromCharCode(head.charCodeAt(0) - 1)
       digs.push(digits.slice(-1))
-    } else {
+    } else if (head >= 'b' && head <= 'z') {
+      h = String.fromCharCode(head.charCodeAt(0) - 1)
       digs.pop()
+    } else if (head >= 'B' && head <= 'Z') {
+      // Legacy uppercase
+      h = String.fromCharCode(head.charCodeAt(0) - 1)
+      digs.push(digits.slice(-1))
+    } else if (head === 'A') {
+      // Legacy uppercase
+      return null
+    } else {
+      throw new Error('invalid head: ' + head)
     }
     return h + digs.join('')
   } else {
@@ -232,6 +298,10 @@ export function generateKeyBetween(a, b, digits = BASE_36_DIGITS) {
 
     const ib = getIntegerPart(b)
     const fb = b.slice(ib.length)
+    if (ib === SMALLEST_KEY) {
+      return ib + midpoint('', fb, digits)
+    }
+    // Legacy check
     if (ib === 'A' + digits[0].repeat(26)) {
       return ib + midpoint('', fb, digits)
     }

test/sort/int.spec.ts

@@ -526,7 +526,7 @@ describe('Sort', () => {
 
         expect(orderable1._order).toBeDefined()
         expect(orderable2._order).toBeDefined()
-        expect(parseInt(orderable1._order, 16)).toBeLessThan(parseInt(orderable2._order, 16))
+        expect(parseInt(orderable1._order, 36)).toBeLessThan(parseInt(orderable2._order, 36))
         expect(ordered.docs[0].id).toStrictEqual(orderable1.id)
         expect(ordered.docs[1].id).toStrictEqual(orderable2.id)
       })
@@ -556,8 +556,8 @@ describe('Sort', () => {
           },
         })
 
-        expect(parseInt(ordered.docs[0]._order, 16)).toBeLessThan(
-          parseInt(ordered.docs[1]._order, 16),
+        expect(parseInt(ordered.docs[0]._order, 36)).toBeLessThan(
+          parseInt(ordered.docs[1]._order, 36),
         )
       })
 
@@ -589,8 +589,8 @@ describe('Sort', () => {
 
         expect(ordered.docs).toHaveLength(2)
 
-        expect(parseInt(ordered.docs[0]._order, 16)).toBeLessThan(
-          parseInt(ordered.docs[1]._order, 16),
+        expect(parseInt(ordered.docs[0]._order, 36)).toBeLessThan(
+          parseInt(ordered.docs[1]._order, 36),
         )
       })
 
@@ -606,7 +606,7 @@ describe('Sort', () => {
           data: {},
         })
         expect(docDuplicated.title).toBe('new document')
-        expect(parseInt(doc._order!, 16)).toBeLessThan(parseInt(docDuplicated._order!, 16))
+        expect(parseInt(doc._order!, 36)).toBeLessThan(parseInt(docDuplicated._order!, 36))
 
         await restClient.POST('/reorder', {
           body: JSON.stringify({
@@ -626,8 +626,8 @@ describe('Sort', () => {
           collection: 'orderable',
           id: docDuplicated.id,
         })
-        expect(parseInt(docAfterReorder._order!, 16)).toBeGreaterThan(
-          parseInt(docDuplicatedAfterReorder._order!, 16),
+        expect(parseInt(docAfterReorder._order!, 36)).toBeGreaterThan(
+          parseInt(docDuplicatedAfterReorder._order!, 36),
         )
       })
 
@@ -701,6 +701,148 @@ describe('Sort', () => {
         expect(orderableIndex).toBeGreaterThan(aAIndex)
         expect(orderableIndex).toBeLessThan(a0Index)
       })
+
+      it('should generate case-insensitive-safe keys when moving to first position', async () => {
+        const collection = orderableSlug
+
+        const { docs: allDocs } = await payload.find({
+          collection,
+          sort: '_order',
+          limit: 1,
+        })
+        expect(allDocs).toHaveLength(1)
+        const firstDoc = allDocs[0]!
+
+        const newDoc = await payload.create({
+          collection,
+          data: { title: 'Move to first test' },
+        })
+
+        const res = await restClient.POST('/reorder', {
+          body: JSON.stringify({
+            collectionSlug: collection,
+            docsToMove: [newDoc.id],
+            newKeyWillBe: 'less',
+            orderableFieldName: '_order',
+            target: {
+              id: firstDoc.id,
+              key: firstDoc._order,
+            },
+          }),
+        })
+
+        expect(res.status).toStrictEqual(200)
+
+        const newDocAfter = await payload.findByID({ collection, id: newDoc.id })
+
+        expect(newDocAfter._order).toMatch(/^\d/)
+        expect(parseInt(newDocAfter._order!, 36)).toBeLessThan(parseInt(firstDoc._order!, 36))
+      })
+
+      it('should allow multiple reorders to absolute first position', async () => {
+        const collection = orderableSlug
+
+        const { docs: initialDocs } = await payload.find({
+          collection,
+          sort: '_order',
+          limit: 1,
+        })
+        expect(initialDocs).toHaveLength(1)
+        const originalFirst = initialDocs[0]!
+
+        const docA = await payload.create({
+          collection,
+          data: { title: 'Multi first A' },
+        })
+
+        const docB = await payload.create({
+          collection,
+          data: { title: 'Multi first B' },
+        })
+
+        await restClient.POST('/reorder', {
+          body: JSON.stringify({
+            collectionSlug: collection,
+            docsToMove: [docA.id],
+            newKeyWillBe: 'less',
+            orderableFieldName: '_order',
+            target: {
+              id: originalFirst.id,
+              key: originalFirst._order,
+            },
+          }),
+        })
+
+        const docAAfter = await payload.findByID({ collection, id: docA.id })
+        expect(docAAfter._order).toMatch(/^\d/)
+
+        const res = await restClient.POST('/reorder', {
+          body: JSON.stringify({
+            collectionSlug: collection,
+            docsToMove: [docB.id],
+            newKeyWillBe: 'less',
+            orderableFieldName: '_order',
+            target: {
+              id: docA.id,
+              key: docAAfter._order,
+            },
+          }),
+        })
+
+        expect(res.status).toStrictEqual(200)
+
+        const docBAfter = await payload.findByID({ collection, id: docB.id })
+
+        expect(docBAfter._order).toMatch(/^\d/)
+        expect(parseInt(docBAfter._order!, 36)).toBeLessThan(parseInt(docAAfter._order!, 36))
+      })
+
+      it('should handle moving doc from first position and back', async () => {
+        const collection = orderableSlug
+
+        const { docs: initialDocs } = await payload.find({
+          collection,
+          sort: '_order',
+          limit: 2,
+        })
+        expect(initialDocs.length).toBeGreaterThanOrEqual(2)
+        const firstDoc = initialDocs[0]!
+        const secondDoc = initialDocs[1]!
+
+        await restClient.POST('/reorder', {
+          body: JSON.stringify({
+            collectionSlug: collection,
+            docsToMove: [firstDoc.id],
+            newKeyWillBe: 'greater',
+            orderableFieldName: '_order',
+            target: {
+              id: secondDoc.id,
+              key: secondDoc._order,
+            },
+          }),
+        })
+
+        const firstDocMoved = await payload.findByID({ collection, id: firstDoc.id })
+        expect(parseInt(firstDocMoved._order!, 36)).toBeGreaterThan(parseInt(secondDoc._order!, 36))
+
+        const res = await restClient.POST('/reorder', {
+          body: JSON.stringify({
+            collectionSlug: collection,
+            docsToMove: [firstDoc.id],
+            newKeyWillBe: 'less',
+            orderableFieldName: '_order',
+            target: {
+              id: secondDoc.id,
+              key: secondDoc._order,
+            },
+          }),
+        })
+
+        expect(res.status).toStrictEqual(200)
+
+        const firstDocBack = await payload.findByID({ collection, id: firstDoc.id })
+        expect(parseInt(firstDocBack._order!, 36)).toBeLessThan(parseInt(secondDoc._order!, 36))
+      })
     })
 
     describe('Orderable join', () => {
@@ -775,7 +917,7 @@ describe('Sort', () => {
           depth: 1,
         })
         const orders = (related.orderableJoinField1 as { docs: Orderable[] }).docs.map((doc) =>
-          parseInt(doc._orderable_orderableJoinField1_order, 16),
+          parseInt(doc._orderable_orderableJoinField1_order, 36),
         ) as [number, number, number]
         expect(orders[0]).toBeLessThan(orders[1])
         expect(orders[1]).toBeLessThan(orders[2])