Skip to content

Commit c060d34

Browse files
dfa1claude
dfa1
and
claude
committed
refactor(core): LE_* layouts move to VortexFormat
Endianness is a property of the wire format — trailer fields, spec-table indexes, scaffolding, and element values are all little-endian — so the shared unaligned LE layouts belong in VortexFormat next to the magic and trailer shape, not in PTypeIO (which keeps its real job: mapping ptypes onto those layouts). Sweeps 116 files onto the single source and deletes the six private copies (Trailer, LazyDecimalArray, GenericArray, ChunkedEncoding- Decoder, PcoTansDecoder, PcoEncodingDecoder) — two of which used reversed names (SHORT_LE) for byte-identical constants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 0c61bd4 commit c060d34

125 files changed

Lines changed: 632 additions & 623 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1717

1818
### Changed
1919

20+
- The little-endian `ValueLayout` constants moved from `PTypeIO.LE_*` to `VortexFormat.LE_*` — endianness is a property of the wire format, not of ptypes, and `VortexFormat` is where format facts live. Six classes that carried private copies (including reversed-name duplicates like `SHORT_LE`) now share the single source; nothing outside `VortexFormat` defines a `withOrder(LITTLE_ENDIAN)` layout. ([72095d9e](https://github.com/dfa1/vortex-java/commit/72095d9e))
2021
- `Chunk.columns()` returns an order-preserving `SequencedMap<ColumnName, Chunk.Column>` — one map instead of two parallel string-keyed maps, with each column's `Array` and `DType` traveling together in the `Column` carrier. `column(String)` stays as boundary sugar (plus a `column(ColumnName)` overload); iteration order is the schema/projection order, now guaranteed even for 1–2 column chunks. Typed names originate in `ScanIterator` from the file's already-certified schema. ([f8ad15d1](https://github.com/dfa1/vortex-java/commit/f8ad15d1))
2122

2223
- `Compute.filteredSum` over a dictionary-encoded filter column is ~20× faster (best runs ~30×): the predicate is resolved against the dictionary's value pool once and the raw `u8` codes are scanned directly from their backing segments, instead of decoding every row through the per-element accessor — a fused `SUM(measure) WHERE category = …` over 100M rows drops from ~760 ms to ~38 ms. ([85e251cc](https://github.com/dfa1/vortex-java/commit/85e251cc))

‎core/src/main/java/io/github/dfa1/vortex/core/compute/PrimitiveArrays.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
import io.github.dfa1.vortex.core.model.PType;
44
import io.github.dfa1.vortex.core.model.EncodingId;
5+
import io.github.dfa1.vortex.core.io.VortexFormat;
56
import io.github.dfa1.vortex.core.io.PTypeIO;
67
import io.github.dfa1.vortex.core.error.VortexException;
78

@@ -97,7 +98,7 @@ public static long[] toLongs(Object data, PType ptype, EncodingId encoding) {
9798
public static MemorySegment fromLongs(long[] longs, PType ptype, SegmentAllocator arena) {
9899
if (ptype == PType.I64 || ptype == PType.U64) {
99100
MemorySegment dst = arena.allocate((long) longs.length * 8);
100-
MemorySegment.copy(MemorySegment.ofArray(longs), ValueLayout.JAVA_LONG, 0L, dst, PTypeIO.LE_LONG, 0L, longs.length);
101+
MemorySegment.copy(MemorySegment.ofArray(longs), ValueLayout.JAVA_LONG, 0L, dst, VortexFormat.LE_LONG, 0L, longs.length);
101102
return dst;
102103
}
103104
int n = longs.length;

‎core/src/main/java/io/github/dfa1/vortex/core/fbs/FbsBuilder.java‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
package io.github.dfa1.vortex.core.fbs;
22

3-
import static io.github.dfa1.vortex.core.io.PTypeIO.LE_DOUBLE;
4-
import static io.github.dfa1.vortex.core.io.PTypeIO.LE_FLOAT;
5-
import static io.github.dfa1.vortex.core.io.PTypeIO.LE_INT;
6-
import static io.github.dfa1.vortex.core.io.PTypeIO.LE_LONG;
7-
import static io.github.dfa1.vortex.core.io.PTypeIO.LE_SHORT;
3+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_DOUBLE;
4+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_FLOAT;
5+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_INT;
6+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_LONG;
7+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_SHORT;
88

99
import java.lang.foreign.MemorySegment;
1010
import java.lang.foreign.ValueLayout;

‎core/src/main/java/io/github/dfa1/vortex/core/fbs/FbsStruct.java‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
package io.github.dfa1.vortex.core.fbs;
22

3-
import static io.github.dfa1.vortex.core.io.PTypeIO.LE_DOUBLE;
4-
import static io.github.dfa1.vortex.core.io.PTypeIO.LE_FLOAT;
5-
import static io.github.dfa1.vortex.core.io.PTypeIO.LE_INT;
6-
import static io.github.dfa1.vortex.core.io.PTypeIO.LE_LONG;
7-
import static io.github.dfa1.vortex.core.io.PTypeIO.LE_SHORT;
3+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_DOUBLE;
4+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_FLOAT;
5+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_INT;
6+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_LONG;
7+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_SHORT;
88

99
import java.lang.foreign.MemorySegment;
1010
import java.lang.foreign.ValueLayout;

‎core/src/main/java/io/github/dfa1/vortex/core/fbs/FbsTable.java‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
package io.github.dfa1.vortex.core.fbs;
22

3-
import static io.github.dfa1.vortex.core.io.PTypeIO.LE_DOUBLE;
4-
import static io.github.dfa1.vortex.core.io.PTypeIO.LE_FLOAT;
5-
import static io.github.dfa1.vortex.core.io.PTypeIO.LE_INT;
6-
import static io.github.dfa1.vortex.core.io.PTypeIO.LE_LONG;
7-
import static io.github.dfa1.vortex.core.io.PTypeIO.LE_SHORT;
3+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_DOUBLE;
4+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_FLOAT;
5+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_INT;
6+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_LONG;
7+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_SHORT;
88

99
import java.lang.foreign.MemorySegment;
1010
import java.lang.foreign.ValueLayout;

‎core/src/main/java/io/github/dfa1/vortex/core/io/PTypeIO.java‎

Lines changed: 8 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,12 @@
99
import java.lang.invoke.MethodHandles;
1010
import java.lang.invoke.MethodType;
1111
import java.lang.invoke.VarHandle;
12-
import java.nio.ByteOrder;
12+
13+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_DOUBLE;
14+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_FLOAT;
15+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_INT;
16+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_LONG;
17+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_SHORT;
1318

1419
/// Bulk I/O helpers for primitive ptypes, backed by `MemorySegment`/`ValueLayout`/`MethodHandle`.
1520
///
@@ -18,21 +23,10 @@
1823
/// the unaligned little-endian VarHandle. This lets hot loops avoid per-element `switch`
1924
/// dispatch on `PType`.
2025
///
21-
/// The LE_* layout constants are public so callers outside this package can share them
22-
/// without duplicating the `withOrder(LITTLE_ENDIAN)` boilerplate.
26+
/// The shared little-endian layouts live in [VortexFormat] — endianness is a property of the
27+
/// wire format, not of ptypes; this class only maps ptypes onto those layouts.
2328
public final class PTypeIO {
2429

25-
/// Unaligned little-endian layout for 16-bit shorts.
26-
public static final ValueLayout.OfShort LE_SHORT = ValueLayout.JAVA_SHORT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
27-
/// Unaligned little-endian layout for 32-bit ints.
28-
public static final ValueLayout.OfInt LE_INT = ValueLayout.JAVA_INT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
29-
/// Unaligned little-endian layout for 64-bit longs.
30-
public static final ValueLayout.OfLong LE_LONG = ValueLayout.JAVA_LONG_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
31-
/// Unaligned little-endian layout for 32-bit floats.
32-
public static final ValueLayout.OfFloat LE_FLOAT = ValueLayout.JAVA_FLOAT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
33-
/// Unaligned little-endian layout for 64-bit doubles.
34-
public static final ValueLayout.OfDouble LE_DOUBLE = ValueLayout.JAVA_DOUBLE_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
35-
3630
private static final MethodHandle[] SETTERS = buildSetters();
3731

3832
private PTypeIO() {

‎core/src/main/java/io/github/dfa1/vortex/core/io/VortexFormat.java‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
package io.github.dfa1.vortex.core.io;
22

33
import java.lang.foreign.MemorySegment;
4+
import java.lang.foreign.ValueLayout;
5+
6+
import java.nio.ByteOrder;
47

58
/// Wire-format constants for the Vortex file format.
69
///
@@ -24,6 +27,22 @@ public final class VortexFormat {
2427
/// Files with any other version are rejected up front rather than silently mis-parsed.
2528
public static final int VERSION = 1;
2629

30+
// All multi-byte integers in the Vortex wire format are little-endian — trailer fields,
31+
// spec-table indexes, buffer scaffolding, and element values alike. These unaligned
32+
// little-endian layouts are the single source for every wire read/write; nothing outside
33+
// this class defines its own withOrder(LITTLE_ENDIAN) copy.
34+
35+
/// Unaligned little-endian layout for 16-bit shorts.
36+
public static final ValueLayout.OfShort LE_SHORT = ValueLayout.JAVA_SHORT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
37+
/// Unaligned little-endian layout for 32-bit ints.
38+
public static final ValueLayout.OfInt LE_INT = ValueLayout.JAVA_INT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
39+
/// Unaligned little-endian layout for 64-bit longs.
40+
public static final ValueLayout.OfLong LE_LONG = ValueLayout.JAVA_LONG_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
41+
/// Unaligned little-endian layout for 32-bit floats.
42+
public static final ValueLayout.OfFloat LE_FLOAT = ValueLayout.JAVA_FLOAT_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
43+
/// Unaligned little-endian layout for 64-bit doubles.
44+
public static final ValueLayout.OfDouble LE_DOUBLE = ValueLayout.JAVA_DOUBLE_UNALIGNED.withOrder(ByteOrder.LITTLE_ENDIAN);
45+
2746
private VortexFormat() {
2847
}
2948
}

‎core/src/main/java/io/github/dfa1/vortex/core/model/TimestampDtype.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
import io.github.dfa1.vortex.core.error.VortexException;
44

5-
import static io.github.dfa1.vortex.core.io.PTypeIO.LE_SHORT;
5+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_SHORT;
66

77
import java.lang.foreign.MemorySegment;
88
import java.lang.foreign.ValueLayout;

‎core/src/main/java/io/github/dfa1/vortex/core/proto/ProtoReader.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
package io.github.dfa1.vortex.core.proto;
22

3-
import static io.github.dfa1.vortex.core.io.PTypeIO.LE_INT;
4-
import static io.github.dfa1.vortex.core.io.PTypeIO.LE_LONG;
3+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_INT;
4+
import static io.github.dfa1.vortex.core.io.VortexFormat.LE_LONG;
55
import java.io.IOException;
66
import java.lang.foreign.MemorySegment;
77
import java.lang.foreign.ValueLayout;

‎core/src/test/java/io/github/dfa1/vortex/core/compute/PrimitiveArraysTest.java‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
package io.github.dfa1.vortex.core.compute;
22

33
import io.github.dfa1.vortex.core.model.EncodingId;
4-
import io.github.dfa1.vortex.core.io.PTypeIO;
4+
import io.github.dfa1.vortex.core.io.VortexFormat;
55

66
import io.github.dfa1.vortex.core.model.PType;
77
import io.github.dfa1.vortex.core.error.VortexException;
@@ -142,7 +142,7 @@ void fromLongs_i64_writesLittleEndian() {
142142

143143
// Then it is stored little-endian (lowest byte first)
144144
assertThat(seg.get(ValueLayout.JAVA_BYTE, 0)).isEqualTo((byte) 0x08);
145-
assertThat(seg.getAtIndex(PTypeIO.LE_LONG, 0)).isEqualTo(0x0102_0304_0506_0708L);
145+
assertThat(seg.getAtIndex(VortexFormat.LE_LONG, 0)).isEqualTo(0x0102_0304_0506_0708L);
146146
}
147147
}
148148

@@ -164,9 +164,9 @@ void fromLongs_narrowWidth_keepsOnlyLowBytes() {
164164
private static long readElement(MemorySegment seg, PType ptype, int i) {
165165
return switch (ptype) {
166166
case I8, U8 -> seg.get(ValueLayout.JAVA_BYTE, i);
167-
case I16, U16 -> seg.getAtIndex(PTypeIO.LE_SHORT, i);
168-
case I32, U32 -> seg.getAtIndex(PTypeIO.LE_INT, i);
169-
case I64, U64 -> seg.getAtIndex(PTypeIO.LE_LONG, i);
167+
case I16, U16 -> seg.getAtIndex(VortexFormat.LE_SHORT, i);
168+
case I32, U32 -> seg.getAtIndex(VortexFormat.LE_INT, i);
169+
case I64, U64 -> seg.getAtIndex(VortexFormat.LE_LONG, i);
170170
default -> throw new IllegalArgumentException("not an integer ptype: " + ptype);
171171
};
172172
}

0 commit comments

Comments
 (0)