Skip to content

Commit c148393

Browse files
dfa1claude
andcommitted
fix: ParquetImporter detects duplicate column names before VortexWriter
Previously the first signal a caller got was VortexWriter's internal uniqueness check, several stack frames removed from the real cause and with no mention of which import or source file triggered it: java.lang.IllegalArgumentException: duplicate field name: A at io.github.dfa1.vortex.writer.VortexWriter.<init>(VortexWriter.java:164) at io.github.dfa1.vortex.parquet.ParquetImporter.importParquet(...) checkNoDuplicateNames now runs right after the schema is built from the Parquet columns, throwing a ParquetImporter-specific message naming the duplicate(s) and the source file. Root cause is always the source Parquet file's own schema (DType.Struct legitimately permits duplicate names in memory; VortexWriter correctly enforces the file format's uniqueness requirement) — this only improves error quality at the import boundary, no auto-disambiguation. Verified against the real repro (Raincloud's uk-price-paid slug, whose source CSV lacked a header row so the Parquet conversion tool used the first data row's values as column names): IllegalArgumentException: Parquet schema has duplicate column name(s): [A]; source file: .../uk-price-paid.parquet Fixes #280. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 parent 85e7d4e commit c148393

3 files changed

Lines changed: 54 additions & 0 deletions

File tree

CHANGELOG.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,12 @@ All notable changes to **vortex-java** are documented here.
55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
66
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77

8+
## [Unreleased]
9+
10+
### Fixed
11+
12+
- `ParquetImporter` detects duplicate column names in the source Parquet schema and throws a clear message naming the duplicate(s) and the source file, instead of a confusing `VortexWriter` internal-invariant error several frames removed from the actual cause. ([f2c05e57](https://github.com/dfa1/vortex-java/commit/f2c05e57))
13+
814
## [0.12.3] — 2026-07-19
915

1016
### Added

parquet/src/main/java/io/github/dfa1/vortex/parquet/ParquetImporter.java

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,9 +22,12 @@
2222
import java.nio.file.StandardOpenOption;
2323
import java.util.ArrayList;
2424
import java.util.Arrays;
25+
import java.util.HashSet;
2526
import java.util.LinkedHashMap;
27+
import java.util.LinkedHashSet;
2628
import java.util.List;
2729
import java.util.Map;
30+
import java.util.Set;
2831

2932
/// Reads a Parquet file and writes a Vortex file.
3033
///
@@ -71,6 +74,7 @@ public static void importParquet(Path parquetPath, Path vortexPath, ImportOption
7174
names.add(ColumnName.of(col.name()));
7275
types.add(mapDType(col));
7376
}
77+
checkNoDuplicateNames(names, parquetPath);
7478
DType.Struct schema = new DType.Struct(names, types, false);
7579
long totalRows = parquet.getFileMetaData().numRows();
7680

@@ -115,6 +119,20 @@ public static void importParquet(Path parquetPath, Path vortexPath, ImportOption
115119
}
116120
}
117121

122+
static void checkNoDuplicateNames(List<ColumnName> names, Path parquetPath) {
123+
Set<ColumnName> seen = new HashSet<>();
124+
Set<ColumnName> duplicates = new LinkedHashSet<>();
125+
for (ColumnName name : names) {
126+
if (!seen.add(name)) {
127+
duplicates.add(name);
128+
}
129+
}
130+
if (!duplicates.isEmpty()) {
131+
throw new IllegalArgumentException(
132+
"Parquet schema has duplicate column name(s): " + duplicates + "; source file: " + parquetPath);
133+
}
134+
}
135+
118136
static DType mapDType(ColumnSchema col) {
119137
boolean nullable = col.repetitionType() == RepetitionType.OPTIONAL;
120138
return switch (col.type()) {

parquet/src/test/java/io/github/dfa1/vortex/parquet/ParquetImporterTest.java

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@
2525
import java.util.List;
2626

2727
import static org.assertj.core.api.Assertions.assertThat;
28+
import static org.assertj.core.api.Assertions.assertThatCode;
2829
import static org.assertj.core.api.Assertions.assertThatThrownBy;
2930

3031
class ParquetImporterTest {
@@ -181,6 +182,35 @@ void unknownColumn_throws() {
181182
}
182183
}
183184

185+
@Nested
186+
class DuplicateNames {
187+
188+
@Test
189+
void noDuplicates_doesNotThrow() {
190+
// Given — a, b, c are all distinct
191+
List<ColumnName> names = List.of(ColumnName.of("a"), ColumnName.of("b"), ColumnName.of("c"));
192+
193+
// When / Then
194+
assertThatCode(() -> ParquetImporter.checkNoDuplicateNames(names, Path.of("source.parquet")))
195+
.doesNotThrowAnyException();
196+
}
197+
198+
@Test
199+
void duplicateName_throwsWithNameAndSourcePath() {
200+
// Given — two columns both named "A", as seen on the Raincloud uk-price-paid slug (#280): a
201+
// headerless source CSV made a Parquet conversion tool use the first data row as column
202+
// names, and two property-type flag columns happened to share the value "A".
203+
List<ColumnName> names = List.of(ColumnName.of("A"), ColumnName.of("B"), ColumnName.of("A"));
204+
Path source = Path.of("uk-price-paid.parquet");
205+
206+
// When / Then
207+
assertThatThrownBy(() -> ParquetImporter.checkNoDuplicateNames(names, source))
208+
.isInstanceOf(IllegalArgumentException.class)
209+
.hasMessageContaining("A")
210+
.hasMessageContaining("uk-price-paid.parquet");
211+
}
212+
}
213+
184214
@Nested
185215
class Import {
186216

0 commit comments

Comments
 (0)