Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spark 3.5: Migrate tests to JUnit5 in data directory #9341

Merged
merged 21 commits into from
Dec 22, 2023
Merged

Spark 3.5: Migrate tests to JUnit5 in data directory #9341

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
0114ec6
spark/data dir: update tests to AssertJ non-parameterized
chinmay-bhat Dec 19, 2023
b63609e
remove redundant import
chinmay-bhat Dec 19, 2023
72063ec
update hasNext() and isExhausted()
chinmay-bhat Dec 19, 2023
d2750d6
update temp to private
chinmay-bhat Dec 20, 2023
2a116de
run spotlessApply
chinmay-bhat Dec 20, 2023
4775c29
fix (actual, expected) for assertj
chinmay-bhat Dec 20, 2023
9a0b90b
revert file - modify this file with other paramaterized files
chinmay-bhat Dec 20, 2023
221692a
fix assumeThat import and methods
chinmay-bhat Dec 20, 2023
436e518
remove byte[] cast
chinmay-bhat Dec 20, 2023
f723e78
update tempDir access for subclass
chinmay-bhat Dec 20, 2023
ca029fd
fix to clear CI
chinmay-bhat Dec 20, 2023
53722e2
fix HiddenField error
chinmay-bhat Dec 20, 2023
dd92bca
reduce CI errors
chinmay-bhat Dec 21, 2023
877c91c
remove commented code
chinmay-bhat Dec 21, 2023
1340faf
remove UUID for filenames
chinmay-bhat Dec 21, 2023
73b47a6
run spotlessApply
chinmay-bhat Dec 21, 2023
9152f98
fix file create to temp.resolve()
chinmay-bhat Dec 21, 2023
ddabe1a
chain temp.resolve()
chinmay-bhat Dec 21, 2023
28d2c78
improve assert statement
chinmay-bhat Dec 21, 2023
eaf5e67
fix test parameterization
nastra Dec 21, 2023
041eca2
remove semicolon
nastra Dec 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestAvroScan.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ public static void stopSpark() {

@Override
protected void writeAndValidate(Schema schema) throws IOException {
File parent = new File(temp.toFile(), "avro");
File parent = temp.resolve("avro").toFile();
File location = new File(parent, "test");
File dataFolder = new File(location, "data");
dataFolder.mkdirs();
Expand Down
8 changes: 4 additions & 4 deletions spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWrites.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ protected void writeAndValidate(Schema schema) throws IOException {
@TestTemplate
public void testWriteWithCustomDataLocation() throws IOException {
File location = createTableFolder();
File tablePropertyDataLocation = new File(temp.toFile(), "test-table-property-data-dir");
File tablePropertyDataLocation = temp.resolve("test-table-property-data-dir").toFile();
Table table = createTable(new Schema(SUPPORTED_PRIMITIVES.fields()), location);
table
.updateProperties()
Expand All @@ -156,7 +156,7 @@ public void testWriteWithCustomDataLocation() throws IOException {
}

private File createTableFolder() throws IOException {
File parent = new File(temp.toFile(), "parquet");
File parent = temp.resolve("parquet").toFile();
File location = new File(parent, "test");
assertThat(location.mkdirs()).as("Mkdir should succeed").isTrue();
return location;
Expand Down Expand Up @@ -289,7 +289,7 @@ public void testNullableWithWriteOption() throws IOException {
.as("Spark 3 rejects writing nulls to a required column")
.startsWith("2");

File location = new File(new File(temp.toFile(), "parquet"), "test");
File location = new File(temp.resolve("parquet").toFile(), "test");
String sourcePath = String.format("%s/nullable_poc/sourceFolder/", location.toString());
String targetPath = String.format("%s/nullable_poc/targetFolder/", location.toString());

Expand Down Expand Up @@ -343,7 +343,7 @@ public void testNullableWithSparkSqlOption() throws IOException {
.as("Spark 3 rejects writing nulls to a required column")
.startsWith("2");

File location = new File(new File(temp.toFile(), "parquet"), "test");
File location = new File(temp.resolve("parquet").toFile(), "test");
chinmay-bhat marked this conversation as resolved.
Show resolved Hide resolved
String sourcePath = String.format("%s/nullable_poc/sourceFolder/", location.toString());
String targetPath = String.format("%s/nullable_poc/targetFolder/", location.toString());

Expand Down
2 changes: 1 addition & 1 deletion spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestParquetScan.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ public void testEmptyTableProjection() throws IOException {
}

private Table createTable(Schema schema) throws IOException {
File parent = new File(temp.toFile(), "parquet");
File parent = temp.resolve("parquet").toFile();
File location = new File(parent, "test");
HadoopTables tables = new HadoopTables(CONF);
return tables.create(schema, PartitionSpec.unpartitioned(), location.toString());
Expand Down