docs: add Iceberg public API documentation to contributor guide #3237

andygrove · 2026-01-22T00:24:21Z

Rationale

Iceberg depends on Comet, therefore Comet has a public API that needs to be maintained. This isn't documented very well.

This PR adds an @IcebergApi annotation to the public API.

There is also a new page in the contributor guide documenting this API.

I generated this information by analyzing the latest from Iceberg's main branch.

🤖 Generated with Claude Code

Add documentation detailing all Comet classes and methods that form the public API used by Apache Iceberg. This helps contributors understand which APIs may affect Iceberg integration and need backward compatibility considerations. The documentation covers: - org.apache.comet.parquet: FileReader, RowGroupReader, ReadOptions, ParquetColumnSpec, column readers, BatchReader, Native JNI methods - org.apache.comet: CometSchemaImporter - org.apache.comet.vector: CometVector - org.apache.comet.shaded.arrow: RootAllocator, ValueVector Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add a custom Java annotation @IcebergApi to mark all classes, methods, constructors, and fields that form the public API used by Apache Iceberg. This makes it easy to identify which APIs need backward compatibility considerations when making changes. The annotation is applied to: - org.apache.comet.parquet: FileReader, RowGroupReader, ReadOptions, WrappedInputFile, ParquetColumnSpec, AbstractColumnReader, ColumnReader, BatchReader, MetadataColumnReader, ConstantColumnReader, Native, TypeUtil, Utils - org.apache.comet: CometSchemaImporter - org.apache.comet.vector: CometVector Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Add annotations to: - AbstractColumnReader.nativeHandle (protected field accessed by Iceberg subclasses) - AbstractCometSchemaImporter.close() (called by Iceberg) Also update documentation to include these APIs. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

codecov-commenter · 2026-01-22T01:17:04Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.04%. Comparing base (f09f8af) to head (7b58965).
⚠️ Report is 864 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3237      +/-   ##
============================================
+ Coverage     56.12%   60.04%   +3.92%     
- Complexity      976     1429     +453     
============================================
  Files           119      170      +51     
  Lines         11743    15774    +4031     
  Branches       2251     2606     +355     
============================================
+ Hits           6591     9472    +2881     
- Misses         4012     4981     +969     
- Partials       1140     1321     +181

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

huaxingao · 2026-01-22T01:42:43Z

common/src/main/java/org/apache/comet/parquet/AbstractColumnReader.java

 import org.apache.comet.vector.CometVector;

 /** Base class for Comet Parquet column reader implementations. */
+@IcebergApi


If we annotate the class with @IcebergApi, do we still need @IcebergApi in L66, L101 and L119?

Thanks for reviewing this @huaxingao. My plan was to annotate every class, method, and field referenced from Iceberg. Some methods are not referenced, so it would be safe to modify them without affecting Iceberg compatibility. The idea is that we know not to modify anything with the @IcebergApi annotation without careful consideration.

Add a new Maven module containing dedicated unit tests for all @IcebergApi annotated classes, ensuring the public API contract with Apache Iceberg remains stable and tested. Key changes: - Add iceberg-public-api module with 169 tests covering all @IcebergApi classes - Fix CometVector constructor visibility (protected -> public) to match API annotation - Add IcebergApiVerificationTest for reflection-based API verification - Add tests for FileReader, BatchReader, ColumnReader, Native, TypeUtil, Utils - Add tests for CometVector, CometSchemaImporter, WrappedInputFile Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

martin-g · 2026-01-22T20:50:18Z

How this will be maintained in the future ?
Is it expected that the Iceberg maintainers and contributors will open a PR here every time they start using a method/class from Comet that is not yet annotated with @IcebergApi? Or to remove @IcebergApi when they stop using some annotated method ?
Or it will be Comet's devs responsibility to "scan" Iceberg code for new/obsolete usage ?

andygrove · 2026-01-22T21:09:51Z

How this will be maintained in the future ? Is it expected that the Iceberg maintainers and contributors will open a PR here every time they start using a method/class from Comet that is not yet annotated with @IcebergApi? Or to remove @IcebergApi when they stop using some annotated method ? Or it will be Comet's devs responsibility to "scan" Iceberg code for new/obsolete usage ?

The next step is #3240 which adds unit tests to ensure that the API is as expected. This will help prevent accidental changes to the current API methods.

We should probably update the release process documentation to suggest auditing the Iceberg API use before creating a new release.

martin-g · 2026-01-22T21:03:19Z

docs/source/contributor-guide/iceberg_public_api.md

+// Methods used by Iceberg
+public void setRequestedSchemaFromSpecs(List<ParquetColumnSpec> specs)
+public RowGroupReader readNextRowGroup() throws IOException
+public void skipNextRowGroup()


Suggested change

public void skipNextRowGroup()

public boolean skipNextRowGroup()

martin-g · 2026-01-22T21:06:28Z

docs/source/contributor-guide/iceberg_public_api.md

+
+```java
+// Constructor
+public WrappedInputFile(org.apache.iceberg.io.InputFile inputFile)


datafusion-comet/common/src/main/java/org/apache/comet/parquet/WrappedInputFile.java

Line 40 in 640dd03

public WrappedInputFile(Object inputFile) {

actually accepts java.lang.Object

martin-g · 2026-01-22T21:12:57Z

iceberg-public-api/src/test/java/org/apache/comet/iceberg/api/AbstractApiTest.java

+  @After
+  public void tearDown() throws IOException {
+    if (tempDir != null && Files.exists(tempDir)) {
+      Files.walk(tempDir).sorted(Comparator.reverseOrder()).map(Path::toFile).forEach(File::delete);


Suggested change

Files.walk(tempDir).sorted(Comparator.reverseOrder()).map(Path::toFile).forEach(File::delete);

try (Stream<Path> stream : Files.walk(tempDir)) {

stream.sorted(Comparator.reverseOrder()).map(Path::toFile).forEach(File::delete);

}

https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/nio/file/Files.html#walk(java.nio.file.Path,java.nio.file.FileVisitOption...) says:

The returned stream contains references to one or more open directories. The directories are closed by closing the stream.

martin-g · 2026-01-22T21:14:12Z

iceberg-public-api/src/test/java/org/apache/comet/iceberg/api/AbstractApiTest.java

+  /** Checks if native library is available. */
+  protected static boolean isNativeLibraryAvailable() {
+    try {
+      Class.forName("org.apache.comet.NativeBase");


This does not guarantee that the native library is available and loadable.

martin-g · 2026-01-22T21:15:55Z

common/src/main/java/org/apache/comet/parquet/BatchReader.java

  }

  /**
   * @deprecated since 0.10.0, will be removed in 0.11.0.


Maybe it should be undeprecated ?!
Currently is is both deprecated and important (used by Iceberg)

martin-g · 2026-01-22T21:35:14Z

iceberg-public-api/src/test/java/org/apache/comet/iceberg/api/parquet/BatchReaderApiTest.java

+  @Test
+  public void testCurrentBatchMethodExists() throws NoSuchMethodException {
+    Method method = BatchReader.class.getMethod("currentBatch");
+    assertThat(method).isNotNull();


Suggested change

assertThat(method).isNotNull();

martin-g · 2026-01-22T21:35:27Z

iceberg-public-api/src/test/java/org/apache/comet/iceberg/api/parquet/BatchReaderApiTest.java

+  @Test
+  public void testCloseMethodExists() throws NoSuchMethodException {
+    Method method = BatchReader.class.getMethod("close");
+    assertThat(method).isNotNull();


Suggested change

assertThat(method).isNotNull();

martin-g · 2026-01-22T21:38:05Z

docs/source/contributor-guide/iceberg_public_api.md

+// Methods used by Iceberg
+public static AbstractColumnReader getColumnReader(
+    DataType sparkType,
+    ColumnDescriptor descriptor,


The parameter types do not match with https://github.com/apache/datafusion-comet/pull/3237/changes#diff-54de18a6f3ec3c2944f1628012f8c0b0af863da30419a8bc989eb6cd8ccb8cd1R39-R46

Suggested change

ColumnDescriptor descriptor,

ParquetColumnSpec columnSpec,

martin-g · 2026-01-22T21:38:34Z

docs/source/contributor-guide/iceberg_public_api.md

+    CometSchemaImporter importer,
+    int batchSize,
+    boolean useDecimal128,
+    boolean isConstant


Suggested change

boolean isConstant

boolean useLazyMaterialization,

boolean useLegacyTimestamp

martin-g · 2026-01-22T21:42:15Z

docs/source/contributor-guide/iceberg_public_api.md

+
+1. Creates `CometSchemaImporter` with a `RootAllocator`
+2. Uses `Utils.getColumnReader()` to create appropriate column readers
+3. Calls `reset()` and `setPageReader()` for each row group


Which reset() method ?
Maybe:

Suggested change

3. Calls `reset()` and `setPageReader()` for each row group

3. Calls `Native.resetBatch()` and `setPageReader()` for each row group

?!

andygrove and others added 2 commits January 21, 2026 17:24

andygrove requested review from huaxingao and parthchandra January 22, 2026 00:34

andygrove and others added 2 commits January 21, 2026 17:35

format

7b58965

huaxingao reviewed Jan 22, 2026

View reviewed changes

This was referenced Jan 22, 2026

test: Add Iceberg API stability tests #3240

Open

Move Iceberg-only API classes to iceberg-api module #3241

Closed

Future of Iceberg Support in Comet #2921

Open

andygrove added this to the 0.13.0 milestone Jan 22, 2026

trigger CI

640dd03

martin-g reviewed Jan 22, 2026

View reviewed changes

	public void skipNextRowGroup()
	public boolean skipNextRowGroup()

-      Files.walk(tempDir).sorted(Comparator.reverseOrder()).map(Path::toFile).forEach(File::delete);
+      try (Stream<Path> stream : Files.walk(tempDir)) {
+        stream.sorted(Comparator.reverseOrder()).map(Path::toFile).forEach(File::delete);
+      }

	boolean isConstant
	boolean useLazyMaterialization,
	boolean useLegacyTimestamp

	3. Calls `reset()` and `setPageReader()` for each row group
	3. Calls `Native.resetBatch()` and `setPageReader()` for each row group

docs: add Iceberg public API documentation to contributor guide #3237

Are you sure you want to change the base?

docs: add Iceberg public API documentation to contributor guide #3237

Uh oh!

Conversation

andygrove commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale

Uh oh!

codecov-commenter commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

martin-g commented Jan 22, 2026

Uh oh!

andygrove commented Jan 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

andygrove commented Jan 22, 2026 •

edited

Loading

codecov-commenter commented Jan 22, 2026 •

edited

Loading