[SPEC] Add fileCount to dataset stats facets

[dolfinus]

Problem

👋 Thanks for opening a pull request! Please include a brief summary of the problem your change is trying to solve, or bug fix. If your change fixes a bug or you'd like to provide context on why you're making the change, please link the issue as follows:

Closes: #2550

Solution

Please describe your change as it relates to the problem, or bug fix, as well as any dependencies. If your change requires a schema change, please describe the schema modification(s) and whether it's a backwards-incompatible or backwards-compatible change, then select one of the following:

Note: All schema changes require discussion. Please link the issue for context.

• Your change modifies the core OpenLineage model
• Your change modifies one or more OpenLineage facets

If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports S3 and GCS filesystem operations, tested with AWS EMR).

One-line summary:

Adds "fileCount" field to DataQualityMetricsInputDatasetFacet and OutputStatisticsOutputDatasetFacet specification

Checklist

• You've signed-off your work
• Your pull request title follows our guidelines
• Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• Your comment includes a one-liner for the changelog about the specific purpose of the change (if necessary)
• You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
• You've added a header to source files (if relevant)

---

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2023 contributors to the OpenLineage project

[dolfinus]

Changing number of fields in OutputStatisticsOutputDatasetFacet from 2 to 3 lead to replacing method .newOutputStatisticsOutputDatasetFacet(rowCount, size) with .newOutputStatisticsOutputDatasetFacetBuilder().rowCount(...).size(...).
This could introduce compatibility issues for users which use Java client.

[JDarDagran]

Changing number of fields in OutputStatisticsOutputDatasetFacet from 2 to 3 lead to replacing method .newOutputStatisticsOutputDatasetFacet(rowCount, size) with .newOutputStatisticsOutputDatasetFacetBuilder().rowCount(...).size(...). This could introduce compatibility issues for users which use Java client.

I don't understand why? I just checked and

public OutputStatisticsOutputDatasetFacet newOutputStatisticsOutputDatasetFacet(Long rowCount,
      Long size, Long fileCount) {
    return new OutputStatisticsOutputDatasetFacet(this.producer, rowCount, size, fileCount);
  }

gets generated as well.

[dolfinus]

Ah, I see, now method accepts 3 arguments instead of 2. Without default value for filesCount, users have to update their code to pass new argument to the method

[mobuchowski]

Having exact number of arguments is by design. If you don't care you can use builder as solution here.

[JDarDagran]

LGTM 👍
@dolfinus, may I kindly ask how did you find the need to add this field?
Would it be worth/possible to add the statistic somewhere in any of the existing integration?

[dolfinus]

I'm developing a ETL tool which allows both manipulating data in DBMS and file systems using PySpark, but also provide a way to download/upload/move raw files in file systems. Using fileCount field I can collect information that some process, for example, downloaded 100 files from SFTP to S3, but next ETL process in the chain read only 98 files from S3, and 2 files are missing for some reason.

Would it be worth/possible to add the statistic somewhere in any of the existing integration?

Yes, but in a separate PR. This may be tricky, for example Spark does not provide metrics for number of read/written files. In general, number of dataframe partitions is equal to number of files, so we can count number of successful tasks and use it as a file count. But Spark's Catalist can merge small partitions to a large one, or read one file in many partitions if file format is splittable, so this can produce wrong results.

[mobuchowski]

Thanks for contribution @dolfinus .