RFC: API & Artifact Compatibility for TFX OSS & Libraries

[zhitaoli]

API & Artifact Compatibility for TFX OSS & Libraries

This RFC will be open for comments until April 7, 2021

Status	Proposed
RFC #	API & Artifact Compatibility for TFX::OSS & Libraries.
Author(s)	Zhitao Li (zhitaoli@google.com), Jiayi Zhao (jyzhao@google.com), Zhuo Peng (zhuo@google.com)
Sponsor	Konstantinos Katsiapis (katsiapis@google.com), Zhitao Li (zhitaoli@google.com)
Updated	2021-03-21

Objective

Defining requirements and processes for TFX::OSS and related libraries to achieve
necessary API & Artifact compatibility.

Goals

• Clearly define a stable API surface of TFX::OSS, and lay out a structure to
  determine what falls into the stable public APIs.
• API stability: TFX::OSS (as pipeline) and all APIs from Key Dependency Libs
  exposed through TFX::OSS should achieve desired backward compatibility.

[casassg]

Another question: Whats the compatibility between Component and Executor versions. If I define a pipeline in version 1.0.0, will it be possible to have garantees that if the TFX version is 1.X will be able to execute the component nicely? This comes to mind on versioning docker images for Kubeflow or AI Platform Training. Will we need to keep version parity between environment defining the pipeline and environment executing components.

[zhitaoli]

Another question: Whats the compatibility between Component and Executor versions. If I define a pipeline in version 1.0.0, will it be possible to have garantees that if the TFX version is 1.X will be able to execute the component nicely? This comes to mind on versioning docker images for Kubeflow or AI Platform Training. Will we need to keep version parity between environment defining the pipeline and environment executing components.

If I understand it correctly, your example is asking if a TFX pipeline authored with 1.0.0 can use container images from TFX 1.1.0, or vice versa.

To guarantee that example to work, we would need to commit that both TFX IR and Container entrypoint interface are fully stable, both of which was out of our initial scope, because they are still seeing active development.

[casassg]

To guarantee that example to work, we would need to commit that both TFX IR and Container entrypoint interface are fully stable, both of which was out of our initial scope, because they are still seeing active development.

Copy that. May be interesting to call it out in the document.

[zhitaoli]

To guarantee that example to work, we would need to commit that both TFX IR and Container entrypoint interface are fully stable, both of which was out of our initial scope, because they are still seeing active development.

Copy that. May be interesting to call it out in the document.

Done.

[casassg]

lgtm. Added a small question, but not blocking for release as we can figure out later and make it stable later.

[casassg]

🎉