Data pipelines

[blairdrummond]

• Figure out best practices
• How do users create containers? (or bypass the need for them)
• Effectively passing data between containers?
• Establish patterns for ingress/egress of data from pipelines to/from storage.

This is just a starting list of objectives, but the overall goal of the epic is to make kf-pipelines and/or Pachyderm a polished experience for data scientists making pipelines.

[chritter]

@blairdrummond Good idea to open these points for discussion. Could you schedule a discussion in Teams or Slack audio?

[brendangadd]

With the change to a non-open-source licence, we need to evaluate alternatives to Pachyderm. Key features we're looking for data pipelines:

• Orchestration (schedule, trigger)
• Parallelization (multi-container)
• Historical metadata (what ran when with what parameters)
• Data snapshotting/lineage (what data was consumed/produced, maybe diffed)

[ca-scribner]

from @justbert : https://dvc.org/blog/cml-release
looks interesting