CRD for scheduling Argo workflows (API spec)

[vicaire]

We named this CRD a "schedule": It schedules workflows for now but we will add support for other K8 resources (e.g., jobs). It supports:

• executing pipelines using a cron schedule
• executing pipelines using a periodic interval
• An optional start date and an end date.
• Backfilling (by specifying a start date in the past).
• A maximum number of concurrent workflows.

Most of the files are automatically generated except for the files in:

./hack/custom-boilerplate.go.txt
./hack/update-codegen.sh
./hack/verify-codegen.sh
./pkg/apis/schedule/register.go
./pkg/apis/schedule/v1alpha1/doc.go
./pkg/apis/schedule/v1alpha1/register.go
./pkg/apis/schedule/v1alpha1/types.go

---

This change is 

[jlewi]

/cc @jessesuen

[k8s-ci-robot]

@jlewi: GitHub didn't allow me to request PR reviews from the following users: jessesuen.

Note that only kubeflow members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @jessesuen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jlewi]

Review status: 0 of 33 files reviewed at latest revision, all discussions resolved.

---

pkg/apis/schedule/register.go, line 18 at r1 (raw file):

const (
	Kind      string = "Schedule"

"Schedule" seems like an incomplete name? Why not WorkflowSchedule since you use Workflow in other places (e.g. the Type).

---

pkg/apis/schedule/v1alpha1/types.go, line 27 at r1 (raw file):

// Schedule phases
const (

I believe phases have been dropped in favor of conditions.
kubeflow/training-operator#223

---

pkg/apis/schedule/v1alpha1/types.go, line 62 at r1 (raw file):

type

What's meant by human readable? Does this mean its not used for anything?

---

pkg/apis/schedule/v1alpha1/types.go, line 65 at r1 (raw file):

	// If the schedule is disabled, it does not create any new workflow.
	Enabled bool `json:"enabled,omitempty"`

Why would you enable/disable the schedule as opposed to just deleting it to disable it and creating it to enable it?

---

pkg/apis/schedule/v1alpha1/types.go, line 79 at r1 (raw file):

	// MaxHistory cannot be larger than 100.
	// +optional
	MaxHistory *int64 `json:"maxHistory,omitempty"`

Why is there a limit on history? Whose keeping track of history?

---

pkg/apis/schedule/v1alpha1/types.go, line 97 at r1 (raw file):

	// [[ScheduledTime]] is substituted by the scheduled time of the workflow (default format)
	// [[CurrentTime]] is substituted by the current time (default format)
	// [[Index]] is substituted by the index of the workflow (e.g. 3 mins that it was the 3rd workflow created)

Why are you exposing Index? Why would operation/parameterization of a workflow depend on this?

---

pkg/apis/schedule/v1alpha1/types.go, line 124 at r1 (raw file):

	// Create workflows periodically.
	PeriodicSchedule *PeriodicSchedule `json:"periodicSchedule,omitempty"`

Do you really need PeriodicSchedule? Is CronSchedule not sufficient?

---

pkg/apis/schedule/v1alpha1/types.go, line 134 at r1 (raw file):

not

no end time

---

pkg/apis/schedule/v1alpha1/types.go, line 207 at r1 (raw file):

	// URL representing this object.
	SelfLink string `json:"selfLink,omitempty"`

What is this for?

---

pkg/apis/schedule/v1alpha1/types.go, line 213 at r1 (raw file):

	// Phase is a high level summary of the status of the workflow.
	Phase v1alpha1.NodePhase `json:phase,omitempty`

Why do you have phase? Why not just conditions as noted above?

---

Comments from Reviewable

[vicaire]

Review status: 0 of 45 files reviewed at latest revision, 10 unresolved discussions, all commit checks successful.

---

pkg/apis/schedule/register.go, line 18 at r1 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

"Schedule" seems like an incomplete name? Why not WorkflowSchedule since you use Workflow in other places (e.g. the Type).

Done. Renamed to ScheduledWorkflow as we discussed.

---

pkg/apis/schedule/v1alpha1/types.go, line 27 at r1 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

I believe phases have been dropped in favor of conditions.
kubeflow/training-operator#223

Done. Switched to using conditions.

---

pkg/apis/schedule/v1alpha1/types.go, line 62 at r1 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

type

What's meant by human readable? Does this mean its not used for anything?

Removing it. I don't think it belongs in a CRD spec.

---

pkg/apis/schedule/v1alpha1/types.go, line 65 at r1 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

Why would you enable/disable the schedule as opposed to just deleting it to disable it and creating it to enable it?

This is needed for the following use case:

• Say you create a ScheduledWorkflow to run from now to the end of time, every hour.
• If you delete the workflow and re-create it, it will restart scheduling based on the time at which the ScheduledWorkflow was re-created.
• If you disable the workflow and re-enable it, it will continue starting from the time at which it was disabled.
  Said another way, the enable/disable option backfills the missing hours.

Here is another example:

• Say you are backfilling. You need to run a workflow for every hour of the previous day (the Argo workflow is parameterized by the scheduled time).
• If you delete/re-create, it will restart from scratch.
• If you disable it will temporarily stop creating new workflows. When you re-enable, it will continue where it left off.

This is similar to the backfill functionality of Airflow.

---

pkg/apis/schedule/v1alpha1/types.go, line 79 at r1 (raw file):

Why is there a limit on history? Whose keeping track of history?
The ScheduledWorkflowStatus.WorkflowHistory structure is providing a list of recently executed workflow (both active and completed)

This is very convenient when using kubectl so that the user does not have to query the list of workflows separately with the correct labels.

This is similar to the way the Argo resource returns the graph of PODs so that users can easily investigate the ones that failed.

We need to limit the size of the history returned by the "kubectl get" call since ScheduledWorkflows may run forever

In the future, we may also decide to garbage collect the workflows that are completed, and only keep the X most recent ones in the K8 key value store. This is what the CronJob does: https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/batch/types.go#L276

---

pkg/apis/schedule/v1alpha1/types.go, line 97 at r1 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

Why are you exposing Index? Why would operation/parameterization of a workflow depend on this?

This is one feature explicitly requested by users working on ML workflows.

They want a value, substituted in the parameters of the workflow, that is unique, short, and indicating the order in which the workflows executed.

A unique time can be very verbose: 20180504121645

A UUID is way too long.

This index (suffixed to the workflow name in the Argo workflow spec), is ideal for these users. They use it to generate unique directories to store data in GCS for instance.

---

pkg/apis/schedule/v1alpha1/types.go, line 124 at r1 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

Do you really need PeriodicSchedule? Is CronSchedule not sufficient?

We only provided Cron at first, but discovered an issue.

Cron does not let you schedule a job, for instance, every 5 hours.

It can handle every 6 hours: 0, 6, 12, 18, 0, 6, 12

But if you try for every 5 hours, you end up with: 0, 5, 10, 15, 20, <--- NOT 5 HOURS --> 0, 5, 10

This may be why Airflow supports both Cron and an interval as well: https://airflow.apache.org/scheduler.html#scheduling-triggers

---

pkg/apis/schedule/v1alpha1/types.go, line 134 at r1 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

not

no end time

Fixed.

---

pkg/apis/schedule/v1alpha1/types.go, line 207 at r1 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

What is this for?

These are fields copied from the Argo status for convenience (and so that we can have a fancy looking CLI like Argo).

The ScheduledWorkflowStatus provides a WorkflowHistory object.

The WorkflowHistory object has a list of WorkflowStatuses for active Argo Workflows.

The Workflow Status contains fields copied directly from the Argo status (except for the scheduledTime).

---

pkg/apis/schedule/v1alpha1/types.go, line 213 at r1 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

Why do you have phase? Why not just conditions as noted above?

The field is copied from Argo so we cannot change the type. It is very convenient for the user to have "kubectl get ScheduledWorkflow" return a list of recently executed workflows and their status.

I changed to a Condition above for our own CRD.

---

Comments from Reviewable

[jlewi]

Review status: 0 of 45 files reviewed at latest revision, 4 unresolved discussions.

---

pkg/apis/schedule/v1alpha1/types.go, line 27 at r1 (raw file):

Previously, vicaire (Pascal Vicaire) wrote…

Done. Switched to using conditions.

Done.

---

pkg/apis/schedule/v1alpha1/types.go, line 62 at r1 (raw file):

Previously, vicaire (Pascal Vicaire) wrote…

Removing it. I don't think it belongs in a CRD spec.

Done.

---

pkg/apis/schedule/v1alpha1/types.go, line 65 at r1 (raw file):

Previously, vicaire (Pascal Vicaire) wrote…

This is needed for the following use case:

• Say you create a ScheduledWorkflow to run from now to the end of time, every hour.
• If you delete the workflow and re-create it, it will restart scheduling based on the time at which the ScheduledWorkflow was re-created.
• If you disable the workflow and re-enable it, it will continue starting from the time at which it was disabled.
  Said another way, the enable/disable option backfills the missing hours.

Here is another example:

• Say you are backfilling. You need to run a workflow for every hour of the previous day (the Argo workflow is parameterized by the scheduled time).
• If you delete/re-create, it will restart from scratch.
• If you disable it will temporarily stop creating new workflows. When you re-enable, it will continue where it left off.

This is similar to the backfill functionality of Airflow.

But I thought the whole point of the CRD and the database was that you are persisting actual job runs to a database.

So when you recreate it you should check the database for existing runs.

I think this goes to the point about the CRD being edge triggered vs. level triggered
https://hackernoon.com/level-triggering-and-reconciliation-in-kubernetes-1f17fe30333d

Suppose a ScheduledWorkflow has a schedule of
Daily at 7am
Starting on May1

If you're level triggered then on May 10 we know we want the following Workflows to exist
May 1 7am
May 2 7am
....
May 10 7am

It doesn't matter who/when/or how those runs were created. If any of them are missing we know that we should go and start them.

Thus, it doesn't matter whether the schedule was deleted and then recreated.

I think your implementation is currently edge triggered. You are relying on observing transitions from one time to another.

I think this is going to be more brittle.

If you with a level design then I don't think you need the on/off bit.

---

pkg/apis/schedule/v1alpha1/types.go, line 97 at r1 (raw file):

Previously, vicaire (Pascal Vicaire) wrote…

This is one feature explicitly requested by users working on ML workflows.

They want a value, substituted in the parameters of the workflow, that is unique, short, and indicating the order in which the workflows executed.

A unique time can be very verbose: 20180504121645

A UUID is way too long.

This index (suffixed to the workflow name in the Argo workflow spec), is ideal for these users. They use it to generate unique directories to store data in GCS for instance.

Why not use a random number? Or it least make it look random?

If you return a monotonically increasing number users will likely start to make inferences about what that index means and that will limit your implementation.

e.g. users will likely make assumptions like the numbers should be continuous and that they should match the trigger times of the runs.

---

Comments from Reviewable

[vicaire]

Review status: 0 of 45 files reviewed at latest revision, 4 unresolved discussions.

---

pkg/apis/schedule/v1alpha1/types.go, line 65 at r1 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

But I thought the whole point of the CRD and the database was that you are persisting actual job runs to a database.

So when you recreate it you should check the database for existing runs.

I think this goes to the point about the CRD being edge triggered vs. level triggered
https://hackernoon.com/level-triggering-and-reconciliation-in-kubernetes-1f17fe30333d

Suppose a ScheduledWorkflow has a schedule of
Daily at 7am
Starting on May1

If you're level triggered then on May 10 we know we want the following Workflows to exist
May 1 7am
May 2 7am
....
May 10 7am

It doesn't matter who/when/or how those runs were created. If any of them are missing we know that we should go and start them.

Thus, it doesn't matter whether the schedule was deleted and then recreated.

I think your implementation is currently edge triggered. You are relying on observing transitions from one time to another.

I think this is going to be more brittle.

If you with a level design then I don't think you need the on/off bit.

We want this implementation to be independent of the DB.

Additionally, we need it to scale. The user could have a ScheduledWorkflow running for 3 years for every hour. As time progresses, verifying the complete list would become less and less efficient and slow down the controller (particularly if many instances of ScheduledWorkflow are running).

Finally, the workflow resources may be garbage collected (they should be). If the ScheduledWorkflow runs for 3 years, I don't think the correct functioning of this CRD should require that all the workflow objects be preserved in the K8 key-value store.

This is why we implemented this in a way similar to CronJob, keeping track only of scheduledTime of the latest workflow.

We can implement something to address your use case later on, but that does not seem to fit this CRD very well. I would see it more as a CRD that executes workflows based on a list of possible values for the parameters of the workflow, where these values are times. The number of items in the list would have to be limited.

---

pkg/apis/schedule/v1alpha1/types.go, line 97 at r1 (raw file):

Previously, jlewi (Jeremy Lewi) wrote…

Why not use a random number? Or it least make it look random?

If you return a monotonically increasing number users will likely start to make inferences about what that index means and that will limit your implementation.

e.g. users will likely make assumptions like the numbers should be continuous and that they should match the trigger times of the runs.

Returning a monotonically increasing number is here the requirement.

Say the user creates a unique directory for each run of the workflow. They want the directories to be:

• my-directory-1
• my-directory-2
• my-directory-3
• etc.

We can easily add an substitution for a predictable random looking number (based on a hash). But we do need the monotically increasing, guaranteed to be unique index.

---

Comments from Reviewable

[jlewi]

Review status: 0 of 45 files reviewed at latest revision, 4 unresolved discussions.

---

pkg/apis/schedule/v1alpha1/types.go, line 65 at r1 (raw file):

Previously, vicaire (Pascal Vicaire) wrote…

We want this implementation to be independent of the DB.

Additionally, we need it to scale. The user could have a ScheduledWorkflow running for 3 years for every hour. As time progresses, verifying the complete list would become less and less efficient and slow down the controller (particularly if many instances of ScheduledWorkflow are running).

Finally, the workflow resources may be garbage collected (they should be). If the ScheduledWorkflow runs for 3 years, I don't think the correct functioning of this CRD should require that all the workflow objects be preserved in the K8 key-value store.

This is why we implemented this in a way similar to CronJob, keeping track only of scheduledTime of the latest workflow.

We can implement something to address your use case later on, but that does not seem to fit this CRD very well. I would see it more as a CRD that executes workflows based on a list of possible values for the parameters of the workflow, where these values are times. The number of items in the list would have to be limited.

As discussed offline the real issue here is making backfilling reliable. Right now we update last scheduled time as soon as a new workflow is started. But we shouldn't update it until that workflow completed (successfully or failed) so that controller can ensure work was actually completed.

As discussed offline please file an issue address this in a future PR.

---

pkg/apis/schedule/v1alpha1/types.go, line 97 at r1 (raw file):

Previously, vicaire (Pascal Vicaire) wrote…

Returning a monotonically increasing number is here the requirement.

Say the user creates a unique directory for each run of the workflow. They want the directories to be:

• my-directory-1
• my-directory-2
• my-directory-3
• etc.

We can easily add an substitution for a predictable random looking number (based on a hash). But we do need the monotically increasing, guaranteed to be unique index.

SGTM

---

Comments from Reviewable

[jlewi]

/lgtm
/approve

Please file an issue to track making backfilling reliable.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlewi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jlewi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[vicaire]

Issue filed to make backfills reliable:

#5