[PROPOSAL] Drone Custom Stage Definitions

[bradrydzewski]

This proposal is to create a new category of plugin for Drone, which I am tentatively describing as Custom Stage Definitions, which are heavily inspired by custom resource definitions for Kubernetes.

Today a pipeline is a series of steps that execute in Docker containers. These pipelines are classified as Docker pipelines. When the type field is empty we assume this is the default pipeline type.

kind: pipeline

The goal is to allow additional types of pipelines with their own custom yaml syntax. For example, we could have a knative plugin that uses knative syntax.

kind: pipeline
+type: knative

or a pipeline that runs commands directly on the host machine:

kind: pipeline
+type: exec

Each custom stage provider would be responsible for parsing the body of the yaml, interpreting and executing the contents. For example:

kind: pipeline
type: exec

platform:
  os: darwin
  arch: amd64

steps:
- name: build
  commands:
  - go build
  - go test

Kinds

The kind field could be used to create stages that do not qualify as pipelines. For example maybe we want stages to handle approvals:

kind: approval

or highly customized deployment strategies:

kind: deployment
type: redblack|canary

Design

I will begin moving the agent <> server code to the runner-go library and I will create a starter project for creating custom stage operators. You can think of this as, effectively, creating your own Drone agent.

Drone will ship with the default Docker pipeline runner only. All other runners will be community contributed and managed. I will create a separate organization, called drone-runners, to help organize community development.

Why?

I realized that the core Drone team is too inexperienced with Kubernetes and is stretched too thin to offer the Kubernetes runtime. Instead we can open this up to the community, and hand over control, just like we did with plugins years ago. Plugins quickly became one of the defining and most loved features in Drone. I believe Custom Stage Definition will have a similar impact on the project and community.

I envision the community creating all sorts of interesting runners:

• exec runner launches builds on the host machine
• chroot runner, similar to exec, but in a chroot
• tekton runner
• virtualbox runner
• vmware vsphere runner (for osx)
• aws runner (launches ad-hoc ephemeral vms)
• gcp runner (launches ad-hoc ephemeral vms)
• digitalocean runner (launches ad-hoc ephemeral vms)

[metalmatze]

I like the idea a lot!

I suppose there could be some sort of community driven but still official Custom Stage Definitions? Just like there are some more official plugins in drone-plugins?
Give the current tight coupling between Drone's agent <> server, how would the interface probably look? Is this somewhat similar to what we already have for Docker and Kubernetes?

[bradrydzewski]

We currently define the below interface in Drone for the agent to request builds from the server and report back updates. We would move this interface to drone-go so anyone could write a client that interacts with the server. We would need to modify Request to include the kind and type of pipelines that the client is requesting, and update the server to filter on this data. Overall though should be a minor change.

	BuildManager interface {
		// Request requests the next available build stage for execution.
		Request(ctx context.Context, args *Request) (*core.Stage, error)

		// Accept accepts the build stage for execution.
		Accept(ctx context.Context, stage int64, machine string) error

		// Netrc returns a valid netrc for execution.
		Netrc(ctx context.Context, repo int64) (*core.Netrc, error)

		// Details fetches build details
		Details(ctx context.Context, stage int64) (*Context, error)

		// Before signals the build step is about to start.
		Before(ctxt context.Context, step *core.Step) error

		// After signals the build step is complete.
		After(ctx context.Context, step *core.Step) error

		// Before signals the build stage is about to start.
		BeforeAll(ctxt context.Context, stage *core.Stage) error

		// After signals the build stage is complete.
		AfterAll(ctx context.Context, stage *core.Stage) error

		// Watch watches for build cancellation requests.
		Watch(ctx context.Context, stage int64) (bool, error)

		// Write writes a line to the build logs
		Write(ctx context.Context, step int64, line *core.Line) error

		// Upload uploads the full logs
		Upload(ctx context.Context, step int64, r io.Reader) error

		// UploadBytes uploads the full logs
		UploadBytes(ctx context.Context, step int64, b []byte) error
	}

[tboerger]

I REALLY like this approach. That would make Drone even much more flexible and can grow lots of different stage implementations.

[bobcatfish]

tekton runner

Very exciting!! 😻 @bradrydzewski if you're ever interested in chatting and/or demoing drone in a Tekton working group we'd love to have you :D

[bradrydzewski]

@bobcatfish thanks, I will definitely reach out once we start working on Tekton support 😄

[metalmatze]

I just signed up for your meetings @bobcatfish. I'll try to join and see what you people are up to, since I've been working on the Kubernetes runtime. See you on Tuesday!

[bradrydzewski]

custom stage definitions are now enabled for kind: pipeline with custom type values. The rpc mechanism is available and necessary changes were made to the scheduler. Closing this issue and will document custom stage definitions in the documentation, and hope to have a couple example runners published next week.