Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(RF)RFC: reusable tasks #7

Open
vito opened this Issue Jul 19, 2018 · 14 comments

Comments

Projects
None yet
6 participants
@vito
Copy link
Member

vito commented Jul 19, 2018

(this is a rough draft/placeholder just to centralize brainstorming before an actual RFC)

ref. concourse/concourse#417
ref. concourse/docker-image-resource#185

There is clearly a need to make tasks just as easy to share, version, and re-use just as resources are.

What would this look like?

Today you can approximate this by storing task .ymls in a Git repo, and then pulling them in as a resource.

One big hole is configuration. You can only really specify params, which for tasks are environment variables. Lots of resources use params for arbitrary, sometimes nested config. Should we extend tasks to have the same ability? It would actually be pretty easy to just pass it to the script on stdin, just like we do for resources. (Would we then rename today's params to env or something to be more clear?)

@jchesterpivotal

This comment has been minimized.

Copy link

jchesterpivotal commented Jul 31, 2018

A few months backed I sketched some notes for these. It goes a bit further than what's immediately needed -- I assume that streaming inputs are a supported case.

@vito

This comment has been minimized.

Copy link
Member Author

vito commented Aug 10, 2018

I wanna pull concourse/concourse#608 into the discussion here too. I think if we had more shareable tasks we'd be more inclined to implement things like the Terraform resource as a task or set of tasks instead of a resource, and then the user is free to use whatever storage backend the like for persisting the task's side effects.

I recognise that this goes against the idea of tasks being pure and side-effect-free, but I think the more important aspect has always been that they're idempotent, and I'm interested in seeing if this change in mindset leads to a more expressive reconfiguration of existing ideas.

I've mocked up a pipeline for this here: https://gist.github.com/vito/3f0ed09648bf117d95a3b7daa99f61c2

Here are some decisions I made with this mockup:

  • Tasks are defined at the top-level, and are referenced by name when used in jobs. Now there's an actual use for the name of the task step! That feels nice an intuitive - task names have been completely cosmetic for a while, unlike their get/put counterparts.
  • Top-level tasks are provided by a resource, similar to resource types. This supports storing them in git repos pretty well. This pattern is already followed by cf-deployment-concourse-tasks for example. I like storing tasks in git repos as it makes versioning and inspection very easy (browse around on GitHub rather than reverse-engineering a Dockerfile).
  • Each top-level task configures the path to the task config file within its resource. Theoretically this could be inferred from the task name and we could have a convention like tasks/(name).yml - having to type it out feels a little clunky compared to resources. Feels kinda magical tho.
  • I snuck in a new feature on get steps: optional: true. This would be necessary for the initial deploy where there's no state yet. This feature would have obviated the need for concourse/s3-resource#21 which was suggested precisely for this very use case. (Also, tasks already support optional inputs so these two features would work nicely together.)
  • On a task referenced by-name, params becomes arbitrary config, not env. This is confusing but we can make it less confusing by replacing the word params with env when it comes to task params. We can do this backwards-compatibly by having params mean env for file:/config:-provided tasks, and have it mean config for pipeline-provided tasks.

One thing I decided against with this mockup was having tasks provided by Docker images (a-la resource types). In analyzing my current use case and cf-deployment-concourse-tasks, I noticed that there tends to be more than one task using a given image. For example, my pipeline already has a case for both terraform apply and terraform output. Forcing those to be entirely separate images feels wasteful.

There'd also be a bit of a circle here because task configs already exist as a top-level task entrypoint, and they can already point to their own image. So it felt more natural and flexible to just start from a task config rather than a Docker image.

@jchesterpivotal

This comment has been minimized.

Copy link

jchesterpivotal commented Aug 10, 2018

A nitpick -- if we do allow inferred directories, I'm keen to push on using the full taskdir pattern. So instead of tasks/task-name.yaml, it's tasks/task-name/task.yaml. The executable by convention is tasks/task-name/run (not task, because tab completion is nice).

In my experience tasks can become quite sophisticated; having all the logic isolated in the subdirectory is easier to navigate and selectively update.

@jchesterpivotal

This comment has been minimized.

Copy link

jchesterpivotal commented Aug 16, 2018

A pattern I am trying is to bundle tasks with a resource image, as they are intended to be used together. See concourse-build-resource for the example.

@ari-becker

This comment has been minimized.

Copy link

ari-becker commented Feb 13, 2019

Internally, we're approaching this problem with Dhall. If the rest of the pipeline is also written in Dhall, then we can write a default "implementation" for the task, pull it in, and pass it configuration by way of function parameters or by overwriting the relevant fields of the default.

More broadly, this may not just be re-usable tasks, but re-usable pipelines - allowing people to write common pipelines for workflows like building, testing, and pushing containers, where the entire pipeline is represented by a function which takes parameters like the image name, registry details, etc.

It would be great if Concourse had better support for Dhall (we're currently using dhall-to-yaml to get it to work with Concourse), but maybe that's a bigger discussion.

@amnk

This comment has been minimized.

Copy link

amnk commented Feb 13, 2019

@ari-becker wow. Are there any public resources about that approach? Maybe some articles you wrote? Or at least a gist with an example? :)

@ari-becker

This comment has been minimized.

Copy link

ari-becker commented Feb 14, 2019

Are there any public resources about that approach? Maybe some articles you wrote? Or at least a gist with an example? :)

@amnk For Concourse, not yet. Right now we're trying to ship our implementation internally, then it needs to be cleaned up / refactored, and then we'll get around to open-sourcing the generic bits and presenting on it. It can't really be summed up in a gist because Dhall ends up forcing you to write types for the entire schema. But you can imagine that if you had a task type:

{ task : Text
, config : Optional 
  { platform : Text
  , image_resource : Optional ImageResource
  , inputs : Optional  (List Input)
  , run : Run
  , rootfs_uri : Optional Text
  , outputs : Optional (List Output)
  , caches : Optional (List Cache)
  , params : Optional (List {mapKey : Text , mapValue : Text})
  }
, file : Optional Text
, privileged : Optional Bool
, params : Optional (List {mapKey: Text, mapValue: Text})
, image : Optional Text
, input_mapping : Optional (List {mapKey : Text, mapValue : Text})
, output_mapping : Optional (List {mapKey : Text, mapValue : Text})
}

where ImageResource, Input, Output, and Cache are defined elsewhere, and you wrote a default task for some computation:

\(_params : { config_params : List {mapKey : Text, mapValue : Text}, task_params : List {mapKey : Text, mapValue: Text}}) ->
{ task = "example-computation"
, config = Some 
  { platform = "linux"
  , image_resource = Some <fill_in_the_gap...>
  , inputs = Some [ <input_dir> ]
  , run = <does_the_computation>
  , rootfs_uri = None Text
  , outputs = Some [ <output_dir> ]
  , caches = None (List Cache)
  , params = Some _params.config_params
  }
, file = None Text
, privileged = None Bool
, params = Some _params.task_params
, image = Some "<image_defined_elsewhere>"
, input_mapping = None (List {mapKey : Text, mapValue : Text})
, output_mapping = None (List {mapKey : Text, mapValue : Text})
} : Task

Then all the consumer has to do is call :

mkDefaultExampleComputationTask { config_params = [ { mapKey = "example_key", mapValue = "example_value" }], task_params =  [ { mapKey = "example_key", mapValue = "example_value" }] } // { image = Some "<other_image>" }

and if you call dhall-to-yaml on that then you get your generated task as yaml, with the given parameters from config_params and task_params, and the image field will show <other_image> instead of <image_defined_elsewhere>.

Now, idiomatic Dhall doesn't really look like that; the types are handled differently, and you can write more than one implementation, e.g. mkParamlessExampleComputationTask which inlines empty lists for params in both the task and the config.... lots of stuff like that. But hopefully that gives some idea of where to get started.

@itsdalmo

This comment has been minimized.

Copy link

itsdalmo commented Mar 13, 2019

I recognise that this goes against the idea of tasks being pure and side-effect-free, but I think the more important aspect has always been that they're idempotent, and I'm interested in seeing if this change in mindset leads to a more expressive reconfiguration of existing ideas.

I think this RFC and some help for reusable tasks is necessary, but I think the distinction between a task and a resource becomes somewhat blurry in this case. Since resources and tasks are both meant to be idempotent, and both can have side-effects, what then is the distinction between a task and a resource?

And what do we mean by idempotent here? Using terraform as an example, strictly speaking it is impossible for the task to be idempotent because the state-file is opaque to Concourse, and the true state is not known until terraform refreshes the state?

One thing I decided against with this mockup was having tasks provided by Docker images (a-la resource types). In analyzing my current use case and cf-deployment-concourse-tasks, I noticed that there tends to be more than one task using a given image. For example, my pipeline already has a case for both terraform apply and terraform output. Forcing those to be entirely separate images feels wasteful.

A simple solution to the above would just be to pass terraform apply or terraform output as arguments to the task, which would then allow us to reuse the same image for multiple tasks. I think for tasks like terraform apply (which has potentially large side-effects) and other tasks where correctness is paramount, it makes the most sense if the task itself is responsible for setting defaults and validating the source configuration and parameters, instead of relying on a task.yml which may/may not get out of sync with the code (especially if the code is delivered as part of a docker image, in the case of e.g. compiled languages).

I think this RFC is nice for shaping up tasks as they exist today, but I feel like we are still missing a solution for "serious" tasks as mentioned above. Essentially a resource which implements only put, which is today considered an anti-pattern, would be perfect for something like terraform apply and friends 🤔

@ari-becker

This comment has been minimized.

Copy link

ari-becker commented Mar 13, 2019

@ari-becker wow. Are there any public resources about that approach? Maybe some articles you wrote? Or at least a gist with an example? :)

Just to link to it from this issue: since replying to @amnk , we've open-sourced dhall-concourse, which provides a limited solution to this problem, and I imagine is a helpful reference point for the maintainers looking at solutions.

I think this RFC and some help for reusable tasks is necessary, but I think the distinction between a task and a resource becomes somewhat blurry in this case. Since resources and tasks are both meant to be idempotent, and both can have side-effects, what then is the distinction between a task and a resource?

A resource defines some/all of three named impure tasks: check, in/get, and out/put, each having fixed, defined input and output schemas so as to be additionally useful to Concourse. I imagine that you could think of resources as a unique, built-in to Concourse, reusable union of those three named impure tasks. That's not quite the same thing as generic, reusable, ideally-pure tasks.

... Terraform ...

Terraform is nearly idempotent when plan files are used - essentially, state can be versioned for the check by hashing the output of terraform plan, state can be fetched for in/get by running terraform plan -out=terraform.plan and making sure that it matches the hash, and state can be pushed for out/put by running terraform apply terraform.plan. It's certainly not a put-only paradigm.

It's still not a great fit for Terraform though, because terraform apply cannot (even in theory) be represented according to Concourse's current version paradigm - for a given invocation of terraform apply, whether using a plan file or not, and for reasons entirely outside of Terraform's or Concourse's control (i.e. provider availability), the output may represent success, failure, or somewhere in the middle (what Terraform calls "tainted"). In essence, Concourse currently thinks of versions as being entirely deterministic, which doesn't fit Terraform's model because Terraform won't know what the "version" of the infrastructure will be until after terraform apply is called, not before. Anyway, Concourse's notions of versioning don't seem so relevant to this RF(RFC).

@itsdalmo

This comment has been minimized.

Copy link

itsdalmo commented Mar 13, 2019

A resource defines some/all of three named impure tasks: check, in/get, and out/put, each having fixed, defined input and output schemas so as to be additionally useful to Concourse. I imagine that you could think of resources as a unique, built-in to Concourse, reusable union of those three named impure tasks. That's not quite the same thing as generic, reusable, ideally-pure tasks.

Thanks for explaining, but I don't feel like this makes it any more clear to me. Currently, I could write a task with side effects, it just would not be versioned by Concourse. Likewise, I could write a resource which is not versioned by Concourse by implementing a no-op check and get, and not emitting any versions from put. Both the task and the resource would behave the same way from a user perspective, but Concourse would treat them differently internally.

Anyway, I'm not saying that Concourse does not need tasks. I'm just thinking that the info for resources v2 would allow us to implement a wider range of resource types, and have Concourse work out internally which keywords map to which operation, and how to use them. So e.g. we could add a task artifact to a resource, and Concourse would know that said resource could be used for tasks, and that it would not output any versions for Concourse to track when used as a task.

Terraform is nearly idempotent when plan files are used - essentially, state can be versioned for the check by hashing the output of terraform plan, state can be fetched for in/get by running terraform plan -out=terraform.plan and making sure that it matches the hash, and state can be pushed for out/put by running terraform apply terraform.plan. It's certainly not a put-only paradigm.

I don't think this is correct/would work for the following reasons:

  • The output of terraform plan and the hash changes on every invocation even if the plan stays the same. Additionally, it is in a binary format and it is documented as having an unstable API and being solely for consumption by terraform.
  • terraform plan does not fetch the state, it does however refresh the current state before producing a plan which describes how to get from current state > desired state. As such, using terraform plan and apply in two separate steps does not really make it any more idempotent since the state lock would be released between plan and apply, which means that both the terraform state and the "true" state could have changed in between.
@vito

This comment has been minimized.

Copy link
Member Author

vito commented Mar 13, 2019

I think this RFC and some help for reusable tasks is necessary, but I think the distinction between a task and a resource becomes somewhat blurry in this case. Since resources and tasks are both meant to be idempotent, and both can have side-effects, what then is the distinction between a task and a resource?

Resources are versioned, tasks are not. Tasks can perform side-effects, but they can only produce artifact outputs, and to persist them or version them, you would then use a resource to persist those outputs. This is a more explicit separation and lets people decide how to persist things like terraform state without the terraform resource having to implement multiple persistence backends.

I see this as our tactic to eliminate put-only resources - especially as we'll soon be removing their ability to even record versions in light of concourse/concourse#3463, as that has led to a lot of user confusion.

And what do we mean by idempotent here? Using terraform as an example, strictly speaking it is impossible for the task to be idempotent because the state-file is opaque to Concourse, and the true state is not known until terraform refreshes the state?

All I mean by idempotent is the task can be re-run with the same inputs and will attempt to do the same thing. Partial successes can be re-tried, and may eventually succeed. It sounds like the conflict here is that even partial successes have to be persisted in some way, though I'm really not sure how to reason about that because you could always fail to persist the state file anyway.

@vito

This comment has been minimized.

Copy link
Member Author

vito commented Mar 13, 2019

I forgot to note it here, but my most recent experiment in this area is the builder task.

To be honest, it already works pretty well! It's been interesting seeing the patterns emerge from how it's used. The task image is pulled in via a resource in our pipeline and used like this:

https://github.com/concourse/concourse/blob/9d2a80765400d89b0214a396b7532b1245675e39/ci/pipelines/concourse.yml#L93-L105

...but then we also have some task .ymls that pull it in via image_resource:

https://github.com/concourse/concourse/blob/master/ci/tasks/build-rc-image.yml#L4-L7

This makes it easy to use fly execute to run the task and allows more of its configuration to be pre-defined.

Here are things I learned from this experiment:

  • Allow images to configure an ENTRYPOINT - then consumers wouldn't need to set run: at all in this case, and Terraform could just have an entrypoint that interprets apply or output arguments.
  • Being able to use JSON instead of env for params: would go a long way in terms of API flexibility. This would bring tasks up to par with resources.
  • Task caches are actually pretty useful, at least in our scenario - it lets us cache intermediate layers between image builds. This was a pleasant surprise.
  • input_mapping and output_mapping work nicely with generic tasks, and being able to list the inputs in the task config helps a lot with fly execute.

It feels like there aren't many missing pieces here, and that by implementing those first two points we might enable some pretty interesting usage patterns that integrate nicely with a lot of today's existing patterns.

@ljfranklin

This comment has been minimized.

Copy link

ljfranklin commented Mar 14, 2019

Really liking the direction this is heading, specifically the ability to ship tasks as docker images rather than repos with random bash scripts and the ability to pass YAML/JSON rather than trying to pack complex structures into env vars.

Bit of a tangent but I recently made some changes to the terraform-resource that seems related. Up to this point the terraform-resource only supported storing state in S3-compatible blobstores which was a pain for users that would rather use GCS/Azure/etc. However Terraform recently added support for storing the statefile in a variety of backends from the CLI itself. There are currently 10+ backends which are supported. Once you configure a backend, when you run terraform apply the CLI automatically persists the statefile changes into the configured storage endpoint. I recently updated the resource to use this backend functionality rather than my home-rolled S3 client.

Along with these backends Terraform also now has a concept of workspaces. This allows Terraform to store multiple statefiles within a single backend, as well as list, download, delete those statefiles. These changes have nicely filled out the terraform-resource so it can more realistically implement the get and check operations. It still falls short in fetching versions other than latest for each environment, but at least now it doesn't have to rely on put producing version numbers (proposal to remove put-only resource functionality).

While the re-usable tasks are going to be super useful, for the Terraform case specifically I'm still liking the resource as an interface. The user doesn't have to worry about getting the ensure blocks just right in order to persist state in the event of a failure and using backends allows you to specify your storage configuration via a first-class Terraform feature so you're more in line with the rest of that community. I'm also still having trouble letting go of my "ideal" task mental model:

  • get on a resource pulls state from the outside world (e.g. the internet)
  • put on a resource modifies state in the outside world
  • tasks are purely functional operations on inputs and outputs, no side-effects
    • also makes tasks easily unit testable if you want to test your test scripts

In practice I've written a ton of tasks that don't conform to this description as it was easier to just write a bunch of wgets than fit what I was doing into a resource, hence the scare quotes on "ideal". All that said, none of this disagrees with the re-usable task proposal above, just thinking out loud around what could/should be modeled as a task vs a resource.

@vito

This comment has been minimized.

Copy link
Member Author

vito commented Mar 14, 2019

@ljfranklin Thanks, that context really helps! 👍 Using Terraform's persistence backends definitely makes it feel more "resource"-like.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.