From 6fd41f9dac479b6e5a3302f10b462f72b614ddfa Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Wed, 3 Apr 2019 15:47:07 -0400 Subject: [PATCH 01/34] initial generalized resources proposal the proposal itself is not yet fleshed out at all - still in the information collecting phase so that the motivations are clear Signed-off-by: Alex Suraci --- 023-generalized-resources/proposal.md | 110 ++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) create mode 100644 023-generalized-resources/proposal.md diff --git a/023-generalized-resources/proposal.md b/023-generalized-resources/proposal.md new file mode 100644 index 00000000..e48a72d1 --- /dev/null +++ b/023-generalized-resources/proposal.md @@ -0,0 +1,110 @@ +# Summary + +This proposal adapts today's 'resource' interface into a more general interface that is less specialized to the 'versioned artifacts' use case, while also introducing versioning to the interface so that it'll be easier to make incremental changes to the interface itself and support mixed resource versions within a user's pipeline. + +Today's resources are closely tied to the 'versioned artifact' use case, so this proposal will also show how the new interface is "interpreted" in order to support this use case. + + +# Motivation + +* Support for deleting versions in `put`: + * https://github.com/concourse/concourse/issues/362 + * https://github.com/concourse/concourse/issues/524 + +* Support for creating multiple versions from `put`: + * https://github.com/concourse/concourse/issues/2660 + +* Having resource metadata immediately available via check: + * https://github.com/concourse/git-resource/issues/193 + * https://github.com/concourse/concourse/issues/1714 + +* Make the `get` after `put` opt-in: + * https://github.com/concourse/concourse/issues/3299 + * https://stackoverflow.com/questions/38964299/why-does-concourse-get-a-resource-after-puting-it + * https://github.com/concourse/registry-image-resource/issues/16 + +* Unifying `source` and `params` as just `config` so that resources don't have to care where configuration is being set in pipelines: + * https://github.com/concourse/git-resource/pull/172 + * https://github.com/concourse/bosh-deployment-resource/issues/13 + * https://github.com/concourse/bosh-deployment-resource/issues/6 + * https://github.com/concourse/cf-resource/pull/20 + * https://github.com/concourse/cf-resource/pull/25 + * https://github.com/concourse/git-resource/pull/210 + +* Generalize interface to support non-versioned state: + * https://github.com/concourse/concourse/issues/739 + +* Support for trigger-only resources that don't result in fetching anything - they just trigger the job: + * https://github.com/concourse/concourse/issues/3572 + * https://github.com/concourse/concourse/issues/3595 + +* Make resource actions more reentrant so that we no longer receive `unexpected EOF` errors upon reattaching to an in-flight build. + * https://github.com/concourse/concourse/issues/1580 + +* Support multi-branch workflows: + * https://github.com/concourse/concourse/issues/1172 + + * Begin phasing out `version: every` in by reframing the problem as 'pipeline per commit': + * https://github.com/concourse/concourse/issues/736 + +* Support notifications in a way that doesn't pollute pipeline config and UI: + * https://github.com/concourse/concourse/issues/1052 + * https://github.com/concourse/rfcs/issues/10 + + +# Proposal + +* unify `source` and `params` into single `config` +* replace `version` with "config fragments" so that the interface isn't biased towards "versioned artifacts" and instead re-configures itself. + * "config fragments" get merged into `config` + * `check` -> config fragment `{ref: abcd}` +* `/info` returns interface version and mapping of `check`/`get`/`put` to commands. +* responses go to files so that: + * chunks of output are not lost if events are emitted to quickly + * responses can be read after the process exits when a build is reattached + + +# Open Questions + +* metadata structure? + * enrich metadata?: https://github.com/concourse/concourse/issues/310 +* TLS? + * https://github.com/concourse/rfcs/issues/9 +* webhooks? + * should these instead be something supported by *Concourse*? +* icons? + * https://github.com/concourse/concourse/pull/3220 + * https://github.com/concourse/concourse/pull/3581 + * can this come from `/info`? would it need `config` passed to it? +* resource-determined triggerability of versions? + * https://github.com/concourse/rfcs/issues/11 + + +# Answered Questions + +* version filtering is probably best left to `config` + + +# New Implications + +many + + +# Yet-to-be-organized notes + +Cataloguing ways in which generalized resources can be composed to accomplish different goals: + +* artifact `check` -> config fragment + artifact `check` + * check from version +* artifact `check` -> config fragment + artifact `get` + * fetch specific version +* artifact `put` -> config fragment + artifact `get` + * fetch just-created version +* spatial `check` -> config fragment + artifact `check` + * check across all spaces +* artifact `get` -> config fragment + notification `put` + * update github status +* trigger `check` -> config fragment -> trigger build if different from last config fragment + * trigger-only resources + * maybe the config fragment could be passed to something to support parameterized triggers? :thinking: + * maybe that could fit nicely with however we approach https://github.com/concourse/concourse/issues/783? :thinking: \ No newline at end of file From 28505007a0ea91861b7de73e2925a956dbc12c10 Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Wed, 3 Apr 2019 15:57:18 -0400 Subject: [PATCH 02/34] i guessed the number wrong :( Signed-off-by: Alex Suraci --- .../proposal.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {023-generalized-resources => 024-generalized-resources}/proposal.md (100%) diff --git a/023-generalized-resources/proposal.md b/024-generalized-resources/proposal.md similarity index 100% rename from 023-generalized-resources/proposal.md rename to 024-generalized-resources/proposal.md From 50a272a72f2a498d2370a8d5841d811de541736e Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Wed, 3 Apr 2019 17:14:30 -0400 Subject: [PATCH 03/34] start defining important terms, tweak wording Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index e48a72d1..a2f2ac0d 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -54,6 +54,23 @@ Today's resources are closely tied to the 'versioned artifact' use case, so this # Proposal +## Glossary + +* **Config**: an arbitrarily nested JSON object containing user-provided configuration + * Examples: `{"uri":"https://github.com/concourse/concourse"}`, `{"interval":"10m"}` +* **Config fragment**: a smaller JSON object intended to be spliced into a **Config** + * Examples: `{"ref":"abcdef"}`, `{"branch":"develop"}` +* **Bits**: a directory containing arbitrary data + * Examples: source code, compiled artifacts, etc. +* **Metadata**: structured data associated to a **config fragment** containing information about the fragment that should be surfaced to the user + * Examples: `[{"name":"committer","value":"Alex Suraci"}]` +* **Resource**: any entity which supports the following verbs: + * `check`: given a **config**, emit **config fragments** + * `get`: given a **config**, populate a directory with **bits** + * `put`: given a **config** and a directory containing **bits**, create or delete **config fragments** + + + * unify `source` and `params` into single `config` * replace `version` with "config fragments" so that the interface isn't biased towards "versioned artifacts" and instead re-configures itself. * "config fragments" get merged into `config` @@ -66,13 +83,14 @@ Today's resources are closely tied to the 'versioned artifact' use case, so this # Open Questions -* metadata structure? - * enrich metadata?: https://github.com/concourse/concourse/issues/310 -* TLS? +* enrich metadata? + * https://github.com/concourse/concourse/issues/310 +* standardize TLS config? * https://github.com/concourse/rfcs/issues/9 * webhooks? * should these instead be something supported by *Concourse*? * icons? + * https://github.com/concourse/concourse/issues/788 * https://github.com/concourse/concourse/pull/3220 * https://github.com/concourse/concourse/pull/3581 * can this come from `/info`? would it need `config` passed to it? From 9554b973d96ba4b00ef02d2176e39afcc64d1849 Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Thu, 4 Apr 2019 15:04:42 -0400 Subject: [PATCH 04/34] glossary updates, add icon support as motivation ...plus a bunch of tweaks to make the markdown linter happy Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 170 +++++++++++++++----------- 1 file changed, 96 insertions(+), 74 deletions(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index a2f2ac0d..57e58b93 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -1,128 +1,150 @@ -# Summary +# Generalized Resources This proposal adapts today's 'resource' interface into a more general interface that is less specialized to the 'versioned artifacts' use case, while also introducing versioning to the interface so that it'll be easier to make incremental changes to the interface itself and support mixed resource versions within a user's pipeline. Today's resources are closely tied to the 'versioned artifact' use case, so this proposal will also show how the new interface is "interpreted" in order to support this use case. - -# Motivation +## Motivation * Support for deleting versions in `put`: - * https://github.com/concourse/concourse/issues/362 - * https://github.com/concourse/concourse/issues/524 + * https://github.com/concourse/concourse/issues/362 + * https://github.com/concourse/concourse/issues/524 * Support for creating multiple versions from `put`: - * https://github.com/concourse/concourse/issues/2660 + * https://github.com/concourse/concourse/issues/2660 * Having resource metadata immediately available via check: - * https://github.com/concourse/git-resource/issues/193 - * https://github.com/concourse/concourse/issues/1714 + * https://github.com/concourse/git-resource/issues/193 + * https://github.com/concourse/concourse/issues/1714 * Make the `get` after `put` opt-in: - * https://github.com/concourse/concourse/issues/3299 - * https://stackoverflow.com/questions/38964299/why-does-concourse-get-a-resource-after-puting-it - * https://github.com/concourse/registry-image-resource/issues/16 + * https://github.com/concourse/concourse/issues/3299 + * https://stackoverflow.com/questions/38964299/why-does-concourse-get-a-resource-after-puting-it + * https://github.com/concourse/registry-image-resource/issues/16 * Unifying `source` and `params` as just `config` so that resources don't have to care where configuration is being set in pipelines: - * https://github.com/concourse/git-resource/pull/172 - * https://github.com/concourse/bosh-deployment-resource/issues/13 - * https://github.com/concourse/bosh-deployment-resource/issues/6 - * https://github.com/concourse/cf-resource/pull/20 - * https://github.com/concourse/cf-resource/pull/25 - * https://github.com/concourse/git-resource/pull/210 + * https://github.com/concourse/git-resource/pull/172 + * https://github.com/concourse/bosh-deployment-resource/issues/13 + * https://github.com/concourse/bosh-deployment-resource/issues/6 + * https://github.com/concourse/cf-resource/pull/20 + * https://github.com/concourse/cf-resource/pull/25 + * https://github.com/concourse/git-resource/pull/210 * Generalize interface to support non-versioned state: - * https://github.com/concourse/concourse/issues/739 + * https://github.com/concourse/concourse/issues/739 * Support for trigger-only resources that don't result in fetching anything - they just trigger the job: - * https://github.com/concourse/concourse/issues/3572 - * https://github.com/concourse/concourse/issues/3595 + * https://github.com/concourse/concourse/issues/3572 + * https://github.com/concourse/concourse/issues/3595 * Make resource actions more reentrant so that we no longer receive `unexpected EOF` errors upon reattaching to an in-flight build. - * https://github.com/concourse/concourse/issues/1580 + * https://github.com/concourse/concourse/issues/1580 * Support multi-branch workflows: - * https://github.com/concourse/concourse/issues/1172 + * https://github.com/concourse/concourse/issues/1172 - * Begin phasing out `version: every` in by reframing the problem as 'pipeline per commit': - * https://github.com/concourse/concourse/issues/736 +* Begin phasing out `version: every` in by reframing the problem as 'pipeline per commit': + * https://github.com/concourse/concourse/issues/736 * Support notifications in a way that doesn't pollute pipeline config and UI: - * https://github.com/concourse/concourse/issues/1052 - * https://github.com/concourse/rfcs/issues/10 - + * https://github.com/concourse/concourse/issues/1052 + * https://github.com/concourse/rfcs/issues/10 -# Proposal +* Support for showing icons for resources in the web UI: + * https://github.com/concourse/concourse/issues/788 + * https://github.com/concourse/concourse/pull/3220 + * https://github.com/concourse/concourse/pull/3581 ## Glossary * **Config**: an arbitrarily nested JSON object containing user-provided configuration - * Examples: `{"uri":"https://github.com/concourse/concourse"}`, `{"interval":"10m"}` -* **Config fragment**: a smaller JSON object intended to be spliced into a **Config** - * Examples: `{"ref":"abcdef"}`, `{"branch":"develop"}` -* **Bits**: a directory containing arbitrary data - * Examples: source code, compiled artifacts, etc. -* **Metadata**: structured data associated to a **config fragment** containing information about the fragment that should be surfaced to the user - * Examples: `[{"name":"committer","value":"Alex Suraci"}]` -* **Resource**: any entity which supports the following verbs: - * `check`: given a **config**, emit **config fragments** - * `get`: given a **config**, populate a directory with **bits** - * `put`: given a **config** and a directory containing **bits**, create or delete **config fragments** - - + * Examples: `{"uri":"https://github.com/concourse/concourse"}`, `{"interval":"10m"}` -* unify `source` and `params` into single `config` -* replace `version` with "config fragments" so that the interface isn't biased towards "versioned artifacts" and instead re-configures itself. - * "config fragments" get merged into `config` - * `check` -> config fragment `{ref: abcd}` -* `/info` returns interface version and mapping of `check`/`get`/`put` to commands. -* responses go to files so that: - * chunks of output are not lost if events are emitted to quickly - * responses can be read after the process exits when a build is reattached +* **Config fragment**: a smaller JSON object intended to be "spliced" into a **config** by assigning each field from the fragment into the config. + * Examples: `{"ref":"abcdef"}`, `{"branch":"develop"}` +* **Bits**: a directory containing arbitrary data + * Examples: source code, compiled artifacts, etc. -# Open Questions +* **Metadata**: structured data associated to a **config fragment** containing information about the fragment that should be surfaced to the user + * Examples: `[{"name":"committer","value":"Alex Suraci"}]` + +* **Resource type**: an implementation of the interface defined by this proposal, typically provided as a container image. Implements the following actions: + * `info`: given a **config**, emit provide the commands to run for the following actions: + * `check`: given a **config**, emit **config fragments** + * `get`: given a **config**, populate a directory with **bits** + * `put`: given a **config** and a directory containing **bits**, create or delete **config fragments** + * Examples: + * `git-branches` resource type for tracking branches in a repo + * `git` resource type for tracking commits in a branch + * `github-status` resource type for emitting build status notifications for commits + * `time` resource type for doing timed job triggers + +* **Resource**: a **resource type** with a user-provided **config**, used together to represent external state. + +## Example Resources + +```yaml +type: git-branches +source: + uri: https://github.com/concourse/concourse +``` + +```yaml +type: git +source: + uri: https://github.com/concourse/concourse + branch: master +``` + +```yaml +type: github-status +source: + repository: concourse/concourse + access_token: abcdef +``` + +```yaml +type: time +source: + interval: 10m +``` + +## Open Questions * enrich metadata? - * https://github.com/concourse/concourse/issues/310 + * https://github.com/concourse/concourse/issues/310 * standardize TLS config? - * https://github.com/concourse/rfcs/issues/9 + * https://github.com/concourse/rfcs/issues/9 * webhooks? - * should these instead be something supported by *Concourse*? -* icons? - * https://github.com/concourse/concourse/issues/788 - * https://github.com/concourse/concourse/pull/3220 - * https://github.com/concourse/concourse/pull/3581 - * can this come from `/info`? would it need `config` passed to it? + * should these instead be something supported by *Concourse*? + * can this come from `/info`? would it need `config` passed to it? * resource-determined triggerability of versions? - * https://github.com/concourse/rfcs/issues/11 - + * https://github.com/concourse/rfcs/issues/11 -# Answered Questions +## Answered Questions -* version filtering is probably best left to `config` +* Version filtering is probably best left to `config`. - -# New Implications +## New Implications many - -# Yet-to-be-organized notes +## Yet-to-be-organized notes Cataloguing ways in which generalized resources can be composed to accomplish different goals: * artifact `check` -> config fragment + artifact `check` - * check from version + * check from version * artifact `check` -> config fragment + artifact `get` - * fetch specific version + * fetch specific version * artifact `put` -> config fragment + artifact `get` - * fetch just-created version + * fetch just-created version * spatial `check` -> config fragment + artifact `check` - * check across all spaces + * check across all spaces * artifact `get` -> config fragment + notification `put` - * update github status + * update github status * trigger `check` -> config fragment -> trigger build if different from last config fragment - * trigger-only resources - * maybe the config fragment could be passed to something to support parameterized triggers? :thinking: - * maybe that could fit nicely with however we approach https://github.com/concourse/concourse/issues/783? :thinking: \ No newline at end of file + * trigger-only resources + * maybe the config fragment could be passed to something to support parameterized triggers? :thinking: + * maybe that could fit nicely with however we approach https://github.com/concourse/concourse/issues/783? :thinking: From 7e78e1ed0d10833f7e6fe26a71b8d906d128c74d Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Thu, 4 Apr 2019 15:06:03 -0400 Subject: [PATCH 05/34] start interface outline, artifacts interpretation Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 69 +++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index 57e58b93..6bac2e6c 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -110,6 +110,75 @@ source: interval: 10m ``` + +## Interface Definition + +### `info`: discover resource type implementation info + +Concourse will first invoke `./info` to discover the commands to run for each resource action. + +### Request + +The `info` script will be given the resource's `config` on `stdin` so that it may interpret any fields necessary to formulate the response payload. + +```go +type InfoRequest struct { + // User-specified configuration. + Config Config `json:"config"` +} +``` + +### Response + +The `info` script must emit the following response on `stdout`: + +```go +type InfoResponse struct { + // The version of the resource interface that this resource type conforms to. + InterfaceVersion string `json:"interface_version"` + + // An optional icon name to show to the user when viewing the resource. + Icon string `json:"icon,omitempty"` + + // Command to run when performing check actions. + Check string `json:"check"` + + // Command to run when performing get actions. + Get string `json:"get"` + + // Command to run when performing put actions. + Put string `json:"put"` +} +``` + +The value of the `icon` field is a short string corresponding to an icon in Concourse's icon set (currently [Material Design Icons](https://materialdesignicons.com)). + +### `check`: monitor a config to discover config fragments + +### `get`: fetch bits for a given config + +### `put`: use bits to perform side-effects corresponding to config fragments + +## Artifact resources with v2 + +Today's v1 resources are effectively "versioned artifact resources", as that is the only way Concourse pipelines support using them. + +With the v2 interface being more general, it makes no mention of versions and is less coupled to a notion of 'versioned artifacts'. In order for today's workflows to transition to v2 resources, we need to define how the general interface is interpreted by Concourse pipelines: + +* Firstly, all **config fragments** are interpreted as **versions** for artifact resources. + +* The `check` action will first be run with a "naked" config, containing only what the user specified. In this situation `check` must return *all* versions discovered in the config, in chronological order. + +* Subsequent calls to `check` will be given a config that has been spliced with the last emitted version. The `check` script must emit any versions that came after the specified version. + * If the specified version is no longer present, the `check` action must emit a `reset` event and then return *all* versions, as if the version was not specified in the first place. + +* The `get` action will always be invoked with a spliced config specifying which version to fetch. + * A `fetched` event must be emitted for all versions that have been fetched into the **bits** directory. Each version will be recorded as an input to the build. + +* The `put` action will be invoked with user-provided configuration. + * A `created` event must be emitted for all versions that have been created by the `put` action. These will be recorded as outputs of the build. + * A `deleted` event must be emitted for all versions that have been deleted by the `put` action. These versions will be marked "deleted" and no longer be available for use in other builds. + ## Open Questions * enrich metadata? From ed3b28ac75504f9da033e1a7e4e597b19aded5cd Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Thu, 4 Apr 2019 15:37:05 -0400 Subject: [PATCH 06/34] fix bare links, + other minor cleanups Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 55 +++++++++++++-------------- 1 file changed, 27 insertions(+), 28 deletions(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index 6bac2e6c..c22c7033 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -7,53 +7,52 @@ Today's resources are closely tied to the 'versioned artifact' use case, so this ## Motivation * Support for deleting versions in `put`: - * https://github.com/concourse/concourse/issues/362 - * https://github.com/concourse/concourse/issues/524 + * [concourse/concourse#362](https://github.com/concourse/concourse/issues/362) + * [concourse/concourse#524](https://github.com/concourse/concourse/issues/524) * Support for creating multiple versions from `put`: - * https://github.com/concourse/concourse/issues/2660 + * [concourse/concourse#2660](https://github.com/concourse/concourse/issues/2660) * Having resource metadata immediately available via check: - * https://github.com/concourse/git-resource/issues/193 - * https://github.com/concourse/concourse/issues/1714 + * [git-resource#193](https://github.com/concourse/git-resource/issues/193) + * [concourse/concourse#1714](https://github.com/concourse/concourse/issues/1714) * Make the `get` after `put` opt-in: - * https://github.com/concourse/concourse/issues/3299 - * https://stackoverflow.com/questions/38964299/why-does-concourse-get-a-resource-after-puting-it - * https://github.com/concourse/registry-image-resource/issues/16 + * [concourse/concourse#3299](https://github.com/concourse/concourse/issues/3299) + * [image-resource#16](https://github.com/concourse/registry-image-resource/issues/16) * Unifying `source` and `params` as just `config` so that resources don't have to care where configuration is being set in pipelines: - * https://github.com/concourse/git-resource/pull/172 - * https://github.com/concourse/bosh-deployment-resource/issues/13 - * https://github.com/concourse/bosh-deployment-resource/issues/6 - * https://github.com/concourse/cf-resource/pull/20 - * https://github.com/concourse/cf-resource/pull/25 - * https://github.com/concourse/git-resource/pull/210 + * [concourse/git-resource#172](https://github.com/concourse/git-resource/pull/172) + * [concourse/bosh-deployment-resource#13](https://github.com/concourse/bosh-deployment-resource/issues/13) + * [concourse/bosh-deployment-resource#6](https://github.com/concourse/bosh-deployment-resource/issues/6) + * [concourse/cf-resource#20](https://github.com/concourse/cf-resource/pull/20) + * [concourse/cf-resource#25](https://github.com/concourse/cf-resource/pull/25) + * [concourse/git-resource#210](https://github.com/concourse/git-resource/pull/210) * Generalize interface to support non-versioned state: - * https://github.com/concourse/concourse/issues/739 + * [concourse/concourse#739](https://github.com/concourse/concourse/issues/739) * Support for trigger-only resources that don't result in fetching anything - they just trigger the job: - * https://github.com/concourse/concourse/issues/3572 - * https://github.com/concourse/concourse/issues/3595 + * [concourse/concourse#3572](https://github.com/concourse/concourse/issues/3572) + * [concourse/concourse#3595](https://github.com/concourse/concourse/issues/3595) * Make resource actions more reentrant so that we no longer receive `unexpected EOF` errors upon reattaching to an in-flight build. - * https://github.com/concourse/concourse/issues/1580 + * [concourse/concourse#1580](https://github.com/concourse/concourse/issues/1580) * Support multi-branch workflows: - * https://github.com/concourse/concourse/issues/1172 + * [concourse/concourse#1172](https://github.com/concourse/concourse/issues/1172) * Begin phasing out `version: every` in by reframing the problem as 'pipeline per commit': - * https://github.com/concourse/concourse/issues/736 + * [concourse/concourse#736](https://github.com/concourse/concourse/issues/736) * Support notifications in a way that doesn't pollute pipeline config and UI: - * https://github.com/concourse/concourse/issues/1052 - * https://github.com/concourse/rfcs/issues/10 + * [concourse/concourse#1052](https://github.com/concourse/concourse/issues/1052) + * [concourse/rfcs#10](https://github.com/concourse/rfcs/issues/10) * Support for showing icons for resources in the web UI: - * https://github.com/concourse/concourse/issues/788 - * https://github.com/concourse/concourse/pull/3220 - * https://github.com/concourse/concourse/pull/3581 + * [concourse/concourse#788](https://github.com/concourse/concourse/issues/788) + * [concourse/concourse#3220](https://github.com/concourse/concourse/pull/3220) + * [concourse/concourse#3581](https://github.com/concourse/concourse/pull/3581) ## Glossary @@ -110,7 +109,6 @@ source: interval: 10m ``` - ## Interface Definition ### `info`: discover resource type implementation info @@ -187,7 +185,6 @@ With the v2 interface being more general, it makes no mention of versions and is * https://github.com/concourse/rfcs/issues/9 * webhooks? * should these instead be something supported by *Concourse*? - * can this come from `/info`? would it need `config` passed to it? * resource-determined triggerability of versions? * https://github.com/concourse/rfcs/issues/11 @@ -197,7 +194,9 @@ With the v2 interface being more general, it makes no mention of versions and is ## New Implications -many +* Notifications +* Spaces +* Triggers ## Yet-to-be-organized notes From 3ca8f962f6fd7abc53d9b5f9e882e52c26d24c58 Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Thu, 4 Apr 2019 15:37:16 -0400 Subject: [PATCH 07/34] clarify reasoning for ./info being relative Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index c22c7033..cd9cfdb1 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -113,7 +113,7 @@ source: ### `info`: discover resource type implementation info -Concourse will first invoke `./info` to discover the commands to run for each resource action. +Concourse will first invoke `./info` to discover the commands to run for each resource action. The path to this script is relative to the image's working directory, so that it isn't coupled to any particular operating system. By not hardcoding an absolute path we can run resource types on platforms which may not support the idea of a "chroot". ### Request From b5e332ebd309c89bdf14c9eaafb558c3d639d3ff Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Thu, 4 Apr 2019 15:37:28 -0400 Subject: [PATCH 08/34] change formatting of 'versioned artifact' section Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 32 +++++++++++++++++---------- 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index cd9cfdb1..ff5501d1 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -153,29 +153,37 @@ The value of the `icon` field is a short string corresponding to an icon in Conc ### `check`: monitor a config to discover config fragments -### `get`: fetch bits for a given config +### `get`: fetch bits for a spliced config ### `put`: use bits to perform side-effects corresponding to config fragments ## Artifact resources with v2 -Today's v1 resources are effectively "versioned artifact resources", as that is the only way Concourse pipelines support using them. +All v1 resources are effectively "versioned artifact resources", as that is the only way Concourse pipelines support using them. -With the v2 interface being more general, it makes no mention of versions and is less coupled to a notion of 'versioned artifacts'. In order for today's workflows to transition to v2 resources, we need to define how the general interface is interpreted by Concourse pipelines: +A v2 resource type can be used as a versioned artifact resource by treating the **config fragments** as **versions** and implementing the following behavior: -* Firstly, all **config fragments** are interpreted as **versions** for artifact resources. +### `check`: discover versions over time -* The `check` action will first be run with a "naked" config, containing only what the user specified. In this situation `check` must return *all* versions discovered in the config, in chronological order. +The `check` action will first be run with a "naked" config, containing only what the user specified. In this situation `check` must return *all* versions discovered in the config, in chronological order. -* Subsequent calls to `check` will be given a config that has been spliced with the last emitted version. The `check` script must emit any versions that came after the specified version. - * If the specified version is no longer present, the `check` action must emit a `reset` event and then return *all* versions, as if the version was not specified in the first place. +Subsequent calls to `check` will be given a config that has been spliced with the last emitted version. The `check` script must emit any versions that came after the specified version. -* The `get` action will always be invoked with a spliced config specifying which version to fetch. - * A `fetched` event must be emitted for all versions that have been fetched into the **bits** directory. Each version will be recorded as an input to the build. +If the specified version is no longer present, the `check` action must emit a `reset` event and then return *all* versions, as if the version was not specified in the first place. -* The `put` action will be invoked with user-provided configuration. - * A `created` event must be emitted for all versions that have been created by the `put` action. These will be recorded as outputs of the build. - * A `deleted` event must be emitted for all versions that have been deleted by the `put` action. These versions will be marked "deleted" and no longer be available for use in other builds. +### `get`: fetch a version of an artifact + +The `get` action will always be invoked with a spliced config specifying which version to fetch. It is given an empty directory in which it should fetch the bits. + +A `fetched` event must be emitted for all versions that have been fetched into the bits directory. Each version will be recorded as an input to the build. + +### `put`: create, update, and/or delete artifact versions + +The `put` action will be invoked with user-provided configuration and arbitrary bits. + +A `created` event must be emitted for all versions that have been created by the `put` action. These will be recorded as outputs of the build. + +A `deleted` event must be emitted for all versions that have been deleted by the `put` action. These versions will be marked "deleted" and no longer be available for use in other builds. ## Open Questions From 0edb683e7b07a1e0cb0893e2f2d407b9f190e7ea Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Thu, 4 Apr 2019 16:22:51 -0400 Subject: [PATCH 09/34] flesh out 'check' interface I decided to stray from the initial v2 doc, which had a 'reset' event type and a 'discovered' event type. Instead 'check' only returns a stream of JSON objects containing config and metadata. Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 70 +++++++++++++++++++++++++-- 1 file changed, 66 insertions(+), 4 deletions(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index ff5501d1..a6b45fe3 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -115,8 +115,6 @@ source: Concourse will first invoke `./info` to discover the commands to run for each resource action. The path to this script is relative to the image's working directory, so that it isn't coupled to any particular operating system. By not hardcoding an absolute path we can run resource types on platforms which may not support the idea of a "chroot". -### Request - The `info` script will be given the resource's `config` on `stdin` so that it may interpret any fields necessary to formulate the response payload. ```go @@ -126,8 +124,6 @@ type InfoRequest struct { } ``` -### Response - The `info` script must emit the following response on `stdout`: ```go @@ -151,8 +147,74 @@ type InfoResponse struct { The value of the `icon` field is a short string corresponding to an icon in Concourse's icon set (currently [Material Design Icons](https://materialdesignicons.com)). +#### Example + +Request sent to `stdin`: + +```json +{ + "config": { + "uri": "https://github.com/concourse/concourse" + } +} +``` + +Response written to `stdout`: + +```json +{ + "interface_version": "2.0", + "icon": "github-circle", + "check": "/opt/resource/check", + "get": "/opt/resource/get", + "put": "/opt/resource/put" +} +``` + ### `check`: monitor a config to discover config fragments +The `check` command specified by `info` will be invoked with the following request piped to `stdin`: + +```go +type CheckRequest struct { + // User-specified configuration. + Config Config `json:"config"` + + // Path to a file into which the action must write its response. + ResponsePath string `json:"response_path"` +} +``` + +The `check` command must write a stream of JSON objects ("events") containing **config fragments** and any associated **metadata** to the specified `response_path`. + +```go +type CheckEvent struct { + Config Config `json:"config"` + Metadata []Metadata `json:"metadata,omitempty"` +} +``` + +#### Example + +Request sent to `stdin`: + +```json +{ + "config": { + "uri": "https://github.com/concourse/concourse", + "branch": "master" + } +} +``` + +Response written to `stdout`: + +```json +{"config":{"ref":"e4be0b367d7bd34580f4842dd09e7b59b6097b25"},"metadata":[{"name":"message","value":"init"}]} +{"config":{"ref":"5a052ba6438d754f73252283c6b6429f2a74dbff"},"metadata":[{"name":"message","value":"add not-very-useful-yet readme"}]} +{"config":{"ref":"2e256c3cb4b077f6fa3c465dd082fa74df8fab0a"},"metadata":[{"name":"message","value":"start fleshing out RFC process"}]} +``` + ### `get`: fetch bits for a spliced config ### `put`: use bits to perform side-effects corresponding to config fragments From f8e7164fc015469488d2f1e80c494d63208d00cc Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Thu, 4 Apr 2019 16:30:46 -0400 Subject: [PATCH 10/34] give 'check' events an event type Since `put` can both create and delete configs, it seems better to keep them all using the same event structure, even if 'check' can only emit one type of event (`discovered`). The alternative to this would be to split deleting out of 'put'. At that point they would each emit a single type of event, and the interface would be less open-ended. There's value in that, but that value has to be weighed against the value of being able to create + delete in one action. It's also not super clear that all resource types would support 'delete'. (But then, not all of them will support 'put', either.) Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 34 ++++++++++++++++++++++++--- 1 file changed, 31 insertions(+), 3 deletions(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index a6b45fe3..69a16b46 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -189,6 +189,7 @@ The `check` command must write a stream of JSON objects ("events") containing ** ```go type CheckEvent struct { + Event string `json:"event"` Config Config `json:"config"` Metadata []Metadata `json:"metadata,omitempty"` } @@ -210,9 +211,36 @@ Request sent to `stdin`: Response written to `stdout`: ```json -{"config":{"ref":"e4be0b367d7bd34580f4842dd09e7b59b6097b25"},"metadata":[{"name":"message","value":"init"}]} -{"config":{"ref":"5a052ba6438d754f73252283c6b6429f2a74dbff"},"metadata":[{"name":"message","value":"add not-very-useful-yet readme"}]} -{"config":{"ref":"2e256c3cb4b077f6fa3c465dd082fa74df8fab0a"},"metadata":[{"name":"message","value":"start fleshing out RFC process"}]} +{ + "event": "discovered", + "config": {"ref": "e4be0b367d7bd34580f4842dd09e7b59b6097b25"}, + "metadata": [ + { + "name": "message", + "value": "init" + } + ] +} +{ + "event": "discovered", + "config": {"ref": "5a052ba6438d754f73252283c6b6429f2a74dbff"}, + "metadata": [ + { + "name": "message", + "value": "add not-very-useful-yet readme" + } + ] +} +{ + "event": "discovered", + "config": {"ref": "2e256c3cb4b077f6fa3c465dd082fa74df8fab0a"}, + "metadata": [ + { + "name": "message", + "value": "start fleshing out RFC process" + } + ] +} ``` ### `get`: fetch bits for a spliced config From e90ed7ff9f3af1f62c34dd1671966b8d995c2704 Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Sun, 7 Apr 2019 13:14:41 -0400 Subject: [PATCH 11/34] make linter happy Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index 69a16b46..fe022ce2 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -277,14 +277,11 @@ A `deleted` event must be emitted for all versions that have been deleted by the ## Open Questions -* enrich metadata? - * https://github.com/concourse/concourse/issues/310 -* standardize TLS config? - * https://github.com/concourse/rfcs/issues/9 +* [enrich metadata?](https://github.com/concourse/concourse/issues/310) +* [standardize TLS config?](https://github.com/concourse/rfcs/issues/9) +* [resource-determined triggerability of versions?](https://github.com/concourse/rfcs/issues/11) * webhooks? * should these instead be something supported by *Concourse*? -* resource-determined triggerability of versions? - * https://github.com/concourse/rfcs/issues/11 ## Answered Questions @@ -313,4 +310,4 @@ Cataloguing ways in which generalized resources can be composed to accomplish di * trigger `check` -> config fragment -> trigger build if different from last config fragment * trigger-only resources * maybe the config fragment could be passed to something to support parameterized triggers? :thinking: - * maybe that could fit nicely with however we approach https://github.com/concourse/concourse/issues/783? :thinking: + * maybe that could fit nicely with however we approach [concourse/concourse#738](https://github.com/concourse/concourse/issues/783)? :thinking: From 836eb6720067523651dde961f94648bc852b62e6 Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Sun, 7 Apr 2019 13:14:53 -0400 Subject: [PATCH 12/34] add 'delete' action, unify action interface this removes 'event types' from the response from each action, and now they all have the same response. event types were needed because 'put' could both create and delete, and we needed a way to tell the difference. so instead, we've split 'delete' out, and now each action has only one way to interpret the events, so the types are no longer needed. this makes the interface a bit less flexible, but it seems worth seeing how far we can go with this constraint for the sake of having a simpler, more focused interface. if we can go the distance with this limitation it feels like stronger validation of the resources concept. along the way I've put the configuration for each command under an 'actions' field in the info response. i'd like to also have it so actions are optional, i.e. you can have a resource that doesn't implement 'delete' or 'put'. this seems like it'll be necessary if we want a common resource interface to be used for things like notifications or triggers where there's nothing to really delete, and even today there are cases where resources are 'read-only' and don't support put at all (e.g. RSS). Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 141 ++++++++++++++++---------- 1 file changed, 86 insertions(+), 55 deletions(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index fe022ce2..06e5b682 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -109,45 +109,77 @@ source: interval: 10m ``` -## Interface Definition +## Interface Types -### `info`: discover resource type implementation info +```go +// Config represents arbitrary user-specified configuration. +type Config map[string]interface{} -Concourse will first invoke `./info` to discover the commands to run for each resource action. The path to this script is relative to the image's working directory, so that it isn't coupled to any particular operating system. By not hardcoding an absolute path we can run resource types on platforms which may not support the idea of a "chroot". +// ConfigFragment represents additional fields that can be spliced into a Config. +type ConfigFragment map[string]interface{} -The `info` script will be given the resource's `config` on `stdin` so that it may interpret any fields necessary to formulate the response payload. +// MetadataField represents a named bit of metadata associated to a ConfigFragment. +type MetadataField struct { + Name string `json:"name"` + Value string `json:"value"` +} -```go +// InfoRequest is the payload written to stdin for the `./info` script. type InfoRequest struct { - // User-specified configuration. - Config Config `json:"config"` + // User-specified configuration. + Config Config `json:"config"` } -``` -The `info` script must emit the following response on `stdout`: - -```go +// InfoResponse is the payload written to stdout from the `./info` script. type InfoResponse struct { - // The version of the resource interface that this resource type conforms to. - InterfaceVersion string `json:"interface_version"` + // The version of the resource interface that this resource type conforms to. + InterfaceVersion string `json:"interface_version"` - // An optional icon name to show to the user when viewing the resource. - Icon string `json:"icon,omitempty"` + // An optional icon name to show to the user when viewing the resource. + Icon string `json:"icon,omitempty"` + // The actions supported by the resource type. + Actions struct { // Command to run when performing check actions. - Check string `json:"check"` + Check string `json:"check,omitempty"` // Command to run when performing get actions. - Get string `json:"get"` + Get string `json:"get,omitempty"` // Command to run when performing put actions. - Put string `json:"put"` + Put string `json:"put,omitempty"` + + // Command to run when performing delete actions. + Delete string `json:"delete,omitempty"` + } `json:"actions"` +} + +// ActionRequest is the payload written to stdin for each action command. +type ActionRequest struct { + // User-specified configuration. + Config Config `json:"config"` + + // Path to a file into which the action must write its response. + ResponsePath string `json:"response_path"` +} + +// ActionResponse is written to the `response_path` by an action for each fragment affected by the action. Multiple respones may be written as a JSON stream. +type ActionResponse struct { + // The fragment. May be used as an identifier, unique within the scope of a Config. + Fragment ConfigFragment `json:"fragment"` + + // Metadata to associate with the fragment. Shown to the user. + Metadata []MetadataField `json:"metadata,omitempty"` } ``` -The value of the `icon` field is a short string corresponding to an icon in Concourse's icon set (currently [Material Design Icons](https://materialdesignicons.com)). +## `info`: discover resource type implementation info + +Prior to running any action, `./info` will be executed with an `InfoRequest` piped to `stdin`. The path to this script will be relative to the image's working directory. + +The resource type must emit a `InfoResponse` to `stdout` in response. This response specifies the resource interface version that the resource conforms to, an optional icon to show in the UI, and the command to run for each supported resource action. -#### Example +### Example `info` request-response Request sent to `stdin`: @@ -165,55 +197,42 @@ Response written to `stdout`: { "interface_version": "2.0", "icon": "github-circle", - "check": "/opt/resource/check", - "get": "/opt/resource/get", - "put": "/opt/resource/put" + "actions": { + "check": "/usr/bin/git-resource check", + "get": "/usr/bin/git-resource get", + "put": "/usr/bin/git-resource put", + "delete": "/usr/bin/git-resource delete" + } } ``` -### `check`: monitor a config to discover config fragments +## Resource Actions -The `check` command specified by `info` will be invoked with the following request piped to `stdin`: +Each action is invoked with a JSON-encoded `ActionRequest` piped to `stdin`. This request contains the **config** and the path to which the response should be written. This path may be relative, and if so it is to be expanded from the current working directory. -```go -type CheckRequest struct { - // User-specified configuration. - Config Config `json:"config"` +All actions will be run in a working directory for the **bits** - either an empty directory to which bits should be written, or a directory containing the bits given to the action. - // Path to a file into which the action must write its response. - ResponsePath string `json:"response_path"` -} -``` - -The `check` command must write a stream of JSON objects ("events") containing **config fragments** and any associated **metadata** to the specified `response_path`. - -```go -type CheckEvent struct { - Event string `json:"event"` - Config Config `json:"config"` - Metadata []Metadata `json:"metadata,omitempty"` -} -``` +All actions respond by performing their side-effect and writing sequential `ActionResponse` JSON objects to the file path specified by `response_path`. How this response is interpreted depends on the action, but typically there should be one response for each external resource affected (`put`, `delete`), discovered (`check`), or fetched (`get`). -#### Example +### Example **action** request/response Request sent to `stdin`: ```json { "config": { - "uri": "https://github.com/concourse/concourse", + "uri": "https://github.com/concourse/rfcs", "branch": "master" - } + }, + "response_path": "../response/response.json" } ``` -Response written to `stdout`: +Response written to `../response/response.json`: ```json { - "event": "discovered", - "config": {"ref": "e4be0b367d7bd34580f4842dd09e7b59b6097b25"}, + "fragment": {"ref": "e4be0b367d7bd34580f4842dd09e7b59b6097b25"}, "metadata": [ { "name": "message", @@ -222,8 +241,7 @@ Response written to `stdout`: ] } { - "event": "discovered", - "config": {"ref": "5a052ba6438d754f73252283c6b6429f2a74dbff"}, + "fragment": {"ref": "5a052ba6438d754f73252283c6b6429f2a74dbff"}, "metadata": [ { "name": "message", @@ -232,8 +250,7 @@ Response written to `stdout`: ] } { - "event": "discovered", - "config": {"ref": "2e256c3cb4b077f6fa3c465dd082fa74df8fab0a"}, + "fragment": {"ref": "2e256c3cb4b077f6fa3c465dd082fa74df8fab0a"}, "metadata": [ { "name": "message", @@ -243,9 +260,23 @@ Response written to `stdout`: } ``` -### `get`: fetch bits for a spliced config +This response would be typical of a `check` that ran against a repo that had three commits. + +### `check`: monitor a config to discover config fragments + +* **bits** is initially empty - the resource may place arbitrary data there to cache state between checks + +### `get`: fetch bits corresponding to a spliced config + +* **bits** is initially empty, and the action must fetch into it + +### `put`: create-or-update config fragments + +* **bits** contains arbitrary user-provided data + +### `delete`: use bits to delete config fragments -### `put`: use bits to perform side-effects corresponding to config fragments +* **bits** contains arbitrary user-provided data ## Artifact resources with v2 From 999a803c1029fe0694f72e7c0bec33df0e49effc Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Mon, 8 Apr 2019 01:16:10 -0400 Subject: [PATCH 13/34] execute default command for image, not ./info we were already relying on image semantics here by expecting it to be in the working directory, so relying on CMD should be no worse. this will allow greater flexibility for resource type authors, allowing for a single-binary entrypoint rather than forcing them to have an 'info' executable in a particular location. Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index 06e5b682..3811dbc2 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -173,13 +173,13 @@ type ActionResponse struct { } ``` -## `info`: discover resource type implementation info +## Resource Info -Prior to running any action, `./info` will be executed with an `InfoRequest` piped to `stdin`. The path to this script will be relative to the image's working directory. +Prior to running any action, Concourse will execute the default command (i.e. [`CMD`](https://docs.docker.com/engine/reference/builder/#cmd)) for the image with an `InfoRequest` piped to `stdin`. -The resource type must emit a `InfoResponse` to `stdout` in response. This response specifies the resource interface version that the resource conforms to, an optional icon to show in the UI, and the command to run for each supported resource action. +The command must write an `InfoResponse` to `stdout` in response. This response specifies the resource interface version that the resource type conforms to, an optional icon to show in the UI, and the command to run for each supported resource action. -### Example `info` request-response +### Example **info** request/response Request sent to `stdin`: From 1e611ee36faa534bd895e52045467a6263a3e7a7 Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Mon, 8 Apr 2019 10:22:51 -0400 Subject: [PATCH 14/34] fix a couple of link titles Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index 3811dbc2..71bf2015 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -14,12 +14,12 @@ Today's resources are closely tied to the 'versioned artifact' use case, so this * [concourse/concourse#2660](https://github.com/concourse/concourse/issues/2660) * Having resource metadata immediately available via check: - * [git-resource#193](https://github.com/concourse/git-resource/issues/193) + * [concourse/git-resource#193](https://github.com/concourse/git-resource/issues/193) * [concourse/concourse#1714](https://github.com/concourse/concourse/issues/1714) * Make the `get` after `put` opt-in: * [concourse/concourse#3299](https://github.com/concourse/concourse/issues/3299) - * [image-resource#16](https://github.com/concourse/registry-image-resource/issues/16) + * [concourse/registry-image-resource#16](https://github.com/concourse/registry-image-resource/issues/16) * Unifying `source` and `params` as just `config` so that resources don't have to care where configuration is being set in pipelines: * [concourse/git-resource#172](https://github.com/concourse/git-resource/pull/172) From aec9032c9ea2ca9155b2953c26ede07a61b255cd Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Mon, 8 Apr 2019 11:25:31 -0400 Subject: [PATCH 15/34] update artifact check semantics, add git example check no longer emits 'reset', it emits the given version first to show it still exists (just as today) Signed-off-by: Alex Suraci --- .../examples/git-v2/Dockerfile | 15 +++ .../examples/git-v2/Gemfile | 4 + .../examples/git-v2/Gemfile.lock | 19 +++ .../examples/git-v2/git-resource.rb | 115 ++++++++++++++++++ 024-generalized-resources/proposal.md | 4 +- 5 files changed, 155 insertions(+), 2 deletions(-) create mode 100644 024-generalized-resources/examples/git-v2/Dockerfile create mode 100644 024-generalized-resources/examples/git-v2/Gemfile create mode 100644 024-generalized-resources/examples/git-v2/Gemfile.lock create mode 100644 024-generalized-resources/examples/git-v2/git-resource.rb diff --git a/024-generalized-resources/examples/git-v2/Dockerfile b/024-generalized-resources/examples/git-v2/Dockerfile new file mode 100644 index 00000000..457d6a4c --- /dev/null +++ b/024-generalized-resources/examples/git-v2/Dockerfile @@ -0,0 +1,15 @@ +FROM alpine + +RUN apk add git ruby ruby-json ruby-bundler build-base ruby-dev cmake openssl-dev + +ENV GEM_HOME="/usr/local/bundle" +ENV PATH $GEM_HOME/bin:$GEM_HOME/gems/bin:$PATH +ADD Gemfile . +RUN bundle install + +ADD git-resource.rb /usr/bin/git-resource +RUN chmod +x /usr/bin/git-resource + +WORKDIR /tmp/request + +CMD ["git-resource", "info"] \ No newline at end of file diff --git a/024-generalized-resources/examples/git-v2/Gemfile b/024-generalized-resources/examples/git-v2/Gemfile new file mode 100644 index 00000000..0259cb5a --- /dev/null +++ b/024-generalized-resources/examples/git-v2/Gemfile @@ -0,0 +1,4 @@ +source :rubygems + +gem 'rugged' +gem 'pry' \ No newline at end of file diff --git a/024-generalized-resources/examples/git-v2/Gemfile.lock b/024-generalized-resources/examples/git-v2/Gemfile.lock new file mode 100644 index 00000000..bf762b7c --- /dev/null +++ b/024-generalized-resources/examples/git-v2/Gemfile.lock @@ -0,0 +1,19 @@ +GEM + remote: http://rubygems.org/ + specs: + coderay (1.1.2) + method_source (0.9.2) + pry (0.12.2) + coderay (~> 1.1.0) + method_source (~> 0.9.0) + rugged (0.28.1) + +PLATFORMS + ruby + +DEPENDENCIES + pry + rugged + +BUNDLED WITH + 1.16.2 diff --git a/024-generalized-resources/examples/git-v2/git-resource.rb b/024-generalized-resources/examples/git-v2/git-resource.rb new file mode 100644 index 00000000..138526b1 --- /dev/null +++ b/024-generalized-resources/examples/git-v2/git-resource.rb @@ -0,0 +1,115 @@ +#!/usr/bin/env ruby + +require "json" +require "rugged" +require "pry" +require "benchmark" + +$request = JSON.parse(STDIN.read, symbolize_names: true) + +def commit_fragment(commit) + JSON.dump({ + config: {ref: commit.oid}, + metadata: [ + {name: "author", value: enc(commit, commit.author[:name])}, + {name: "committer", value: enc(commit, commit.committer[:name])}, + {name: "message", value: enc(commit, commit.message)} + ] + }) +end + +def icon(uri) + case uri + when /github/ + "github-circle" + when /gitlab/ + "gitlab" + when /bitbucket/ + "bitbucket" + else + "git" + end +end + +def bench(label, &blk) + time = Benchmark.realtime(&blk) + $stderr.puts "#{label}: #{time}s" +end + +def enc(commit, str) + str = str.force_encoding("ISO-8859-1") unless commit.header_field("Encoding") + str.encode("UTF-8") +end + +case ARGV[0] +when "info" + puts JSON.dump({ + "interface_version": "2.0", + "icon": icon($request[:config][:uri]), + "actions": { + "check": "git-resource check", + "get": "git-resource get", + "put": "git-resource put" + # delete is unsupported + } + }) + +when "check" + repo = + if File.exists?("HEAD") + Rugged::Repository.new(".").tap do |r| + r.fetch("origin") + end + else + Rugged::Repository.clone_at( + $request[:config][:uri], + ".", + checkout_branch: $request[:config][:branch], + bare: true, + progress: lambda { |t| $stderr.print t }) + end + + walker = Rugged::Walker.new(repo) + walker.sorting(Rugged::SORT_TOPO|Rugged::SORT_REVERSE) + walker.simplify_first_parent + walker.push(repo.head.target) + + response = File.new $request[:response_path], 'w' + total_commits = 0 + + from = $request[:config][:ref] + if from && repo.include?(from) + commit = repo.lookup(from) + walker.hide(commit) + + response.puts commit_fragment(commit) + total_commits += 1 + end + + bench("walk") do + walker.walk do |c| + response.puts commit_fragment(c) + total_commits += 1 + end + end + + $stderr.puts "commits: #{total_commits}" + + response.close + +when "get" + repo = + Rugged::Repository.clone_at( + $request[:config][:uri], + ".", + checkout_branch: $request[:config][:branch]) + + repo.checkout($request[:config][:ref]) + + response = File.new $request[:response_path], 'w' + response.puts commit_fragment(repo.head.target) + response.close + +when "put" + puts "putting" +end \ No newline at end of file diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index 71bf2015..6750d1fa 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -288,9 +288,9 @@ A v2 resource type can be used as a versioned artifact resource by treating the The `check` action will first be run with a "naked" config, containing only what the user specified. In this situation `check` must return *all* versions discovered in the config, in chronological order. -Subsequent calls to `check` will be given a config that has been spliced with the last emitted version. The `check` script must emit any versions that came after the specified version. +Subsequent calls to `check` will be given a config that has been spliced with the last emitted version. The `check` script must emit the given version if it still exists, followed by any versions that came after it. -If the specified version is no longer present, the `check` action must emit a `reset` event and then return *all* versions, as if the version was not specified in the first place. +If the specified version is no longer present, the `check` action must return *all* versions, as if the version was not specified in the first place. Concourse will detect this scenario by noticing that the first event emitted does not match the requested version. ### `get`: fetch a version of an artifact From 0b42c119424e42fe57bac42566d822cf668840df Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Mon, 8 Apr 2019 12:25:51 -0400 Subject: [PATCH 16/34] bring artifact resources description up to date Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 40 ++++++++++----------------- 1 file changed, 15 insertions(+), 25 deletions(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index 6750d1fa..7bc2d792 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -262,49 +262,39 @@ Response written to `../response/response.json`: This response would be typical of a `check` that ran against a repo that had three commits. -### `check`: monitor a config to discover config fragments - -* **bits** is initially empty - the resource may place arbitrary data there to cache state between checks - -### `get`: fetch bits corresponding to a spliced config - -* **bits** is initially empty, and the action must fetch into it - -### `put`: create-or-update config fragments - -* **bits** contains arbitrary user-provided data - -### `delete`: use bits to delete config fragments - -* **bits** contains arbitrary user-provided data - ## Artifact resources with v2 All v1 resources are effectively "versioned artifact resources", as that is the only way Concourse pipelines support using them. A v2 resource type can be used as a versioned artifact resource by treating the **config fragments** as **versions** and implementing the following behavior: -### `check`: discover versions over time +### `check`: discover versions in order + +The `check` action will first be run with a "naked" config, containing only what the user specified. In this situation `check` must emit an `ActionResponse` for all versions discovered in the config, in chronological order. -The `check` action will first be run with a "naked" config, containing only what the user specified. In this situation `check` must return *all* versions discovered in the config, in chronological order. +Subsequent calls to `check` will be given a config that has been spliced with the last emitted version config fragment. The `check` script must an `ActionResponse` for the given version if it still exists, followed by a response for any versions that came after it. -Subsequent calls to `check` will be given a config that has been spliced with the last emitted version. The `check` script must emit the given version if it still exists, followed by any versions that came after it. +If the specified version is no longer present, the `check` action must go back to returning all versions, as if the version was not specified in the first place. Concourse will detect this scenario by noticing that the first `ActionResponse` emitted does not match the requested version. All versions that existed before that were emitted will be automatically marked "deleted". -If the specified version is no longer present, the `check` action must return *all* versions, as if the version was not specified in the first place. Concourse will detect this scenario by noticing that the first event emitted does not match the requested version. +The `check` action can use the **bits** directory to cache state between runs of the `check` on that worker. On the first run, the directory will be empty. ### `get`: fetch a version of an artifact -The `get` action will always be invoked with a spliced config specifying which version to fetch. It is given an empty directory in which it should fetch the bits. +The `get` action will always be invoked with a spliced config specifying which version to fetch. It is given an empty **bits** directory in which to fetch the data. -A `fetched` event must be emitted for all versions that have been fetched into the bits directory. Each version will be recorded as an input to the build. +An `ActionResponse` must be emitted for all versions that have been fetched into the bits directory. Each version will be recorded as an input to the build. -### `put`: create, update, and/or delete artifact versions +### `put`: idempotently create artifact versions The `put` action will be invoked with user-provided configuration and arbitrary bits. -A `created` event must be emitted for all versions that have been created by the `put` action. These will be recorded as outputs of the build. +An `ActionResponse` must be emitted for all versions that have been created/updated. Each version will be recorded as an output of the build. + +### `delete`: idempotently destroy artifact versions + +The `delete` action will be invoked with user-provided configuration and arbitrary bits. -A `deleted` event must be emitted for all versions that have been deleted by the `put` action. These versions will be marked "deleted" and no longer be available for use in other builds. +An `ActionResponse` must be emitted for all versions that have been destroyed. These versions will be marked "deleted" and no longer be available for use in other builds. ## Open Questions From 80f3ce1f8920a2a528fad946cbadfd4f94a67cfd Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Mon, 8 Apr 2019 12:54:20 -0400 Subject: [PATCH 17/34] add proposals for notifications/spaces/triggers these all go hand-in-hand so they have the same RFC number and are in the same PR, but they each deserve their own proposals Signed-off-by: Alex Suraci --- .../examples/git-v2/Dockerfile | 0 .../examples/git-v2/Gemfile | 0 .../examples/git-v2/Gemfile.lock | 0 .../examples/git-v2/git-resource.rb | 0 024-artifact-resources/proposal.md | 53 +++++++++++ 024-generalized-resources/proposal.md | 91 +++---------------- 024-notification-resources/proposal.md | 36 ++++++++ 024-spatial-resources/proposal.md | 38 ++++++++ 024-trigger-resources/proposal.md | 36 ++++++++ 9 files changed, 175 insertions(+), 79 deletions(-) rename {024-generalized-resources => 024-artifact-resources}/examples/git-v2/Dockerfile (100%) rename {024-generalized-resources => 024-artifact-resources}/examples/git-v2/Gemfile (100%) rename {024-generalized-resources => 024-artifact-resources}/examples/git-v2/Gemfile.lock (100%) rename {024-generalized-resources => 024-artifact-resources}/examples/git-v2/git-resource.rb (100%) create mode 100644 024-artifact-resources/proposal.md create mode 100644 024-notification-resources/proposal.md create mode 100644 024-spatial-resources/proposal.md create mode 100644 024-trigger-resources/proposal.md diff --git a/024-generalized-resources/examples/git-v2/Dockerfile b/024-artifact-resources/examples/git-v2/Dockerfile similarity index 100% rename from 024-generalized-resources/examples/git-v2/Dockerfile rename to 024-artifact-resources/examples/git-v2/Dockerfile diff --git a/024-generalized-resources/examples/git-v2/Gemfile b/024-artifact-resources/examples/git-v2/Gemfile similarity index 100% rename from 024-generalized-resources/examples/git-v2/Gemfile rename to 024-artifact-resources/examples/git-v2/Gemfile diff --git a/024-generalized-resources/examples/git-v2/Gemfile.lock b/024-artifact-resources/examples/git-v2/Gemfile.lock similarity index 100% rename from 024-generalized-resources/examples/git-v2/Gemfile.lock rename to 024-artifact-resources/examples/git-v2/Gemfile.lock diff --git a/024-generalized-resources/examples/git-v2/git-resource.rb b/024-artifact-resources/examples/git-v2/git-resource.rb similarity index 100% rename from 024-generalized-resources/examples/git-v2/git-resource.rb rename to 024-artifact-resources/examples/git-v2/git-resource.rb diff --git a/024-artifact-resources/proposal.md b/024-artifact-resources/proposal.md new file mode 100644 index 00000000..c80ca5eb --- /dev/null +++ b/024-artifact-resources/proposal.md @@ -0,0 +1,53 @@ +# Artifact Resources + +This proposal uses the [generalized resource interface](../024-generalized-resources/proposal.md) to show how the interface would be implemented and interpreted by Concourse to support "artifact resources", today's primary use case for v1 resources. + +## Motivation + +* Support for creating multiple versions from `put`: [concourse/concourse#2660](https://github.com/concourse/concourse/issues/2660) + +* Support for deleting versions: [concourse/concourse#362](https://github.com/concourse/concourse/issues/362), [concourse/concourse#524](https://github.com/concourse/concourse/issues/524) + +* Make the `get` after `put` opt-in: [concourse/concourse#3299](https://github.com/concourse/concourse/issues/3299), [concourse/registry-image-resource#16](https://github.com/concourse/registry-image-resource/issues/16) + +## Examples + +* A basic implementation of the `git` resource can be found in [`git-v2`](examples/git-v2). + +## Proposal + +A v2 resource type can be used as a versioned artifact resource by treating the **config fragments** as **versions** and emitting them in chronological order from `check`. This way the resource type is used to model change in an external resource over time. + +### `check`: discover versions in order + +The `check` action will first be run with a "naked" config, containing only what the user specified. In this situation `check` must emit an `ActionResponse` for all versions discovered in the config, in chronological order. + +Subsequent calls to `check` will be given a config that has been spliced with the last emitted version config fragment. The `check` script must an `ActionResponse` for the given version if it still exists, followed by a response for any versions that came after it. + +If the specified version is no longer present, the `check` action must go back to returning all versions, as if the version was not specified in the first place. Concourse will detect this scenario by noticing that the first `ActionResponse` emitted does not match the requested version. All versions that existed before that were emitted will be automatically marked "deleted". + +The `check` action can use the **bits** directory to cache state between runs of the `check` on that worker. On the first run, the directory will be empty. + +### `get`: fetch a version of an artifact + +The `get` action will always be invoked with a spliced config specifying which version to fetch. It is given an empty **bits** directory in which to fetch the data. + +An `ActionResponse` must be emitted for all versions that have been fetched into the bits directory. Each version will be recorded as an input to the build. + +### `put`: idempotently create artifact versions + +The `put` action will be invoked with user-provided configuration and arbitrary bits. + +An `ActionResponse` must be emitted for all versions that have been created/updated. Each version will be recorded as an output of the build. + +### `delete`: idempotently destroy artifact versions + +The `delete` action will be invoked with user-provided configuration and arbitrary bits. + +An `ActionResponse` must be emitted for all versions that have been destroyed. These versions will be marked "deleted" and no longer be available for use in other builds. + +## Open Questions + +* [resource-determined triggerability of versions?](https://github.com/concourse/rfcs/issues/11) +* webhooks? + * should these instead be something supported by *Concourse*? diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index 7bc2d792..191bb582 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -6,53 +6,23 @@ Today's resources are closely tied to the 'versioned artifact' use case, so this ## Motivation -* Support for deleting versions in `put`: - * [concourse/concourse#362](https://github.com/concourse/concourse/issues/362) - * [concourse/concourse#524](https://github.com/concourse/concourse/issues/524) +* Support for creating multiple versions from `put`: [concourse/concourse#2660](https://github.com/concourse/concourse/issues/2660) -* Support for creating multiple versions from `put`: - * [concourse/concourse#2660](https://github.com/concourse/concourse/issues/2660) +* Support for deleting versions: [concourse/concourse#362](https://github.com/concourse/concourse/issues/362), [concourse/concourse#524](https://github.com/concourse/concourse/issues/524) -* Having resource metadata immediately available via check: - * [concourse/git-resource#193](https://github.com/concourse/git-resource/issues/193) - * [concourse/concourse#1714](https://github.com/concourse/concourse/issues/1714) +* Having resource metadata immediately available via check: [concourse/git-resource#193](https://github.com/concourse/git-resource/issues/193), [concourse/concourse#1714](https://github.com/concourse/concourse/issues/1714) -* Make the `get` after `put` opt-in: - * [concourse/concourse#3299](https://github.com/concourse/concourse/issues/3299) - * [concourse/registry-image-resource#16](https://github.com/concourse/registry-image-resource/issues/16) +* Unifying `source` and `params` as just `config` so that resources don't have to care where configuration is being set in pipelines: [concourse/git-resource#172](https://github.com/concourse/git-resource/pull/172), [concourse/bosh-deployment-resource#13](https://github.com/concourse/bosh-deployment-resource/issues/13), [concourse/bosh-deployment-resource#6](https://github.com/concourse/bosh-deployment-resource/issues/6), [concourse/cf-resource#20](https://github.com/concourse/cf-resource/pull/20), [concourse/cf-resource#25](https://github.com/concourse/cf-resource/pull/25), [concourse/git-resource#210](https://github.com/concourse/git-resource/pull/210) -* Unifying `source` and `params` as just `config` so that resources don't have to care where configuration is being set in pipelines: - * [concourse/git-resource#172](https://github.com/concourse/git-resource/pull/172) - * [concourse/bosh-deployment-resource#13](https://github.com/concourse/bosh-deployment-resource/issues/13) - * [concourse/bosh-deployment-resource#6](https://github.com/concourse/bosh-deployment-resource/issues/6) - * [concourse/cf-resource#20](https://github.com/concourse/cf-resource/pull/20) - * [concourse/cf-resource#25](https://github.com/concourse/cf-resource/pull/25) - * [concourse/git-resource#210](https://github.com/concourse/git-resource/pull/210) +* Make resource actions more reentrant so that we no longer receive `unexpected EOF` errors upon reattaching to an in-flight build: [concourse/concourse#1580](https://github.com/concourse/concourse/issues/1580) -* Generalize interface to support non-versioned state: - * [concourse/concourse#739](https://github.com/concourse/concourse/issues/739) +* Support for showing icons for resources in the web UI: [concourse/concourse#788](https://github.com/concourse/concourse/issues/788), [concourse/concourse#3220](https://github.com/concourse/concourse/pull/3220), [concourse/concourse#3581](https://github.com/concourse/concourse/pull/3581) -* Support for trigger-only resources that don't result in fetching anything - they just trigger the job: - * [concourse/concourse#3572](https://github.com/concourse/concourse/issues/3572) - * [concourse/concourse#3595](https://github.com/concourse/concourse/issues/3595) +* Support [trigger resources](../024-trigger-resources/proposal.md). -* Make resource actions more reentrant so that we no longer receive `unexpected EOF` errors upon reattaching to an in-flight build. - * [concourse/concourse#1580](https://github.com/concourse/concourse/issues/1580) +* Support [spatial resources](../024-spatial-resources/proposal.md). -* Support multi-branch workflows: - * [concourse/concourse#1172](https://github.com/concourse/concourse/issues/1172) - -* Begin phasing out `version: every` in by reframing the problem as 'pipeline per commit': - * [concourse/concourse#736](https://github.com/concourse/concourse/issues/736) - -* Support notifications in a way that doesn't pollute pipeline config and UI: - * [concourse/concourse#1052](https://github.com/concourse/concourse/issues/1052) - * [concourse/rfcs#10](https://github.com/concourse/rfcs/issues/10) - -* Support for showing icons for resources in the web UI: - * [concourse/concourse#788](https://github.com/concourse/concourse/issues/788) - * [concourse/concourse#3220](https://github.com/concourse/concourse/pull/3220) - * [concourse/concourse#3581](https://github.com/concourse/concourse/pull/3581) +* Support [notification resources](../024-notification-resources/proposal.md). ## Glossary @@ -262,47 +232,10 @@ Response written to `../response/response.json`: This response would be typical of a `check` that ran against a repo that had three commits. -## Artifact resources with v2 - -All v1 resources are effectively "versioned artifact resources", as that is the only way Concourse pipelines support using them. - -A v2 resource type can be used as a versioned artifact resource by treating the **config fragments** as **versions** and implementing the following behavior: - -### `check`: discover versions in order - -The `check` action will first be run with a "naked" config, containing only what the user specified. In this situation `check` must emit an `ActionResponse` for all versions discovered in the config, in chronological order. - -Subsequent calls to `check` will be given a config that has been spliced with the last emitted version config fragment. The `check` script must an `ActionResponse` for the given version if it still exists, followed by a response for any versions that came after it. - -If the specified version is no longer present, the `check` action must go back to returning all versions, as if the version was not specified in the first place. Concourse will detect this scenario by noticing that the first `ActionResponse` emitted does not match the requested version. All versions that existed before that were emitted will be automatically marked "deleted". - -The `check` action can use the **bits** directory to cache state between runs of the `check` on that worker. On the first run, the directory will be empty. - -### `get`: fetch a version of an artifact - -The `get` action will always be invoked with a spliced config specifying which version to fetch. It is given an empty **bits** directory in which to fetch the data. - -An `ActionResponse` must be emitted for all versions that have been fetched into the bits directory. Each version will be recorded as an input to the build. - -### `put`: idempotently create artifact versions - -The `put` action will be invoked with user-provided configuration and arbitrary bits. - -An `ActionResponse` must be emitted for all versions that have been created/updated. Each version will be recorded as an output of the build. - -### `delete`: idempotently destroy artifact versions - -The `delete` action will be invoked with user-provided configuration and arbitrary bits. - -An `ActionResponse` must be emitted for all versions that have been destroyed. These versions will be marked "deleted" and no longer be available for use in other builds. - ## Open Questions * [enrich metadata?](https://github.com/concourse/concourse/issues/310) * [standardize TLS config?](https://github.com/concourse/rfcs/issues/9) -* [resource-determined triggerability of versions?](https://github.com/concourse/rfcs/issues/11) -* webhooks? - * should these instead be something supported by *Concourse*? ## Answered Questions @@ -310,9 +243,9 @@ An `ActionResponse` must be emitted for all versions that have been destroyed. T ## New Implications -* Notifications -* Spaces -* Triggers +* Support [trigger resources](../024-trigger-resources/proposal.md). +* Support [spatial resources](../024-spatial-resources/proposal.md). +* Support [notification resources](../024-notification-resources/proposal.md). ## Yet-to-be-organized notes diff --git a/024-notification-resources/proposal.md b/024-notification-resources/proposal.md new file mode 100644 index 00000000..8ba4d642 --- /dev/null +++ b/024-notification-resources/proposal.md @@ -0,0 +1,36 @@ +# Notification Resources + +This proposal uses the [generalized resource interface](../024-generalized-resources/proposal.md) to show how the interface would be implemented and interpreted by Concourse to support "notification resources", which are composed with another resource to emit notifications for build status. + +## Motivation + +* Support notifications in a way that doesn't pollute pipeline config and UI: [concourse/concourse#1052](https://github.com/concourse/concourse/issues/1052), [concourse/rfcs#10](https://github.com/concourse/rfcs/issues/10) + +## Proposal + +> Describe your proposal. +> +> Things that can help: clearly defining terms, providing example content, +> pseudocode, etc. +> +> Feel free to mention key implementation concerns. + +## Open Questions + +> Raise any concerns here for things you aren't sure about yet. + +## Answered Questions + +> If there were any major concerns that have already (or eventually, through +> the RFC process) reached consensus, it can still help to include them along +> with their resolution, if it's otherwise unclear. +> +> This can be especially useful for RFCs that have taken a long time and there +> were some subtle yet important details to get right. +> +> This may very well be empty if the proposal is simple enough. + +## New Implications + +> What is the impact of this change, outside of the change itself? How might it +> change peoples' workflows today, good or bad? diff --git a/024-spatial-resources/proposal.md b/024-spatial-resources/proposal.md new file mode 100644 index 00000000..d167bf29 --- /dev/null +++ b/024-spatial-resources/proposal.md @@ -0,0 +1,38 @@ +# Spatial Resources + +This proposal uses the [generalized resource interface](../024-generalized-resources/proposal.md) to show how the interface would be implemented and interpreted by Concourse to support "spatial resources", which model change over *space*, not time (as with [artifact resources](../024-artifact-resources/proposal.md)). + +## Motivation + +* Support multi-branch workflows: [concourse/concourse#1172](https://github.com/concourse/concourse/issues/1172) + +* Begin phasing out `version: every` in by reframing the problem as 'pipeline per commit': [concourse/concourse#736](https://github.com/concourse/concourse/issues/736) + +## Proposal + +> Describe your proposal. +> +> Things that can help: clearly defining terms, providing example content, +> pseudocode, etc. +> +> Feel free to mention key implementation concerns. + +## Open Questions + +> Raise any concerns here for things you aren't sure about yet. + +## Answered Questions + +> If there were any major concerns that have already (or eventually, through +> the RFC process) reached consensus, it can still help to include them along +> with their resolution, if it's otherwise unclear. +> +> This can be especially useful for RFCs that have taken a long time and there +> were some subtle yet important details to get right. +> +> This may very well be empty if the proposal is simple enough. + +## New Implications + +> What is the impact of this change, outside of the change itself? How might it +> change peoples' workflows today, good or bad? diff --git a/024-trigger-resources/proposal.md b/024-trigger-resources/proposal.md new file mode 100644 index 00000000..c46657d3 --- /dev/null +++ b/024-trigger-resources/proposal.md @@ -0,0 +1,36 @@ +# Trigger Resources + +This proposal uses the [generalized resource interface](../024-generalized-resources/proposal.md) to show how the interface would be implemented and interpreted by Concourse to support "trigger resources", are tied to a job and perform `check`s to determine whether the job should trigger. + +## Motivation + +* Support for trigger-only resources that don't result in fetching anything - they just trigger the job: [concourse/concourse#3572](https://github.com/concourse/concourse/issues/3572), [concourse/concourse#3595](https://github.com/concourse/concourse/issues/3595) + +## Proposal + +> Describe your proposal. +> +> Things that can help: clearly defining terms, providing example content, +> pseudocode, etc. +> +> Feel free to mention key implementation concerns. + +## Open Questions + +> Raise any concerns here for things you aren't sure about yet. + +## Answered Questions + +> If there were any major concerns that have already (or eventually, through +> the RFC process) reached consensus, it can still help to include them along +> with their resolution, if it's otherwise unclear. +> +> This can be especially useful for RFCs that have taken a long time and there +> were some subtle yet important details to get right. +> +> This may very well be empty if the proposal is simple enough. + +## New Implications + +> What is the impact of this change, outside of the change itself? How might it +> change peoples' workflows today, good or bad? From 13035108fe0e9d7ab7198d75d1141aad9c66a834 Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Mon, 8 Apr 2019 14:46:31 -0400 Subject: [PATCH 18/34] give overview of other workflows and link to them Signed-off-by: Alex Suraci --- 024-artifact-resources/proposal.md | 4 ++ 024-generalized-resources/proposal.md | 71 +++++++++++++++++---------- 024-trigger-resources/proposal.md | 3 +- 3 files changed, 50 insertions(+), 28 deletions(-) diff --git a/024-artifact-resources/proposal.md b/024-artifact-resources/proposal.md index c80ca5eb..b9625772 100644 --- a/024-artifact-resources/proposal.md +++ b/024-artifact-resources/proposal.md @@ -51,3 +51,7 @@ An `ActionResponse` must be emitted for all versions that have been destroyed. T * [resource-determined triggerability of versions?](https://github.com/concourse/rfcs/issues/11) * webhooks? * should these instead be something supported by *Concourse*? + +## Answered Questions + +* Version filtering is probably best left to `config`. \ No newline at end of file diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index 191bb582..eace5c57 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -232,36 +232,53 @@ Response written to `../response/response.json`: This response would be typical of a `check` that ran against a repo that had three commits. -## Open Questions +## Interpreting the Resource Interface -* [enrich metadata?](https://github.com/concourse/concourse/issues/310) -* [standardize TLS config?](https://github.com/concourse/rfcs/issues/9) +The resource interface itself is now a general way of expressing interactions with external state. It is no longer restricted to versioning of artifacts. -## Answered Questions +Concourse will now codify things like "versioned artifacts" and "spatial resources" as interpretations of this general interface, composing resource types with one another via **config fragments**. -* Version filtering is probably best left to `config`. +By leaving the interface general, resource authors don't know how their resource type will be used. This gives Concourse flexibility in defining new workflows without requiring resource authors to implement these new workflows themselves, and allows a resource type to be used for multiple use cases. For example, notifications and triggers are somewhat complementary and may both be supported by a resource type that implements the full interface. -## New Implications +These interpretations are outlined in the following proposals: -* Support [trigger resources](../024-trigger-resources/proposal.md). -* Support [spatial resources](../024-spatial-resources/proposal.md). -* Support [notification resources](../024-notification-resources/proposal.md). +### [Artifact resources](../024-artifact-resources/proposal.md) + +* `check`: return versions in order +* `get`: fetch a version of the resource +* `put`: push versions of a resource +* `delete`: delete versions of a resource + +Examples: `git` + +### [Spatial resources](../024-spatial-resources/proposal.md) + +* `check`: return a fragment for each space, no order +* `get`: fetch whatever metadata is useful for a given space +* `put`: create or update spaces +* `delete`: delete spaces + +Examples: `git-branch`, `github-pr` -## Yet-to-be-organized notes - -Cataloguing ways in which generalized resources can be composed to accomplish different goals: - -* artifact `check` -> config fragment + artifact `check` - * check from version -* artifact `check` -> config fragment + artifact `get` - * fetch specific version -* artifact `put` -> config fragment + artifact `get` - * fetch just-created version -* spatial `check` -> config fragment + artifact `check` - * check across all spaces -* artifact `get` -> config fragment + notification `put` - * update github status -* trigger `check` -> config fragment -> trigger build if different from last config fragment - * trigger-only resources - * maybe the config fragment could be passed to something to support parameterized triggers? :thinking: - * maybe that could fit nicely with however we approach [concourse/concourse#738](https://github.com/concourse/concourse/issues/783)? :thinking: +### [Notification resources](../024-notification-resources/proposal.md) + +* `check`: not used +* `get`: fetch bits pertaining to the notification +* `put`: emit a notification +* `delete`: clear github status? + +Examples: `github-status`, `slack` + +### [Trigger resources](../024-trigger-resources/proposal.md) + +* `check`: check against last fragment used for job +* `get`: fetch bits pertaining to the trigger +* `put`: manual trigger? +* `delete`: not useful + +Examples: `time` + +## Open Questions + +* [enrich metadata?](https://github.com/concourse/concourse/issues/310) +* [standardize TLS config?](https://github.com/concourse/rfcs/issues/9) \ No newline at end of file diff --git a/024-trigger-resources/proposal.md b/024-trigger-resources/proposal.md index c46657d3..c06869f7 100644 --- a/024-trigger-resources/proposal.md +++ b/024-trigger-resources/proposal.md @@ -17,7 +17,8 @@ This proposal uses the [generalized resource interface](../024-generalized-resou ## Open Questions -> Raise any concerns here for things you aren't sure about yet. +* could the config fragment be passed to something to support parameterized triggers? :thinking: +* could this be related to how we approach [concourse/concourse#738](https://github.com/concourse/concourse/issues/783)? :thinking: ## Answered Questions From 4b3c95d09de0b60cf46cd49cda766bf51e2df3af Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Mon, 8 Apr 2019 15:20:28 -0400 Subject: [PATCH 19/34] bring TLS configuration into the interface this will allow us to have a common mechanism for configuring TLS across all resource types. having it be a formal part of the interface feels sensible as all resources are meant to interact with external state - and in doing so, TLS will be a commonplace. a later RFC (or perhaps another proposal within this one) will demonstrate how users configure TLS - it may be something like the pipeline-level, or another field alongside 'source' on the resource definitions. Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index eace5c57..cf7a383d 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -18,6 +18,8 @@ Today's resources are closely tied to the 'versioned artifact' use case, so this * Support for showing icons for resources in the web UI: [concourse/concourse#788](https://github.com/concourse/concourse/issues/788), [concourse/concourse#3220](https://github.com/concourse/concourse/pull/3220), [concourse/concourse#3581](https://github.com/concourse/concourse/pull/3581) +* Standardize TLS configuration so every resource doesn't implement their own way: [concourse/rfcs#9](https://github.com/concourse/rfcs/issues/9) + * Support [trigger resources](../024-trigger-resources/proposal.md). * Support [spatial resources](../024-spatial-resources/proposal.md). @@ -94,6 +96,15 @@ type MetadataField struct { Value string `json:"value"` } +// TLSConfig captures common configuration for communicating with servers over TLS. +type TLSConfig struct { + // An array of CA certificates to trust. + CAs []string `json:"ca_certs,omitempty"` + + // Disable TLS, effectively making communication over TLS insecure. + InsecureSkipVerify bool `json:"insecure_skip_verify,omitempty"` +} + // InfoRequest is the payload written to stdin for the `./info` script. type InfoRequest struct { // User-specified configuration. @@ -129,6 +140,9 @@ type ActionRequest struct { // User-specified configuration. Config Config `json:"config"` + // Configuration for handling TLS. + TLS TLSConfig `json:"tls,omitempty"` + // Path to a file into which the action must write its response. ResponsePath string `json:"response_path"` } @@ -280,5 +294,4 @@ Examples: `time` ## Open Questions -* [enrich metadata?](https://github.com/concourse/concourse/issues/310) -* [standardize TLS config?](https://github.com/concourse/rfcs/issues/9) \ No newline at end of file +* [enrich metadata?](https://github.com/concourse/concourse/issues/310) \ No newline at end of file From d6eb993734ad1dc5dddfa2585a40cff1f94349b5 Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Mon, 8 Apr 2019 15:23:12 -0400 Subject: [PATCH 20/34] leave rich metadata out-of-scope this is not well-defined yet and this is easy enough to do as a later iteration of the resource interface, now that it's versioned. Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index cf7a383d..8d11a8b1 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -292,6 +292,6 @@ Examples: `github-status`, `slack` Examples: `time` -## Open Questions +## Out of scope -* [enrich metadata?](https://github.com/concourse/concourse/issues/310) \ No newline at end of file +* [Richer metadata](https://github.com/concourse/concourse/issues/310) - this hasn't gained much traction and probably needs more investigation before it can be incorporated. This should be easy enough to add as a later RFC. \ No newline at end of file From 0a805ec8425344ff1e5202a707b9c42d706c4388 Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Mon, 13 May 2019 15:00:05 -0400 Subject: [PATCH 21/34] add 'previous discussions' section Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index 8d11a8b1..4cf3c9d0 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -4,6 +4,12 @@ This proposal adapts today's 'resource' interface into a more general interface Today's resources are closely tied to the 'versioned artifact' use case, so this proposal will also show how the new interface is "interpreted" in order to support this use case. +## Previous Discussions + +* [RFC #1](https://github.com/concourse/rfcs/pull/1), now defunct, is similar to this proposal but had the "spaces" concept integrated into it. + * **Recommended reading**: [this comment](https://github.com/concourse/rfcs/pull/1#issuecomment-477749314) outlines the thought process that led to this RFC. +* [concourse/concourse#534](https://github.com/concourse/concourse/issues/534) was the first 'new resource interface' proposal which pre-dated the RFC process. + ## Motivation * Support for creating multiple versions from `put`: [concourse/concourse#2660](https://github.com/concourse/concourse/issues/2660) From 2cd1d64e27d8c211a39dc62732fcd1def153a6c6 Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Tue, 14 May 2019 12:26:12 -0400 Subject: [PATCH 22/34] namespace icons fixes concourse/concourse#3789 Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index 4cf3c9d0..896501ec 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -123,6 +123,9 @@ type InfoResponse struct { InterfaceVersion string `json:"interface_version"` // An optional icon name to show to the user when viewing the resource. + // + // Icons must be namespaced by in order to explicitly reference an icon set + // supported by Concourse, e.g. 'mdi:' for Material Design Icons. Icon string `json:"icon,omitempty"` // The actions supported by the resource type. @@ -186,7 +189,7 @@ Response written to `stdout`: ```json { "interface_version": "2.0", - "icon": "github-circle", + "icon": "mdi:github-circle", "actions": { "check": "/usr/bin/git-resource check", "get": "/usr/bin/git-resource get", From b2b5b1c5e914d82234a63c6d0eb679b419280443 Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Wed, 15 May 2019 12:15:31 -0400 Subject: [PATCH 23/34] clearer wording Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index 896501ec..18a07285 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -6,7 +6,7 @@ Today's resources are closely tied to the 'versioned artifact' use case, so this ## Previous Discussions -* [RFC #1](https://github.com/concourse/rfcs/pull/1), now defunct, is similar to this proposal but had the "spaces" concept integrated into it. +* [RFC #1](https://github.com/concourse/rfcs/pull/1), now defunct, is similar to this proposal but had a concept of "spaces" baked into the interface. This concept has been decoupled from the resource interface and will be re-introduced in a later RFC that is compatible with v1 and v2 resources. * **Recommended reading**: [this comment](https://github.com/concourse/rfcs/pull/1#issuecomment-477749314) outlines the thought process that led to this RFC. * [concourse/concourse#534](https://github.com/concourse/concourse/issues/534) was the first 'new resource interface' proposal which pre-dated the RFC process. @@ -20,7 +20,7 @@ Today's resources are closely tied to the 'versioned artifact' use case, so this * Unifying `source` and `params` as just `config` so that resources don't have to care where configuration is being set in pipelines: [concourse/git-resource#172](https://github.com/concourse/git-resource/pull/172), [concourse/bosh-deployment-resource#13](https://github.com/concourse/bosh-deployment-resource/issues/13), [concourse/bosh-deployment-resource#6](https://github.com/concourse/bosh-deployment-resource/issues/6), [concourse/cf-resource#20](https://github.com/concourse/cf-resource/pull/20), [concourse/cf-resource#25](https://github.com/concourse/cf-resource/pull/25), [concourse/git-resource#210](https://github.com/concourse/git-resource/pull/210) -* Make resource actions more reentrant so that we no longer receive `unexpected EOF` errors upon reattaching to an in-flight build: [concourse/concourse#1580](https://github.com/concourse/concourse/issues/1580) +* Make resource actions reentrant so that we no longer receive `unexpected EOF` errors when reattaching to an in-flight build whose resource action completed while we weren't attached: [concourse/concourse#1580](https://github.com/concourse/concourse/issues/1580) * Support for showing icons for resources in the web UI: [concourse/concourse#788](https://github.com/concourse/concourse/issues/788), [concourse/concourse#3220](https://github.com/concourse/concourse/pull/3220), [concourse/concourse#3581](https://github.com/concourse/concourse/pull/3581) @@ -47,10 +47,11 @@ Today's resources are closely tied to the 'versioned artifact' use case, so this * Examples: `[{"name":"committer","value":"Alex Suraci"}]` * **Resource type**: an implementation of the interface defined by this proposal, typically provided as a container image. Implements the following actions: - * `info`: given a **config**, emit provide the commands to run for the following actions: + * `info`: given a **config**, emit a response specifying the command to run for each action * `check`: given a **config**, emit **config fragments** * `get`: given a **config**, populate a directory with **bits** - * `put`: given a **config** and a directory containing **bits**, create or delete **config fragments** + * `put`: given a **config** and a directory containing **bits**, create or update **config fragments** + * `delete`: given a **config** and a directory containing **bits**, delete **config fragments** * Examples: * `git-branches` resource type for tracking branches in a repo * `git` resource type for tracking commits in a branch From e63bd67571e117f17ff69e8e758366a060e1487b Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Tue, 21 May 2019 16:24:58 -0400 Subject: [PATCH 24/34] flesh out artifact resources proposal * introduce artifacts: and how it can be used as a migration path for explicit `get` after `put` * update open/closed questions (they are all now closed) Signed-off-by: Alex Suraci --- 024-artifact-resources/proposal.md | 66 +++++++++++++++++++++++++----- 1 file changed, 56 insertions(+), 10 deletions(-) diff --git a/024-artifact-resources/proposal.md b/024-artifact-resources/proposal.md index b9625772..e8c72e19 100644 --- a/024-artifact-resources/proposal.md +++ b/024-artifact-resources/proposal.md @@ -12,13 +12,19 @@ This proposal uses the [generalized resource interface](../024-generalized-resou ## Examples -* A basic implementation of the `git` resource can be found in [`git-v2`](examples/git-v2). +A basic implementation of the `git` resource can be found in [`git-v2`](examples/git-v2). ## Proposal -A v2 resource type can be used as a versioned artifact resource by treating the **config fragments** as **versions** and emitting them in chronological order from `check`. This way the resource type is used to model change in an external resource over time. +All resources as of Concourse v5.2 are technically artifact resources. They are now explicitly named artifact resources to disambiguate from other interpretations of the resource interface (e.g. spatial resources, notification resources, trigger resources). -### `check`: discover versions in order +This proposal describes how the new resource interface will be interpreted in order to support artifacts, and also proposes pipeline changes in order to support a backwards-compatible transition to the now-explicit "`get` after `put`" semantics. + +### Resource interface v2 interpretation + +A v2 resource type can be used as an artifact resource by treating the **config fragments** as **versions** and emitting them in chronological order from `check`. This way the resource type is used to model change in an external resource over time. + +#### `check`: discover versions in order The `check` action will first be run with a "naked" config, containing only what the user specified. In this situation `check` must emit an `ActionResponse` for all versions discovered in the config, in chronological order. @@ -28,30 +34,70 @@ If the specified version is no longer present, the `check` action must go back t The `check` action can use the **bits** directory to cache state between runs of the `check` on that worker. On the first run, the directory will be empty. -### `get`: fetch a version of an artifact +#### `get`: fetch a version of an artifact The `get` action will always be invoked with a spliced config specifying which version to fetch. It is given an empty **bits** directory in which to fetch the data. An `ActionResponse` must be emitted for all versions that have been fetched into the bits directory. Each version will be recorded as an input to the build. -### `put`: idempotently create artifact versions +#### `put`: idempotently create artifact versions The `put` action will be invoked with user-provided configuration and arbitrary bits. An `ActionResponse` must be emitted for all versions that have been created/updated. Each version will be recorded as an output of the build. -### `delete`: idempotently destroy artifact versions +#### `delete`: idempotently destroy artifact versions The `delete` action will be invoked with user-provided configuration and arbitrary bits. An `ActionResponse` must be emitted for all versions that have been destroyed. These versions will be marked "deleted" and no longer be available for use in other builds. +### Making `get` after `put` explicit + +Artifact resources will be defined at the top-level in the pipeline, under a new field called `artifacts:`. This field replaces `resources:` and has exactly the same structure. + +Example: + +```yaml +artifacts: # instead of 'resources:' +- name: booklit + type: git + source: {uri: "https://github.com/vito/booklit"} + +jobs: +- name: unit + plan: + - get: booklit + trigger: true + - task: test + file: booklit/ci/test.yml +``` + +When `artifacts:` is defined instead of `resources:`, `put` steps will no longer imply an automatic `get` step. Instead, a `get` field must be explicitly added to the step: + +```yaml +artifacts: +- name: my-resource + type: git + source: # ... + +jobs: +- name: push-pull-resource + plan: + - put: my-resource + get: input-name + # the created artifact will be fetched as 'input-name' +``` + +This change will be backwards-compatible - `resources:` will be treated as `artifacts:` and continue to use the "automatic `get` after `put`" behavior. Configuring pipelines with `resources:` will result in a deprecation warning instructing them to switch to `artifacts:` and migrate to the new "explicit `get` after `put`" behavior. + ## Open Questions -* [resource-determined triggerability of versions?](https://github.com/concourse/rfcs/issues/11) -* webhooks? - * should these instead be something supported by *Concourse*? +n/a ## Answered Questions -* Version filtering is probably best left to `config`. \ No newline at end of file +* [Version filtering is best left to `config`.](https://github.com/concourse/concourse/issues/1176#issuecomment-472111623) +* [Resource-determined triggerability of versions](https://github.com/concourse/rfcs/issues/11) will be addressed by the "trigger resource" RFC. +* Webhooks are left out of this first pass of the interface. I would like to investigate alternative approaches before baking it in. + * For example, could Concourse itself integrate with these services and map webhooks to resource checks intelligently? \ No newline at end of file From 5ae7100797b0a4733b2f641591e875686f290737 Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Tue, 21 May 2019 16:26:59 -0400 Subject: [PATCH 25/34] remove proposals for resource interpretations each of these proposals should be submitted as separate RFCs. each RFC should show how the v2 *and* v1 interfaces are to be interpreted. i've also removed the artifact resources proposal, which I initially included in order to provide a concrete example relating today's resource usage to the v2 interface. however, it can technically be done independently of the v2 interface, if not just as a way of achieving 'explicit `get` after `put`' in a backwards-compatible way. so let's just propose it separately and link to all four to demonstrate how it is just one of four proposed interpretations. Signed-off-by: Alex Suraci --- .../examples/git-v2/Dockerfile | 15 --- .../examples/git-v2/Gemfile | 4 - .../examples/git-v2/Gemfile.lock | 19 --- .../examples/git-v2/git-resource.rb | 115 ------------------ 024-artifact-resources/proposal.md | 103 ---------------- 024-generalized-resources/proposal.md | 8 +- 024-notification-resources/proposal.md | 36 ------ 024-spatial-resources/proposal.md | 38 ------ 024-trigger-resources/proposal.md | 37 ------ 9 files changed, 4 insertions(+), 371 deletions(-) delete mode 100644 024-artifact-resources/examples/git-v2/Dockerfile delete mode 100644 024-artifact-resources/examples/git-v2/Gemfile delete mode 100644 024-artifact-resources/examples/git-v2/Gemfile.lock delete mode 100644 024-artifact-resources/examples/git-v2/git-resource.rb delete mode 100644 024-artifact-resources/proposal.md delete mode 100644 024-notification-resources/proposal.md delete mode 100644 024-spatial-resources/proposal.md delete mode 100644 024-trigger-resources/proposal.md diff --git a/024-artifact-resources/examples/git-v2/Dockerfile b/024-artifact-resources/examples/git-v2/Dockerfile deleted file mode 100644 index 457d6a4c..00000000 --- a/024-artifact-resources/examples/git-v2/Dockerfile +++ /dev/null @@ -1,15 +0,0 @@ -FROM alpine - -RUN apk add git ruby ruby-json ruby-bundler build-base ruby-dev cmake openssl-dev - -ENV GEM_HOME="/usr/local/bundle" -ENV PATH $GEM_HOME/bin:$GEM_HOME/gems/bin:$PATH -ADD Gemfile . -RUN bundle install - -ADD git-resource.rb /usr/bin/git-resource -RUN chmod +x /usr/bin/git-resource - -WORKDIR /tmp/request - -CMD ["git-resource", "info"] \ No newline at end of file diff --git a/024-artifact-resources/examples/git-v2/Gemfile b/024-artifact-resources/examples/git-v2/Gemfile deleted file mode 100644 index 0259cb5a..00000000 --- a/024-artifact-resources/examples/git-v2/Gemfile +++ /dev/null @@ -1,4 +0,0 @@ -source :rubygems - -gem 'rugged' -gem 'pry' \ No newline at end of file diff --git a/024-artifact-resources/examples/git-v2/Gemfile.lock b/024-artifact-resources/examples/git-v2/Gemfile.lock deleted file mode 100644 index bf762b7c..00000000 --- a/024-artifact-resources/examples/git-v2/Gemfile.lock +++ /dev/null @@ -1,19 +0,0 @@ -GEM - remote: http://rubygems.org/ - specs: - coderay (1.1.2) - method_source (0.9.2) - pry (0.12.2) - coderay (~> 1.1.0) - method_source (~> 0.9.0) - rugged (0.28.1) - -PLATFORMS - ruby - -DEPENDENCIES - pry - rugged - -BUNDLED WITH - 1.16.2 diff --git a/024-artifact-resources/examples/git-v2/git-resource.rb b/024-artifact-resources/examples/git-v2/git-resource.rb deleted file mode 100644 index 138526b1..00000000 --- a/024-artifact-resources/examples/git-v2/git-resource.rb +++ /dev/null @@ -1,115 +0,0 @@ -#!/usr/bin/env ruby - -require "json" -require "rugged" -require "pry" -require "benchmark" - -$request = JSON.parse(STDIN.read, symbolize_names: true) - -def commit_fragment(commit) - JSON.dump({ - config: {ref: commit.oid}, - metadata: [ - {name: "author", value: enc(commit, commit.author[:name])}, - {name: "committer", value: enc(commit, commit.committer[:name])}, - {name: "message", value: enc(commit, commit.message)} - ] - }) -end - -def icon(uri) - case uri - when /github/ - "github-circle" - when /gitlab/ - "gitlab" - when /bitbucket/ - "bitbucket" - else - "git" - end -end - -def bench(label, &blk) - time = Benchmark.realtime(&blk) - $stderr.puts "#{label}: #{time}s" -end - -def enc(commit, str) - str = str.force_encoding("ISO-8859-1") unless commit.header_field("Encoding") - str.encode("UTF-8") -end - -case ARGV[0] -when "info" - puts JSON.dump({ - "interface_version": "2.0", - "icon": icon($request[:config][:uri]), - "actions": { - "check": "git-resource check", - "get": "git-resource get", - "put": "git-resource put" - # delete is unsupported - } - }) - -when "check" - repo = - if File.exists?("HEAD") - Rugged::Repository.new(".").tap do |r| - r.fetch("origin") - end - else - Rugged::Repository.clone_at( - $request[:config][:uri], - ".", - checkout_branch: $request[:config][:branch], - bare: true, - progress: lambda { |t| $stderr.print t }) - end - - walker = Rugged::Walker.new(repo) - walker.sorting(Rugged::SORT_TOPO|Rugged::SORT_REVERSE) - walker.simplify_first_parent - walker.push(repo.head.target) - - response = File.new $request[:response_path], 'w' - total_commits = 0 - - from = $request[:config][:ref] - if from && repo.include?(from) - commit = repo.lookup(from) - walker.hide(commit) - - response.puts commit_fragment(commit) - total_commits += 1 - end - - bench("walk") do - walker.walk do |c| - response.puts commit_fragment(c) - total_commits += 1 - end - end - - $stderr.puts "commits: #{total_commits}" - - response.close - -when "get" - repo = - Rugged::Repository.clone_at( - $request[:config][:uri], - ".", - checkout_branch: $request[:config][:branch]) - - repo.checkout($request[:config][:ref]) - - response = File.new $request[:response_path], 'w' - response.puts commit_fragment(repo.head.target) - response.close - -when "put" - puts "putting" -end \ No newline at end of file diff --git a/024-artifact-resources/proposal.md b/024-artifact-resources/proposal.md deleted file mode 100644 index e8c72e19..00000000 --- a/024-artifact-resources/proposal.md +++ /dev/null @@ -1,103 +0,0 @@ -# Artifact Resources - -This proposal uses the [generalized resource interface](../024-generalized-resources/proposal.md) to show how the interface would be implemented and interpreted by Concourse to support "artifact resources", today's primary use case for v1 resources. - -## Motivation - -* Support for creating multiple versions from `put`: [concourse/concourse#2660](https://github.com/concourse/concourse/issues/2660) - -* Support for deleting versions: [concourse/concourse#362](https://github.com/concourse/concourse/issues/362), [concourse/concourse#524](https://github.com/concourse/concourse/issues/524) - -* Make the `get` after `put` opt-in: [concourse/concourse#3299](https://github.com/concourse/concourse/issues/3299), [concourse/registry-image-resource#16](https://github.com/concourse/registry-image-resource/issues/16) - -## Examples - -A basic implementation of the `git` resource can be found in [`git-v2`](examples/git-v2). - -## Proposal - -All resources as of Concourse v5.2 are technically artifact resources. They are now explicitly named artifact resources to disambiguate from other interpretations of the resource interface (e.g. spatial resources, notification resources, trigger resources). - -This proposal describes how the new resource interface will be interpreted in order to support artifacts, and also proposes pipeline changes in order to support a backwards-compatible transition to the now-explicit "`get` after `put`" semantics. - -### Resource interface v2 interpretation - -A v2 resource type can be used as an artifact resource by treating the **config fragments** as **versions** and emitting them in chronological order from `check`. This way the resource type is used to model change in an external resource over time. - -#### `check`: discover versions in order - -The `check` action will first be run with a "naked" config, containing only what the user specified. In this situation `check` must emit an `ActionResponse` for all versions discovered in the config, in chronological order. - -Subsequent calls to `check` will be given a config that has been spliced with the last emitted version config fragment. The `check` script must an `ActionResponse` for the given version if it still exists, followed by a response for any versions that came after it. - -If the specified version is no longer present, the `check` action must go back to returning all versions, as if the version was not specified in the first place. Concourse will detect this scenario by noticing that the first `ActionResponse` emitted does not match the requested version. All versions that existed before that were emitted will be automatically marked "deleted". - -The `check` action can use the **bits** directory to cache state between runs of the `check` on that worker. On the first run, the directory will be empty. - -#### `get`: fetch a version of an artifact - -The `get` action will always be invoked with a spliced config specifying which version to fetch. It is given an empty **bits** directory in which to fetch the data. - -An `ActionResponse` must be emitted for all versions that have been fetched into the bits directory. Each version will be recorded as an input to the build. - -#### `put`: idempotently create artifact versions - -The `put` action will be invoked with user-provided configuration and arbitrary bits. - -An `ActionResponse` must be emitted for all versions that have been created/updated. Each version will be recorded as an output of the build. - -#### `delete`: idempotently destroy artifact versions - -The `delete` action will be invoked with user-provided configuration and arbitrary bits. - -An `ActionResponse` must be emitted for all versions that have been destroyed. These versions will be marked "deleted" and no longer be available for use in other builds. - -### Making `get` after `put` explicit - -Artifact resources will be defined at the top-level in the pipeline, under a new field called `artifacts:`. This field replaces `resources:` and has exactly the same structure. - -Example: - -```yaml -artifacts: # instead of 'resources:' -- name: booklit - type: git - source: {uri: "https://github.com/vito/booklit"} - -jobs: -- name: unit - plan: - - get: booklit - trigger: true - - task: test - file: booklit/ci/test.yml -``` - -When `artifacts:` is defined instead of `resources:`, `put` steps will no longer imply an automatic `get` step. Instead, a `get` field must be explicitly added to the step: - -```yaml -artifacts: -- name: my-resource - type: git - source: # ... - -jobs: -- name: push-pull-resource - plan: - - put: my-resource - get: input-name - # the created artifact will be fetched as 'input-name' -``` - -This change will be backwards-compatible - `resources:` will be treated as `artifacts:` and continue to use the "automatic `get` after `put`" behavior. Configuring pipelines with `resources:` will result in a deprecation warning instructing them to switch to `artifacts:` and migrate to the new "explicit `get` after `put`" behavior. - -## Open Questions - -n/a - -## Answered Questions - -* [Version filtering is best left to `config`.](https://github.com/concourse/concourse/issues/1176#issuecomment-472111623) -* [Resource-determined triggerability of versions](https://github.com/concourse/rfcs/issues/11) will be addressed by the "trigger resource" RFC. -* Webhooks are left out of this first pass of the interface. I would like to investigate alternative approaches before baking it in. - * For example, could Concourse itself integrate with these services and map webhooks to resource checks intelligently? \ No newline at end of file diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index 18a07285..da2f38b0 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -266,7 +266,7 @@ By leaving the interface general, resource authors don't know how their resource These interpretations are outlined in the following proposals: -### [Artifact resources](../024-artifact-resources/proposal.md) +### [Artifact resources](https://github.com/concourse/rfcs/pull/26) * `check`: return versions in order * `get`: fetch a version of the resource @@ -275,7 +275,7 @@ These interpretations are outlined in the following proposals: Examples: `git` -### [Spatial resources](../024-spatial-resources/proposal.md) +### [Spatial resources](https://github.com/concourse/rfcs/pull/29) * `check`: return a fragment for each space, no order * `get`: fetch whatever metadata is useful for a given space @@ -284,7 +284,7 @@ Examples: `git` Examples: `git-branch`, `github-pr` -### [Notification resources](../024-notification-resources/proposal.md) +### [Notification resources](https://github.com/concourse/rfcs/pull/28) * `check`: not used * `get`: fetch bits pertaining to the notification @@ -293,7 +293,7 @@ Examples: `git-branch`, `github-pr` Examples: `github-status`, `slack` -### [Trigger resources](../024-trigger-resources/proposal.md) +### [Trigger resources](https://github.com/concourse/rfcs/pull/27) * `check`: check against last fragment used for job * `get`: fetch bits pertaining to the trigger diff --git a/024-notification-resources/proposal.md b/024-notification-resources/proposal.md deleted file mode 100644 index 8ba4d642..00000000 --- a/024-notification-resources/proposal.md +++ /dev/null @@ -1,36 +0,0 @@ -# Notification Resources - -This proposal uses the [generalized resource interface](../024-generalized-resources/proposal.md) to show how the interface would be implemented and interpreted by Concourse to support "notification resources", which are composed with another resource to emit notifications for build status. - -## Motivation - -* Support notifications in a way that doesn't pollute pipeline config and UI: [concourse/concourse#1052](https://github.com/concourse/concourse/issues/1052), [concourse/rfcs#10](https://github.com/concourse/rfcs/issues/10) - -## Proposal - -> Describe your proposal. -> -> Things that can help: clearly defining terms, providing example content, -> pseudocode, etc. -> -> Feel free to mention key implementation concerns. - -## Open Questions - -> Raise any concerns here for things you aren't sure about yet. - -## Answered Questions - -> If there were any major concerns that have already (or eventually, through -> the RFC process) reached consensus, it can still help to include them along -> with their resolution, if it's otherwise unclear. -> -> This can be especially useful for RFCs that have taken a long time and there -> were some subtle yet important details to get right. -> -> This may very well be empty if the proposal is simple enough. - -## New Implications - -> What is the impact of this change, outside of the change itself? How might it -> change peoples' workflows today, good or bad? diff --git a/024-spatial-resources/proposal.md b/024-spatial-resources/proposal.md deleted file mode 100644 index d167bf29..00000000 --- a/024-spatial-resources/proposal.md +++ /dev/null @@ -1,38 +0,0 @@ -# Spatial Resources - -This proposal uses the [generalized resource interface](../024-generalized-resources/proposal.md) to show how the interface would be implemented and interpreted by Concourse to support "spatial resources", which model change over *space*, not time (as with [artifact resources](../024-artifact-resources/proposal.md)). - -## Motivation - -* Support multi-branch workflows: [concourse/concourse#1172](https://github.com/concourse/concourse/issues/1172) - -* Begin phasing out `version: every` in by reframing the problem as 'pipeline per commit': [concourse/concourse#736](https://github.com/concourse/concourse/issues/736) - -## Proposal - -> Describe your proposal. -> -> Things that can help: clearly defining terms, providing example content, -> pseudocode, etc. -> -> Feel free to mention key implementation concerns. - -## Open Questions - -> Raise any concerns here for things you aren't sure about yet. - -## Answered Questions - -> If there were any major concerns that have already (or eventually, through -> the RFC process) reached consensus, it can still help to include them along -> with their resolution, if it's otherwise unclear. -> -> This can be especially useful for RFCs that have taken a long time and there -> were some subtle yet important details to get right. -> -> This may very well be empty if the proposal is simple enough. - -## New Implications - -> What is the impact of this change, outside of the change itself? How might it -> change peoples' workflows today, good or bad? diff --git a/024-trigger-resources/proposal.md b/024-trigger-resources/proposal.md deleted file mode 100644 index c06869f7..00000000 --- a/024-trigger-resources/proposal.md +++ /dev/null @@ -1,37 +0,0 @@ -# Trigger Resources - -This proposal uses the [generalized resource interface](../024-generalized-resources/proposal.md) to show how the interface would be implemented and interpreted by Concourse to support "trigger resources", are tied to a job and perform `check`s to determine whether the job should trigger. - -## Motivation - -* Support for trigger-only resources that don't result in fetching anything - they just trigger the job: [concourse/concourse#3572](https://github.com/concourse/concourse/issues/3572), [concourse/concourse#3595](https://github.com/concourse/concourse/issues/3595) - -## Proposal - -> Describe your proposal. -> -> Things that can help: clearly defining terms, providing example content, -> pseudocode, etc. -> -> Feel free to mention key implementation concerns. - -## Open Questions - -* could the config fragment be passed to something to support parameterized triggers? :thinking: -* could this be related to how we approach [concourse/concourse#738](https://github.com/concourse/concourse/issues/783)? :thinking: - -## Answered Questions - -> If there were any major concerns that have already (or eventually, through -> the RFC process) reached consensus, it can still help to include them along -> with their resolution, if it's otherwise unclear. -> -> This can be especially useful for RFCs that have taken a long time and there -> were some subtle yet important details to get right. -> -> This may very well be empty if the proposal is simple enough. - -## New Implications - -> What is the impact of this change, outside of the change itself? How might it -> change peoples' workflows today, good or bad? From fb6238086da9a794312ad04719859819f9359b8a Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Tue, 21 May 2019 18:25:08 -0400 Subject: [PATCH 26/34] don't list trigger/spatial/notification resources these are being made independent of v1 vs. v2 Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index da2f38b0..729b47ec 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -26,12 +26,6 @@ Today's resources are closely tied to the 'versioned artifact' use case, so this * Standardize TLS configuration so every resource doesn't implement their own way: [concourse/rfcs#9](https://github.com/concourse/rfcs/issues/9) -* Support [trigger resources](../024-trigger-resources/proposal.md). - -* Support [spatial resources](../024-spatial-resources/proposal.md). - -* Support [notification resources](../024-notification-resources/proposal.md). - ## Glossary * **Config**: an arbitrarily nested JSON object containing user-provided configuration @@ -304,4 +298,4 @@ Examples: `time` ## Out of scope -* [Richer metadata](https://github.com/concourse/concourse/issues/310) - this hasn't gained much traction and probably needs more investigation before it can be incorporated. This should be easy enough to add as a later RFC. \ No newline at end of file +* [Richer metadata](https://github.com/concourse/concourse/issues/310) - this hasn't gained much traction and probably needs more investigation before it can be incorporated. This should be easy enough to add as a later RFC. From b974dc2e79bf4a6afa93b09e0a3aba43ac59d1ac Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Wed, 22 May 2019 09:00:35 -0400 Subject: [PATCH 27/34] insecure_skip_verify -> skip_verification calling this something that sounds like better english until someone has a good reason otherwise Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index 729b47ec..5f55af3a 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -102,8 +102,8 @@ type TLSConfig struct { // An array of CA certificates to trust. CAs []string `json:"ca_certs,omitempty"` - // Disable TLS, effectively making communication over TLS insecure. - InsecureSkipVerify bool `json:"insecure_skip_verify,omitempty"` + // Skip certificate verification, effectively making communication insecure. + SkipVerification bool `json:"skip_verification,omitempty"` } // InfoRequest is the payload written to stdin for the `./info` script. From 504a13a700a17ba6a610ca0951f8876619fc804d Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Wed, 22 May 2019 10:26:25 -0400 Subject: [PATCH 28/34] flesh out summary and interpretation sections Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 89 ++++++++++++++++----------- 1 file changed, 52 insertions(+), 37 deletions(-) diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md index 5f55af3a..85df92d3 100644 --- a/024-generalized-resources/proposal.md +++ b/024-generalized-resources/proposal.md @@ -1,8 +1,14 @@ # Generalized Resources -This proposal adapts today's 'resource' interface into a more general interface that is less specialized to the 'versioned artifacts' use case, while also introducing versioning to the interface so that it'll be easier to make incremental changes to the interface itself and support mixed resource versions within a user's pipeline. +This proposal adapts today's 'resource' interface ("v1") into a more general resource interface that is less specialized to the 'versioned artifacts' use case currently supported by Concourse pipelines. -Today's resources are closely tied to the 'versioned artifact' use case, so this proposal will also show how the new interface is "interpreted" in order to support this use case. +The proposed interface is versioned, starting at `2.0`. Each resource type will have its [resource info](#resource-info) fetched to discover its version. + +Concourse will support use of mixed resource interface versions. When a resource type does not support the new 'info' request flow it will be assumed to be v1. + +[Resource actions](#resource-actions) have all been updated to fit the exact same request/response interface. This proposal defines four core actions: `check`, `get`, `put`, and `delete`. + +Each action emits **config fragments**, which replace 'versions' from the v1 interface. By using a more general name the interface can be used for use cases beyond just versioning artifacts. See [Interpreting the Resource Interface](#interpreting-the-resource-interface) for more details. ## Previous Discussions @@ -54,34 +60,6 @@ Today's resources are closely tied to the 'versioned artifact' use case, so this * **Resource**: a **resource type** with a user-provided **config**, used together to represent external state. -## Example Resources - -```yaml -type: git-branches -source: - uri: https://github.com/concourse/concourse -``` - -```yaml -type: git -source: - uri: https://github.com/concourse/concourse - branch: master -``` - -```yaml -type: github-status -source: - repository: concourse/concourse - access_token: abcdef -``` - -```yaml -type: time -source: - interval: 10m -``` - ## Interface Types ```go @@ -252,16 +230,29 @@ This response would be typical of a `check` that ran against a repo that had thr ## Interpreting the Resource Interface -The resource interface itself is now a general way of expressing interactions with external state. It is no longer restricted to versioning of artifacts. +The resource interface itself is now a general way of expressing interactions with external state. It is no longer based on versioning of artifacts. + +Concourse will now codify things like "artifact resources" and "spatial resources" as interpretations of this general interface, composing resource types with one another via **config fragments**. -Concourse will now codify things like "versioned artifacts" and "spatial resources" as interpretations of this general interface, composing resource types with one another via **config fragments**. +By leveraging composition instead of having a monolithic interface, this approach encourages narrowly scoped resource type implementations. Implementations that are small in scope are more likely to be correct and 'finished' at some point, and only become more powerful as Concourse enables new workflows at the pipeline-level. -By leaving the interface general, resource authors don't know how their resource type will be used. This gives Concourse flexibility in defining new workflows without requiring resource authors to implement these new workflows themselves, and allows a resource type to be used for multiple use cases. For example, notifications and triggers are somewhat complementary and may both be supported by a resource type that implements the full interface. +For example, as an author of a `git` artifact resource type, I just have to implement a linear, versioned artifact interface. In `git`, linear versioning (assuming `--first-parent`) occurs on a branch, so that can be specified in `config`. What if users want to run against all branches? As a resource type author, that's not my problem! Let them use a `git-branches` spatial resource and compose it with my resource to provide the `branch` config. -These interpretations are outlined in the following proposals: +By not baking the workflow directly into the interface, this also allows a single resource type implementation to be used for multiple interpretations. For example, notifications and triggers are complementary and can both be supported by a resource type that supports the required actions (`put` for notifications and `check`/`get` for triggers). + +Four interpretations are outlined in the following proposals. The RFCs linked below show how the v2 interface can be used to support various pipeline workflows. The described workflows can also work with v1 resources, however, and so they are not explicitly dependent on the v2 interface. This is done intentionally so that our roadmap doesn't have to be so linear. ### [Artifact resources](https://github.com/concourse/rfcs/pull/26) +```yaml +artifacts: +- name: concourse + type: git + source: + uri: https://github.com/concourse/concourse + branch: master +``` + * `check`: return versions in order * `get`: fetch a version of the resource * `put`: push versions of a resource @@ -271,27 +262,51 @@ Examples: `git` ### [Spatial resources](https://github.com/concourse/rfcs/pull/29) +```yaml +spaces: +- name: concourse-branches + type: git-branches + source: + uri: https://github.com/concourse/concourse +``` + * `check`: return a fragment for each space, no order * `get`: fetch whatever metadata is useful for a given space * `put`: create or update spaces * `delete`: delete spaces -Examples: `git-branch`, `github-pr` +Examples: `git-branches`, `github-prs` ### [Notification resources](https://github.com/concourse/rfcs/pull/28) +```yaml +notifications: +- name: concourse-status + type: github-status + source: + repository: concourse/concourse + access_token: abcdef +``` + * `check`: not used * `get`: fetch bits pertaining to the notification * `put`: emit a notification -* `delete`: clear github status? +* `delete`: not sure - clear github status? Examples: `github-status`, `slack` ### [Trigger resources](https://github.com/concourse/rfcs/pull/27) +```yaml +triggers: +- name: every-10m + type: time + source: {interval: 10m} +``` + * `check`: check against last fragment used for job * `get`: fetch bits pertaining to the trigger -* `put`: manual trigger? +* `put`: not useful * `delete`: not useful Examples: `time` From 66c9d09cb00fa5be16e07f7fd8dc9ef25074279e Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Mon, 9 Sep 2019 10:03:47 -0400 Subject: [PATCH 29/34] rework 'generalized resources' into 'prototypes' Signed-off-by: Alex Suraci --- 024-generalized-resources/proposal.md | 316 --------------- 037-prototypes/proposal.md | 544 ++++++++++++++++++++++++++ 2 files changed, 544 insertions(+), 316 deletions(-) delete mode 100644 024-generalized-resources/proposal.md create mode 100644 037-prototypes/proposal.md diff --git a/024-generalized-resources/proposal.md b/024-generalized-resources/proposal.md deleted file mode 100644 index 85df92d3..00000000 --- a/024-generalized-resources/proposal.md +++ /dev/null @@ -1,316 +0,0 @@ -# Generalized Resources - -This proposal adapts today's 'resource' interface ("v1") into a more general resource interface that is less specialized to the 'versioned artifacts' use case currently supported by Concourse pipelines. - -The proposed interface is versioned, starting at `2.0`. Each resource type will have its [resource info](#resource-info) fetched to discover its version. - -Concourse will support use of mixed resource interface versions. When a resource type does not support the new 'info' request flow it will be assumed to be v1. - -[Resource actions](#resource-actions) have all been updated to fit the exact same request/response interface. This proposal defines four core actions: `check`, `get`, `put`, and `delete`. - -Each action emits **config fragments**, which replace 'versions' from the v1 interface. By using a more general name the interface can be used for use cases beyond just versioning artifacts. See [Interpreting the Resource Interface](#interpreting-the-resource-interface) for more details. - -## Previous Discussions - -* [RFC #1](https://github.com/concourse/rfcs/pull/1), now defunct, is similar to this proposal but had a concept of "spaces" baked into the interface. This concept has been decoupled from the resource interface and will be re-introduced in a later RFC that is compatible with v1 and v2 resources. - * **Recommended reading**: [this comment](https://github.com/concourse/rfcs/pull/1#issuecomment-477749314) outlines the thought process that led to this RFC. -* [concourse/concourse#534](https://github.com/concourse/concourse/issues/534) was the first 'new resource interface' proposal which pre-dated the RFC process. - -## Motivation - -* Support for creating multiple versions from `put`: [concourse/concourse#2660](https://github.com/concourse/concourse/issues/2660) - -* Support for deleting versions: [concourse/concourse#362](https://github.com/concourse/concourse/issues/362), [concourse/concourse#524](https://github.com/concourse/concourse/issues/524) - -* Having resource metadata immediately available via check: [concourse/git-resource#193](https://github.com/concourse/git-resource/issues/193), [concourse/concourse#1714](https://github.com/concourse/concourse/issues/1714) - -* Unifying `source` and `params` as just `config` so that resources don't have to care where configuration is being set in pipelines: [concourse/git-resource#172](https://github.com/concourse/git-resource/pull/172), [concourse/bosh-deployment-resource#13](https://github.com/concourse/bosh-deployment-resource/issues/13), [concourse/bosh-deployment-resource#6](https://github.com/concourse/bosh-deployment-resource/issues/6), [concourse/cf-resource#20](https://github.com/concourse/cf-resource/pull/20), [concourse/cf-resource#25](https://github.com/concourse/cf-resource/pull/25), [concourse/git-resource#210](https://github.com/concourse/git-resource/pull/210) - -* Make resource actions reentrant so that we no longer receive `unexpected EOF` errors when reattaching to an in-flight build whose resource action completed while we weren't attached: [concourse/concourse#1580](https://github.com/concourse/concourse/issues/1580) - -* Support for showing icons for resources in the web UI: [concourse/concourse#788](https://github.com/concourse/concourse/issues/788), [concourse/concourse#3220](https://github.com/concourse/concourse/pull/3220), [concourse/concourse#3581](https://github.com/concourse/concourse/pull/3581) - -* Standardize TLS configuration so every resource doesn't implement their own way: [concourse/rfcs#9](https://github.com/concourse/rfcs/issues/9) - -## Glossary - -* **Config**: an arbitrarily nested JSON object containing user-provided configuration - * Examples: `{"uri":"https://github.com/concourse/concourse"}`, `{"interval":"10m"}` - -* **Config fragment**: a smaller JSON object intended to be "spliced" into a **config** by assigning each field from the fragment into the config. - * Examples: `{"ref":"abcdef"}`, `{"branch":"develop"}` - -* **Bits**: a directory containing arbitrary data - * Examples: source code, compiled artifacts, etc. - -* **Metadata**: structured data associated to a **config fragment** containing information about the fragment that should be surfaced to the user - * Examples: `[{"name":"committer","value":"Alex Suraci"}]` - -* **Resource type**: an implementation of the interface defined by this proposal, typically provided as a container image. Implements the following actions: - * `info`: given a **config**, emit a response specifying the command to run for each action - * `check`: given a **config**, emit **config fragments** - * `get`: given a **config**, populate a directory with **bits** - * `put`: given a **config** and a directory containing **bits**, create or update **config fragments** - * `delete`: given a **config** and a directory containing **bits**, delete **config fragments** - * Examples: - * `git-branches` resource type for tracking branches in a repo - * `git` resource type for tracking commits in a branch - * `github-status` resource type for emitting build status notifications for commits - * `time` resource type for doing timed job triggers - -* **Resource**: a **resource type** with a user-provided **config**, used together to represent external state. - -## Interface Types - -```go -// Config represents arbitrary user-specified configuration. -type Config map[string]interface{} - -// ConfigFragment represents additional fields that can be spliced into a Config. -type ConfigFragment map[string]interface{} - -// MetadataField represents a named bit of metadata associated to a ConfigFragment. -type MetadataField struct { - Name string `json:"name"` - Value string `json:"value"` -} - -// TLSConfig captures common configuration for communicating with servers over TLS. -type TLSConfig struct { - // An array of CA certificates to trust. - CAs []string `json:"ca_certs,omitempty"` - - // Skip certificate verification, effectively making communication insecure. - SkipVerification bool `json:"skip_verification,omitempty"` -} - -// InfoRequest is the payload written to stdin for the `./info` script. -type InfoRequest struct { - // User-specified configuration. - Config Config `json:"config"` -} - -// InfoResponse is the payload written to stdout from the `./info` script. -type InfoResponse struct { - // The version of the resource interface that this resource type conforms to. - InterfaceVersion string `json:"interface_version"` - - // An optional icon name to show to the user when viewing the resource. - // - // Icons must be namespaced by in order to explicitly reference an icon set - // supported by Concourse, e.g. 'mdi:' for Material Design Icons. - Icon string `json:"icon,omitempty"` - - // The actions supported by the resource type. - Actions struct { - // Command to run when performing check actions. - Check string `json:"check,omitempty"` - - // Command to run when performing get actions. - Get string `json:"get,omitempty"` - - // Command to run when performing put actions. - Put string `json:"put,omitempty"` - - // Command to run when performing delete actions. - Delete string `json:"delete,omitempty"` - } `json:"actions"` -} - -// ActionRequest is the payload written to stdin for each action command. -type ActionRequest struct { - // User-specified configuration. - Config Config `json:"config"` - - // Configuration for handling TLS. - TLS TLSConfig `json:"tls,omitempty"` - - // Path to a file into which the action must write its response. - ResponsePath string `json:"response_path"` -} - -// ActionResponse is written to the `response_path` by an action for each fragment affected by the action. Multiple respones may be written as a JSON stream. -type ActionResponse struct { - // The fragment. May be used as an identifier, unique within the scope of a Config. - Fragment ConfigFragment `json:"fragment"` - - // Metadata to associate with the fragment. Shown to the user. - Metadata []MetadataField `json:"metadata,omitempty"` -} -``` - -## Resource Info - -Prior to running any action, Concourse will execute the default command (i.e. [`CMD`](https://docs.docker.com/engine/reference/builder/#cmd)) for the image with an `InfoRequest` piped to `stdin`. - -The command must write an `InfoResponse` to `stdout` in response. This response specifies the resource interface version that the resource type conforms to, an optional icon to show in the UI, and the command to run for each supported resource action. - -### Example **info** request/response - -Request sent to `stdin`: - -```json -{ - "config": { - "uri": "https://github.com/concourse/concourse" - } -} -``` - -Response written to `stdout`: - -```json -{ - "interface_version": "2.0", - "icon": "mdi:github-circle", - "actions": { - "check": "/usr/bin/git-resource check", - "get": "/usr/bin/git-resource get", - "put": "/usr/bin/git-resource put", - "delete": "/usr/bin/git-resource delete" - } -} -``` - -## Resource Actions - -Each action is invoked with a JSON-encoded `ActionRequest` piped to `stdin`. This request contains the **config** and the path to which the response should be written. This path may be relative, and if so it is to be expanded from the current working directory. - -All actions will be run in a working directory for the **bits** - either an empty directory to which bits should be written, or a directory containing the bits given to the action. - -All actions respond by performing their side-effect and writing sequential `ActionResponse` JSON objects to the file path specified by `response_path`. How this response is interpreted depends on the action, but typically there should be one response for each external resource affected (`put`, `delete`), discovered (`check`), or fetched (`get`). - -### Example **action** request/response - -Request sent to `stdin`: - -```json -{ - "config": { - "uri": "https://github.com/concourse/rfcs", - "branch": "master" - }, - "response_path": "../response/response.json" -} -``` - -Response written to `../response/response.json`: - -```json -{ - "fragment": {"ref": "e4be0b367d7bd34580f4842dd09e7b59b6097b25"}, - "metadata": [ - { - "name": "message", - "value": "init" - } - ] -} -{ - "fragment": {"ref": "5a052ba6438d754f73252283c6b6429f2a74dbff"}, - "metadata": [ - { - "name": "message", - "value": "add not-very-useful-yet readme" - } - ] -} -{ - "fragment": {"ref": "2e256c3cb4b077f6fa3c465dd082fa74df8fab0a"}, - "metadata": [ - { - "name": "message", - "value": "start fleshing out RFC process" - } - ] -} -``` - -This response would be typical of a `check` that ran against a repo that had three commits. - -## Interpreting the Resource Interface - -The resource interface itself is now a general way of expressing interactions with external state. It is no longer based on versioning of artifacts. - -Concourse will now codify things like "artifact resources" and "spatial resources" as interpretations of this general interface, composing resource types with one another via **config fragments**. - -By leveraging composition instead of having a monolithic interface, this approach encourages narrowly scoped resource type implementations. Implementations that are small in scope are more likely to be correct and 'finished' at some point, and only become more powerful as Concourse enables new workflows at the pipeline-level. - -For example, as an author of a `git` artifact resource type, I just have to implement a linear, versioned artifact interface. In `git`, linear versioning (assuming `--first-parent`) occurs on a branch, so that can be specified in `config`. What if users want to run against all branches? As a resource type author, that's not my problem! Let them use a `git-branches` spatial resource and compose it with my resource to provide the `branch` config. - -By not baking the workflow directly into the interface, this also allows a single resource type implementation to be used for multiple interpretations. For example, notifications and triggers are complementary and can both be supported by a resource type that supports the required actions (`put` for notifications and `check`/`get` for triggers). - -Four interpretations are outlined in the following proposals. The RFCs linked below show how the v2 interface can be used to support various pipeline workflows. The described workflows can also work with v1 resources, however, and so they are not explicitly dependent on the v2 interface. This is done intentionally so that our roadmap doesn't have to be so linear. - -### [Artifact resources](https://github.com/concourse/rfcs/pull/26) - -```yaml -artifacts: -- name: concourse - type: git - source: - uri: https://github.com/concourse/concourse - branch: master -``` - -* `check`: return versions in order -* `get`: fetch a version of the resource -* `put`: push versions of a resource -* `delete`: delete versions of a resource - -Examples: `git` - -### [Spatial resources](https://github.com/concourse/rfcs/pull/29) - -```yaml -spaces: -- name: concourse-branches - type: git-branches - source: - uri: https://github.com/concourse/concourse -``` - -* `check`: return a fragment for each space, no order -* `get`: fetch whatever metadata is useful for a given space -* `put`: create or update spaces -* `delete`: delete spaces - -Examples: `git-branches`, `github-prs` - -### [Notification resources](https://github.com/concourse/rfcs/pull/28) - -```yaml -notifications: -- name: concourse-status - type: github-status - source: - repository: concourse/concourse - access_token: abcdef -``` - -* `check`: not used -* `get`: fetch bits pertaining to the notification -* `put`: emit a notification -* `delete`: not sure - clear github status? - -Examples: `github-status`, `slack` - -### [Trigger resources](https://github.com/concourse/rfcs/pull/27) - -```yaml -triggers: -- name: every-10m - type: time - source: {interval: 10m} -``` - -* `check`: check against last fragment used for job -* `get`: fetch bits pertaining to the trigger -* `put`: not useful -* `delete`: not useful - -Examples: `time` - -## Out of scope - -* [Richer metadata](https://github.com/concourse/concourse/issues/310) - this hasn't gained much traction and probably needs more investigation before it can be incorporated. This should be easy enough to add as a later RFC. diff --git a/037-prototypes/proposal.md b/037-prototypes/proposal.md new file mode 100644 index 00000000..d1cbbee8 --- /dev/null +++ b/037-prototypes/proposal.md @@ -0,0 +1,544 @@ +# Concourse Prototype Protocol + +This proposal outlines a small protocol for invoking conceptually related +commands in a container image with a JSON-based request-response. This protocol +is to be used as a foundation for both a new resource type interface and as a +way to package and reuse arbitrary functionality (i.e. tasks). + +The protocol is versioned, starting at `1.0`. + +An implementation of this protocol is called a **prototype**. ([Why +'prototype'?](#etymology)) Conceptually, a prototype handles **messages** sent +to **objects** which are provided by the user as configuration or produced by +the prototype itself in response to a prior message. + +For example, a `git` prototype may handle a `check` message against a +repository object to produce commit objects, which then can then handle a `get` +message to fetch the commit. + +An object's supported messages are discovered through an [Info +request](#prototype-info). For example, the `git` prototype may support `get` +against a commit object, but not against a repository object. The Info request +is also a good place to perform validation. + +A message is handled by invoking its corresponding command in the container +with a [Message request](#prototype-message). A message handler responds by +emitting objects and accompanying metadata. These objects are written to a +response path specified by the request. + +While a prototype can handle any number of messages with any name, certain +messages names will have semantic meaning in a Concourse pipeline. For example, +a prototype which supports `check` and `get` messages can be used as a resource +in a pipeline. These messages will be sent by the Concourse pipeline scheduler +if a user configures the prototype and config as a resource. + + +## Previous Discussions + +* [RFC #24][rfc-24] was a previous iteration of this proposal. It had a fixed + set of "actions" (`check`, `get`, `put`, `delete`) and was still centered in + the 'resources' concept and terminology. This RFC generalizes the design + further, switches to prototype-based terminology and allows prototypes to + handle arbitrary messages. + +* [RFC #1][rfc-1], now defunct, is similar to this proposal but had a concept + of "spaces" baked into the interface. This concept has been decoupled from + the resource interface and will be re-introduced in a later RFC that is + compatible with v1 and v2 resources. + * **Recommended reading**: [this comment][rfc-1-comment] outlines the thought + process that led to this RFC. + +* [concourse/concourse#534](https://github.com/concourse/concourse/issues/534) + was the first 'new resource interface' proposal which pre-dated the RFC + process. + + +## Motivation + +* Support for 'reusable tasks' in order to stop the shoehorning of + functionality into the resource interface: + [concourse/rfcs#7](https://github.com/concourse/rfcs/issues/7). + +* Support for creating multiple versions from `put`: + [concourse/concourse#2660](https://github.com/concourse/concourse/issues/2660) + +* Support for deleting versions: + [concourse/concourse#362](https://github.com/concourse/concourse/issues/362), + [concourse/concourse#524](https://github.com/concourse/concourse/issues/524) + +* Having resource metadata immediately available via check: + [concourse/git-resource#193](https://github.com/concourse/git-resource/issues/193), + [concourse/concourse#1714](https://github.com/concourse/concourse/issues/1714) + +* Unifying `source` and `params` as just `config` so that resources don't have + to care where configuration is being set in pipelines: + [concourse/git-resource#172](https://github.com/concourse/git-resource/pull/172), + [concourse/bosh-deployment-resource#13](https://github.com/concourse/bosh-deployment-resource/issues/13), + [concourse/bosh-deployment-resource#6](https://github.com/concourse/bosh-deployment-resource/issues/6), + [concourse/cf-resource#20](https://github.com/concourse/cf-resource/pull/20), + [concourse/cf-resource#25](https://github.com/concourse/cf-resource/pull/25), + [concourse/git-resource#210](https://github.com/concourse/git-resource/pull/210) + +* Make resource actions reentrant so that we no longer receive `unexpected EOF` + errors when reattaching to an in-flight build whose resource action completed + while we weren't attached: + [concourse/concourse#1580](https://github.com/concourse/concourse/issues/1580) + +* Support for showing icons for resources in the web UI: + [concourse/concourse#788](https://github.com/concourse/concourse/issues/788), + [concourse/concourse#3220](https://github.com/concourse/concourse/pull/3220), + [concourse/concourse#3581](https://github.com/concourse/concourse/pull/3581) + +* Standardize TLS configuration so every resource doesn't implement their own + way: [concourse/rfcs#9](https://github.com/concourse/rfcs/issues/9) + + +## Glossary + +* **Prototype**: an implementation of the interface defined by this proposal, + typically packaged as an OCI container image. + + Examples: + + * a `git` prototype for interacting with commits and branches in a repo. + * a `time` prototype for performing interval triggers. + +* **Message**: an operation to perform against the object. Corresponds to a + command to run in the container image. + + Examples: + + * `check` + * `get` + +* **Object**: a logical entity, encoded as a JSON object, acted on and emitted + by a prototype in response to messages. + + Examples: + + * `{"uri":"https://github.com/concourse/rfcs"}` + * `{"ref":"e4be0b367d7bd34580f4842dd09e7b59b6097b25"}` + +* **Metadata**: additional information about an object to surface to users. + + Examples: + + * `[{"name":"committer","value":"Alex Suraci"}]` + +* **Bits**: a directory containing arbitrary data. + + Examples: + + * source code checked out from a repo + * compiled artifacts created by a task or another prototype + +* **Resource**: a **prototype** which implements the following messages: + + * `check`: discover objects representing versions, in order + * `get`: fetch an object version + + +## Interface Types + +```go +// Object is an object receiving messages and being emitted in response to +// messages. +type Object map[string]interface{} + +// InfoRequest is the payload written to stdin for the `./info` script. +type InfoRequest struct { + // The properties specifying the object to act on. + Object Object `json:"object"` + + // Configuration for handling TLS. + TLS TLSConfig `json:"tls,omitempty"` +} + +// InfoResponse is the payload written to stdout from the `./info` script. +type InfoResponse struct { + // The version of the prototype interface that this prototype conforms to. + InterfaceVersion string `json:"interface_version"` + + // An optional icon to show to the user. + // + // Icons must be namespaced by in order to explicitly reference an icon set + // supported by Concourse, e.g. 'mdi:' for Material Design Icons. + Icon string `json:"icon,omitempty"` + + // The messages supported by the object. + Messages []string `json:"messages,omitempty"` +} + +// MessageRequest is the payload written to stdin for a message. +type MessageRequest struct { + // The properties specifying the object to act on. + Object Object `json:"object"` + + // Configuration for handling TLS. + TLS TLSConfig `json:"tls,omitempty"` + + // Path to a file into which the message handler must write its response. + ResponsePath string `json:"response_path"` +} + +// MessageResponse is written to the `response_path` for each object returned +// by the message. Multiple responses may be written to the same file, +// concatenated as a JSON stream. +type MessageResponse struct { + // The identifying properties for the object. + Object Object `json:"object"` + + // Metadata to associate with the properties. Shown to the user. + Metadata []MetadataField `json:"metadata,omitempty"` +} + +// TLSConfig captures common configuration for communicating with servers over +// TLS. +type TLSConfig struct { + // An array of CA certificates to trust. + CACerts []string `json:"ca_certs,omitempty"` + + // Skip certificate verification, effectively making communication insecure. + SkipVerification bool `json:"skip_verification,omitempty"` +} + +// MetadataField represents a named bit of metadata associated to an object. +type MetadataField struct { + Name string `json:"name"` + Value string `json:"value"` +} +``` + + +## Prototype Info + +Prior to sending any message, the default command (i.e. +[`CMD`](https://docs.docker.com/engine/reference/builder/#cmd)) for the image +will be executed with an `InfoRequest` piped to `stdin`. This request contains +an **object**. + +The command must write an `InfoResponse` to `stdout` in response. This response +specifies the prototype interface version that the prototype conforms to, an +optional icon to show in the UI, and the messages supported by the given +object. + +### Example **info** request/response + +Request sent to `stdin`: + +```json +{ "object": { "uri": "https://github.com/concourse/concourse" } } +``` + +Response written to `stdout`: + +```json +{ + "interface_version": "1.0", + "icon": "mdi:github-circle", + "messages": ["check"] +} +``` + + +## Prototype Messages + +A message handler is invoked by executing the command named after the message +with a JSON-encoded `MessageRequest` piped to `stdin`. This request contains +the **object** which is conceptually receiving the message. + +All message commands will be run in a working directory layout containing the +**bits** provided to the message handler, as well as empty directories to which +bits should be written by the message handler. + +A message handler writes its `MessageResponse`(s) to the file path specified by +`response_path` in the `MessageRequest`. This path may be relative to the +initial working directory. + +How this response is interpreted depends on the message, but typically there +should be one response for each object affected (`put`, `delete`), discovered +(`check`), or fetched (`get`). + +### Example **message** request/response + +Request sent to `stdin`: + +```json +{ + "object": { + "uri": "https://github.com/concourse/rfcs", + "branch": "master" + }, + "response_path": "../response/response.json" +} +``` + +Response written to `../response/response.json`: + +```json +{ + "object": {"ref": "e4be0b367d7bd34580f4842dd09e7b59b6097b25"}, + "metadata": [ { "name": "message", "value": "init" } ] +} +{ + "object": {"ref": "5a052ba6438d754f73252283c6b6429f2a74dbff"}, + "metadata": [ { "name": "message", "value": "add not-very-useful-yet readme" } ] +} +{ + "object": {"ref": "2e256c3cb4b077f6fa3c465dd082fa74df8fab0a"}, + "metadata": [ { "name": "message", "value": "start fleshing out RFC process" } ] +} +``` + +This response would be typical of a `check` that ran against a `git` repository +that had three commits. + + +## Object Cloning + +Technically, all parts of the prototocol have been specified, but there is a +particular usage pattern that is worth outlining here as it relates to the +'prototype' terminology and may be common across a few use cases for this +protocol. + +**Objects** are just arbitrary JSON data. This data is consumed by a prototype +action and may be emitted by the prototype in response to a message. + +One example of a message that will yield new objects is `check` (part of the +[resource interface](https://github.com/concourse/rfcs/pull/26)). This message +will be sent to prototypes that are used as a resource in a Concourse pipeline. + +Per the resource interface, the `check` message returns an object for each +version. However, these objects aren't self-contained - instead, they are an +implicit *clone* of the original object, with the returned object's fields +merged in. + +For example, the `git` prototype may be initially receive the following +user-configured properties in a `check` message: + +```json +{ + "object": { + "uri": "https://github.com/concourse/rfcs", + "branch": "master" + } +} +``` + +Let's say the `check` handler responds with the following objects, representing +two commits in the repository (with `metadata` omitted for brevity): + +```json +{ + "object": { + "ref": "e4be0b367d7bd34580f4842dd09e7b59b6097b25" + } +} +{ + "object": { + "ref": "5a052ba6438d754f73252283c6b6429f2a74dbff" + } +} +``` + +Per the resource interface, these object properties would be saved as versions +of the resource in the order returned by `check`. + +When it comes time to fetch a version in a build, the `git` prototype would +receive a `get` message with the following object: + +```json +{ + "object": { + "uri": "https://github.com/concourse/rfcs", + "branch": "master", + "ref": "e4be0b367d7bd34580f4842dd09e7b59b6097b25" + } +} +``` + +Note that the object properties have been merged into the original set of +properties. + +## Migrating Resources to Prototypes + +All existing Concourse resources can map directly to a prototype +implementation, as they implement a set of commands which act on the resource +type's defining entity. + +Concourse will support use of resources alongside prototypes. When a resource +type does not support the new 'info' request flow it will be assumed to be v1. + +* `resource_types`: v1 resources +* `prototypes:` prototypes! + +## Direct Pipeline Usage + +Now for the fun part! + +So far this proposal has been waxing theoretical and using a bunch of +fancy-pants terminology. But how will it look when it goes up to the user? + +For starters, I *don't* think we should necessarily bubble up the same +terminology. Users writing pipelines may appreciate the mental model underlying +everything, but it's more likely that when they're writing build plans they +just want to run something. Not "send something a message." + +There are a lot of options here. Some of them add a bit of syntactic sugar +(e.g. `oci-image.build`) which is uncharted territory but does feel a bit +nicer. + +```yaml +prototypes: +- name: oci-image + type: registry-image + source: + repository: vito/oci-image-prototype + +jobs: +- name: build-image + plan: + - get: concourse + + # manually run 'get' action anonymously + # + # NOTE: has no checking or caching semantics + - run: get + type: git + params: + uri: https://github.com/concourse/concourse + ref: v5.0.0 + + # 'get' handlers always produce an output named 'resource' + output_mapping: {resource: concourse} + + - run: build + type: oci-image + params: {context: concourse} + outputs: [image] +``` + +### Running a Message Handler + +A new `run` step type will be introduced. It takes the name of a message to +send, and a `type` denoting the prototype to use: + +```yaml +prototypes: +- name: some-prototype + type: registry-image + source: + repository: vito/some-prototype + +jobs: +- name: run-prototype + plan: + - run: some-message + type: some-prototype +``` + +So far this example is not very useful. There aren't many things I can think to +run that don't take any inputs or configuration or produce any outputs. Let's +try giving it an input and collecting its resulting output. This can be done +with `inputs`/`input_mapping` and `outputs`/`output_mapping`: + +```yaml +prototypes: +- name: oci-image + type: registry-image + source: + repository: vito/oci-image-prototype + +resources: +- name: my-image-src + type: git + source: + uri: https://example.com/my-image-src + +jobs: +- name: build-my-image + plan: + - get: my-image-src + - run: build + type: oci-image + input_mapping: {context: my-image-src} + outputs: [image] +``` + +For this example we've also snuck in a pipeline resource definition +(`my-image-src`). Under the hood, resources are just prototypes which support +`check` and `get` and have the following behavior: + +* A resource prototype's `check` handler finds new versions of the resource. +* A resource prototype's `get` handler produces an output directory named + `resource`. +* Concourse will `check` periodically to find new versions of the resource. + * When a job has a `get` step with `trigger: true`, Concourse will + automatically trigger new builds of the job when new versions of the + resource are found. +* The `get` step will execute the `get` handler and map its `resource` output + to the name of the resource, making it available to subsequent steps. +* Concourse will cache the `resource` output so that subsequent builds do not + have to run the `get` handler on a worker which has already fetched the same + version. + +Removing the syntactic sugar, you could also execute the `get` handler +manually, like so: + +```yaml +run: get +type: git +params: + uri: https://example.com/my-image-src + ref: v1.5.0 +output_mapping: + resource: my-image-src +``` + +This will circumvent the caching and auto-triggering and just run the `get` +handler every time the build runs, as if it were a task. + + +## Out of scope + +* [Richer metadata](https://github.com/concourse/concourse/issues/310) - this + hasn't gained much traction and probably needs more investigation before it + can be incorporated. This should be easy enough to add as a later RFC. + + +## Etymology + +* The root of the name comes from [Prototype-based + programming](https://en.wikipedia.org/wiki/Prototype-based_programming), a + flavor of object-oriented programming, which the concepts introduced in this + proposal mirror somewhat closely. There are also some hints of actor-based + programming - actors handling messages and emitting more actors - but that + felt like more of a stretch. + + This is also an opportunity to frame Concourse as "object-oriented CI/CD," if + we want to. + +* We've been conflating the terms 'resource' and 'resource type' pretty + everywhere which seems like it would be confusing to newcomers. For example, + all of our repos are typically named 'X-resource' when really the repo + provides a resource *type*. + + Adopting a new name which still has the word 'type' in it feels like it fixes + this problem, while preserving 'resource' terminology for its original use + case (pipeline resources). + +* It's a pun: 'protocol type' => 'prototype'. + +* 'Prototype' has a pretty nifty background in aerospace engineering. This is a + completely different meaning, but we never use it that way, so it doesn't + seem like there should be much cause for confusion. + + +## Open Questions + +* Is this terminology unrelatable? + + +[rfc-1]: https://github.com/concourse/rfcs/pull/1 +[rfc-1-comment]: https://github.com/concourse/rfcs/pull/1#issuecomment-477749314 +[rfc-24]: https://github.com/concourse/rfcs/pull/24 From 8ca3c12431b5dcab57b35a2fa8776f7cd181cd60 Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Thu, 19 Sep 2019 16:40:18 -0400 Subject: [PATCH 30/34] defer to resource prototypes RFC Signed-off-by: Alex Suraci --- 037-prototypes/proposal.md | 107 ++++++------------------------------- 1 file changed, 15 insertions(+), 92 deletions(-) diff --git a/037-prototypes/proposal.md b/037-prototypes/proposal.md index d1cbbee8..7828b88b 100644 --- a/037-prototypes/proposal.md +++ b/037-prototypes/proposal.md @@ -360,65 +360,7 @@ receive a `get` message with the following object: Note that the object properties have been merged into the original set of properties. -## Migrating Resources to Prototypes - -All existing Concourse resources can map directly to a prototype -implementation, as they implement a set of commands which act on the resource -type's defining entity. - -Concourse will support use of resources alongside prototypes. When a resource -type does not support the new 'info' request flow it will be assumed to be v1. - -* `resource_types`: v1 resources -* `prototypes:` prototypes! - -## Direct Pipeline Usage - -Now for the fun part! - -So far this proposal has been waxing theoretical and using a bunch of -fancy-pants terminology. But how will it look when it goes up to the user? - -For starters, I *don't* think we should necessarily bubble up the same -terminology. Users writing pipelines may appreciate the mental model underlying -everything, but it's more likely that when they're writing build plans they -just want to run something. Not "send something a message." - -There are a lot of options here. Some of them add a bit of syntactic sugar -(e.g. `oci-image.build`) which is uncharted territory but does feel a bit -nicer. - -```yaml -prototypes: -- name: oci-image - type: registry-image - source: - repository: vito/oci-image-prototype - -jobs: -- name: build-image - plan: - - get: concourse - - # manually run 'get' action anonymously - # - # NOTE: has no checking or caching semantics - - run: get - type: git - params: - uri: https://github.com/concourse/concourse - ref: v5.0.0 - - # 'get' handlers always produce an output named 'resource' - output_mapping: {resource: concourse} - - - run: build - type: oci-image - params: {context: concourse} - outputs: [image] -``` - -### Running a Message Handler +## Pipeline Usage A new `run` step type will be introduced. It takes the name of a message to send, and a `type` denoting the prototype to use: @@ -455,48 +397,28 @@ resources: source: uri: https://example.com/my-image-src +- name: my-image + type: registry-image + source: + repository: example/my-image + username: some-username + password: some-password + jobs: -- name: build-my-image +- name: build-and-push plan: - get: my-image-src - run: build type: oci-image input_mapping: {context: my-image-src} outputs: [image] + - put: my-image + params: {image: image/image.tar} ``` -For this example we've also snuck in a pipeline resource definition -(`my-image-src`). Under the hood, resources are just prototypes which support -`check` and `get` and have the following behavior: - -* A resource prototype's `check` handler finds new versions of the resource. -* A resource prototype's `get` handler produces an output directory named - `resource`. -* Concourse will `check` periodically to find new versions of the resource. - * When a job has a `get` step with `trigger: true`, Concourse will - automatically trigger new builds of the job when new versions of the - resource are found. -* The `get` step will execute the `get` handler and map its `resource` output - to the name of the resource, making it available to subsequent steps. -* Concourse will cache the `resource` output so that subsequent builds do not - have to run the `get` handler on a worker which has already fetched the same - version. - -Removing the syntactic sugar, you could also execute the `get` handler -manually, like so: - -```yaml -run: get -type: git -params: - uri: https://example.com/my-image-src - ref: v1.5.0 -output_mapping: - resource: my-image-src -``` - -This will circumvent the caching and auto-triggering and just run the `get` -handler every time the build runs, as if it were a task. +This example also makes use of resources, which are also implemented as +prototypes with special pipeline pipeline semantics and step syntax. These +"resource prototypes" are described in [RFC #38][rfc-38]. ## Out of scope @@ -542,3 +464,4 @@ handler every time the build runs, as if it were a task. [rfc-1]: https://github.com/concourse/rfcs/pull/1 [rfc-1-comment]: https://github.com/concourse/rfcs/pull/1#issuecomment-477749314 [rfc-24]: https://github.com/concourse/rfcs/pull/24 +[rfc-38]: https://github.com/concourse/rfcs/pull/38 From b1bada0dfcb6ed2e98827e7585541322c988d3fa Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Tue, 24 Sep 2019 10:44:17 -0400 Subject: [PATCH 31/34] reword Signed-off-by: Alex Suraci --- 037-prototypes/proposal.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/037-prototypes/proposal.md b/037-prototypes/proposal.md index 7828b88b..b79c231f 100644 --- a/037-prototypes/proposal.md +++ b/037-prototypes/proposal.md @@ -147,7 +147,7 @@ type Object map[string]interface{} // InfoRequest is the payload written to stdin for the `./info` script. type InfoRequest struct { - // The properties specifying the object to act on. + // The object to act on. Object Object `json:"object"` // Configuration for handling TLS. @@ -171,7 +171,7 @@ type InfoResponse struct { // MessageRequest is the payload written to stdin for a message. type MessageRequest struct { - // The properties specifying the object to act on. + // The object to act on. Object Object `json:"object"` // Configuration for handling TLS. @@ -185,10 +185,10 @@ type MessageRequest struct { // by the message. Multiple responses may be written to the same file, // concatenated as a JSON stream. type MessageResponse struct { - // The identifying properties for the object. + // The object. Object Object `json:"object"` - // Metadata to associate with the properties. Shown to the user. + // Metadata to associate with the object. Shown to the user. Metadata []MetadataField `json:"metadata,omitempty"` } From c14e284ed7733f442b3d99d1335311517bd5f9da Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Wed, 16 Oct 2019 11:15:36 -0400 Subject: [PATCH 32/34] don't provide TLS configuration to Info Signed-off-by: Alex Suraci --- 037-prototypes/proposal.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/037-prototypes/proposal.md b/037-prototypes/proposal.md index b79c231f..240ea15b 100644 --- a/037-prototypes/proposal.md +++ b/037-prototypes/proposal.md @@ -149,9 +149,6 @@ type Object map[string]interface{} type InfoRequest struct { // The object to act on. Object Object `json:"object"` - - // Configuration for handling TLS. - TLS TLSConfig `json:"tls,omitempty"` } // InfoResponse is the payload written to stdout from the `./info` script. From 7c6073b42c48b8555079e82149e71529832d242b Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Wed, 16 Oct 2019 11:17:26 -0400 Subject: [PATCH 33/34] Info writes its response to a file Signed-off-by: Alex Suraci --- 037-prototypes/proposal.md | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/037-prototypes/proposal.md b/037-prototypes/proposal.md index 240ea15b..38be3e0a 100644 --- a/037-prototypes/proposal.md +++ b/037-prototypes/proposal.md @@ -149,9 +149,13 @@ type Object map[string]interface{} type InfoRequest struct { // The object to act on. Object Object `json:"object"` + + // Path to a file into which the prototype must write its InfoResponse. + ResponsePath string `json:"response_path"` } -// InfoResponse is the payload written to stdout from the `./info` script. +// InfoResponse is the payload written to the `response_path` in response to an +// InfoRequest. type InfoResponse struct { // The version of the prototype interface that this prototype conforms to. InterfaceVersion string `json:"interface_version"` @@ -174,7 +178,7 @@ type MessageRequest struct { // Configuration for handling TLS. TLS TLSConfig `json:"tls,omitempty"` - // Path to a file into which the message handler must write its response. + // Path to a file into which the message handler must write its MessageResponses. ResponsePath string `json:"response_path"` } @@ -214,20 +218,25 @@ Prior to sending any message, the default command (i.e. will be executed with an `InfoRequest` piped to `stdin`. This request contains an **object**. -The command must write an `InfoResponse` to `stdout` in response. This response -specifies the prototype interface version that the prototype conforms to, an -optional icon to show in the UI, and the messages supported by the given -object. +The command must write an `InfoResponse` to the file path specified by +`response_path` in the `InfoRequest`. This response specifies the prototype +interface version that the prototype conforms to, an optional icon to show in +the UI, and the messages supported by the given object. ### Example **info** request/response Request sent to `stdin`: ```json -{ "object": { "uri": "https://github.com/concourse/concourse" } } +{ + "object": { + "uri": "https://github.com/concourse/concourse" + }, + "response_path": "../info/response.json" +} ``` -Response written to `stdout`: +Response written to `../info/response.json`: ```json { From aaf3d6e913cafb27ba971ee9d218b9bbb3bed3a6 Mon Sep 17 00:00:00 2001 From: Alex Suraci Date: Sat, 6 Jun 2020 16:32:56 -0400 Subject: [PATCH 34/34] add initial support for encrypted responses Signed-off-by: Alex Suraci --- 037-prototypes/proposal.md | 135 ++++++++++++++++++++++++++++++++++++- 1 file changed, 134 insertions(+), 1 deletion(-) diff --git a/037-prototypes/proposal.md b/037-prototypes/proposal.md index 38be3e0a..7cbd0599 100644 --- a/037-prototypes/proposal.md +++ b/037-prototypes/proposal.md @@ -175,9 +175,12 @@ type MessageRequest struct { // The object to act on. Object Object `json:"object"` - // Configuration for handling TLS. + // Configuration for establishing TLS connections. TLS TLSConfig `json:"tls,omitempty"` + // A base64-encoded 32-byte encryption key for use with AES-GCM. + Encryption EncryptionConfig `json:"encryption,omitempty"` + // Path to a file into which the message handler must write its MessageResponses. ResponsePath string `json:"response_path"` } @@ -189,6 +192,9 @@ type MessageResponse struct { // The object. Object Object `json:"object"` + // Encrypted fields of the object. + Encrypted EncryptedObject `json:"encrypted"` + // Metadata to associate with the object. Shown to the user. Metadata []MetadataField `json:"metadata,omitempty"` } @@ -203,6 +209,28 @@ type TLSConfig struct { SkipVerification bool `json:"skip_verification,omitempty"` } +type EncryptionConfig struct { + // The encryption algorithm for the prototype to use. + // + // This value will be static, and changing it will imply a major bump to the + // Prototype protocol version. It is included here as a helpful indicator so + // that prototype authors don't have to guess at the payload. + Algorithm string `json:"algorithm"` + + // A base64-encoded 32-length key, unique to each message. + Key []byte `json:"key"` +} + +// EncryptedObject contains an AES-GCM encrypted JSON payload containing +// additional fields of the object. +type EncryptedObject struct { + // The base64-encoded encrypted payload. + Payload []byte `json:"payload"` + + // The base64-encrypted nonce. + Nonce []byte `json:"nonce"` +} + // MetadataField represents a named bit of metadata associated to an object. type MetadataField struct { Name string `json:"name"` @@ -300,6 +328,111 @@ This response would be typical of a `check` that ran against a `git` repository that had three commits. +## Encryption + +In order to use Prototypes for credential acquisition, there must be a way to +return object attributes which contain sensitive data without writing the data +to disk in plaintext. + +A Prototype's `MessageRequest` may contain an `EncryptionConfig` which +specifies the encryption algorithm to use and any other necessary data for use +with the algorithm (such as a key). + +The Prototypes protcol will only support one encryption algorithm at a time, +and if it needs to be changed, this will imply a **major** bump to the protocol +version. This is to encourage phasing out support for no-longer-adequate +security algorithms. + +The decision as to which algorithm to use for the initial version is currently +an open question, but the most likely candidate right now is AES-GCM, which is +the same algorithm used for database encryption in Concourse. Another candidate +may be NaCL. Note that whichever one we choose must be available in various +languages so that Prototype authors aren't restricted to any particular library +or language. + +Assuming AES-GCM, the `EncryptionConfig` in the request will include a `key` +field containing a base64-encoded 32-length key, and a `nonce_size` field +indicating the size of the nonce necessary to encrypt/decrypt: + +```json +{ + "object": { + "uri": "https://vault.example.com", + "client_token": "01234567889abcdef" + }, + "encryption": { + "algorithm": "AES-GCM", + "key": "aXzsY7eK/Jmn4L36eZSwAisyl6Q4LPFIVSGEE4XH0hA=", + "nonce_size": 12 + } +} +``` + + +It is the responsibility of the Prototype implementation to generate a nonce +value of the specified length. The Prototype would then marshal a JSON object +containing fields to be encrypted, encrypt it with the key and nonce, and +return the encrypted payload along with the nonce value in a `EncryptedObject` +in the `MessageResponse` - both as base64-encoded values: + +```json +{ + "object": { + "public": "fields" + }, + "encrypted": { + "nonce": "6rYKFHXh43khqsVs", + "payload": "St5pRZumCx75d2x2s3vIjsClUi9DqgnIoG2Slt2RoCvz" + } +} +``` + +The encrypted payload above is `{"some":"secret"}`, so the above response +ultimately describes the following object: + +```json +{ + "public": "fields", + "some": "secret" +} +``` + +### Rationale for encryption technique + +A few alternatives to this approach were considered: + +* Initially, the use of HTTPS was appealing as it would avoid placing data on + disk entirely. + + However this is a much more complicated architecture that raises many more + questions: + + * What are the API endpoints? + * How are the requests routed? + * How is TLS configured? + * How is the response delivered? + * Regular HTTP response? If the `web` node detaches, how can we re-attach? + * Callbacks? What happens when the callback endpoint is down? + * Is it safe to retry requests? + +* We could write the responses to `tmpfs` to ensure they only ever exist in + RAM. + + The main downside is that operators would have to make sure that swap is + disabled so that data is never written to disk. This seems like a safe + assumption for Kubernetes but seems like an easy mistake to make for + operators managing their own VMs. + +One advantage of the current approach is that it tells Concourse which fields +can be public and which fields contain sensitive information, while requiring +the sensitive information to be encrypted. + +While this could be supported either way by specifying a field such as +`expose":["public"]`, it seems valuable to force Prototype authors to "do the +right thing" and encrypt the sensitive data, rather than allowing them to +simply hide it from the UI. + + ## Object Cloning Technically, all parts of the prototocol have been specified, but there is a