Skip to content

Latest commit

 

History

History
316 lines (233 loc) · 13.9 KB

proposal.md

File metadata and controls

316 lines (233 loc) · 13.9 KB
Raw

Generalized Resources

This proposal adapts today's 'resource' interface ("v1") into a more general resource interface that is less specialized to the 'versioned artifacts' use case currently supported by Concourse pipelines.

The proposed interface is versioned, starting at 2.0. Each resource type will have its resource info fetched to discover its version.

Concourse will support use of mixed resource interface versions. When a resource type does not support the new 'info' request flow it will be assumed to be v1.

Resource actions have all been updated to fit the exact same request/response interface. This proposal defines four core actions: check, get, put, and delete.

Each action emits config fragments, which replace 'versions' from the v1 interface. By using a more general name the interface can be used for use cases beyond just versioning artifacts. See Interpreting the Resource Interface for more details.

Previous Discussions

  • RFC #1, now defunct, is similar to this proposal but had a concept of "spaces" baked into the interface. This concept has been decoupled from the resource interface and will be re-introduced in a later RFC that is compatible with v1 and v2 resources.
    • Recommended reading: this comment outlines the thought process that led to this RFC.
  • concourse/concourse#534 was the first 'new resource interface' proposal which pre-dated the RFC process.

Motivation

Glossary

  • Config: an arbitrarily nested JSON object containing user-provided configuration

    • Examples: {"uri":"https://github.com/concourse/concourse"}, {"interval":"10m"}

  • Config fragment: a smaller JSON object intended to be "spliced" into a config by assigning each field from the fragment into the config.

    • Examples: {"ref":"abcdef"}, {"branch":"develop"}

  • Bits: a directory containing arbitrary data

    • Examples: source code, compiled artifacts, etc.

  • Metadata: structured data associated to a config fragment containing information about the fragment that should be surfaced to the user

    • Examples: [{"name":"committer","value":"Alex Suraci"}]

  • Resource type: an implementation of the interface defined by this proposal, typically provided as a container image. Implements the following actions:

    • info: given a config, emit a response specifying the command to run for each action
    • check: given a config, emit config fragments
    • get: given a config, populate a directory with bits
    • put: given a config and a directory containing bits, create or update config fragments
    • delete: given a config and a directory containing bits, delete config fragments
    • Examples:
      • git-branches resource type for tracking branches in a repo
      • git resource type for tracking commits in a branch
      • github-status resource type for emitting build status notifications for commits
      • time resource type for doing timed job triggers

  • Resource: a resource type with a user-provided config, used together to represent external state.

Interface Types

// Config represents arbitrary user-specified configuration.
type Config map[string]interface{}

// ConfigFragment represents additional fields that can be spliced into a Config.
type ConfigFragment map[string]interface{}

// MetadataField represents a named bit of metadata associated to a ConfigFragment.
type MetadataField struct {
  Name  string `json:"name"`
  Value string `json:"value"`
}

// TLSConfig captures common configuration for communicating with servers over TLS.
type TLSConfig struct {
  // An array of CA certificates to trust.
  CAs []string `json:"ca_certs,omitempty"`

  // Skip certificate verification, effectively making communication insecure.
  SkipVerification bool `json:"skip_verification,omitempty"`
}

// InfoRequest is the payload written to stdin for the `./info` script.
type InfoRequest struct {
  // User-specified configuration.
  Config Config `json:"config"`
}

// InfoResponse is the payload written to stdout from the `./info` script.
type InfoResponse struct {
  // The version of the resource interface that this resource type conforms to.
  InterfaceVersion string `json:"interface_version"`

  // An optional icon name to show to the user when viewing the resource.
  //
  // Icons must be namespaced by in order to explicitly reference an icon set
  // supported by Concourse, e.g. 'mdi:' for Material Design Icons.
  Icon string `json:"icon,omitempty"`

  // The actions supported by the resource type.
  Actions struct {
    // Command to run when performing check actions.
    Check string `json:"check,omitempty"`

    // Command to run when performing get actions.
    Get string `json:"get,omitempty"`

    // Command to run when performing put actions.
    Put string `json:"put,omitempty"`

    // Command to run when performing delete actions.
    Delete string `json:"delete,omitempty"`
  } `json:"actions"`
}

// ActionRequest is the payload written to stdin for each action command.
type ActionRequest struct {
  // User-specified configuration.
  Config Config `json:"config"`

  // Configuration for handling TLS.
  TLS TLSConfig `json:"tls,omitempty"`

  // Path to a file into which the action must write its response.
  ResponsePath string `json:"response_path"`
}

// ActionResponse is written to the `response_path` by an action for each fragment affected by the action. Multiple respones may be written as a JSON stream.
type ActionResponse struct {
  // The fragment. May be used as an identifier, unique within the scope of a Config.
  Fragment ConfigFragment `json:"fragment"`

  // Metadata to associate with the fragment. Shown to the user.
  Metadata []MetadataField `json:"metadata,omitempty"`
}

Resource Info

Prior to running any action, Concourse will execute the default command (i.e. CMD) for the image with an InfoRequest piped to stdin.

The command must write an InfoResponse to stdout in response. This response specifies the resource interface version that the resource type conforms to, an optional icon to show in the UI, and the command to run for each supported resource action.

Example info request/response

Request sent to stdin:

{
  "config": {
    "uri": "https://github.com/concourse/concourse"
  }
}

Response written to stdout:

{
  "interface_version": "2.0",
  "icon": "mdi:github-circle",
  "actions": {
    "check": "/usr/bin/git-resource check",
    "get": "/usr/bin/git-resource get",
    "put": "/usr/bin/git-resource put",
    "delete": "/usr/bin/git-resource delete"
  }
}

Resource Actions

Each action is invoked with a JSON-encoded ActionRequest piped to stdin. This request contains the config and the path to which the response should be written. This path may be relative, and if so it is to be expanded from the current working directory.

All actions will be run in a working directory for the bits - either an empty directory to which bits should be written, or a directory containing the bits given to the action.

All actions respond by performing their side-effect and writing sequential ActionResponse JSON objects to the file path specified by response_path. How this response is interpreted depends on the action, but typically there should be one response for each external resource affected (put, delete), discovered (check), or fetched (get).

Example action request/response

Request sent to stdin:

{
  "config": {
    "uri": "https://github.com/concourse/rfcs",
    "branch": "master"
  },
  "response_path": "../response/response.json"
}

Response written to ../response/response.json:

{
  "fragment": {"ref": "e4be0b367d7bd34580f4842dd09e7b59b6097b25"},
  "metadata": [
    {
      "name": "message",
      "value": "init"
    }
  ]
}
{
  "fragment": {"ref": "5a052ba6438d754f73252283c6b6429f2a74dbff"},
  "metadata": [
    {
      "name": "message",
      "value": "add not-very-useful-yet readme"
    }
  ]
}
{
  "fragment": {"ref": "2e256c3cb4b077f6fa3c465dd082fa74df8fab0a"},
  "metadata": [
    {
      "name": "message",
      "value": "start fleshing out RFC process"
    }
  ]
}

This response would be typical of a check that ran against a repo that had three commits.

Interpreting the Resource Interface

The resource interface itself is now a general way of expressing interactions with external state. It is no longer based on versioning of artifacts.

Concourse will now codify things like "artifact resources" and "spatial resources" as interpretations of this general interface, composing resource types with one another via config fragments.

By leveraging composition instead of having a monolithic interface, this approach encourages narrowly scoped resource type implementations. Implementations that are small in scope are more likely to be correct and 'finished' at some point, and only become more powerful as Concourse enables new workflows at the pipeline-level.

For example, as an author of a git artifact resource type, I just have to implement a linear, versioned artifact interface. In git, linear versioning (assuming --first-parent) occurs on a branch, so that can be specified in config. What if users want to run against all branches? As a resource type author, that's not my problem! Let them use a git-branches spatial resource and compose it with my resource to provide the branch config.

By not baking the workflow directly into the interface, this also allows a single resource type implementation to be used for multiple interpretations. For example, notifications and triggers are complementary and can both be supported by a resource type that supports the required actions (put for notifications and check/get for triggers).

Four interpretations are outlined in the following proposals. The RFCs linked below show how the v2 interface can be used to support various pipeline workflows. The described workflows can also work with v1 resources, however, and so they are not explicitly dependent on the v2 interface. This is done intentionally so that our roadmap doesn't have to be so linear.

Artifact resources

artifacts:
- name: concourse
  type: git
  source:
    uri: https://github.com/concourse/concourse
    branch: master
  • check: return versions in order
  • get: fetch a version of the resource
  • put: push versions of a resource
  • delete: delete versions of a resource

Examples: git

Spatial resources

spaces:
- name: concourse-branches
  type: git-branches
  source:
    uri: https://github.com/concourse/concourse
  • check: return a fragment for each space, no order
  • get: fetch whatever metadata is useful for a given space
  • put: create or update spaces
  • delete: delete spaces

Examples: git-branches, github-prs

Notification resources

notifications:
- name: concourse-status
  type: github-status
  source:
    repository: concourse/concourse
    access_token: abcdef
  • check: not used
  • get: fetch bits pertaining to the notification
  • put: emit a notification
  • delete: not sure - clear github status?

Examples: github-status, slack

Trigger resources

triggers:
- name: every-10m
  type: time
  source: {interval: 10m}
  • check: check against last fragment used for job
  • get: fetch bits pertaining to the trigger
  • put: not useful
  • delete: not useful

Examples: time

Out of scope

  • Richer metadata - this hasn't gained much traction and probably needs more investigation before it can be incorporated. This should be easy enough to add as a later RFC.