[Proposal] - Bootstrapping + support for different resource types

[chungers]

There are more types of resources in a computing infrastructure landscape than just instances (compute). The current focus on managing instances have led to the development of top level concepts like groups, instances, and flavors. While these primitives and their operations (rolling update, scale, destroy) are quite general, we need to see how they can be applied to other resource types and if they don't fit, define new abstractions and models where they make sense.

The resource types that are important and need to be addressed include

• networks
• volumes
• security groups / firewalls
• load balancers

As state convergence of more resource types are addressed, it will also solve the bootstrapping problem. Bootstrapping is defined as the process to set up the environment enough such that InfraKit can then take over and ensure the convergence of infrastructure state to user specification. Carried to the extreme, the bootstrapping process can be reduced to the creation of a single node, the seed, from which everything else will be created, as InfraKit drives the entire infrastructure towards the user's specification.

[chungers]

In terms of implementation, I see a few immediate tasks:

• We need to define a schema for specifying a collection of resources. This specification will also need to support dependencies amongst resources to be created. This will be composed into a larger specification document with the existing specifications of groups.
• Specify a new plugin type / API for creating different resources. Do we want a single plugin that can handle different types of resources (e.g. subnets, security groups) or one for each type? My sense is the former is simpler and better. We can call it the Resource plugin.
• Implement an engine that can perform dependency analysis and create a plan of actions such as creating resources via calls to the resource plugin.

Thoughts? @nwt @fermayo

[nwt]

We need to define a schema for specifying a collection of resources. This specification will also need to support dependencies amongst resources to be created. This will be composed into a larger specification document with the existing specifications of groups.

For the collection of resources, I think we want something like

{
  "Resources": {
    "MyVpc": {
      "Plugin": "infrakit-resource-aws",
      "Type": "Vpc",
      "Properties": {
        "CreateVpcInput": {
          "CidrBlock": "10.0.0.0/16"
        }
      }
    },
    "MySecurityGroup": {
      "Plugin": "infrakit-resource-aws",
      "Type": "SecurityGroup",
      "Properties": {
        "CreateSecurityGroupInput": {
          "GroupName": "MySecurityGroup",
          "VpcId":  "{{.Resources.MyVpc.ID}}"
        }
      }
    },
}

where {{.Resources.MyVpc.ID}} is a dynamic value referring to the provider-assigned
identifier for that resource. (It also indicates a dependency.)

Specify a new plugin type / API for creating different resources. Do we want a single plugin that can handle different types of resources (e.g. subnets, security groups) or one for each type? My sense is the former is simpler and better. We can call it the Resource plugin.

As you can see from the schema above, I also prefer a single plugin. Here's a Resource plugin SPI that can support it. (With the exception of Type/resourceType, it's a subset of the Instance plugin SPI.)

package resource

type ID string

type Description struct {
	ID   ID
	Tags map[string]string
}

type Spec struct {
	Type       string
	Properties *json.RawMessage
	Tags       map[string]string
}

type Plugin interface {
	Validate(req json.RawMessage) error
	Provision(spec Spec) (*ID, error)
	DescribeResources(resourceType string, tags map[string]string) ([]Description, error)
}

Implement an engine that can perform dependency analysis and create a plan of actions such as creating resources via calls to the resource plugin.

Are we ready for a pull request, or should we settle the schema and SPI first?

[chungers]

type Plugin interface {
	Validate(req json.RawMessage) error
	Provision(spec Spec) (*ID, error)
	DescribeResources(resourceType string, tags map[string]string) ([]Description, error)
}

I think we should have a Destroy(*ID) as well? The semantic of updates isn't obvious to me right now, but I think we can start with this Go interface?

Are we ready for a pull request, or should we settle the schema and SPI first?

I think so. Let's start with a PR and we can work on that iteratively? Curious to see about the package layout, etc.