Config template format does not work well in large environments #887

Closed
jjongsma opened this Issue Feb 14, 2014 · 12 comments

Projects

None yet

6 participants

@jjongsma

The strict JSON template format does not work well for managing hundreds of images that need to share some common pieces (builders, etc). It is missing capabilities for includes and comments, both of which are critical in large environments. Without these, maintaining a large library of server images is a painful exercise in undocumented copy and paste.

Currently we are using a python wrapper around Packer that parses our own YAML config files (with an "!include" constructor extension) and converts the results to a throwaway JSON template for each build. I can't imagine using Packer without this right now.

@mitchellh
Owner

This is one of the primary reasons we chose JSON as the configuration format: it is highly convenient to write a script to generate the configuration. I think your approach is correct.

We're considering other formats (in addition to JSON), but aren't sure what the benefits are over letting the user choose the best tool for this (Ruby, Python, etc.). What do you think?

@mitchellh mitchellh closed this Feb 21, 2014
@jjongsma

Ongoing maintenance and simplicity is my main concern. I wouldn't expect the config files to be everything for everybody, but I think there's a small but important set of features everyone will need if they're maintaining more than 10 images or so (mainly includes and comments.) Asking every user to hand-roll their own configuration generators in order to meet that baseline increases the barrier to entry and also makes it hard to share build configurations.

For me, this minimum require feature set resulted in a YAML processor that allows configurations that look like this, which makes it much easier to do mass updates of multiple images:

variables:
  image_name: karaf-3_0_0
  ssh_name: ubuntu
  ssh_pass: ubuntu
  hostname: ubuntu
  ec2_instance_type: m1.medium
  disk_size: 5000
  karaf_version: 1.5.1
builders:
  - !include ../../builders/ubuntu-12.04-server-amd64/ubuntu-12.04-server-amd64.kvm.yaml
  - !include ../../builders/ubuntu-12.04-server-amd64/ubuntu-12.04-server-amd64.aws.yaml
  - !include ../../builders/ubuntu-12.04-server-amd64/ubuntu-12.04-server-amd64.docker.yaml
provisioners:
  - !include ../../provisioners/ubuntu-12.04-base/ubuntu-12.04-base.yaml
  - !include ../../provisioners/ops-deploy/ops-deploy.yaml
  - !include ../../provisioners/app-java7/app-java7.yaml
  - type: file
    source: apps/karaf-3.0.0/files/
    destination: /tmp/provision
  - type: shell
    environment_vars: [ 'KARAF_VERSION={{user `karaf_version`}}' ]
    script: apps/karaf-3.0.0/karaf-3.0.0.sh 

Basically it is just standard YAML with an !include extension, and some post-processing that flattens the provisioners section in order to allow multiple provisioners in those includes before exporting to JSON.

@mitchellh
Owner

Thanks. This helps. And I'm not asking everyone to handroll a config generator. I'm rather hoping instead someone will care enough to publish one that can be shared by people.

@jjongsma

I'm happy to share what I'm doing, but the downside to a third party generator like this is that it still requires either a wrapper to execute packer, or a separate config generation step before running the build. This workflow is better for us than before, but I don't think it's ideal yet.

https://gist.github.com/jjongsma/9158547

With this script we run builds with build.py path/to/image (the argument is just a directory, it finds the JSON/YAML config file having the same name as the directory).

@mitchellh
Owner

Right, but Packer does let things come over stdin. So it would be pretty easy to do a

packer-gen-thing thing.yaml | packer build -

Or even:

packer-gen-thing thing.yaml | packer fix - | packer build -

@jjongsma

Ok, I streamlined my implementation for input piping.

https://gist.github.com/jjongsma/9160719

Is there a third-party scripts module you want to put this in?

@kontrafiktion

another advantage: you can "merge" nodes to reduce duplication:

below a &basic_builder is defined for all the keys that are common two both builders.
this is then used with <<: *basic_builder.
This yaml syntax is supported by Pythons's yaml implementation

It is just necessary to remove the "macros" key in the python script.

macros:
- &basic_builder
  boot_command:
  - <tab> text ks=http://{{ .HTTPIP }}:{{ .HTTPPort}}/ks6.cfg<enter>
  disk_size: 10140
  http_directory: http
  iso_checksum: '32c7695b97f7dcd1f59a77a71f64f2957dddf738'
  iso_checksum_type: sha1
  iso_url: 'http://mirrors.kernel.org/centos/6.5/isos/x86_64/CentOS-6.5-x86_64-bin-DVD1.iso'
  shutdown_command: echo 'vagrant'|sudo -S shutdown -P now
  ssh_password: vagrant
  ssh_username: vagrant
  ssh_wait_timeout: 10000s
  vm_name: centos65-docker

builders:
  - <<: *basic_builder
    type: vmware-iso
    tools_upload_flavor: linux
    guest_os_type: centos-64
    vmx_data:
      cpuid.coresPerSocket: '1'
      memsize: '512'
      numvcpus: '1'
  - <<: *basic_builder
    type: virtualbox-iso
    guest_additions_path: VBoxGuestAdditions_{{.Version}}.iso
    guest_os_type: RedHat_64
    vboxmanage:
    - - modifyvm
      - '{{.Name}}'
      - --memory
      - '512'
    - - modifyvm
      - '{{.Name}}'
      - --cpus
      - '1'
    virtualbox_version_file: .vbox_version
@kontrafiktion

the script and some documentation can be found here: http://victorvolle.wordpress.com/2014/06/06/245/

@gaborcsardi

Having standard ways to generate the templates would indeed be great. I am creating a bunch of very similar boxes, and planning to use box-cutter (https://github.com/box-cutter), so ideally I would just modify box-cutter templates via some include mechanism.

The YAML -> JSON converter is also very useful, but now I am between using that or the box-cutter templates or the converter.

Or I would need to write another converter to convert box-cutter JSON to YAML, and then include that into my own YAML. Might work, but it is probably error-prone and cumbersome.

@rickard-von-essen
Collaborator

@gaborcsardi Translating to/from yaml - json is simple, see http://jsontoyaml.com/#ruby

@jszwedko
Contributor

I'm in agreement that JSON is a poor choice for human generated configuration files (though I understand the motivation to use it if you expect your configuration to always be generated). I do, however, think it would be prudent to also support the use case of user written configuration and allow for YAML to also be used as a configuration language. I personally haven't had to do anything complicated enough to require generating configuration, but have done enough to miss some of the niceties of YAML like commenting.

@mitchellh would a pull request adding YAML support be considered?

@jszwedko
Contributor

Never mind, I see support for HCL is in the works!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment