Template file is json and hence doesnt have support for comments. #283

Closed
rluiten opened this Issue Aug 14, 2013 · 44 comments

Comments

Projects
None yet
@rluiten

rluiten commented Aug 14, 2013

Comments are something I already miss.
We discussed briefly on IRC and you suggested I raise issue.
You also suggested maybe can add support for rifled "comment": for some comment support.

I am in windows 7 64 bit, using packer v0.3.1

@mitchellh

This comment has been minimized.

Show comment
Hide comment
@mitchellh

mitchellh Aug 16, 2013

Member

Thanks for opening this issue. It'll at least hopefully allow Google to pick it up. For now, however, I think I'm not going to pursue this. Closing. Thanks

Member

mitchellh commented Aug 16, 2013

Thanks for opening this issue. It'll at least hopefully allow Google to pick it up. For now, however, I think I'm not going to pursue this. Closing. Thanks

@mitchellh mitchellh closed this Aug 16, 2013

@jtopper

This comment has been minimized.

Show comment
Hide comment
@jtopper

jtopper Sep 25, 2013

Contributor

That's a shame - provisioning via AWS is full of magic values like AMI and security group IDs, and it'd be nice to be able to clarify these inline, though I can't immediately see a nice way to do this in vanilla JSON and supporting config file pre-processing feels like the wrong approach too.

Contributor

jtopper commented Sep 25, 2013

That's a shame - provisioning via AWS is full of magic values like AMI and security group IDs, and it'd be nice to be able to clarify these inline, though I can't immediately see a nice way to do this in vanilla JSON and supporting config file pre-processing feels like the wrong approach too.

@jasongilman

This comment has been minimized.

Show comment
Hide comment
@jasongilman

jasongilman Nov 20, 2013

Agreed. There needs to be some kind of workaround at least. This feels like an issue which has to be fixed. I understand that JSON doesn't support comments but then it feels like it's the wrong choice for a config file.

Agreed. There needs to be some kind of workaround at least. This feels like an issue which has to be fixed. I understand that JSON doesn't support comments but then it feels like it's the wrong choice for a config file.

@dpehrson

This comment has been minimized.

Show comment
Hide comment
@dpehrson

dpehrson Nov 24, 2013

I just wanted to throw my two cents in here as well. Been using packer for the past week and absolutely love it, however as I have gotten further and further into making packer mirror existing technology, the need for commenting my template has grown, mostly because I have many builders with lots of "only" provisioners.

An alternative to comments would be some sort of way of splitting packer configs into multiple files that can be included in each other.

I just wanted to throw my two cents in here as well. Been using packer for the past week and absolutely love it, however as I have gotten further and further into making packer mirror existing technology, the need for commenting my template has grown, mostly because I have many builders with lots of "only" provisioners.

An alternative to comments would be some sort of way of splitting packer configs into multiple files that can be included in each other.

@mwhooker

This comment has been minimized.

Show comment
Hide comment
@mwhooker

mwhooker Nov 25, 2013

Collaborator

As with most things JSON, I find it necessary and with great value to generate it using some scripting language. That provides a much better way to structure, comment, and reuse parts of it.

On Nov 24, 2013, at 13:30, Daniel Pehrson notifications@github.com wrote:

I just wanted to throw my two cents in here as well. Been using packer for the past week and absolutely love it, however as I have gotten further and further into making packer mirror existing technology, the need for commenting my template has grown, mostly because I have many builders with lots of "only" provisioners.

An alternative to comments would be some sort of way of splitting packer configs into multiple files that can be included in each other.


Reply to this email directly or view it on GitHub.

Collaborator

mwhooker commented Nov 25, 2013

As with most things JSON, I find it necessary and with great value to generate it using some scripting language. That provides a much better way to structure, comment, and reuse parts of it.

On Nov 24, 2013, at 13:30, Daniel Pehrson notifications@github.com wrote:

I just wanted to throw my two cents in here as well. Been using packer for the past week and absolutely love it, however as I have gotten further and further into making packer mirror existing technology, the need for commenting my template has grown, mostly because I have many builders with lots of "only" provisioners.

An alternative to comments would be some sort of way of splitting packer configs into multiple files that can be included in each other.


Reply to this email directly or view it on GitHub.

@dpehrson

This comment has been minimized.

Show comment
Hide comment
@dpehrson

dpehrson Nov 25, 2013

@mwhooker That’s the technique I use with CloudFormation templates, I build them in a DSL of some sort and then serialize down to JSON. While it works well, it introduces another layer of processing which adds another step to the process.

I mostly wanted to see if there was any intent to address this issue directly before I go build another piece of machinery.

@mwhooker That’s the technique I use with CloudFormation templates, I build them in a DSL of some sort and then serialize down to JSON. While it works well, it introduces another layer of processing which adds another step to the process.

I mostly wanted to see if there was any intent to address this issue directly before I go build another piece of machinery.

@mwhooker

This comment has been minimized.

Show comment
Hide comment
@mwhooker

mwhooker Nov 25, 2013

Collaborator

That's the exact use case I was thinking of. See https://github.com/SimpleFinance/agi

That's the problem with json. Tries to be both human and machine readable but doesn't do either very well. (A worthy successor to XML!). At least it's simple

Plaese forgive typos -- sent via phone

On Nov 25, 2013, at 5:14, Daniel Pehrson notifications@github.com wrote:

@mwhooker That’s the technique I use with CloudFormation templates, I build them in a DSL of some sort and then serialize down to JSON. While it works well, it introduces another layer of processing which adds another step to the process.

I mostly wanted to see if there was any intent to address this issue directly before I go build another piece of machinery.


Reply to this email directly or view it on GitHub.

Collaborator

mwhooker commented Nov 25, 2013

That's the exact use case I was thinking of. See https://github.com/SimpleFinance/agi

That's the problem with json. Tries to be both human and machine readable but doesn't do either very well. (A worthy successor to XML!). At least it's simple

Plaese forgive typos -- sent via phone

On Nov 25, 2013, at 5:14, Daniel Pehrson notifications@github.com wrote:

@mwhooker That’s the technique I use with CloudFormation templates, I build them in a DSL of some sort and then serialize down to JSON. While it works well, it introduces another layer of processing which adds another step to the process.

I mostly wanted to see if there was any intent to address this issue directly before I go build another piece of machinery.


Reply to this email directly or view it on GitHub.

@mitchellh

This comment has been minimized.

Show comment
Hide comment
@mitchellh

mitchellh Nov 25, 2013

Member

I've mentioned before, I am interested in perhaps adding a meta-key such as "comment" that is ignored. But I'm still thinking about how to do this best.

Member

mitchellh commented Nov 25, 2013

I've mentioned before, I am interested in perhaps adding a meta-key such as "comment" that is ignored. But I'm still thinking about how to do this best.

@jvoorhis

This comment has been minimized.

Show comment
Hide comment
@jvoorhis

jvoorhis Nov 25, 2013

Contributor

This is a long shot, but would you be open to pull requests for alternate formats like YAML, TOML or .ini?

Contributor

jvoorhis commented Nov 25, 2013

This is a long shot, but would you be open to pull requests for alternate formats like YAML, TOML or .ini?

@dpehrson

This comment has been minimized.

Show comment
Hide comment
@dpehrson

dpehrson Nov 25, 2013

@mitchellh I was actually thinking that you could just allow any object node to have a comment property, such as:

{
  "builders": [
    {
      "comment": "The VMWare box we use for local development.",
      "type": "vmware",
      ...
    }
  ],
  "provisioners": [
    {
      "comment": "This provisioner is used to ensure the machine has a home .ssh directory before uploading keys.",
      "type": "shell",
      ...
    }
  ],
  "post-processors": [
    {
      "comment": "Only machines used for local development are vagranted.",
      "type": "vagrant",
      ...
    }
  ]
}

Additionally, it might not be a terrible idea to support an ignore property on any object node which has the effect of "commenting out" the item, such as:

{
  "builders": [
    {
      "comment": "We're not currently building this machine for some unknown reason.",
      "ignore": true,
      "type": "vmware",
      ...
    },
    ...
  ]
}

@mitchellh I was actually thinking that you could just allow any object node to have a comment property, such as:

{
  "builders": [
    {
      "comment": "The VMWare box we use for local development.",
      "type": "vmware",
      ...
    }
  ],
  "provisioners": [
    {
      "comment": "This provisioner is used to ensure the machine has a home .ssh directory before uploading keys.",
      "type": "shell",
      ...
    }
  ],
  "post-processors": [
    {
      "comment": "Only machines used for local development are vagranted.",
      "type": "vagrant",
      ...
    }
  ]
}

Additionally, it might not be a terrible idea to support an ignore property on any object node which has the effect of "commenting out" the item, such as:

{
  "builders": [
    {
      "comment": "We're not currently building this machine for some unknown reason.",
      "ignore": true,
      "type": "vmware",
      ...
    },
    ...
  ]
}
@jvoorhis

This comment has been minimized.

Show comment
Hide comment
@jvoorhis

jvoorhis Nov 25, 2013

Contributor

The per-node comment is a nice idea, but it prevents comments on particular attributes from being placed within close proximity to the attribute they are describing.

Contributor

jvoorhis commented Nov 25, 2013

The per-node comment is a nice idea, but it prevents comments on particular attributes from being placed within close proximity to the attribute they are describing.

@dpehrson

This comment has been minimized.

Show comment
Hide comment
@dpehrson

dpehrson Nov 25, 2013

@jvoorhis I think to a certain extent that as long as we are limited by JSON, it's probably the best we can do.

At a higher level, while I appreciate the simplicity of JSON, I believe it makes for a very poor configuration format for anything even marginally complex, which in my opinion packer is, and then some. In the long term, as packer is more widely used and is doing more and more, an alternative to JSON config will be needed whether it is provided by packer, or a third party.

@jvoorhis I think to a certain extent that as long as we are limited by JSON, it's probably the best we can do.

At a higher level, while I appreciate the simplicity of JSON, I believe it makes for a very poor configuration format for anything even marginally complex, which in my opinion packer is, and then some. In the long term, as packer is more widely used and is doing more and more, an alternative to JSON config will be needed whether it is provided by packer, or a third party.

@MaggieLeber

This comment has been minimized.

Show comment
Hide comment
@MaggieLeber

MaggieLeber Jan 14, 2014

Well, I see I'm not the only person struggling with this issue...starting with my very first template I'd hoped I could simply put comment values in and have them be ignored, but even that fails validation. A value namespace that could be used for unprocessed comment annotations would be most welcome.

Crockford's response to people who want comments to be valid JSON is

https://plus.google.com/+DouglasCrockfordEsq/posts/RK8qyGVaGSr

Jerk.

Well, I see I'm not the only person struggling with this issue...starting with my very first template I'd hoped I could simply put comment values in and have them be ignored, but even that fails validation. A value namespace that could be used for unprocessed comment annotations would be most welcome.

Crockford's response to people who want comments to be valid JSON is

https://plus.google.com/+DouglasCrockfordEsq/posts/RK8qyGVaGSr

Jerk.

@rclilly

This comment has been minimized.

Show comment
Hide comment
@rclilly

rclilly Jan 26, 2014

Why is this issue closed? It's a need that still needs a solution. I vote for either using something other than JSON for configuration or doing something along the lines of what @dpehrson suggested in #283 (comment)

rclilly commented Jan 26, 2014

Why is this issue closed? It's a need that still needs a solution. I vote for either using something other than JSON for configuration or doing something along the lines of what @dpehrson suggested in #283 (comment)

@kalefranz

This comment has been minimized.

Show comment
Hide comment
@kalefranz

kalefranz Feb 12, 2014

+1 for suggesting indexed by google, and
+1 for having some type of comment capability like @dpehrson's comment and ignore properties or @mwhooker's pre-process filtering

Given the Vagrantfile and Dockerfile precedents, is a non-json Packerfile so out of the question?

+1 for suggesting indexed by google, and
+1 for having some type of comment capability like @dpehrson's comment and ignore properties or @mwhooker's pre-process filtering

Given the Vagrantfile and Dockerfile precedents, is a non-json Packerfile so out of the question?

@MaggieLeber

This comment has been minimized.

Show comment
Hide comment
@MaggieLeber

MaggieLeber Feb 12, 2014

Here's my solution for now; other folks using Scala and Play may find it handy:

https://gist.github.com/MaggieLeber/8953905

Here's my solution for now; other folks using Scala and Play may find it handy:

https://gist.github.com/MaggieLeber/8953905

@rasa

This comment has been minimized.

Show comment
Hide comment
@rasa

rasa Feb 12, 2014

Collaborator

How about having the validator ignore any tag that starts (or ends, or both) with an underscore? That would allow extreme flexibility:

{
  "builders": [
    {
      "type": "vmware",
      "_comment1": "The VMWare box we use for local development.",
      "_comment2": "The VMWare box we use for local development.",
      ...
    }
  ],
  "provisioners": [
    {
      "sort_me_1st_comment_": "This provisioner is used to ensure the machine has a home .ssh               directory before uploading keys.",
      "sort_me_2nd_comment_": "",
      "type": "shell",
      ...
    }
  ],
  "post-processors": [
    {
      "_comment": "Only machines used for local development are vagranted.",
      "type": "vagrant",
      ...
    }
  ]
}
Collaborator

rasa commented Feb 12, 2014

How about having the validator ignore any tag that starts (or ends, or both) with an underscore? That would allow extreme flexibility:

{
  "builders": [
    {
      "type": "vmware",
      "_comment1": "The VMWare box we use for local development.",
      "_comment2": "The VMWare box we use for local development.",
      ...
    }
  ],
  "provisioners": [
    {
      "sort_me_1st_comment_": "This provisioner is used to ensure the machine has a home .ssh               directory before uploading keys.",
      "sort_me_2nd_comment_": "",
      "type": "shell",
      ...
    }
  ],
  "post-processors": [
    {
      "_comment": "Only machines used for local development are vagranted.",
      "type": "vagrant",
      ...
    }
  ]
}
@themartorana

This comment has been minimized.

Show comment
Hide comment
@themartorana

themartorana Feb 24, 2014

I've been following this issue. The Universal Configuration Language library (libucl) has been gaining some traction, and seems like an interesting and possibly useful solution to retain JSON output for config files while enabling the features people want here.

https://github.com/vstakhov/libucl

I've been following this issue. The Universal Configuration Language library (libucl) has been gaining some traction, and seems like an interesting and possibly useful solution to retain JSON output for config files while enabling the features people want here.

https://github.com/vstakhov/libucl

@tehgeekmeister

This comment has been minimized.

Show comment
Hide comment
@tehgeekmeister

tehgeekmeister Mar 25, 2014

👍 for YAML. Might be able to spend some work time on a pull request to make that happen, too. It solves this, and another problem, I have with using JSON, which is that I can't break long string literals into multiple lines.

👍 for YAML. Might be able to spend some work time on a pull request to make that happen, too. It solves this, and another problem, I have with using JSON, which is that I can't break long string literals into multiple lines.

@ccope

This comment has been minimized.

Show comment
Hide comment
@ccope

ccope Apr 29, 2014

I just ran into this problem while creating a Solaris template. There are some odd things that really should be documented in comments (like where to get the iso, since you can't automatically download it).

ccope commented Apr 29, 2014

I just ran into this problem while creating a Solaris template. There are some odd things that really should be documented in comments (like where to get the iso, since you can't automatically download it).

@timsutton timsutton referenced this issue in timsutton/osx-vm-templates May 26, 2014

Merged

`vagrant` user may need `vagrant` group #13

@timc3

This comment has been minimized.

Show comment
Hide comment
@timc3

timc3 May 27, 2014

I think just a prefix on the key would be good enough, and then packer ignores that key value pair.

timc3 commented May 27, 2014

I think just a prefix on the key would be good enough, and then packer ignores that key value pair.

@sparkprime

This comment has been minimized.

Show comment
Hide comment
@sparkprime

sparkprime Aug 28, 2014

Contributor

If you just want JSON with comments, you can preprocess your JSON with Jsonnet: http://google.github.io/jsonnet/doc/

This also gives you a whole lot of other language features that you may or may not want.

The same goes for Cloud Formation templates.

Contributor

sparkprime commented Aug 28, 2014

If you just want JSON with comments, you can preprocess your JSON with Jsonnet: http://google.github.io/jsonnet/doc/

This also gives you a whole lot of other language features that you may or may not want.

The same goes for Cloud Formation templates.

@sparkprime

This comment has been minimized.

Show comment
Hide comment
@sparkprime

sparkprime Aug 28, 2014

Contributor

Jsonnet allows you to break string literals onto multiple lines, but you can also use the importstr construct to read the string from a file, which is a lot more appropriate for things like embedding bash scripts / config files.

Contributor

sparkprime commented Aug 28, 2014

Jsonnet allows you to break string literals onto multiple lines, but you can also use the importstr construct to read the string from a file, which is a lot more appropriate for things like embedding bash scripts / config files.

@nchammas

This comment has been minimized.

Show comment
Hide comment
@nchammas

nchammas Oct 17, 2014

Contributor

+1

As a Packer n00b, I often find myself wanting to comment out/in blocks of a template during testing and development.

As a user, not being able to do this hurts.

Contributor

nchammas commented Oct 17, 2014

+1

As a Packer n00b, I often find myself wanting to comment out/in blocks of a template during testing and development.

As a user, not being able to do this hurts.

@nchammas

This comment has been minimized.

Show comment
Hide comment
@nchammas

nchammas Oct 20, 2014

Contributor

@MaggieLeber pointed to Douglas Crockford's comments on what to do if you want to comment JSON, and I think the suggestion is actually potentially useful:

Suppose you are using JSON to keep configuration files, which you would like to annotate. Go ahead and insert all the comments you like. Then pipe it through JSMin before handing it to your JSON parser.

Instead of coming up with a home-brew implementation of JSON comments or switching to a new config format altogether, why not just take Crockford's suggestion and allow people to comment their JSON templates like they were JavaScript files?

Packer can just take the file as-is, pass it through JSMin (or something equivalent) internally, and then proceed as normal.

Wouldn't that be the easiest, lowest friction way to allow template comments? What are the drawbacks?

Contributor

nchammas commented Oct 20, 2014

@MaggieLeber pointed to Douglas Crockford's comments on what to do if you want to comment JSON, and I think the suggestion is actually potentially useful:

Suppose you are using JSON to keep configuration files, which you would like to annotate. Go ahead and insert all the comments you like. Then pipe it through JSMin before handing it to your JSON parser.

Instead of coming up with a home-brew implementation of JSON comments or switching to a new config format altogether, why not just take Crockford's suggestion and allow people to comment their JSON templates like they were JavaScript files?

Packer can just take the file as-is, pass it through JSMin (or something equivalent) internally, and then proceed as normal.

Wouldn't that be the easiest, lowest friction way to allow template comments? What are the drawbacks?

@tehgeekmeister

This comment has been minimized.

Show comment
Hide comment
@tehgeekmeister

tehgeekmeister Oct 20, 2014

I kinda like @nchammas' suggestion, but also I would like to avoid another dependency, as there's already a ton of stuff for me to explain to coworkers to set up when they need to use packer with our internal tooling around it. I guess the homebrew formula (which we always use) could be updated to account for this, but I imagine it'll still be counter to what seems to be a design goal of no external dependencies.

I kinda like @nchammas' suggestion, but also I would like to avoid another dependency, as there's already a ton of stuff for me to explain to coworkers to set up when they need to use packer with our internal tooling around it. I guess the homebrew formula (which we always use) could be updated to account for this, but I imagine it'll still be counter to what seems to be a design goal of no external dependencies.

@nchammas

This comment has been minimized.

Show comment
Hide comment
@nchammas

nchammas Oct 20, 2014

Contributor

Yeah, it would be best if we avoided having an external dependency. We also don't want to implement our own JavaScript minifier if possible; that would be annoying.

What if we found an existing JavaScript minifier with a permissive license that we could just include as part of Packer? Users wouldn't have to know about it as it would just be used internally by Packer.

For example, YUI Compressor (BSD license) and Go-Yui (MIT license) look like promising options. There are also several Python options with permissive licenses, some of which don't depend on YUI and therefore don't need Java installed.

Could we just package one of those with Packer, allowing Packer to strip JSON templates of comments, without requiring users to manually manage this dependency?

@mitchellh What do you think of this approach?

Other potential options include simply porting the original JSMin from C to Go (it's quite short), or asking Crockford if he is open to relicensing JSMin under a more permissive license evaluating whether the existing license (which seems quite permissive) allows us to package JSMin safely.

Contributor

nchammas commented Oct 20, 2014

Yeah, it would be best if we avoided having an external dependency. We also don't want to implement our own JavaScript minifier if possible; that would be annoying.

What if we found an existing JavaScript minifier with a permissive license that we could just include as part of Packer? Users wouldn't have to know about it as it would just be used internally by Packer.

For example, YUI Compressor (BSD license) and Go-Yui (MIT license) look like promising options. There are also several Python options with permissive licenses, some of which don't depend on YUI and therefore don't need Java installed.

Could we just package one of those with Packer, allowing Packer to strip JSON templates of comments, without requiring users to manually manage this dependency?

@mitchellh What do you think of this approach?

Other potential options include simply porting the original JSMin from C to Go (it's quite short), or asking Crockford if he is open to relicensing JSMin under a more permissive license evaluating whether the existing license (which seems quite permissive) allows us to package JSMin safely.

@dpehrson

This comment has been minimized.

Show comment
Hide comment
@dpehrson

dpehrson Oct 20, 2014

Another option is to just move away from JSON. It's appropriateness as a solution to configuration management scales very poorly past very basic needs.

It's a format for transferring data between computers, it's not really designed for human management.

Another option is to just move away from JSON. It's appropriateness as a solution to configuration management scales very poorly past very basic needs.

It's a format for transferring data between computers, it's not really designed for human management.

@tehgeekmeister

This comment has been minimized.

Show comment
Hide comment
@tehgeekmeister

tehgeekmeister Oct 20, 2014

I actually like JSON, at least as an option. I'm considering building a layer in front of packer that might do dynamic mixing and matching of provisioners/etc., and so a standard interchange format as an option is important.

I actually like JSON, at least as an option. I'm considering building a layer in front of packer that might do dynamic mixing and matching of provisioners/etc., and so a standard interchange format as an option is important.

@sparkprime

This comment has been minimized.

Show comment
Hide comment
@sparkprime

sparkprime Oct 20, 2014

Contributor

@tehgeekmeister I have been doing that with Jsonnet, I am planning to release something very soon and if you're interested I could prioritize that. It'd be great to consider more use-cases.

http://google.github.io/jsonnet/doc/

Contributor

sparkprime commented Oct 20, 2014

@tehgeekmeister I have been doing that with Jsonnet, I am planning to release something very soon and if you're interested I could prioritize that. It'd be great to consider more use-cases.

http://google.github.io/jsonnet/doc/

@sparkprime

This comment has been minimized.

Show comment
Hide comment
@sparkprime

sparkprime Oct 20, 2014

Contributor

In general this question is a difficult one. Even in the constraint of maintaining backwards compatibility with JSON there are many options. The most obvious extension of JSON is YAML it has been around quite a while and lots of people know it. I'm surprised by how many people do not realize that YAML is an extension of JSON. But regardless, you can write idiomatic JSON with comments by using YAML. Since packer is written by Mitchell Hashimoto, the most likely extension of JSON he would use is going to be HCL, which he wrote. UCL, which is very similar, is also an option. I would say YAML is the most powerful of those syntaxes because it does support referencing and a bunch of other things (that seem to not be used that much). But even YAML is not considered powerful enough, as evidenced by the number of systems that combine it (rather inelegantly in my opinion) with Jinja. For this reason I would be wary of adding anything to packer itself, because you then never be able to take it away again. And if you decide to go in HCL direction, you cannot change your mind later and go YAML (at least without having multiple front-ends). Another problem with directly integrating a language is it would have to be bindable from Go, preferably without a native bridge.

More to the point, you don't even need to do that. I assume most people run packer from a makefile, because then you can build multiple images in parallel and chain it into your build process. With that in mind, you can run your own pre-preprocessor / generator in front of it and that works very well. You don't even need to be backwards-compatible with JSON if you go in that direction. There are Python-ish options like Coil and Pystachio. Personally I would recommend taking a look at Jsonnet though :)

Contributor

sparkprime commented Oct 20, 2014

In general this question is a difficult one. Even in the constraint of maintaining backwards compatibility with JSON there are many options. The most obvious extension of JSON is YAML it has been around quite a while and lots of people know it. I'm surprised by how many people do not realize that YAML is an extension of JSON. But regardless, you can write idiomatic JSON with comments by using YAML. Since packer is written by Mitchell Hashimoto, the most likely extension of JSON he would use is going to be HCL, which he wrote. UCL, which is very similar, is also an option. I would say YAML is the most powerful of those syntaxes because it does support referencing and a bunch of other things (that seem to not be used that much). But even YAML is not considered powerful enough, as evidenced by the number of systems that combine it (rather inelegantly in my opinion) with Jinja. For this reason I would be wary of adding anything to packer itself, because you then never be able to take it away again. And if you decide to go in HCL direction, you cannot change your mind later and go YAML (at least without having multiple front-ends). Another problem with directly integrating a language is it would have to be bindable from Go, preferably without a native bridge.

More to the point, you don't even need to do that. I assume most people run packer from a makefile, because then you can build multiple images in parallel and chain it into your build process. With that in mind, you can run your own pre-preprocessor / generator in front of it and that works very well. You don't even need to be backwards-compatible with JSON if you go in that direction. There are Python-ish options like Coil and Pystachio. Personally I would recommend taking a look at Jsonnet though :)

@nchammas

This comment has been minimized.

Show comment
Hide comment
@nchammas

nchammas Oct 22, 2014

Contributor

Here is a work-around I am using during development to allow comments in Packer templates. All it requires is that you have JSMin or something similar installed. On OS X with Homebrew this is as simple as:

brew install jsmin

Then, as Crockford suggests:

packer validate <(jsmin < packer-template.json)

That's it. This allows comments in your templates. For example:

<snipped>

  "provisioners": [
    {
      "type": "shell",
      "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh -x '{{ .Path }}'",
      "scripts": [
        "./tools-setup.sh",
        "./python-setup.sh"    // Python 3 doesn't work, so we're stuck with 2.7
        // "./python3-setup.sh"
      ]
      // "pause_before": "10s"
    },

<snipped>
Contributor

nchammas commented Oct 22, 2014

Here is a work-around I am using during development to allow comments in Packer templates. All it requires is that you have JSMin or something similar installed. On OS X with Homebrew this is as simple as:

brew install jsmin

Then, as Crockford suggests:

packer validate <(jsmin < packer-template.json)

That's it. This allows comments in your templates. For example:

<snipped>

  "provisioners": [
    {
      "type": "shell",
      "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh -x '{{ .Path }}'",
      "scripts": [
        "./tools-setup.sh",
        "./python-setup.sh"    // Python 3 doesn't work, so we're stuck with 2.7
        // "./python3-setup.sh"
      ]
      // "pause_before": "10s"
    },

<snipped>
@nchammas

This comment has been minimized.

Show comment
Hide comment
@nchammas

nchammas Oct 30, 2014

Contributor

After discussing this with my buddy @lumberj, we came up with another potential approach that does not require modifying core.

We can develop a Packer command plugin that introduces a couple of new commands, say x-validate and x-build (x- for extended), that pre-process the template using JSMin before handling it as usual.

For the actual minification, there is apparently already a Go port of JSMin out there that we could use.

What do people think of this approach? If it takes off, I guess down the road it could be incorporated into Packer core.

Contributor

nchammas commented Oct 30, 2014

After discussing this with my buddy @lumberj, we came up with another potential approach that does not require modifying core.

We can develop a Packer command plugin that introduces a couple of new commands, say x-validate and x-build (x- for extended), that pre-process the template using JSMin before handling it as usual.

For the actual minification, there is apparently already a Go port of JSMin out there that we could use.

What do people think of this approach? If it takes off, I guess down the road it could be incorporated into Packer core.

@rickard-von-essen

This comment has been minimized.

Show comment
Hide comment
@rickard-von-essen

rickard-von-essen Oct 30, 2014

Collaborator

@nchammas before you do that see #1629.

Collaborator

rickard-von-essen commented Oct 30, 2014

@nchammas before you do that see #1629.

@rickard-von-essen

This comment has been minimized.

Show comment
Hide comment
@rickard-von-essen

rickard-von-essen Oct 30, 2014

Collaborator

@nchammas I suggest use Ruby or something to convert Yaml to json and write your config in Yaml instead to have a more human friendly template format.

Collaborator

rickard-von-essen commented Oct 30, 2014

@nchammas I suggest use Ruby or something to convert Yaml to json and write your config in Yaml instead to have a more human friendly template format.

@sparkprime

This comment has been minimized.

Show comment
Hide comment
@sparkprime

sparkprime Oct 30, 2014

Contributor

Note that the following is valid YAML:

{
  "provisioners": [
    {
      "type": "shell",
      "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh -x '{{ .Path }}'",
      "scripts": [
        "./tools-setup.sh",
        "./python-setup.sh"    # Python 3 doesn't work, so we're stuck with 2.7
        # "./python3-setup.sh"
      ]
      # "pause_before": "10s"
    }
  ]
}

So using YAML does not necessarily mean using a meaningful whitespace syntax.

Contributor

sparkprime commented Oct 30, 2014

Note that the following is valid YAML:

{
  "provisioners": [
    {
      "type": "shell",
      "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh -x '{{ .Path }}'",
      "scripts": [
        "./tools-setup.sh",
        "./python-setup.sh"    # Python 3 doesn't work, so we're stuck with 2.7
        # "./python3-setup.sh"
      ]
      # "pause_before": "10s"
    }
  ]
}

So using YAML does not necessarily mean using a meaningful whitespace syntax.

@nchammas

This comment has been minimized.

Show comment
Hide comment
@nchammas

nchammas Nov 3, 2014

Contributor

Thanks for the heads up @rickard-von-essen.

Contributor

nchammas commented Nov 3, 2014

Thanks for the heads up @rickard-von-essen.

@delitescere delitescere referenced this issue Dec 16, 2014

Closed

Use HCL #1768

@nchammas

This comment has been minimized.

Show comment
Hide comment
@nchammas

nchammas Dec 25, 2014

Contributor

To add to the laundry list of suggestions here, there is also JSON5, a simple extension to JSON that allows for comments as well as other tweaks that make JSON more human-friendly.

From the project website:

JSON5 is a proposed extension to JSON that aims to make it easier for humans to write and maintain by hand. It does this by adding some minimal syntax features directly from ECMAScript 5.

Some of the interesting features:

  • Object keys can be unquoted if they're valid identifiers.
  • Objects and arrays can have trailing commas.
  • Both inline (single-line) and block (multi-line) comments are allowed.

@sethvargo You mentioned HCL in #1768.

Has it been decided that that is what Packer is moving to for configuration?

Contributor

nchammas commented Dec 25, 2014

To add to the laundry list of suggestions here, there is also JSON5, a simple extension to JSON that allows for comments as well as other tweaks that make JSON more human-friendly.

From the project website:

JSON5 is a proposed extension to JSON that aims to make it easier for humans to write and maintain by hand. It does this by adding some minimal syntax features directly from ECMAScript 5.

Some of the interesting features:

  • Object keys can be unquoted if they're valid identifiers.
  • Objects and arrays can have trailing commas.
  • Both inline (single-line) and block (multi-line) comments are allowed.

@sethvargo You mentioned HCL in #1768.

Has it been decided that that is what Packer is moving to for configuration?

@wolfhechel

This comment has been minimized.

Show comment
Hide comment
@wolfhechel

wolfhechel Jan 17, 2015

I would argue that this is something that should not be put on packer. Pre-processing is just another step that could easily be managed by some other script or tool prior to inputting it to packer.

If anyone needs a simple way to feed template with // comments into packer, here's my approach.

sed -e 's@//.*@@g' build.json | packer build -

I would argue that this is something that should not be put on packer. Pre-processing is just another step that could easily be managed by some other script or tool prior to inputting it to packer.

If anyone needs a simple way to feed template with // comments into packer, here's my approach.

sed -e 's@//.*@@g' build.json | packer build -
@mfischer-zd

This comment has been minimized.

Show comment
Hide comment
@mfischer-zd

mfischer-zd Jan 30, 2015

The simplest solution seems like the best solution to me: ignore any keys that start with _.

The simplest solution seems like the best solution to me: ignore any keys that start with _.

@jcoutch

This comment has been minimized.

Show comment
Hide comment
@jcoutch

jcoutch Apr 30, 2015

The simplest solution would be just to pick something and implement it...rather than dragging on this discussion for 2 years. Put an underscore in, implement an enabled flag, implement HCL/YAML/etc...just pick something!

jcoutch commented Apr 30, 2015

The simplest solution would be just to pick something and implement it...rather than dragging on this discussion for 2 years. Put an underscore in, implement an enabled flag, implement HCL/YAML/etc...just pick something!

@sparkprime

This comment has been minimized.

Show comment
Hide comment
@sparkprime

sparkprime Apr 30, 2015

Contributor

I think the value is actually kinda low because a lot of people use Packer as part of a larger toolchain. By the time your configs are large enough to need comments, you're probably generating the JSON anyway. And even if you're not in that situation, you can use whatever language you like, as long as you're prepared to write a 10 line Python wrapper.

Contributor

sparkprime commented Apr 30, 2015

I think the value is actually kinda low because a lot of people use Packer as part of a larger toolchain. By the time your configs are large enough to need comments, you're probably generating the JSON anyway. And even if you're not in that situation, you can use whatever language you like, as long as you're prepared to write a 10 line Python wrapper.

@knope

This comment has been minimized.

Show comment
Hide comment
@knope

knope Aug 25, 2015

comments. json needs them.

json sucks until it has them. something as simple as an enclosing \ \ or ## ## or ... be creative... would be awesome... testing and running and running and testing sucks when a line (key) can't be commented out and back in for science. . . .

+1 for spamming: https://tools.ietf.org/html/rfc7159 ; in order to get json comments...

knope commented Aug 25, 2015

comments. json needs them.

json sucks until it has them. something as simple as an enclosing \ \ or ## ## or ... be creative... would be awesome... testing and running and running and testing sucks when a line (key) can't be commented out and back in for science. . . .

+1 for spamming: https://tools.ietf.org/html/rfc7159 ; in order to get json comments...

@kidbrax

This comment has been minimized.

Show comment
Hide comment
@kidbrax

kidbrax May 3, 2017

people obviously want YAML

kidbrax commented May 3, 2017

people obviously want YAML

@hashicorp hashicorp locked and limited conversation to collaborators May 3, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.