Adapt to the not-deprecated checksum syntax.

[timschumi]

Starting with packer 1.6, the iso_checksum_type and iso_checksum_url fields have been deprecated. Instead, the checksum type is now written before the actual checksum in iso_checksum.

[ladar]

If you run ./robox.sh validate you'll see there are a number of template issues. Mostly minor. Duplicate keys, and a few renamed key values. I updated the templates once upon a time, but discovered there was a bug with the Hyper-V template processing that caused packer to crash, so I reverted, and have been using an older version of packer since. I never did get around to reporting the bug, so I don't know if it's fixed.

[timschumi]

If you run ./robox.sh validate you'll see there are a number of template issues. Mostly minor. Duplicate keys, and a few renamed key values.

Since none of those issues have been introduced by this Pull Request, I fixed the rest of them in a different PR.

EDIT: I just got what you meant. Starting with Packer 1.6, this is a hard-fail, not just a recommendation, which is why I chose to do the other (not-fatal) warnings/notices in a seperate PR.

I updated the templates once upon a time, but discovered there was a bug with the Hyper-V template processing that caused packer to crash, so I reverted, and have been using an older version of packer since. I never did get around to reporting the bug, so I don't know if it's fixed.

Hyper-V definitely isn't broken here, since I originally noticed and fixed this on Windows (where I freshly downloaded Packer 1.6.0, which now hard-fails when it sees iso_checksum_type). But if you say there have been issues in the past, I'll test a few more combinations of Host OS and builders.

Also, could you make sure that the setup on your side (and on the build servers) understands this syntax? I haven't yet tracked down when this was introduced (the "new syntax", not the deprecation of the old one). I'm pretty sure that it does, but the documentation for that has been updated only recently.

As soon as everything is sorted out, I'll catch up with recent checksum changes.

[timschumi]

I now successfully tested the following configurations (together with #157 ):

• Linux (Packer v1.6.0) building virtualbox images.
• Linux (Packer v1.6.0) building libvirt images.
• Linux (Packer v1.6.0) building vmware images.
• Windows (Packer v1.6.0) building hyperv images.
• macOS (Packer v1.6.0) building virtualbox images.

Parallels builds I could not test because my test version ran out, and packer apparently only works with the Pro version (starting with Parallels 11).

Windows VirtualBox builds I could not test either, because something in my Windows environment breaks virtualization, instantly crashing VirtualBox when spinning up a VM (also happens on "normal" VMs, so I doubt that this is caused by Packer).

[timschumi]

@ladar Is there anything (apart from the merge conflicts around updated checksums) that is holding this and #157 back?

[ladar]

Yeah, all of the build machines are stuck using packer 1.4.5 until I can overhaul all of the JSON files at once, and bring them up to the current syntax. I tried previously but ran into problems with the Hyper-V template failing to validate and never went back.

I recently did an experiment using packer fix which seemed to work, but rearranged all of the JSON. And I haven't had a chance to go back and fix it, and/or try updating the syntax by hand.

[timschumi]

Yeah, all of the build machines are stuck using packer 1.4.5 until I can overhaul all of the JSON files at once, and bring them up to the current syntax. I tried previously but ran into problems with the Hyper-V template failing to validate and never went back.

I recently did an experiment using packer fix which seemed to work, but rearranged all of the JSON. And I haven't had a chance to go back and fix it, and/or try updating the syntax by hand.

The changes here seem to have been the only thing that was required to make packer 1.6.0+ happy again (i.e. this is the only actual incompatibility that I could spot).

The other PR contains the remaining stuff to make the validate script happy, and I don't remember any leftover warnings after that (at least at the time when I wrote the patches, it might have changed by now).

But yeah, since that's actually infra-related and probably not that many people self-build and use packer 1.6+, I guess that this isn't critical at the moment.

[ladar]

Yeah, it's a tricky update too, because by the time I get chance to review, all the files have been changed. Below is how far I got trying to automate the update. I forgot to mention above, but because everything gets reordered, it's impossible to review the changes because a diff is useless.

# packer fix developer-hyperv.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > developer-hyperv.json.new
# packer fix developer-libvirt.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > developer-libvirt.json.new
# packer fix developer-ova.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > developer-ova.json.new
# packer fix developer-virtualbox.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > developer-virtualbox.json.new
# packer fix developer-vmware.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > developer-vmware.json.new
# packer fix generic-docker.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > generic-docker.json.new
# packer fix generic-hyperv.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > generic-hyperv.json.new
# packer fix generic-libvirt.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > generic-libvirt.json.new
# packer fix generic-libvirt-x32.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > generic-libvirt-x32.json.new
# packer fix generic-parallels.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > generic-parallels.json.new
# packer fix generic-virtualbox.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > generic-virtualbox.json.new
# packer fix generic-vmware.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > generic-vmware.json.new
# packer fix lineage-hyperv.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > lineage-hyperv.json.new
# packer fix lineage-libvirt.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > lineage-libvirt.json.new
# packer fix lineage-virtualbox.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > lineage-virtualbox.json.new
# packer fix lineage-vmware.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > lineage-vmware.json.new
# packer fix magma-docker.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > magma-docker.json.new
# packer fix magma-hyperv.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > magma-hyperv.json.new
# packer fix magma-libvirt.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > magma-libvirt.json.new
# packer fix magma-virtualbox.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > magma-virtualbox.json.new
# packer fix magma-vmware.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > magma-vmware.json.new
# packer fix packer-cache.json | jq "{ variables: .variables, provisioners: [ .provisioners | .[] | { only: .only, scripts: .scripts, type: .type, pause_before: .pause_before, \"start_retry_timeout\": .\"start_retry_timeout\", \"expect_disconnect\": .\"expect_disconnect\", \"execute_command\": .\"execute_command\" } ], builders: .builders, \"post-processors\": .\"post-processors\" }" | grep -v ': null$' | grep -v ': null,$' > packer-cache.json.new

[ladar]

I don't suppose you could convert the template changes into a series of a sed commands? Then I can review and run them at my liesure, and we won't have to worry about a patch getting invalidated everytime I update an ISO, etc.

[timschumi]

If I might ask, is there actually such a big need for having packer fix fully automated? For me, it seems like making up a sed command (iirc, the specific one for this PR was a two-liner) for an upcoming issue and validating that is more manageable than making packer fix generate diffable JSON automatically (not that it is infallible anyways). Since new boxes are usually added through add.sh, I don't see many cases where such issues would be reintroduced, so it would most likely be a one-time thing for each issue.

And since your second response came in just as I was writing this:

Sure, I'll see if I can replicate the sed commands that I used. I'll post them on their relevant PR respectively.

[timschumi]

The second commit of this PR (the one handling JSON) is made using the following commands:

sed -i '/"iso_checksum_type": "sha256"/d' *.json
sed -i 's/"iso_checksum": "/"iso_checksum": "sha256:/g' *.json

The other PR will have to wait until I added back all my RAM (so probably tomorrow), 32GB doesn't seem to be enough to run a full validation.

[hswong3i]

From #156 (comment):

Yeah, all of the build machines are stuck using packer 1.4.5 until I can overhaul all of the JSON files at once, and bring them up to the current syntax. I tried previously but ran into problems with the Hyper-V template failing to validate and never went back.

From our README.md:

The templates in this repo require a current version of packer, (1.3.4+) and in some cases, make use of features which haven't been officially merged and/or released yet. Use the res/providers/packer.sh script to build an appropriately patched packer binary

@ladar Shall I install packer 1.4.5, or running res/provider/packer.sh for a custom patched version (anyway, tried but failed)?

[timschumi]

The templates in this repo require a current version of packer, (1.3.4+) and in some cases, make use of features which haven't been officially merged and/or released yet. Use the res/providers/packer.sh script to build an appropriately patched packer binary

@ladar Shall I install packer 1.4.5, or running res/provider/packer.sh for a custom patched version (anyway, tried but failed)?

The corresponding pull request (hashicorp/packer#7330) has long been merged and the branch was deleted, so I don't think the script will still work.

That said, GitHub indicates that this feature branch is included in releases since v1.5.0, so you should be fine if you are using that version or anything newer.

[ladar]

@timschumi I incorporated the changes. It wasn't as painful as I remembered... in that there were only a small number of duplicate keys, and and the sed commands made it easy to update the ISO keys.

I thought there were other more involved changes with how cpus/mem/disks, etc were defined, as packer has been trying to normalize those... but I must have made those changes already.

Everything validates on my workstation, if I increase the max file limit... aka ulimit -n 16384.

I'll probably let the 3.1.2 boxes finish building with packer 1.4.5 and try updating the build robots to packer 1.6.5 before I start the 3.1.4 robox build.

[timschumi]

@ladar Since the script changes are still missing, could you maybe reopen this (or alternatively, pick over commit 700aebf)?

Otherwise, res/scripts/add.sh and robox.sh iso will produce broken configs using the old syntax.

[ladar]

@timschumi I didn't notice those other changes. I thought this only dealt with the JSON schema updates which caused newer packer versions to fail the validation check.

To cherry pick a question from above, the packer.sh script is called into service whenever the templates in the repo require packer to have specific patches to build. That said, I haven't felt the need to patch packer in awhile, nor had the time to work on it, so it hasn't been used in awhile.

Rather, you can setup the providers by using the providers.sh script. Specifically ./providers.sh packer will download and install the latest version. If you run the script the without any params it will give you some usage help. But that script is geared towards CentOS 7, so it may not work as well on other platforms.

[ladar]

I snagged that single commit as a patch:

https://github.com/lavabit/robox/commit/700aebfce8cebe1375d15c35ed7879a3ec4194be.diff

And merged it.

[ladar]

In case I wasn't clear, all of the build robots use a vanilla 1.4.5 release. But I'm trying to update the templates, so I can get back to keeping pace with packer. With your sed commands, the templates seem to be validating but I haven't tried compiling them yet. I'm saving that for the next robox release.

[ladar]

So I just realized I missed the sed commands for updating disk size, and ssh timeout. I ran them, and pushed the result, but strangely enough the old the syntax wasn't triggering a validation error. It has me worried I might of missed something important.

@timschumi let me know if I missed anything...

[timschumi]

So I just realized I missed the sed commands for updating disk size, and ssh timeout. I ran them, and pushed the result, but strangely enough the old the syntax wasn't triggering a validation error. It has me worried I might of missed something important.

@timschumi let me know if I missed anything...

I'll take a look.

What version have you been validating locally with? iirc at least version 1.6.0+ was terribly mad about both the disk size and SSH timeout.

[timschumi]

Huh, I just tested validation again using the current tip of this PR, so there should be quite a few warnings (precisely the ones fixed in #157 ). But it doesn't find any (apart from the actual duplicated keys).

I'm sure that those warnings were actually there, and I'm pretty sure that I was able to get those warnings on two different PCs, but I can't test on my original systems since I recently reinstalled my OS.

Weird.