Allow for plan-only testing flows

[esword]

My team is starting to write terraform modules that will be used by many other groups. I have been looking for a way to rapidly test them and discovered this project. Many thanks for putting it together!

One idea we had is that you can do a lot of "unit like" testing of modules just by executing "terraform plan" and checking the resulting contents. This approach assumes that terraform will produce what it says in the plan, so it is not complete end-to-end testing, but it is very useful for rapid, first-pass testing of changes. Sort of like using mocks in unit tests before going on to full integration testing.

My thought was to add a "planOnly" prop to the driver or provisioner. When executing converge, it would only create the plan (and destroy would also short circuit). Then make the plan available as some data structure in the tests so that it can be easily traversed and queried. e.g. Does an aws_instance have the proper tags and values? Is it the right instance type? Does it have the right security roles?

I'm willing to make a first pass at this if you think it's a good idea.

[ncs-alane]

Hi @esword!

I'm happy that the project is helping you and your team have a better experience working with Terraform!

I have actually experimented with writing a framework just like you're describing. While I think it has merrit, I also think it would be better suited to be packaged as a separate RSpec plugin. It would be a good excuse to break the Terraform client related logic in to a third gem that both projects could depend on.

Additionally, I will be implementing support for interface testing soon, which will theoretically enable the testing of all resources in a given Terraform configuration, assuming they are exposed by the provider and testable with Inspec.

[esword]

As I was writing my original issue, I was thinking close to the same thing about separating it. Since there isn't really a need for full lifecycle management (i.e. destroy), it would be a bit of a hack to add in the "planOnly" option. Plus, with the plan-related checks, you don't need to ensure ssh access either since you won't be connecting to machines directly. This is actually a big hassle for us since there isn't direct ssh access to our AWS machines from the machine which makes the terraform calls.

It will be interesting to see what you come up with. We will experiment with some stuff over the next few days as well. If it's not directly tied to kitchen, it may make more sense to use some TF code directly and do the framework in go.

[ncs-alane]

The last time I looked, Inspec did not support the use of bastion hosts for SSH forwarding, but if they have implemented that then I'll explore supporting it because your scenario is certainly not unique. One work around is to create a fixture project that uses the project under test as a module; you could add an extra test security group that allows direct SSH access to instances without complicating your production code.

Writing in Go certainly makes sense given that any Ruby gem is just going to be shelling out to the Terraform client, unless there is some Ruby Go bridge gem...

[nictrix]

Looks like it will be a supported feature in the near future: inspec/inspec#431

[ncs-alane]

@nictrix that doesn't seem related

[ncs-alane]

Closed because unit testing is not a responsibility of Test Kitchen.

[esword]

Just as a final piece for anyone else looking in the future - we built a tiny go utility that reads a plan file and outputs JSON. This TF issue is an example of a very verbose format, though we condensed it down to a more readable/fluent set of json objects. We are now to writing rspec tests against that. E.g.

RSpec.describe "Json parse" do
  let(:tf) {JSON.parse(contents)}

  it "host2 module uses proper ami" do
    expect(tf["host_2"]["aws_instance.instance"]["ami"]).to eq("ami-13be557e")
  end
end

[jamtur01]

@esword Is the Go utility available somewhere? Thanks!

[esword]

It's a very, very simple tool. I'll check if we can move it to our open source repo. If not, we can hopefully at least just paste in the primary code.

[esword]

@jamtur01 - we are releasing the tool. Just have to get one more sign off. Again, it's pretty underwhelming in what it does, but is handy if you haven't already written something to transform to json. I'll post when it goes up.

[jamtur01]

Thanks!

[esword]

@jamtur01 - it only took 3 weeks, but finally got the tfjson moved to our OSS org - https://github.com/palantir/tfjson.

[jamtur01]

Neat! Thanks again!