[RFC] Terraform roster module to be used with terraform-provider-salt

[dmacvicar]

• Feature Name: Roster module for Terraform to be used with salt-ssh
• Start Date: 2018-07-27
• RFC PR:
• Salt Issue:

Summary

This is a proposal for a roster module to be used with terraform. It solves the use-case of a self-contained folder in git with infrastructure defined with terraform, where one wants
to provision the infrastructure using salt-ssh. The idea is that the salt-ssh roster is automatically filled with terraform resources.

$ cd myproject
$ terraform apply
$ salt-ssh '*' state.apply

Motivation

The goal is very simple: self-contained directories with infrastructure created by terraform and configuration using salt-ssh. Example here.

Some time ago I proposed a solution for this, which never got merged (for different reasons).
While I did not know about it at the time, the design was very similar to terraform-inventory module for Ansible, and it suffers from the same problems.

It worked by lookin at terraform state file, and recognizing certain types of resources, like libvirt hosts and AWS instances, then doing things like looking at the public_ip and creating a roster based on that.

However, when I got feedback, the approach drawbacks were made evident:

• "Can you use the second ip instead of the first ip? what about the private one?
• "Azure is not supported!"

These drawbacks affected the motivation to get it merged. However it let me thinking on the problem, until I found terraform-ansible-provider and read the inspirational post about it.

I believe this is a much better approach for a terraform/salt-ssh integration roster module.

Design

The design, when compared to the original proposal, is to make the salt side (terraform.py) very agnostic. It does not know about all terraform resource types like AWS or Azure, but only one type of resource: salt_host.

the salt_host resource is provided by another provider: terraform-provider-salt. It is just a logical resource. It does not do much than gluing terraform resources to the roster:

So for a libvirt resource that is instantiated two times:

resource "libvirt_domain" "domain" {
  name = "domain-${count.index}"
  memory = 1024
  disk {
    volume_id = "${element(libvirt_volume.volume.*.id, count.index)}"
  }
  
  network_interface {
    network_name = "default"
    wait_for_lease = true
  }
  count = 2
}

One could create one salt_host resource per libvirt domain, and bind the roster host to the first ip address:

resource "salt_host" "example" {
  salt_id = "minion${count.index}"
  host = "${element(libvirt_domain.domain.*.network_interface.0.addresses[count.index], 0)}"
  user = "root"
  passwd = "linux"
  count = 2
}

As you can see, there are no limits on what you can glue into the roster. You have absolute freedom on how many hosts to expose to Salt, and what address to use in there.

This means terraform.py is much simpler. It scans the salt_host resources from the state, and collects the properties, which are 1:1 mapped to the roster attributes.
The roster module would not need to be updated as terraform adds features and resource types.

Alternatives

• A roster module that does not require a terraform provider. It was described in the motivation and my previous PR. With the drawbacks described above: need to know about specific terraform resources.

• Another idea/alternative is to keep the terraform provider side but alter the terraform.py included in this script and make it a more generic "script" roster module. So that if the module finds a salt-ssh.roster executable, or it is mentioned in the Saltfile, it gets the roster from executing that script. In this case, the terraform-go-provider could implement the script and provide itself the roster (however this may require some knowledge of how to abuse a terraform provider executable to perform another task when called with a different option, and to guess where it is, as providers are not in PATH but in terraform.plugins.d).
  In any case this is an improvement over this RFC, and could be done in a second iteration/step.
  Update: this could be more feasible than I thought: https://twitter.com/dmacvicar/status/1024800245325213697 opinions?

Unresolved questions

• Where to host terraform-provider-salt.

Drawbacks

The disadvantage is that the roster module needs and depends on a specific salt provider on the terraform side. But this provider (prototype) is very simple, as it is pure logical, and will not need much changes if the roster architecture stays the same.

[gtmanfred]

This looks good, just one thing that I would like to see fixed, and another that is a nitpick.

[gtmanfred]

Why is this needed, it is just passing opts to a class in the file that should already have access to opts?

I don't see what this gives us.

[gtmanfred]

Ahh, ok, so what i think we should do here instead is move RosterMatcher to __utils__ and and write a function that interacts with the RosterMatcher, and just returns rmatcher.targets() Instead of having to pass __opts__ around. This way, __opts__ will already be available in the __utils__ modules.

Utils is even already packed inside of rosters, for Fluorine.

https://github.com/saltstack/salt/blob/develop/salt/loader.py#L475

[dmacvicar]

I will refactor this in a next PR.

[gtmanfred]

This should use salt.utils.json.load

[dmacvicar]

Before moving onto fixing the issues in the roster code, I would focus on the RFC.

I would like some feedback on the 2nd alternative. Because if we instead of merging the roster module add a more generic script roster module, then for terraform one would only need the terraform salt provider, and the roster generic script module can be reused for other integrations.

In summary, instead of terraform.py and master having roster: terraform, having something like roster_script: terraform-provider-salt --roster and move all this functionality to the provider. Which would act as a terraform provider plugin when called by terraform and as a roster provider when called by Salt.

[isbm]

@terminalmage @cachedout @rallytime any thoughs on design here?

[isbm]

@dmacvicar whoa, I find this very nice idea (the main one) and the right step towards resources [re]use!

While the idea as it is now is already great and can be left as is, but looking at this from mere user perspective, I find they likely would be even more happy to just call salt-ssh \* state.apply alone. As I would imagine, a particular Salt state could be/do the following:

• Describe resources in the SLS state something like Salt Cloud already doing, which is similar to how Terraform is describing resource. Maybe we could find one generic syntax for SLS on this regard and make sure it is applicable everywhere?
• Render terraform cf from that state structure (extend terraform.py with this ability, perhaps?) so no one needs to describe things in two different places. Also we could separate resource description from state processing, and so Salt Cloud part would leverage from this a lot (and vice versa).
• Plan/apply terraform to setup everything
• Apply the final state on top of it to end the infrastructure setup

Maybe we should also teach Salt to read Terraform state to let Salt know where it is. I am really excited where it goes! 👍 👍

[dmacvicar]

Describe resources in the SLS state something like Salt Cloud already doing, which is similar to how Terraform is describing resource. Maybe we could find one generic syntax for SLS on this regard and make sure it is applicable everywhere?

I think this is true for Salt users. I doubt Terraform users want to change the way they describe resources and that is completely out of the scope of this PR. It belongs to salt-cloud future.

In any case, it is not a simple problem. Terraform graph and Salt states/sls are different. Terraform automatically figures dependencies between resources as you use interpolation between them, unlike Salt require:.

For sake of moving the discussion I beg to keep this topic out. We don't want to saltify terraform. We want to let terraform users use Salt. Everything else is an effort that can be done independently of this PR and should be tackled by someone having a) deep knowledge about limitations of both Terraform and Salt b) some say in the future of salt-cloud.

[dmacvicar]

Something that may be a nice addition, but unrelated to this PR would be a companion ext_pillar so that the terraform-salt-provider could interpolate and expose infrastructure details in the pillar.

To give an example: meta-salt. Create a salt cluster with salt. I use terraform to bring up 4 VMs. I use salt-ssh to configure them, 1 as a salt-master and 3 as a salt-minion. I need salt-ssh to know the ip of the VM that was configured as master to change /etc/salt/minion master key with salt-ssh on the minions. If I have access to the pillar, my "minion" state could use pillar data coming from terraform.

[isbm]

For sake of moving the discussion I beg to keep this topic out. We don't want to saltify terraform. We want to let terraform users use Salt.

Yes, agreed. As I said, current PR perfectly go in as-is and to me current design makes lots of sense.

[rallytime]

I have some small requests. :)

[rallytime]

Can we break these imports up into 2 sections? We usually do something like:

# Import Python libs
from __future__ import absolute_import, unicode_literals
import logging
import os.path

# Import Salt libs
import salt.utils.json

etc.

That makes the import list easier to maintain moving forward.

[dmacvicar]

Done

[rallytime]

Rather than importing salt.utils can you import the individual files directly? Otherwise you'll see deprecation notices as the big utils init file was broken up in 2018.3.

[dmacvicar]

Done

[dmacvicar]

Rebased

[gtmanfred]

Can we move this to __utils__ and then call it from a __utils__ function, like roster.targets() which calls RosterMatcher, instead of having to pass in __opts__?

[gtmanfred]

Overall, i really like this approach, i would just rather see the RosterMatcher stuff moved to utils, instead of importing from another roster, and having to pass opts around.

[dmacvicar]

@gtmanfred I am trying to get this refactored, but after moving it to utils, if I call it using __utils__['roster_matcher.targets'] i get KeyErroras it is not inutils, and if I import it and call it as salt.utils.roster_matcher.targetsI getopts` not defined inside it. So I am missing something.

notworking.diff.txt

[gtmanfred]

if you remove salt.utils.roster_matcher from the front of calling RosterTargets, does it fix the problem?

def targets(conditioned_raw, tgt, tgt_type, ipv='ipv4'):
    rmatcher = RosterMatcher(conditioned_raw, tgt, tgt_type, ipv)
    return rmatcher.targets()

[gtmanfred]

Installing to check this.

[gtmanfred]

With your patch, i get this

[root@salt salt]# salt-ssh \* test.ping
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
AttributeError: 'module' object has no attribute 'roster_matcher'
Traceback (most recent call last):
  File "/bin/salt-ssh", line 8, in <module>
    execfile(__file__)
  File "/root/src/salt/scripts/salt-ssh", line 10, in <module>
    salt_ssh()
  File "/root/src/salt/salt/scripts.py", line 425, in salt_ssh
    client.run()
  File "/root/src/salt/salt/cli/ssh.py", line 23, in run
    ssh = salt.client.ssh.SSH(self.config)
  File "/root/src/salt/salt/client/ssh/__init__.py", line 231, in __init__
    self.tgt_type)
  File "/root/src/salt/salt/roster/__init__.py", line 105, in targets
    targets.update(self.rosters[f_str](tgt, tgt_type))
  File "/root/src/salt/salt/roster/flat.py", line 36, in targets
    return __utils__['roster_matcher.targets'](conditioned_raw, tgt, tgt_type, 'ipv4')
  File "/root/src/salt/salt/utils/roster_matcher.py", line 32, in targets
    rmatcher = salt.utils.roster_matcher.RosterMatcher(conditioned_raw, tgt,
AttributeError: 'module' object has no attribute 'roster_matcher'
Traceback (most recent call last):
  File "/bin/salt-ssh", line 8, in <module>
    execfile(__file__)
  File "/root/src/salt/scripts/salt-ssh", line 10, in <module>
    salt_ssh()
  File "/root/src/salt/salt/scripts.py", line 425, in salt_ssh
    client.run()
  File "/root/src/salt/salt/cli/ssh.py", line 23, in run
    ssh = salt.client.ssh.SSH(self.config)
  File "/root/src/salt/salt/client/ssh/__init__.py", line 231, in __init__
    self.tgt_type)
  File "/root/src/salt/salt/roster/__init__.py", line 105, in targets
    targets.update(self.rosters[f_str](tgt, tgt_type))
  File "/root/src/salt/salt/roster/flat.py", line 36, in targets
    return __utils__['roster_matcher.targets'](conditioned_raw, tgt, tgt_type, 'ipv4')
  File "/root/src/salt/salt/utils/roster_matcher.py", line 32, in targets
    rmatcher = salt.utils.roster_matcher.RosterMatcher(conditioned_raw, tgt,
AttributeError: 'module' object has no attribute 'roster_matcher'

Adding

diff --git a/salt/utils/roster_matcher.py b/salt/utils/roster_matcher.py
index 145a047..40e7533 100644
--- a/salt/utils/roster_matcher.py
+++ b/salt/utils/roster_matcher.py
@@ -29,8 +29,7 @@ log = logging.getLogger(__name__)


 def targets(conditioned_raw, tgt, tgt_type, ipv='ipv4'):
-    rmatcher = salt.utils.roster_matcher.RosterMatcher(conditioned_raw, tgt,
-                                                       tgt_type, ipv)
+    rmatcher = RosterMatcher(conditioned_raw, tgt, tgt_type, ipv)
     return rmatcher.targets()

And it works

[root@salt salt]# salt-ssh \* test.ping
minion:
    True

[dmacvicar]

I fixed the spurious namespace when using RosterMatcher.

I still get undefined __opts__:

  File "/space/git/salt/salt/salt/utils/roster_matcher.py", line 130, in get_data
    ret = copy.deepcopy(__opts__.get('roster_defaults', {}))
NameError: global name '__opts__' is not defined

Which confuses me. Other util modules, like azurerm do a explicit definition of __opts__:

__opts__ = salt.config.minion_config('/etc/salt/minion')
__salt__ = salt.loader.minion_mods(__opts__)

namecheap and vault in salt.utils attemps to manually define __salt___ if missing, but it assumes __opts__ exist:

    global __salt__
    if not __salt__:
        __salt__ = salt.loader.minion_mods(__opts__)

napalm module does a weird thing with wrapped_global_namespace.
Appart of those, I don't see a massive usage of __opts__ in the utils code.

This error happens both in the testcase and runtime.

[dmacvicar]

Adding

__opts__ = salt.config.minion_config('/etc/salt/minion')

in roster_matcher.py fixes the problem for me. Here is my working diff attached.

working.diff.txt

[gtmanfred]

switch it so you use __utils__

Try this diff

https://gist.github.com/gtmanfred/854724d07ae240005f326a52bdb62172

[dmacvicar]

I think I already mentioned it above, but if I do that, I get:

   -> unit.roster.test_terraform.TerraformTestCase.test_default_output  ............
       Traceback (most recent call last):
         File "/space/git/salt/salt/tests/unit/roster/test_terraform.py", line 62, in test_default_output
           ret = terraform.targets('*')
         File "/space/git/salt/salt/salt/roster/terraform.py", line 174, in targets
           return __utils__['roster_matcher.targets'](raw, tgt, tgt_type, 'ipv4')
       KeyError: u'roster_matcher.targets'

It is working on runtime (invoking salt-ssh). It happens when running unit tests. I am looking into other tests to see what kind of magic is used to insert __utils__ into the test.

[gtmanfred]

Ahh, yeah, it should just be something with setup_loader_modules that just needs to inject the opts into the utils modules.

Check out the boto module tests, they use that a lot.

[dmacvicar]

Done

[gtmanfred]

🎉

[dmacvicar]

So, what do I need to do now? :-)

[gtmanfred]

Just waiting on reviews, I moved this over to the fluorine branch to make sure it gets in.

[terminalmage]

I would rather this be reviewed by someone familiar with Terraform, as I know nothing about it.

[dubb-b]

@dmacvicar I have tested this out using the examples from your GH repo using libvirt. I really like the way you implemented this. I am working on a final few items here and I will be done with my review. One thing that may be kind of cool is using the example code that you have in your GH repo is great, although a working example in a Docker container might also be nice. This would allow people to see how this is all wired up in case they have issues with any of the steps listed. Overall this is a very nice feature to Salt and thank you for making it. I should be ready to give the thumbs up tomorrow.

[dmacvicar]

Thanks for the feedback @dubb-b .

As the roster plugin needs the provider counter-part, it will be quite important to make sure people can use it. I am already providing builds of the code in my repo, and once the Salt part is released I will consider doing stable/tagged releases of it.

The installation should be as as easy as copying the builds to terraform.d/plugins. As I am building everything in OBS (including Ubuntu builds), and OBS supports building containers, it should be feasible providing a more plug and play example.

The only thing that bothers me is that the official roster module depends on a very specific provider that for now is unreleased and hosted in my home project, however:

• Anybody implementing a provider that uses the same resource names can provide alternative implementations of the salt provider
• I am open to discuss how to make this provider "official", eg: moving it to the _terraform-providers` project so that it is auto-installed by terraform once it is used.
• All the above steps are worthy only if the roster module is part of Salt
• With all the disadvantages, I still think the new design/approach outweighs any disadvantage vs the old PR that had resource knowledge.

I think I could also create some kind of diagram and add it to the documentation later. It would also help understanding all the wiring, as you called it.

[dubb-b]

I am finished with the testing, it is working quite well. @rallytime 👍

[cachedout]

@dmacvicar There are a few issues identified by CodeClimate. It's not required that they be fixed but could you please take a look and see if any of them seem reasonable to you?

https://codeclimate.com/github/saltstack/salt/pull/48873

[dmacvicar]

I have reviewed those.

I think all of the issues validate my view on why I don't use or like CodeClimate. They are false alarms and the code is fine.

[MalloZup]

awesome 👍 i can't wait for using it 🌞