Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] Terraform roster module to be used with terraform-provider-salt #48873

Merged
merged 12 commits into from Aug 18, 2018

Conversation

Projects
None yet
8 participants
@dmacvicar
Copy link
Contributor

commented Aug 1, 2018

  • Feature Name: Roster module for Terraform to be used with salt-ssh
  • Start Date: 2018-07-27
  • RFC PR:
  • Salt Issue:

Summary

This is a proposal for a roster module to be used with terraform. It solves the use-case of a self-contained folder in git with infrastructure defined with terraform, where one wants
to provision the infrastructure using salt-ssh. The idea is that the salt-ssh roster is automatically filled with terraform resources.

$ cd myproject
$ terraform apply
$ salt-ssh '*' state.apply

Motivation

The goal is very simple: self-contained directories with infrastructure created by terraform and configuration using salt-ssh. Example here.

Some time ago I proposed a solution for this, which never got merged (for different reasons).
While I did not know about it at the time, the design was very similar to terraform-inventory module for Ansible, and it suffers from the same problems.

It worked by lookin at terraform state file, and recognizing certain types of resources, like libvirt hosts and AWS instances, then doing things like looking at the public_ip and creating a roster based on that.

However, when I got feedback, the approach drawbacks were made evident:

  • "Can you use the second ip instead of the first ip? what about the private one?
  • "Azure is not supported!"

These drawbacks affected the motivation to get it merged. However it let me thinking on the problem, until I found terraform-ansible-provider and read the inspirational post about it.

I believe this is a much better approach for a terraform/salt-ssh integration roster module.

Design

The design, when compared to the original proposal, is to make the salt side (terraform.py) very agnostic. It does not know about all terraform resource types like AWS or Azure, but only one type of resource: salt_host.

the salt_host resource is provided by another provider: terraform-provider-salt. It is just a logical resource. It does not do much than gluing terraform resources to the roster:

So for a libvirt resource that is instantiated two times:

resource "libvirt_domain" "domain" {
  name = "domain-${count.index}"
  memory = 1024
  disk {
    volume_id = "${element(libvirt_volume.volume.*.id, count.index)}"
  }
  
  network_interface {
    network_name = "default"
    wait_for_lease = true
  }
  count = 2
}

One could create one salt_host resource per libvirt domain, and bind the roster host to the first ip address:

resource "salt_host" "example" {
  salt_id = "minion${count.index}"
  host = "${element(libvirt_domain.domain.*.network_interface.0.addresses[count.index], 0)}"
  user = "root"
  passwd = "linux"
  count = 2
}

As you can see, there are no limits on what you can glue into the roster. You have absolute freedom on how many hosts to expose to Salt, and what address to use in there.

This means terraform.py is much simpler. It scans the salt_host resources from the state, and collects the properties, which are 1:1 mapped to the roster attributes.
The roster module would not need to be updated as terraform adds features and resource types.

Alternatives

  • A roster module that does not require a terraform provider. It was described in the motivation and my previous PR. With the drawbacks described above: need to know about specific terraform resources.

  • Another idea/alternative is to keep the terraform provider side but alter the terraform.py included in this script and make it a more generic "script" roster module. So that if the module finds a salt-ssh.roster executable, or it is mentioned in the Saltfile, it gets the roster from executing that script. In this case, the terraform-go-provider could implement the script and provide itself the roster (however this may require some knowledge of how to abuse a terraform provider executable to perform another task when called with a different option, and to guess where it is, as providers are not in PATH but in terraform.plugins.d).
    In any case this is an improvement over this RFC, and could be done in a second iteration/step.
    Update: this could be more feasible than I thought: https://twitter.com/dmacvicar/status/1024800245325213697 opinions?

Unresolved questions

  • Where to host terraform-provider-salt.

Drawbacks

The disadvantage is that the roster module needs and depends on a specific salt provider on the terraform side. But this provider (prototype) is very simple, as it is pure logical, and will not need much changes if the roster architecture stays the same.

@salt-jenkins salt-jenkins requested a review from saltstack/team-ssh Aug 1, 2018

@gtmanfred
Copy link
Contributor

left a comment

This looks good, just one thing that I would like to see fixed, and another that is a nitpick.

@@ -47,19 +47,20 @@ def targets(tgt, tgt_type='glob', **kwargs):
conditioned_raw = {}
for minion in raw:
conditioned_raw[six.text_type(minion)] = salt.config.apply_sdb(raw[minion])
rmatcher = RosterMatcher(conditioned_raw, tgt, tgt_type, 'ipv4')
rmatcher = RosterMatcher(conditioned_raw, tgt, tgt_type, 'ipv4', opts=__opts__)

This comment has been minimized.

Copy link
@gtmanfred

gtmanfred Aug 1, 2018

Contributor

Why is this needed, it is just passing opts to a class in the file that should already have access to opts?

I don't see what this gives us.

import json
import os.path
import salt.utils
from salt.roster.flat import RosterMatcher

This comment has been minimized.

Copy link
@gtmanfred

gtmanfred Aug 1, 2018

Contributor

Ahh, ok, so what i think we should do here instead is move RosterMatcher to __utils__ and and write a function that interacts with the RosterMatcher, and just returns rmatcher.targets() Instead of having to pass __opts__ around. This way, __opts__ will already be available in the __utils__ modules.

Utils is even already packed inside of rosters, for Fluorine.

https://github.com/saltstack/salt/blob/develop/salt/loader.py#L475

This comment has been minimized.

Copy link
@dmacvicar

dmacvicar Aug 2, 2018

Author Contributor

I will refactor this in a next PR.

'''
ret = {}
with salt.utils.files.fopen(state_file_path, 'r') as fh_:
tfstate = json.load(fh_)

This comment has been minimized.

Copy link
@gtmanfred

gtmanfred Aug 1, 2018

Contributor

This should use salt.utils.json.load

@dmacvicar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 1, 2018

Before moving onto fixing the issues in the roster code, I would focus on the RFC.

I would like some feedback on the 2nd alternative. Because if we instead of merging the roster module add a more generic script roster module, then for terraform one would only need the terraform salt provider, and the roster generic script module can be reused for other integrations.

In summary, instead of terraform.py and master having roster: terraform, having something like roster_script: terraform-provider-salt --roster and move all this functionality to the provider. Which would act as a terraform provider plugin when called by terraform and as a roster provider when called by Salt.

@isbm

This comment has been minimized.

Copy link
Contributor

commented Aug 2, 2018

@terminalmage @cachedout @rallytime any thoughs on design here?

@isbm

This comment has been minimized.

Copy link
Contributor

commented Aug 2, 2018

@dmacvicar whoa, I find this very nice idea (the main one) and the right step towards resources [re]use!

While the idea as it is now is already great and can be left as is, but looking at this from mere user perspective, I find they likely would be even more happy to just call salt-ssh \* state.apply alone. As I would imagine, a particular Salt state could be/do the following:

  • Describe resources in the SLS state something like Salt Cloud already doing, which is similar to how Terraform is describing resource. Maybe we could find one generic syntax for SLS on this regard and make sure it is applicable everywhere?
  • Render terraform cf from that state structure (extend terraform.py with this ability, perhaps?) so no one needs to describe things in two different places. Also we could separate resource description from state processing, and so Salt Cloud part would leverage from this a lot (and vice versa).
  • Plan/apply terraform to setup everything
  • Apply the final state on top of it to end the infrastructure setup

Maybe we should also teach Salt to read Terraform state to let Salt know where it is. I am really excited where it goes! 👍 👍

@dmacvicar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 2, 2018

Describe resources in the SLS state something like Salt Cloud already doing, which is similar to how Terraform is describing resource. Maybe we could find one generic syntax for SLS on this regard and make sure it is applicable everywhere?

I think this is true for Salt users. I doubt Terraform users want to change the way they describe resources and that is completely out of the scope of this PR. It belongs to salt-cloud future.

In any case, it is not a simple problem. Terraform graph and Salt states/sls are different. Terraform automatically figures dependencies between resources as you use interpolation between them, unlike Salt require:.

For sake of moving the discussion I beg to keep this topic out. We don't want to saltify terraform. We want to let terraform users use Salt. Everything else is an effort that can be done independently of this PR and should be tackled by someone having a) deep knowledge about limitations of both Terraform and Salt b) some say in the future of salt-cloud.

@dmacvicar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 2, 2018

Something that may be a nice addition, but unrelated to this PR would be a companion ext_pillar so that the terraform-salt-provider could interpolate and expose infrastructure details in the pillar.

To give an example: meta-salt. Create a salt cluster with salt. I use terraform to bring up 4 VMs. I use salt-ssh to configure them, 1 as a salt-master and 3 as a salt-minion. I need salt-ssh to know the ip of the VM that was configured as master to change /etc/salt/minion master key with salt-ssh on the minions. If I have access to the pillar, my "minion" state could use pillar data coming from terraform.

@isbm

This comment has been minimized.

Copy link
Contributor

commented Aug 2, 2018

For sake of moving the discussion I beg to keep this topic out. We don't want to saltify terraform. We want to let terraform users use Salt.

Yes, agreed. As I said, current PR perfectly go in as-is and to me current design makes lots of sense.

@rallytime
Copy link
Contributor

left a comment

I have some small requests. :)

'''
from __future__ import absolute_import, unicode_literals
import logging
import salt.utils.json

This comment has been minimized.

Copy link
@rallytime

rallytime Aug 2, 2018

Contributor

Can we break these imports up into 2 sections? We usually do something like:

# Import Python libs
from __future__ import absolute_import, unicode_literals
import logging
import os.path

# Import Salt libs
import salt.utils.json

etc.

That makes the import list easier to maintain moving forward.

This comment has been minimized.

Copy link
@dmacvicar

dmacvicar Aug 2, 2018

Author Contributor

Done

import logging
import salt.utils.json
import os.path
import salt.utils

This comment has been minimized.

Copy link
@rallytime

rallytime Aug 2, 2018

Contributor

Rather than importing salt.utils can you import the individual files directly? Otherwise you'll see deprecation notices as the big utils init file was broken up in 2018.3.

This comment has been minimized.

Copy link
@dmacvicar

dmacvicar Aug 2, 2018

Author Contributor

Done

@rallytime rallytime requested review from cachedout and terminalmage Aug 2, 2018

@dmacvicar dmacvicar force-pushed the dmacvicar:terraform_roster_3 branch from c9aef2f to 38df2cd Aug 2, 2018

@dmacvicar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 2, 2018

Rebased

# Import Salt libs
import salt.utils.files
import salt.utils.json
from salt.roster.flat import RosterMatcher

This comment has been minimized.

Copy link
@gtmanfred

gtmanfred Aug 2, 2018

Contributor

Can we move this to __utils__ and then call it from a __utils__ function, like roster.targets() which calls RosterMatcher, instead of having to pass in __opts__?

@gtmanfred

This comment has been minimized.

Copy link
Contributor

commented Aug 2, 2018

Overall, i really like this approach, i would just rather see the RosterMatcher stuff moved to utils, instead of importing from another roster, and having to pass opts around.

@dmacvicar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 3, 2018

@gtmanfred I am trying to get this refactored, but after moving it to utils, if I call it using __utils__['roster_matcher.targets'] i get KeyErroras it is not inutils, and if I import it and call it as salt.utils.roster_matcher.targetsI getopts` not defined inside it. So I am missing something.

notworking.diff.txt

@gtmanfred

This comment has been minimized.

Copy link
Contributor

commented Aug 3, 2018

if you remove salt.utils.roster_matcher from the front of calling RosterTargets, does it fix the problem?

def targets(conditioned_raw, tgt, tgt_type, ipv='ipv4'):
    rmatcher = RosterMatcher(conditioned_raw, tgt, tgt_type, ipv)
    return rmatcher.targets()
@gtmanfred

This comment has been minimized.

Copy link
Contributor

commented Aug 3, 2018

Installing to check this.

@gtmanfred

This comment has been minimized.

Copy link
Contributor

commented Aug 3, 2018

With your patch, i get this

[root@salt salt]# salt-ssh \* test.ping
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
AttributeError: 'module' object has no attribute 'roster_matcher'
Traceback (most recent call last):
  File "/bin/salt-ssh", line 8, in <module>
    execfile(__file__)
  File "/root/src/salt/scripts/salt-ssh", line 10, in <module>
    salt_ssh()
  File "/root/src/salt/salt/scripts.py", line 425, in salt_ssh
    client.run()
  File "/root/src/salt/salt/cli/ssh.py", line 23, in run
    ssh = salt.client.ssh.SSH(self.config)
  File "/root/src/salt/salt/client/ssh/__init__.py", line 231, in __init__
    self.tgt_type)
  File "/root/src/salt/salt/roster/__init__.py", line 105, in targets
    targets.update(self.rosters[f_str](tgt, tgt_type))
  File "/root/src/salt/salt/roster/flat.py", line 36, in targets
    return __utils__['roster_matcher.targets'](conditioned_raw, tgt, tgt_type, 'ipv4')
  File "/root/src/salt/salt/utils/roster_matcher.py", line 32, in targets
    rmatcher = salt.utils.roster_matcher.RosterMatcher(conditioned_raw, tgt,
AttributeError: 'module' object has no attribute 'roster_matcher'
Traceback (most recent call last):
  File "/bin/salt-ssh", line 8, in <module>
    execfile(__file__)
  File "/root/src/salt/scripts/salt-ssh", line 10, in <module>
    salt_ssh()
  File "/root/src/salt/salt/scripts.py", line 425, in salt_ssh
    client.run()
  File "/root/src/salt/salt/cli/ssh.py", line 23, in run
    ssh = salt.client.ssh.SSH(self.config)
  File "/root/src/salt/salt/client/ssh/__init__.py", line 231, in __init__
    self.tgt_type)
  File "/root/src/salt/salt/roster/__init__.py", line 105, in targets
    targets.update(self.rosters[f_str](tgt, tgt_type))
  File "/root/src/salt/salt/roster/flat.py", line 36, in targets
    return __utils__['roster_matcher.targets'](conditioned_raw, tgt, tgt_type, 'ipv4')
  File "/root/src/salt/salt/utils/roster_matcher.py", line 32, in targets
    rmatcher = salt.utils.roster_matcher.RosterMatcher(conditioned_raw, tgt,
AttributeError: 'module' object has no attribute 'roster_matcher'

Adding

diff --git a/salt/utils/roster_matcher.py b/salt/utils/roster_matcher.py
index 145a047..40e7533 100644
--- a/salt/utils/roster_matcher.py
+++ b/salt/utils/roster_matcher.py
@@ -29,8 +29,7 @@ log = logging.getLogger(__name__)


 def targets(conditioned_raw, tgt, tgt_type, ipv='ipv4'):
-    rmatcher = salt.utils.roster_matcher.RosterMatcher(conditioned_raw, tgt,
-                                                       tgt_type, ipv)
+    rmatcher = RosterMatcher(conditioned_raw, tgt, tgt_type, ipv)
     return rmatcher.targets()


And it works

[root@salt salt]# salt-ssh \* test.ping
minion:
    True
@dmacvicar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 3, 2018

I fixed the spurious namespace when using RosterMatcher.

I still get undefined __opts__:

  File "/space/git/salt/salt/salt/utils/roster_matcher.py", line 130, in get_data
    ret = copy.deepcopy(__opts__.get('roster_defaults', {}))
NameError: global name '__opts__' is not defined

Which confuses me. Other util modules, like azurerm do a explicit definition of __opts__:

__opts__ = salt.config.minion_config('/etc/salt/minion')
__salt__ = salt.loader.minion_mods(__opts__)

namecheap and vault in salt.utils attemps to manually define __salt___ if missing, but it assumes __opts__ exist:

    global __salt__
    if not __salt__:
        __salt__ = salt.loader.minion_mods(__opts__)

napalm module does a weird thing with wrapped_global_namespace.
Appart of those, I don't see a massive usage of __opts__ in the utils code.

This error happens both in the testcase and runtime.

@dmacvicar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 4, 2018

Adding

__opts__ = salt.config.minion_config('/etc/salt/minion')

in roster_matcher.py fixes the problem for me. Here is my working diff attached.

working.diff.txt

@gtmanfred

This comment has been minimized.

Copy link
Contributor

commented Aug 4, 2018

switch it so you use __utils__

Try this diff

https://gist.github.com/gtmanfred/854724d07ae240005f326a52bdb62172

@dmacvicar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 4, 2018

I think I already mentioned it above, but if I do that, I get:

   -> unit.roster.test_terraform.TerraformTestCase.test_default_output  ............
       Traceback (most recent call last):
         File "/space/git/salt/salt/tests/unit/roster/test_terraform.py", line 62, in test_default_output
           ret = terraform.targets('*')
         File "/space/git/salt/salt/salt/roster/terraform.py", line 174, in targets
           return __utils__['roster_matcher.targets'](raw, tgt, tgt_type, 'ipv4')
       KeyError: u'roster_matcher.targets'

It is working on runtime (invoking salt-ssh). It happens when running unit tests. I am looking into other tests to see what kind of magic is used to insert __utils__ into the test.

@gtmanfred

This comment has been minimized.

Copy link
Contributor

commented Aug 4, 2018

Ahh, yeah, it should just be something with setup_loader_modules that just needs to inject the opts into the utils modules.

Check out the boto module tests, they use that a lot.

@dmacvicar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 6, 2018

Done

@gtmanfred
Copy link
Contributor

left a comment

🎉

@dmacvicar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 7, 2018

So, what do I need to do now? :-)

@gtmanfred gtmanfred changed the base branch from develop to fluorine Aug 7, 2018

@gtmanfred gtmanfred added the Fluorine label Aug 7, 2018

@gtmanfred

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2018

Just waiting on reviews, I moved this over to the fluorine branch to make sure it gets in.

@terminalmage
Copy link
Contributor

left a comment

I would rather this be reviewed by someone familiar with Terraform, as I know nothing about it.

@rallytime rallytime requested review from dubb-b and removed request for cachedout Aug 9, 2018

@dubb-b

This comment has been minimized.

Copy link

commented Aug 16, 2018

@dmacvicar I have tested this out using the examples from your GH repo using libvirt. I really like the way you implemented this. I am working on a final few items here and I will be done with my review. One thing that may be kind of cool is using the example code that you have in your GH repo is great, although a working example in a Docker container might also be nice. This would allow people to see how this is all wired up in case they have issues with any of the steps listed. Overall this is a very nice feature to Salt and thank you for making it. I should be ready to give the thumbs up tomorrow.

@dmacvicar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 16, 2018

Thanks for the feedback @dubb-b .

As the roster plugin needs the provider counter-part, it will be quite important to make sure people can use it. I am already providing builds of the code in my repo, and once the Salt part is released I will consider doing stable/tagged releases of it.

The installation should be as as easy as copying the builds to terraform.d/plugins. As I am building everything in OBS (including Ubuntu builds), and OBS supports building containers, it should be feasible providing a more plug and play example.

The only thing that bothers me is that the official roster module depends on a very specific provider that for now is unreleased and hosted in my home project, however:

  • Anybody implementing a provider that uses the same resource names can provide alternative implementations of the salt provider
  • I am open to discuss how to make this provider "official", eg: moving it to the _terraform-providers` project so that it is auto-installed by terraform once it is used.
  • All the above steps are worthy only if the roster module is part of Salt
  • With all the disadvantages, I still think the new design/approach outweighs any disadvantage vs the old PR that had resource knowledge.

I think I could also create some kind of diagram and add it to the documentation later. It would also help understanding all the wiring, as you called it.

@dubb-b

dubb-b approved these changes Aug 16, 2018

Copy link

left a comment

I am finished with the testing, it is working quite well. @rallytime 👍

@cachedout

This comment has been minimized.

Copy link
Collaborator

commented Aug 17, 2018

@dmacvicar There are a few issues identified by CodeClimate. It's not required that they be fixed but could you please take a look and see if any of them seem reasonable to you?

https://codeclimate.com/github/saltstack/salt/pull/48873

@dmacvicar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 17, 2018

I have reviewed those.

I think all of the issues validate my view on why I don't use or like CodeClimate. They are false alarms and the code is fine.

@gtmanfred gtmanfred merged commit 02823e5 into saltstack:fluorine Aug 18, 2018

6 of 9 checks passed

codeclimate 10 issues to fix
Details
jenkins/pr/py2-centos-7 The py2-centos-7 job has failed
Details
jenkins/pr/py3-ubuntu-1604 The py3-ubuntu-1604 job has failed
Details
WIP ready for review
Details
continuous-integration/jenkins/pr-merge This commit looks good
Details
jenkins/pr/docs The docs job has passed
Details
jenkins/pr/lint The lint job has passed
Details
jenkins/pr/py2-ubuntu-1604 The py2-ubuntu-1604 job has passed
Details
jenkins/pr/py3-centos-7 The py3-centos-7 job has passed
Details
@MalloZup

This comment has been minimized.

Copy link
Contributor

commented Aug 20, 2018

awesome 👍 i can't wait for using it 🌞

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.