/
pyobjects.py
462 lines (342 loc) · 14.5 KB
/
pyobjects.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
# -*- coding: utf-8 -*-
'''
Python renderer that includes a Pythonic Object based interface
:maintainer: Evan Borgstrom <evan@borgstrom.ca>
Let's take a look at how you use pyobjects in a state file. Here's a quick
example that ensures the ``/tmp`` directory is in the correct state.
.. code-block:: python
:linenos:
#!pyobjects
File.managed("/tmp", user='root', group='root', mode='1777')
Nice and Pythonic!
By using the "shebang" syntax to switch to the pyobjects renderer we can now
write our state data using an object based interface that should feel at home
to python developers. You can import any module and do anything that you'd
like (with caution, importing sqlalchemy, django or other large frameworks has
not been tested yet). Using the pyobjects renderer is exactly the same as
using the built-in Python renderer with the exception that pyobjects provides
you with an object based interface for generating state data.
Creating state data
^^^^^^^^^^^^^^^^^^^
Pyobjects takes care of creating an object for each of the available states on
the minion. Each state is represented by an object that is the CamelCase
version of its name (i.e. ``File``, ``Service``, ``User``, etc), and these
objects expose all of their available state functions (i.e. ``File.managed``,
``Service.running``, etc).
The name of the state is split based upon underscores (``_``), then each part
is capitalized and finally the parts are joined back together.
Some examples:
* ``postgres_user`` becomes ``PostgresUser``
* ``ssh_known_hosts`` becomes ``SshKnownHosts``
Context Managers and requisites
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
How about something a little more complex. Here we're going to get into the
core of how to use pyobjects to write states.
.. code-block:: python
:linenos:
#!pyobjects
with Pkg.installed("nginx"):
Service.running("nginx", enable=True)
with Service("nginx", "watch_in"):
File.managed("/etc/nginx/conf.d/mysite.conf",
owner='root', group='root', mode='0444',
source='salt://nginx/mysite.conf')
The objects that are returned from each of the magic method calls are setup to
be used a Python context managers (``with``) and when you use them as such all
declarations made within the scope will **automatically** use the enclosing
state as a requisite!
The above could have also been written use direct requisite statements as.
.. code-block:: python
:linenos:
#!pyobjects
Pkg.installed("nginx")
Service.running("nginx", enable=True, require=Pkg("nginx"))
File.managed("/etc/nginx/conf.d/mysite.conf",
owner='root', group='root', mode='0444',
source='salt://nginx/mysite.conf',
watch_in=Service("nginx"))
You can use the direct requisite statement for referencing states that are
generated outside of the current file.
.. code-block:: python
:linenos:
#!pyobjects
# some-other-package is defined in some other state file
Pkg.installed("nginx", require=Pkg("some-other-package"))
The last thing that direct requisites provide is the ability to select which
of the SaltStack requisites you want to use (require, require_in, watch,
watch_in, use & use_in) when using the requisite as a context manager.
.. code-block:: python
:linenos:
#!pyobjects
with Service("my-service", "watch_in"):
...
The above example would cause all declarations inside the scope of the context
manager to automatically have their ``watch_in`` set to
``Service("my-service")``.
Including and Extending
^^^^^^^^^^^^^^^^^^^^^^^
To include other states use the ``include()`` function. It takes one name per
state to include.
To extend another state use the ``extend()`` function on the name when creating
a state.
.. code-block:: python
:linenos:
#!pyobjects
include('http', 'ssh')
Service.running(extend('apache'),
watch=[File('/etc/httpd/extra/httpd-vhosts.conf')])
Importing from other state files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Like any Python project that grows you will likely reach a point where you want
to create reusability in your state tree and share objects between state files,
Map Data (described below) is a perfect example of this.
To facilitate this Python's ``import`` statement has been augmented to allow
for a special case when working with a Salt state tree. If you specify a Salt
url (``salt://...``) as the target for importing from then the pyobjects
renderer will take care of fetching the file for you, parsing it with all of
the pyobjects features available and then place the requested objects in the
global scope of the template being rendered.
This works for all types of import statements; ``import X``,
``from X import Y``, and ``from X import Y as Z``.
.. code-block:: python
:linenos:
#!pyobjects
import salt://myfile.sls
from salt://something/data.sls import Object
from salt://something/data.sls import Object as Other
See the Map Data section for a more practical use.
Caveats:
* Imported objects are ALWAYS put into the global scope of your template,
regardless of where your import statement is.
Salt object
^^^^^^^^^^^
In the spirit of the object interface for creating state data pyobjects also
provides a simple object interface to the ``__salt__`` object.
A function named ``salt`` exists in scope for your sls files and will dispatch
its attributes to the ``__salt__`` dictionary.
The following lines are functionally equivalent:
.. code-block:: python
:linenos:
#!pyobjects
ret = salt.cmd.run(bar)
ret = __salt__['cmd.run'](bar)
Pillar, grain, mine & config data
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Pyobjects provides shortcut functions for calling ``pillar.get``,
``grains.get``, ``mine.get`` & ``config.get`` on the ``__salt__`` object. This
helps maintain the readability of your state files.
Each type of data can be access by a function of the same name: ``pillar()``,
``grains()``, ``mine()`` and ``config()``.
The following pairs of lines are functionally equivalent:
.. code-block:: python
:linenos:
#!pyobjects
value = pillar('foo:bar:baz', 'qux')
value = __salt__['pillar.get']('foo:bar:baz', 'qux')
value = grains('pkg:apache')
value = __salt__['grains.get']('pkg:apache')
value = mine('os:Fedora', 'network.interfaces', 'grain')
value = __salt__['mine.get']('os:Fedora', 'network.interfaces', 'grain')
value = config('foo:bar:baz', 'qux')
value = __salt__['config.get']('foo:bar:baz', 'qux')
Map Data
^^^^^^^^
When building complex states or formulas you often need a way of building up a
map of data based on grain data. The most common use of this is tracking the
package and service name differences between distributions.
To build map data using pyobjects we provide a class named Map that you use to
build your own classes with inner classes for each set of values for the
different grain matches.
.. code-block:: python
:linenos:
#!pyobjects
class Samba(Map):
merge = 'samba:lookup'
class Debian:
server = 'samba'
client = 'samba-client'
service = 'samba'
class Ubuntu:
__grain__ = 'os'
service = 'smbd'
class RedHat:
server = 'samba'
client = 'samba'
service = 'smb'
To use this new data you can import it into your state file and then access
your attributes. To access the data in the map you simply access the attribute
name on the base class that is extending Map. Assuming the above Map was in the
file ``samba/map.sls``, you could do the following.
.. code-block:: python
:linenos:
#!pyobjects
from salt://samba/map.sls import Samba
with Pkg.installed("samba", names=[Samba.server, Samba.client]):
Service.running("samba", name=Samba.service)
TODO
^^^^
* Interface for working with reactor files
'''
# Import Python Libs
from __future__ import absolute_import
import logging
import os
import re
# Import Salt Libs
from salt.ext.six import exec_
import salt.utils
import salt.loader
from salt.fileclient import get_file_client
from salt.utils.pyobjects import Registry, StateFactory, SaltObject, Map
import salt.ext.six as six
# our import regexes
FROM_RE = re.compile(r'^\s*from\s+(salt:\/\/.*)\s+import (.*)$')
IMPORT_RE = re.compile(r'^\s*import\s+(salt:\/\/.*)$')
FROM_AS_RE = re.compile(r'^(.*) as (.*)$')
log = logging.getLogger(__name__)
try:
__context__['pyobjects_loaded'] = True
except NameError:
__context__ = {}
class PyobjectsModule(object):
'''This provides a wrapper for bare imports.'''
def __init__(self, name, attrs):
self.name = name
self.__dict__ = attrs
def __repr__(self):
return "<module '{0!s}' (pyobjects)>".format(self.name)
def load_states():
'''
This loads our states into the salt __context__
'''
states = {}
# the loader expects to find pillar & grain data
__opts__['grains'] = salt.loader.grains(__opts__)
__opts__['pillar'] = __pillar__
lazy_funcs = salt.loader.minion_mods(__opts__)
lazy_utils = salt.loader.utils(__opts__)
lazy_states = salt.loader.states(__opts__, lazy_funcs, lazy_utils)
# TODO: some way to lazily do this? This requires loading *all* state modules
for key, func in six.iteritems(lazy_states):
if '.' not in key:
continue
mod_name, func_name = key.split('.', 1)
if mod_name not in states:
states[mod_name] = {}
states[mod_name][func_name] = func
__context__['pyobjects_states'] = states
def render(template, saltenv='base', sls='', salt_data=True, **kwargs):
if 'pyobjects_states' not in __context__:
load_states()
# these hold the scope that our sls file will be executed with
_globals = {}
# create our StateFactory objects
mod_globals = {'StateFactory': StateFactory}
for mod in __context__['pyobjects_states']:
mod_locals = {}
mod_camel = ''.join([
part.capitalize()
for part in mod.split('_')
])
valid_funcs = "','".join(
__context__['pyobjects_states'][mod]
)
mod_cmd = "{0} = StateFactory('{1!s}', valid_funcs=['{2}'])".format(
mod_camel,
mod,
valid_funcs
)
exec_(mod_cmd, mod_globals, mod_locals)
_globals[mod_camel] = mod_locals[mod_camel]
# add our include and extend functions
_globals['include'] = Registry.include
_globals['extend'] = Registry.make_extend
# add our map class
Map.__salt__ = __salt__
_globals['Map'] = Map
# add some convenience methods to the global scope as well as the "dunder"
# format of all of the salt objects
try:
_globals.update({
# salt, pillar & grains all provide shortcuts or object interfaces
'salt': SaltObject(__salt__),
'pillar': __salt__['pillar.get'],
'grains': __salt__['grains.get'],
'mine': __salt__['mine.get'],
'config': __salt__['config.get'],
# the "dunder" formats are still available for direct use
'__salt__': __salt__,
'__pillar__': __pillar__,
'__grains__': __grains__
})
except NameError:
pass
# if salt_data is not True then we just return the global scope we've
# built instead of returning salt data from the registry
if not salt_data:
return _globals
# this will be used to fetch any import files
client = get_file_client(__opts__)
# process our sls imports
#
# we allow pyobjects users to use a special form of the import statement
# so that they may bring in objects from other files. while we do this we
# disable the registry since all we're looking for here is python objects,
# not salt state data
Registry.enabled = False
def process_template(template, template_globals):
template_data = []
state_globals = {}
for line in template.readlines():
line = line.rstrip('\r\n')
matched = False
for RE in (IMPORT_RE, FROM_RE):
matches = RE.match(line)
if not matches:
continue
import_file = matches.group(1).strip()
try:
imports = matches.group(2).split(',')
except IndexError:
# if we don't have a third group in the matches object it means
# that we're importing everything
imports = None
state_file = client.cache_file(import_file, saltenv)
if not state_file:
raise ImportError("Could not find the file {0!r}".format(import_file))
state_locals = {}
with salt.utils.fopen(state_file) as state_fh:
state_contents, state_locals = process_template(state_fh, template_globals)
exec_(state_contents, template_globals, state_locals)
# if no imports have been specified then we are being imported as: import salt://foo.sls
# so we want to stick all of the locals from our state file into the template globals
# under the name of the module -> i.e. foo.MapClass
if imports is None:
import_name = os.path.splitext(os.path.basename(state_file))[0]
state_globals[import_name] = PyobjectsModule(import_name, state_locals)
else:
for name in imports:
name = alias = name.strip()
matches = FROM_AS_RE.match(name)
if matches is not None:
name = matches.group(1).strip()
alias = matches.group(2).strip()
if name not in state_locals:
raise ImportError("{0!r} was not found in {1!r}".format(
name,
import_file
))
state_globals[alias] = state_locals[name]
matched = True
break
if not matched:
template_data.append(line)
return "\n".join(template_data), state_globals
# process the template that triggered the render
final_template, final_locals = process_template(template, _globals)
_globals.update(final_locals)
# re-enable the registry
Registry.enabled = True
# now exec our template using our created scopes
exec_(final_template, _globals)
return Registry.salt_data()