This section explains Satellite Populate data populate.
Contents
Using $ satellite-populate
you can run the populate
and validate
commands.
That commands are used to read data description from YAML file and
populate the system or validate populated entities.
Having test_data.yaml
with the following content.
vars:
org_label_suffix = inc
actions:
- model: Organization
log: The first organization...
register: org_1
data:
name: MyOrg
label: MyOrg{{org_label_suffix}}
To populate the system
(satellite_env)[you@host]$ satellite-populate test_data.yaml -v -o validation_data.yaml
2017-01-04 04:31:17 - satellite_populate.base - INFO - CREATE: The first organization...
2017-01-04 04:31:19 - satellite_populate.base - INFO - search: Organization {'query': {'search': 'name=MyOrg,label=MyOrg'}} found unique item
2017-01-04 04:31:19 - satellite_populate.base - INFO - create: Entity already exists: Organization 36
2017-01-04 04:31:19 - satellite_populate.base - INFO - registry: org_1 registered
To validate the system use the file generated by population validation_data.yaml
(satellite_env)[you@host]$ satellite-populate validation_data.yaml
(satellite_env)[you@host]$ echo $?
0 # system validated else 1
Use $ satellite-populate --help
for more info
Pass -h --hostname, -p --password, -u --username to the command, or this arguments to decorator:
@populate_with(data, username='x', password='y', hostname='server.com')
NOTE:
"validation data can also be included in config section"
Other way to use populate is via decorator, it is useful to decorate a test_case forcing a populate or validate operation to be performed.
Having a data_file like:
actions: - model: Organization register: organization_1 data: name: My OrgThen you can use in decorators:
@populate_with('file.yaml') def test_case_(self): 'My Org exists in system test anything here'And getting the populated entities inside the test_case:
@populate_with('file.yaml', context_name='my_context') def test_case_(self, my_context=None): assert my_context.organization_1.name == 'My Org' You can also set a customized context wrapper to the context_wrapper argument:: def my_custom_context_wrapper(result): # create an object using result my_context = MyResultContext(result) return my_context @populate_with('file.yaml', context_name='my_context', content_wrapper=my_custom_context_wrapper) def test_case_(self, my_context=None): # assert with some expression using my_context object returned # my_custom_context_wrapper assert some_expression
And if you don't want to have YAML file you can provide a dict:
data_in_dict = { 'actions': [ { 'model': 'Organization', 'register': 'organization_1', 'data': { 'name': 'My Organization 1', 'label': 'my_organization_1' } }, ] } @populate_with(data_in_dict, context_name='my_context', verbose=1) def test_org_1(my_context=None): """a test with populated data""" assert my_context.organization_1.name == "MyOrganization1"
And finally it also accepts bare YAML string for testing purposes:
data_in_string = """ actions: - model: Organization registry: organization_3 data: name: My Organization 3 label: my_organization_3 """ @populate_with(data_in_string, context_name='context', verbose=1) def test_org_3(context=None): """a test with populated data""" assert context.organization_3.name == "My Organization 3" assert context.organization_3.label == "my_organization_3"
NOTE:
"That is important thatcontext_name
argument always be declared using either a default valuemy_context=None
or handle in**kwargs
Otherwisepy.test
may try to use this as a fixture placeholder. And if context_wrapper is set to None, my_context will be the pure unmodified result of populate function."
Decorating UnitTest setUp and test_cases:
class MyTestCase(TestCase): """ This test populates data in setUp and also in individual tests """ @populate_with(data_in_string, context_name='context') def setUp(self, context=None): self.context = context def test_with_setup_data(self): self.assertEqual( self.context.organization_3.name, "My Organization 3" ) @populate_with(data_in_dict, context_name='test_context') def test_with_isolated_data(self, test_context=None): self.assertEqual( test_context.organization_1.name, "My Organization 1" )
In the YAML data file it is possible to specify 3 sections, config
, vars
and actions
.
The config may be used to define special behavior of populator and its keys are:
example:
config:
verbose: 3
populator: api
populators:
api:
module: satellite_populate.api.APIPopulator
cli:
module: satellite_populate.cli.CLIPopulator
Config variables:
config:
# Set verbosity to -v, -vv, -vvv, -vvvv, -vvvvv
# int
# range(0, 5)
verbose: 1
# define the default active populator name
# str
populator: foo
# specify available populators
# dict(<name>=dict(module='module_path'))
populators:
foo:
module: mypack.mymodule.MyPopulatorClass
other:
module: otherpath.OtherClass
# define the mode (override by argument)
# str
# choices: validate | populate
mode: validate
# http or https ? (override by argument)
schema: http
# Satellite system port (override by argument)
port: 443
# hostname (without scheme) (override by argument)
hostname: server.com
# Admin username (override by argument)
username: admin
# admin password (override by argument)
password: changeme
# User for ssh login (override by argument)
ssh_user: root
# Ssh auth (override by argument)
# if None local ~/.ssh pub key is used
# or password
# or keyfile
ssh_auth:
password: 123456
key_file: path/to/file.pub
# raw search rules is a dict of rules
# to force some transformations over nailgun
# EntitySearchMixin
# in the example below we are removing the password
# field from search queries for User entity
raw_search_rules:
user:
password:
remove: true
# In some cases a GPGKey is needed for nailgun
gpgkey:
content: skjfsdhbgbsdhbgsdjbg=
docker_url: system.com:dockerport
# inject following modules to context (import)
add_to_context:
path: os.path
shortname: package.module.module.module.object
# the above will available as {{ shortname }}
Variables to be available in the rendering context of the YAML data
every var defined here is available to be referenced using Jinja
syntax in
any action.
vars:
admin_username: admin
admin_password: changeme
org_name_list:
- company7
- company8
prefix: aaaa
suffix: bbbb
my_name: me
The actions is the most important section of the YAML, it is a list of actions being each action a dictionary containing special keys depending on the action type.
Actions are executed in the defined order and order is very important because
each action can register
its result to the internal registry to be referenced
later in any other action.
The action type is defined in action
key and available actions are:
Crud actions takes a model
argument, any from nailgun.entities
is a valid model,
models are passed as CamelCasedName
of the antity class, then, depending on
the populator being used, that CRUD action can be performed by API, CLI or UI.
List of possible variables for crud actions:
# action name - create | delete | update action: create # entity class model: User # name to register register: my_user # log message to output log: Creating a new user .... # Must iterate a list to repeat the same action? with_items: - item1 - item2 ... # The data to perform a search for the entity data: # base types - int, str, list etc.. name: Foo bar # from an available Python object url: from_object: somemodule.constants.REPO_URL # from a search in the system organization: from_search: model: Organization data: name: SomeCompanyName # from specific ID product: from_read: model: Product data: id: 1 # from registered action user: from_registry: already_existing_user # from fauxfactory generator password: from_factory: alphanumeric # If needed specify data to be used only for search (in validation) search_query: field: something # If needed custom options can be passed to nailgun search search_options: filter: {} # should force a raw search or use attribute search? # note: some entities such as Organization will always be raw searched force_raw: true | false # Choose which populator to use for this specific action # NotImplementedYet via: api | cli | ui | custom_populator # Should errors be silenced and None registered if error? silent_errors: true | false # Run async? # NotImplementedYet async: true | false wait: other_action_register_name # Run only in the case of following condition # Python allowed, registered objects allowed # should be a Boolean operation when: object_a == object_b and 1 > 0
Search for the new entity and creates if not found, else only register the object.
- If no action is informed create will be always the default
- In populate perform search then create
- In validate perform only search
Required variables:
- model: Nailgun Entity Class name
- data: a dictionary to search or populate the entity
Creating a simple Organization:
# a list of dictionaries actions: - model: Organization # the nailgun Entity class # The message to output in the log log: This is the first organization # The name which this object will be registered # to be referenced in other actions. register: my_organization # The data to search or populate the entity data: name: My Company label: mycompany
Creating 2 organizations and 2 users from lists and referencing objects from the registry:
vars: # a list with data for 2 users user_list: - firstname: Michael lastname: Scott - firstname: David lastname: Brent # a list of company names company_names: - Dunder Mifflin - Wernham Hogg actions: # create all the organizations listed above - model: Organization # iterate specified list and repeats the action for each with_items: company_names # include the result in registry # if `with_items` is used, the registered object will be a list register: companies # give the data data: name: "{{item}}" label: "{{item.replace(' ', '')}}" # transform name in a valid label # Create one user as admin for each organization - model: User with_items: user_list data: admin: true # refer to loop iteration using `items` object firstname: "{{item.firstname}}" lastname: "{{item.lastname}}" # Use object methods and Jinja filters to transform data # the following gives us mscott and dbrent login: "{{ '{0}{1}'.format(item.firstname[0], item.lastname) | lower }}" # generate a random password using builtin fauxfactory password: from_factory: alpha # Set the organizations to existing list of orgs organization: from_registry: companies # Set as default org the same positioned in the loop default_organization: from_registry: companies[loop_index]
Get some existing entity and updates it with provided data.
- Executed only in populate mode
- In validate mode it only searches for updated entity
Required variables:
- model: Nailgun Entity Class name
- registry The name registry object
- data: a dictionary to search
Updating the product named old_name with new_name:
actions: - action: update model: Product register: some_product data: name: new_name search_query: name: old_name organization: from_search: model: Organization data: name: Default Organization
If the some_product already exists in registry you can omit the search:
actions: - action: update model: Product register: some_product data: name: new_name
Deletes existing entity.
- Executed only in populate mode
- In validate mode it only searches for updated entity
Required variables:
- model: Nailgun Entity Class name
- registry The name registry object
- data: a dictionary to search
Deleting the product named new_name:
actions: - action: delete model: Product search_query: name: new_name organization: from_search: model: Organization data: name: Default Organization
If the some_product already exists in registry you can omit the search:
actions: - action: delete model: Product register: some_product
Note:
"delete action perform a DELETE call to the api and removes the entity from the system, while unregister action only removes it from runtime registry"
This are other built-in actions
Outputs a message to the LOG and also to stdout.
Required variables:
- log: The message to be logged
Examples:
actions: - action: echo log: Hello World - action: echo log: This an error level: error - action echo log: This message goes also to the stdout print: true - action: echo log: I can read variables, you are {{ env.USER }}
Which outputs:
2017-01-20 00:10:53 - satellite_populate.base - INFO - ECHO: Hello World 2017-01-20 00:10:53 - satellite_populate.base - ERROR - ECHO: This an error 2017-01-20 00:10:53 - satellite_populate.base - INFO - ECHO: This message goes also to the stdout This message goes also to the stdout 2017-01-20 00:10:53 - satellite_populate.base - INFO - ECHO: I can read variables, you are root
Register variables to the runtime registry
Required variables:
- data: A dictionary
Examples:
- action: register data: name: Michael Scott preferred_organization: from_search: model: Organization data: name: My prefered Organization repo_url: from_object: "http://" + file.constants.REPO_BASE_URL
All variables registered above will be available for the next executed actions.
Removes variables from runtime register.
Required variables:
- data: A list of variable names
Examples:
- action: unregister data: - name - preferred_organization - repo_url
All variables unregistered above will be not available for the next executed actions.
Unregister is useful for actions using when: conditions.
Execute predefined assertion operations and fails the validation if assertion returns False.
Required variables:
- operator: Logical operator mapped to a function returning Boolean
- data: A list of two elements to be tested
Built in operators:
- eq # the default
- ne
- gt
- lt
- gte
- lte
- identity
Examples:
- action: assertion log: Check if current user is root operator: eq data: - root - "{{ env.USER }}"
If returns False, the validation ends with exit code 1
Custom Populators can also include custom operators for assertion.
And you can also have special actions defined in a custom populator.
Lets say you have this python module in your project, properly available on PYTHONPATH:
from satellite_populate.api import APIPopulator class MyPopulator(APIPopulator): def action_writeinfile(self, rendered_data, action_data): with open(rendered_data['path'], 'w') as output: output.write(rendered_data['content'])
Now go to your test.yaml
and write:
config: populator: mine populators: mine: module: mypath.mymodule.MyPopulator actions: - action: writeinfile path: /tmp/test.txt content: Hello World!!!
and run:
$ satellite-populate test.yaml -v
There are some ways to fetch dynamic data in action definitions, it depends on the action type.
For any key you can use Jinja
to provide a dynamic value as in:
value: "{{ get_something }}" value: "{{ fauxfactory.gen_string('alpha') }}" value: user_{{ item }}
For some actions you can provide a data
key, that data is used to create
new entities and also to perform searches or build the action function.
Every data
key accepts 4 special reference directives in its sub-keys.
from_registry
Gets anything from registry:
data: organization: from_registry: default_org name: from_registry: my_name
from_object
Gets any Python object available in the environment:
data: url: from_object: name: robottelo.constants.FAKE_0_YUM_REPO
from_search
Perform a search and return its result:
data: organization: from_search: model: Organization data: name: Default Organization
from_read
Perform a read operation, which is useful when we have unique data or id:
data: organization: from_read: model: Organization data: id: 1
Every action which returns a result can write its result to the registry, so it is available to be accessed by other actions.
Provide a register
unique name in action
definition.
The actions that support register
are:
- create
- update
- register
- assertion
All dynamic directives from_*
supports the use of register
Example:
- action: create model: Organization register: my_org data: name: my_org - model: User log: Creating user under {{ register.my_org.name }} data: organization: from_registry: my_org