redesign variable interface

[bcoca]

Proposal: redesign vars

Author: Brian Coca <@bcoca> IRC: bcoca

Date: 2018/31/31

• Status: New
• Proposal type: core design
• Targeted Release: future release
• Estimated time to implement: weeks

Motivation

Simplify variables for both users and developers

Problems

• Currently 'variable precedence' is very confusing and error prone.
• We need more flexibility and predictable interactions
• Unclear scopes makes it hard to know if a variable is available or not

Solution proposal

Redesign the variable system

• Scopes

  • Global: currently ‘stats’ and extra vars
    Subsections? Should be _ prefixed? Saved namespaces?:
    * cli:  CLI options
    * config: config settings
    * extra: extra vars?
    *ansible: built ins? I.e playbook_python, version, etc

  • Play: (playbook=>play=>block/role=>task subscopes): mainly ‘vars:’
    Subsections:
    * ansible: playbook_dir, play_hosts, roles, role_name, etc

  • Host: mainly from inventory and vars plugins
    Subsections:
    * facts: module facts
    * local: module facts labeled from ‘local’
    *ansible: magic built in? Inventory_hostname, inventory_dir...

    • Defaults? Special scope definable at any of others but lowest priority

      Accessible naked and/or via specific dicts:
      (Global[varname], Play[varname], Host[varname]), Defaults[varname]

• Precedence is simple:
  * Host > Play > Global, and in each, internally based on ‘last defined’
  * vars: sub-scope is to ‘current object and contained objects’ (current already mostly works this way)

• Make role vars private by default:
  Add facility for roles to ‘import vars to global or play or 1st container’?

• Remove set_fact in favor of:
  - set: varname= value= scope= update=overwrite|merge|aggregate

• Deprecate/remove vars_files and include_vars. In favor of varname={{lookup(‘file’...}}?

• Deprecate/remove vars_prompt and ‘pause prompt hack’.  prompt action/lookup?

• Add nonlazy facility? Is ‘set’ enough?

Testing (optional)

much simpler than current as precedence and scopes are clear and simple.

Documentation (optional)

Also much simpler than current, hopefully the new system is much more intuitive

[barlik]

This is such an amazing proposal.

Variable scoping is IMHO the weakest point of Ansible. I hate the fact that I can't set a local variable inside a role (E.g. using register on a task or using set_fact module) and I agree that variable precedence rules are error prone and confusing for everybody.

Make role vars private by default:
Add facility for roles to ‘import vars to global or play or 1st container’?

For this, I would prefer having an option to return a variable from a role:

For example:

- include_role:
    name: myrole
  vars:
    myrole_variable1: var1
  register: myrole_out

- set: varname=play_var value=myrole_out.private_var1  

[alikins]

ansible/ansible#28477 is a pr that implemented tracking of where variables "came from" and displaying that info at higher verbosity levels for troubleshooting. It's not an implementation of the proposal here, but it was rooted in the frustration that the variable scoping and precedence can cause.

The two main aspects of it were:

1. Specifying a 'scope_name' when using combine_vars() to update variable manager. Additional info about the scope to provide extra context via the 'scope_info'. For ex, for a variable that comes from defaults/main.yml, the 'scope_name' is 'play_roles_defaults', and the 'scope_info' is the dict {'role_name': 'myrole', 'role_path': '/etc/ansible/roles/myrole'}

2. A data structure to keep track of where the variables come from and the scope_name and scope_info associated with them. The tracking data knows all the scopes that provided a value for a variable and what the the final result was. For an implementation more tightly integrated into variable_manager, this could be done in variable_manager itself.

That branch is intended as a way to debug and troubleshoot the existing very complicated variable precedence rules. A simpler and more straightforward variable precedence scheme would be better in general, but it could still be useful to support introspection of the origins of the final set of variables.

[eugene-bright]

Relates to ansible/ansible#6189.