New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python ansiballz replacement for faster core modules #36275

Closed
MartinNowak opened this Issue Feb 16, 2018 · 6 comments

Comments

Projects
None yet
5 participants
@MartinNowak

MartinNowak commented Feb 16, 2018

Performance issues have been reported a couple of times, but the fundamental design issue responsible for slow playbooks apparently hasn't been addressed nor even been mentioned.
In fact the switch to AnsiballZ made the problem even worse.
Ansible is clearly the simplest IT automation solution due to it's agent-less mode, but the slow execution makes it tough to stick with Ansible.

Here is a reasonable lower bound expectation for ansible performane (with persistent ssh control master).

time { for i in $(seq 0 9); do ssh example.com echo $i; done }

This runs in about 800ms against a host with 30ms ping latency.

Running the equivalent in ansible

---
- hosts: example.com
  become: no
  gather_facts: no
  tasks:
    - command: echo 0
    - command: echo 1
    - command: echo 2
    - command: echo 3
    - command: echo 4
    - command: echo 5
    - command: echo 6
    - command: echo 7
    - command: echo 8
    - command: echo 9

takes 8s (10x slower; not accounting for ansible-playbook startup (~2s); tested with pipelining/ssh control master).

This has nothing to do with inherent agent-less ssh latency, but all with the inefficient command execution model.
Since the introduction of AnsiballZ, the following steps are executed for every single command (see Ansiballz).

  • [local] pack python command module (and dependencies) as zip archive (cached for a ansible-playbook run)
  • [local] base64 encode zip archive and transfer it as self-extracting python module to remote
  • [remote] start python interpreter to decode base64 payload and write out a temporary zip archive (
    zipped_mod = os.path.join(temp_path, 'ansible_modlib.zip')
    modlib = open(zipped_mod, 'wb')
    modlib.write(base64.b64decode(ZIPDATA))
    modlib.close()
    )
  • [remote] unzip the command module from the temporary zip archive
    f.write(z.read('ansible_module_%(ansible_module)s.py'))
  • [remote] start another python interpreter to execute command module (with dependencies still in zip archive on PYTHONPATH)
    p = subprocess.Popen([%(interpreter)s, module], env=os.environ, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)

Thus easily taking 200+ms for the python executions and quite some time to transfer the Ansiball.

A reasonable approach to gain a massive speedup.

  • Rewrite basic/core commands in a native language (already supported) using a busybox multi-command approach.
  • Copy binary for target platform once and cache it under a content-addressable tmp file (e.g. /tmp/ansible-busybox/988881adc9fc3655077dc2d4d757d480b5ea0e11/core). Cleanup old binaries versions (mtime) to not occupy too much space.
  • Run core commands in native speed using the cached multi-command binary.

Some suiteable languages for almost static binaries would be C, C++, D, Go, Rust, and Swift (in alphabetic order). Cross-compiling for multiple targets/platforms is broadly supported.
Matching binaries could be downloaded and cached on-demand on the master from an Ansible CDN, so to avoid distributing all binary targets/platforms with Ansible.

More complex commands (non-core) are used less often (maybe you have some stats) or are inherently slow anyhow (e.g. apt, gce). Those could keep the AnsiballZ approach as the relative slowdown is negligable.
It's the bread-and-butter commands like lineinfile, stat, copy, uri, command, shell, template that need a speedup.

Supporting this scheme shouldn't be too much effort.
It's already possible to override built-in modules with custom binaries. What's really missing is binary caching and using a single binary for multiple modules (could maybe be done via symlink on target).
Most people run x64 linux servers anyhow, so support for other platforms could be defered and initial development could be done entirely as an optional module plugin (with manual installation).

ISSUE TYPE
  • Feature Idea
COMPONENT NAME
  • Ansiballz
ANSIBLE VERSION

ansible-2.4.3.0

CONFIGURATION
OS / ENVIRONMENT

linux

SUMMARY

slow ansible

STEPS TO REPRODUCE

see intro above

EXPECTED RESULTS

Ansible is about as fast as running commands directly via ssh.

ACTUAL RESULTS

Ansible is confusingly about 10x slower and hardly anyone seems to wonder.
Running big playbooks with 100s of tasks takes several minutes, making clean development of a playbook against a (fresh) server unfeasible.
This in fact undermines the sole goal of IT automation, reproducible servers.

@sivel

This comment has been minimized.

Member

sivel commented Feb 16, 2018

Any solution to switch modules away from python is going to be a non starter.

@MartinNowak

This comment has been minimized.

MartinNowak commented Feb 16, 2018

Any solution to switch modules away from python is going to be a non starter.

For what reason? Lack of non-Python developers?

A reasonable approach to gain a massive speedup.

That was just one possible approach, caching unpacked python modules on the target would at least remove the zip/b64/b64d/unzip overhead, but still leave interpreter startup and loading.

@bcoca

This comment has been minimized.

Member

bcoca commented Feb 16, 2018

you can use non python modules right now, Ansible works with interpreted and compiled language that can produce a module that accepts and returns the expected JSON.

@MartinNowak

This comment has been minimized.

MartinNowak commented Feb 16, 2018

you can use non python modules right now, Ansible works with interpreted and compiled language that can produce a module that accepts and returns the expected JSON

I know and it's mentioned above.

Supporting this scheme shouldn't be too much effort.
It's already possible to override built-in modules with custom binaries. What's really missing is binary caching and using a single binary for multiple modules (could maybe be done via symlink on target).
Most people run x64 linux servers anyhow, so support for other platforms could be defered and initial development could be done entirely as an optional module plugin (with manual installation).

@mkrizek mkrizek removed the needs_triage label Feb 16, 2018

@sivel

This comment has been minimized.

Member

sivel commented Feb 16, 2018

We have had an internal discussion, and have decided that at this current time we are going to close this issue.

We do have plans to evaluate some other methods, and plan to have discussions. As these discussions start to kick off, and we are prepared to begin engaging the wider community to solidify plans, we will reach out to you at that point.

If you have further questions please stop by IRC or the mailing list:

@sivel sivel closed this Feb 16, 2018

@MartinNowak

This comment has been minimized.

MartinNowak commented Feb 17, 2018

The mailing list has some interesting information on the topic.
https://groups.google.com/forum/#!searchin/ansible-project/performance
Validating performance expectations: 10x overhead on ssh
Someone is testing another approach, running an agent for the duration of a playbook run.
Seeking testers for a (radically) performance-improving Ansible plug-in
Looks more complex than caching modules and avoiding interpreter startup, but tries to stick completely with python.

@ansibot ansibot added feature and removed feature_idea labels Mar 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment