New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High level API #455

dw opened this Issue Dec 14, 2018 · 0 comments


None yet
1 participant
Copy link

dw commented Dec 14, 2018

Current Mitogen use cases aren't particularly complicated, but the API to achieve those simple things is amazingly complicated, and requires reimplementing some things the library should probably handle out of the box -- like references to existent connections, parallelized connection setup, and handling unexpected disconnection.

Use cases (WIP):

  • Transfer files from client->server or server->client
  • Run a Python function on one or more remote hosts
  • Connect to remote hosts, and keep those connections around
  • Connect to remote user accounts over sudo, using those persistent connections

Completely new desires:

  • Natural interface for handling disconnection and/or stuck hosts.
  • Possibly something task-oriented, where Mitogen's presence in the consuming code is limited to a handful of function calls, and can basically be forgotten about otherwise.
  • Some neat abstraction for streamy results generation? Like OpsMop status updates during function run, etc.

Stretch goal: make it possible for all of the Ansible extension to work in terms of the high level API

The initial idea is to provide a kind of 'Session' object that does pretty much everything internally, that can be stashed away by the consumer code, and each time some activity is desired, some child object is created that specifies the connection configurations to use, activities to execute, retry/failure/timeout policies, and finally uses the Session for long-term caching/persistence of the connections it creates.

Very early/nasty API sketch:

# Thread-safe process global object, wraps up service Pool, FileService,
# Broker, Router, etc.
session = mitogen.session.Session()

# ??
blah = session.connection(method='ssh', hostname='blah')
sudo = session.connection(method='sudo', via=blah)

# Thread-safe(?) per-task object, wraps CallChain, connection error management,
# and the subset of connections that will actually be used.

# The objects returned by add_connect() could be Context object in future, once
# async connect is done, but today might be placeholder objects used only by
# the high-level API. Calling connection() with same args as previously returns
# the same object, or maybe arranges for it to be recreated if it got
# disconnected.
txn = session.start_transaction()
txn.set_hosts([blah, sudo])

txn.upload_file('/path/to/file', dest='..', overwrite=True, error_if_exists=False, preserve_mtime=True, ..)
txn.start_upload_file(..) # async version, works like async funcs below

txn.push_directory('/path/to/dir', dest='..')  # rsync-alike
txn.start_push_directory(..) # async

txn.start_download_file(..) # async

txn.start_pull_directory(..) # async

# Array of CallResult objects? With .connection, .retries, .finished, .result, .. ?
results =, ...)

# Simpler async interface?
txn.start_call(func, ...)
for status_update in txn.iter_status_update():   # use some funky neat status update abstraction
    print(status_update.hostname, status_update.value)

for failure in txn.get_failures():
    print('Failed: ', failure.hostname, failure.reason)

Prehistoric Mitogen had an 'ultra simple' API, where the session object was implicit. Maybe it's going too far?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment