New connection plugin for HTTP(S)-based APIs

[Qalthos]

Proposal: HTTP(S)-transport connection plugin

Author: Nathaniel Case <@Qalthos>

Date: 2018-04-10

• Status: New
• Proposal type: core
• Targeted Release: 2.6
• Estimated time to implement: 6 weeks

Motivation

Implement a first order connection plugin like connection: network_cli & connection: netconf for HTTP(S) transports.
Allows full deprecation of connection: local & provider: for network devices with eAPI and NX-API implementations.
This proposal covers the connection plugin and creation of an eAPI implementation plugin, replacing #102

Problems

• As a user, I want to manage Arista devices that can only be accessed via eAPI
• As a user, I want to connect to an Arista device over eAPI without using connection: local and provider so I can use basic Ansible connection options instead, including specifying http(s) port and encrypted or non-encrypted transport type (HTTP/HTTPS)
• As a user, I want existing playbooks using connection: local to continue to function as before without user intervention
• As a user, I want to securely connect to eos devices via a proxy
• As a user, I want to securely connect using using username & password or a certificate
• As a user, I want to have enough information to debug when the connection/authentication fails
• As a user, I want to be able to control my operating mode (such as privilege mode)
• As a user, I want to be able to reset the connection using the Ansible meta handler reset_connection

Solution proposal

This connection plugin works in concert with a new API plugin system that determines the specifics of the API in question. This new plugin has the following method:

• send_request(self, data, **message_kwargs)
  • data should correspond to the actual message being sent to the device
  • message_kwargs should specify how the message is sent. This will vary according to the particular API in use, but might include message type, destination, or optional headers
  • data and **message_kwargs are interpreted to create a payload to send over HTTP
  • This payload is sent to the connection's send() method, defined below.
  • The message response is unpacked and either sent back to the module, or an appropriate exception can be raised to indicate a failure

Additionally, there is an optional login(self) method which can be used to acquire auth tokens. This method should call connection.send() with the apropriate parameters.

The connection plugin itself works like the existing network_cli and netconf plugins, and is tasked with maintaining the session information for the module.

• When the connection is created
  • check that network_os is set
  • load appropriate cliconf plugin, if one exists
  • load appropriate api implementation plugin
• When _connect(self) is called
  • ensure socket_path is valid
  • call api plugin's login() method
  • set _connected to indicate connection is valid
• When close(self) is called
  • clear any auth token saved to force new login
  • unset _connected
• reset(self) calls close() if socket_path is set
• put_file, fetch_file, and exec_command all redirect to an internal local connection, to allow non-networking modules to function on the local host, as is done in existing network connection plugins.
• __getattr__ is defined so method calls made to the connection plugin not defined in the connection plugin will search first the registered API plugin, then the registered cliconf plugin, if one exists, for like-named methods to execute. In this way, the methods of those plugins are available to consumers of the connection plugin.
• When send(self, path, data, **request_kwargs) is called
  • saved session information (if any) are applied to request_kwargs
  • the request's full URL is built from {protocol}://{host}:{port}/{path}. protocol is 'https' if connection option use_ssl is true (default), otherwise 'http'
  • open_url(url, data, **request_args) is called
  • session information (if any) is saved from the response
  • the response is returned

Finally, some changes to the eos and nxos cliconf plugins will be necessary to use send_request instead of send when communicating over this new connection plugin. This is to allow using the cliconf plugin without overly burdening the API plugins with an interface designed for command-driven connections

Testing

• Add eAPI connection case to existing eos integration tests
• Move all connection: local tests to single test covering both transports
• Each user story should be defended
• High code coverage for new code

Documentation

• EOS docs fragments need to be updated to remove mentions of "This option is only required if you are using eAPI" with reference to connection: local
• EOS platform specific page needs to be altered to replace any reference to connection: local with the new connection type
• The network section of the become guide should be updated to include the new connection type

[caphrim007]

is this proposal only in the context of Arista?

[sivel]

Just wanted to point to a PR that I have been working on that will enable creating a Request object, that can handle cookies/headers/etc from a global perspective across multiple requests.

I believe the Request PR can improve functionality in such a plugin as mentioned here.

ansible/ansible#37622

[privateip]

@Qalthos i have some questions / doubts that his will be generic enough to be handled in the connection plugin if we are targetting any request of http.

build_request(self, commands, output_format, **kwargs) returns a tuple of (data, headers) suitable for being used with open_url()

This method would only be uses in specific instances of network devices sending CLI commands and would be better handled in a plugin that provides messaging structure. That would allow the connection plugin here to be generic enough to handle all http transport use cases as discussed #102 (comment)

[Qalthos]

@caphrim007 Not exactly. This proposal covers eAPI as the first consumer of the plugin (and to reuse as much as possible from #102), but if you have needs not covered by the Problems heading, please note them here so that can be tracked.

@sivel That PR looks like it would make this a lot easier, yes.

[caphrim007]

@Qalthos most of the items in the Problems section are also applicable to F5. Additionally, I would include

• I need to be able to reset the connection within the module.

The reason for the above is that some of the work I need to do involves the device actually rebooting, and I cant have the module fail with a connection terminated when that happens.

[bcoca]

Proposals that try to solve similar or related issues:

auth plugins #24
stackable connection plugins #25
special namespaced credential vars #81
cloud shared credentials #90

[Qalthos]

@caphrim007 I can add that detail. Connections already have a reset() method, which I have neglected t add, which in this case would just call close()

[caphrim007]

@Qalthos does the reset work from inside the module itself? Would it handle an authentication case where,

• user/pass are provided to an endpoint
• return value is stored in a new header
• future communication with other endpoints preserves this header
• if connection is reset (manually for my cases) then the first 3 bullets need to re-occur

[Qalthos]

Based on conversations with @privateip, I've reorganized the flow of this proposal to try to reduce the need to copy network_cli so closely.

@caphrim007 Yes, clearing that header and indicating that it should be re-acquired is one of the two functions of close, along with closing the persistent connection socket. This will force connect() to be called again on next use, which should restore the information.