WIP: bypass ansible-connection for existing connections

[nitzmahone]

SUMMARY

fixes #75813

Creating new ansible-connection processes to use existing persisted connections is extremely wasteful and slow, since the newly-created ansible-connection effectively exits immediately in that case. This PR significantly reduces connection overhead for all networking connection plugins, but especially speeds up local actions under host contexts where a persistent connection is in use (since TE was blindly running ansible-connection in cases where no connection was used).

• skip creation of (expensive!) new process once a persistent connection is going
• move persistent connection detection logic into main worker
• defer persistent connection creation/update until the connection is used
• clean up special cases in TE, speed up dispatch of local-only actions under persistent connection hosts

KNOWN ISSUES:

• Deferred PlayContext updates have incorrect values (just commented that out for now)- maybe a race where we need to snapshot the playcontext immediately? eg, explicit inventory ansible_user is wrong on playcontext remote_user on subsequent connection round-trips- seems to default to current UNIX user instead. Not sure how much longer we'll need to serialize playcontext anyway, since we're already sending over all the connection vars.
• Breaks networking's experimental direct module execution without a small patch to the base networking action direct exec case to ensure the connection is activated (since TE used to do this preemptively even when it wasn't necessary):

            if hasattr(self._connection, '_ensure_socket_path'):
                self._connection._ensure_socket_path()

This could be done in the base persistent connection's _connect method , except that it's defined as abstractmethod, so none of the subclasses are currently calling super()._connect().

ISSUE TYPE

• Bugfix Pull Request

COMPONENT NAME

ansible-connection (et al)

ADDITIONAL INFORMATION

cc @cidrblock @Qalthos

[bcoca]

related #77509

[bcoca]

The problem is that you start the connection before we know the connection details for sure, so getting the data earlier will make it even worse. In some cases (when all is known before task templating) you should have no issues as play_context carries the correct info, but if you need info to change at the task, or worse, loop/delegation levels you'll run into trouble much sooner.

Most of the time this problem can be ignored as plays tend to reuse the same information w/o changing it, specially networking ones.

As you note, since we are passing the connection information and using set_options/get_option on them, we should be able to ignore most of the play_context data, but this requires all connection plugins to correctly source the data (all those in core do, but many of those in collections do not). Why I was planning to add deprecation notices to play_context soon.

[bcoca]

instead of 'starting connection' here we might want to 'reserve socket path', to create it if it doesn't exist and to 'touch it' if it does (making a-connection timeout reset so we avoid a race condition)

[ryanmerolle]

If this was ready soon, would it wait for a .0 release/major release?

[webknjaz]

ansibot added needs_ci and removed needs_rebase labels

@mkrizek what's happening here?