Run Mitogen as a Module

[jmcgrath207]

I am trying to run Mitogen as a module for my environment but I am having an issue that is similiar too @366 where it had a issue with 'Router' object has no attribute 'importer'.

Is there any way not to use if __name__ == "__main__":?

• Have you tried the latest master version from Git?
  yes
• Mention your host and target OS and versions
  host: Arch Linux
  target: Red Hat 6
• Mention your host and target Python versions
  Both Python 3.6.4

CODE

import mitogen.master

import mitogen.core
from MitogenService import MitogenParentContextType
import logging
import mitogen.utils





class MitogenSshClass(object):

    def __init__(self,debug=False):

        self.broker = mitogen.master.Broker()
        self.router = mitogen.master.Router(self.broker)
        if debug:
            logging.basicConfig(level=logging.DEBUG)




    def closeAllConnections(self):
        self.broker.shutdown()



    def createConnection(self,hostname: str, username: str, password: str,
                 python_path: str = 'customPython',via: MitogenParentContextType = None):


        return self.router.ssh(
            hostname=hostname,
            username=username,
            password=password,
            python_path=python_path,
            via=via
        )

    def __enter__(self):
        None

    def __exit__(self,*args):
        if mitogen.core.CallError in args:
            print('Call failed:')
            for x in args:
                print(x)
            self.broker.shutdown()
        elif KeyboardInterrupt in args:
            for x in args:
                print(x)
            self.broker.shutdown()
        elif Exception in args:
            for x in args:
                print(x)
            self.broker.shutdown()
        else:
            None



prodIP = 'prod'
prodUsername = 'username'
prodPassword = 'password'


hostIP='host'
hostUserName='username'
hostPassword='password'

mitogenSshClass = MitogenSshClass(debug=True)

prod = mitogenSshClass.createConnection( hostname=prodIP, username=prodUsername, password=prodPassword,
                 python_path= 'customPython')

host = mitogenSshClass.createConnection( hostname=hostIP, username=hostUserName, password=hostPassword,
                 python_path= 'customPython',via= prod)


def Call_func():
    print("something")


with mitogenSshClass:
    host.call(Call_func)

[dw]

Apologies for the delay replying to this. Your actual problem is the same as in #366.

I've yet to find any general approach to solving this. So far, the options seem to be:

• go the crazy hacky route of carving up the module source code even more than already (the __main__ guard removal is already quite a hack!)
• shipping bytecode/source for individual objects around. This doesn't work across major Python versions, and requires Mitogen to be able to find all dependencies for a particular object. That's very hard, and can't be solved in the general case (e.g. a function that uses exec()).
• better guards against importing __main__ in particular. That's actually a tiny amount of code and what Importer should warn for __main__ imports lacking execution guard #366 is about.

I suspect most of the pain here is just the missing error/warning message when the __main__ guard is absent. Happy to hear your ideas :)

Thanks for reporting!

[dw]

So I've thought about this over the past few days and really there isn't much to be improved upon if the __main__ guard is absent. There is no general way through static analysis to know either:

• The set of statements that create only module globals as opposed to implement the main program flow. Even a human struggles at interpreting this, a machine would fail miserably.
• The full set of dependencies a module object needs in order to be reconstituted on the remote end. Given globals(), vars(), locals() and exec(), it is generally impossible to implement this robustly with observing the running program.

While partial solutions to both of these are possible, at least for the time I'm not willing to spend time on them. The existing import mechanism is already fragile enough, and it is only dealing with whole modules at a time :)

Meanwhile there is now a __main__ guard warning implemented and #366 is closed, so I'm marking this one closed too.

---

This is now on the master branch and will make it into the next release. To be updated when a new release is made, subscribe to https://networkgenomics.com/mail/mitogen-announce/

Thanks for reporting this!