Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run Mitogen as a Module #380

Closed
jmcgrath207 opened this issue Sep 28, 2018 · 2 comments
Closed

Run Mitogen as a Module #380

jmcgrath207 opened this issue Sep 28, 2018 · 2 comments

Comments

@jmcgrath207
Copy link

@jmcgrath207 jmcgrath207 commented Sep 28, 2018

I am trying to run Mitogen as a module for my environment but I am having an issue that is similiar too @366 where it had a issue with 'Router' object has no attribute 'importer'.

Is there any way not to use if __name__ == "__main__":?

  • Have you tried the latest master version from Git?
    yes
  • Mention your host and target OS and versions
    host: Arch Linux
    target: Red Hat 6
  • Mention your host and target Python versions
    Both Python 3.6.4

CODE

import mitogen.master

import mitogen.core
from MitogenService import MitogenParentContextType
import logging
import mitogen.utils





class MitogenSshClass(object):

    def __init__(self,debug=False):

        self.broker = mitogen.master.Broker()
        self.router = mitogen.master.Router(self.broker)
        if debug:
            logging.basicConfig(level=logging.DEBUG)




    def closeAllConnections(self):
        self.broker.shutdown()



    def createConnection(self,hostname: str, username: str, password: str,
                 python_path: str = 'customPython',via: MitogenParentContextType = None):


        return self.router.ssh(
            hostname=hostname,
            username=username,
            password=password,
            python_path=python_path,
            via=via
        )

    def __enter__(self):
        None

    def __exit__(self,*args):
        if mitogen.core.CallError in args:
            print('Call failed:')
            for x in args:
                print(x)
            self.broker.shutdown()
        elif KeyboardInterrupt in args:
            for x in args:
                print(x)
            self.broker.shutdown()
        elif Exception in args:
            for x in args:
                print(x)
            self.broker.shutdown()
        else:
            None



prodIP = 'prod'
prodUsername = 'username'
prodPassword = 'password'


hostIP='host'
hostUserName='username'
hostPassword='password'

mitogenSshClass = MitogenSshClass(debug=True)

prod = mitogenSshClass.createConnection( hostname=prodIP, username=prodUsername, password=prodPassword,
                 python_path= 'customPython')

host = mitogenSshClass.createConnection( hostname=hostIP, username=hostUserName, password=hostPassword,
                 python_path= 'customPython',via= prod)


def Call_func():
    print("something")


with mitogenSshClass:
    host.call(Call_func)
@dw
Copy link
Member

@dw dw commented Oct 1, 2018

Apologies for the delay replying to this. Your actual problem is the same as in #366.

I've yet to find any general approach to solving this. So far, the options seem to be:

  • go the crazy hacky route of carving up the module source code even more than already (the __main__ guard removal is already quite a hack!)
  • shipping bytecode/source for individual objects around. This doesn't work across major Python versions, and requires Mitogen to be able to find all dependencies for a particular object. That's very hard, and can't be solved in the general case (e.g. a function that uses exec()).
  • better guards against importing __main__ in particular. That's actually a tiny amount of code and what #366 is about.

I suspect most of the pain here is just the missing error/warning message when the __main__ guard is absent. Happy to hear your ideas :)

Thanks for reporting!

@dw
Copy link
Member

@dw dw commented Oct 3, 2018

So I've thought about this over the past few days and really there isn't much to be improved upon if the __main__ guard is absent. There is no general way through static analysis to know either:

  • The set of statements that create only module globals as opposed to implement the main program flow. Even a human struggles at interpreting this, a machine would fail miserably.
  • The full set of dependencies a module object needs in order to be reconstituted on the remote end. Given globals(), vars(), locals() and exec(), it is generally impossible to implement this robustly with observing the running program.

While partial solutions to both of these are possible, at least for the time I'm not willing to spend time on them. The existing import mechanism is already fragile enough, and it is only dealing with whole modules at a time :)

Meanwhile there is now a __main__ guard warning implemented and #366 is closed, so I'm marking this one closed too.


This is now on the master branch and will make it into the next release. To be updated when a new release is made, subscribe to https://networkgenomics.com/mail/mitogen-announce/

Thanks for reporting this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants