New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New interprocess communication foundation #221

Merged
merged 33 commits into from Feb 28, 2017

Conversation

Projects
None yet
4 participants
@mottosso
Member

mottosso commented Feb 16, 2017

Goal

Increase stability and intuitiveness.

  • Support OS returning from sleep (multiple times)
  • Support long-running publishes (hours, days)
  • Support simultaneous use of multiple logged on users
  • Support multiple concurrent publishes from many hosts

Resolves #199

Motivation

pyblish-qml is written using PyQt5 but needs to run in an environment where that isn't available.

To tackle this, it was developed in such a way that the a host, such as Maya, could interact with an application written in PyQt5 by passing messages across process boundaries. Across two independently running processes.

Prior to this PR, the method in which messages were passed to each other was via network communication. A web server was setup in both the host and pyblish-qml so as for them to communicate. The protocol (or "language") in which they communicated was called XML-RPC. It's an excellent protocol, minimal and built-in to Python.

The problem with this approach however is reliability.

Like any web server, the recipient cannot know about the receiver and vice versa. The communication happens entirely asynchronously and independently. Instead, one must transmit a message and hope that an answer is returned within a timely limit.

On top of this, there is the issue of sockets. Sockets are managed by the operating system and may fail at any second. (On Windows) Multiple parties may attempt to send messages over the same port, in which case the OS decides who is the receiver and who isn't.

Implementation

This PR implements a new method for communicating with a host. Rather than passing messages over sockets, messages are now passed via standard in and out.

popen.stdin.write("Message to QML")
message_from_qml = popen.stdout.read()

The benefits of this approach is that there is one less man-in-the-middle. The OS is no longer part of the equation, which means reliability is fully under our control.

The disadvantage is that the only way to communicate with pyblish-qml is through the process that initially started it, such as a host. In practice, this shouldn't be an issue, as the same also applies to any software running in any host that isn't using interprocess communication.

Maya

As of this writing, the features are incomplete and to be considered ALPHA. Here is how you can try it in Maya.

from pyblish_qml.ipc import server, service
from pyblish_qml.host import install_host

install_host()

popen = server.PopenServer(
    service=service.Service(),
    python="C:/Python27/python.exe",
    pyqt5="C:/modules/python-qt5"
)

As you can see, you are required to pass both a location to Python and PyQt5. Pyblish libraries are assumed to be available already, and it will inherit those.

A moment after running this, you should see the familiar pyblish-qml GUI appear. On closing this window, the process dies and you can call it again.

Next step

As showing the GUI can take considerable time (seconds), I intend on implementing a minimal splash screen to give the user an indication of something happening.

mottosso added some commits Feb 16, 2017

Enable support for threaded dispatch wrapper
Required for Maya, Nuke and other hosts to properly run commands from a thread.

@mottosso mottosso added the feature label Feb 16, 2017

@BigRoy

This comment has been minimized.

Show comment
Hide comment
@BigRoy

BigRoy Feb 16, 2017

Member

Related to the boot time could this potentially be done?

  1. Run a command like pyblish_qml.setup() which would asynchronously load Pyblish (Python and PyQt5) from the host and keep it alive.

This could run at launch time of the host application. This way it's connected to the host and boot time happens at a time when publishes usually aren't immediately done.

  1. Show the gui afterwards with: pyblish_qml.show()

Since this would launch it from the host would this also allow multiple publishes at the same time from different hosts?

Member

BigRoy commented Feb 16, 2017

Related to the boot time could this potentially be done?

  1. Run a command like pyblish_qml.setup() which would asynchronously load Pyblish (Python and PyQt5) from the host and keep it alive.

This could run at launch time of the host application. This way it's connected to the host and boot time happens at a time when publishes usually aren't immediately done.

  1. Show the gui afterwards with: pyblish_qml.show()

Since this would launch it from the host would this also allow multiple publishes at the same time from different hosts?

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Feb 16, 2017

Member

Since this would launch it from the host would this also allow multiple publishes at the same time from different hosts?

Yes, it would.

Run a command like pyblish_qml.setup() which would from the host asynchronously load of Pyblish and keep it alive.

Yes, of course. That's a good idea. That could work. Basically what we did initially with the original RPC method, but with all the benefits of not using RPC.

Member

mottosso commented Feb 16, 2017

Since this would launch it from the host would this also allow multiple publishes at the same time from different hosts?

Yes, it would.

Run a command like pyblish_qml.setup() which would from the host asynchronously load of Pyblish and keep it alive.

Yes, of course. That's a good idea. That could work. Basically what we did initially with the original RPC method, but with all the benefits of not using RPC.

@tokejepsen

This comment has been minimized.

Show comment
Hide comment
@tokejepsen

tokejepsen Feb 16, 2017

Member

Just to dumb it down for me 😄

pyblish-qml would be a subprocess of the application (Maya)?

Member

tokejepsen commented Feb 16, 2017

Just to dumb it down for me 😄

pyblish-qml would be a subprocess of the application (Maya)?

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Feb 16, 2017

Member

Yes, that's right. :)

Maya would be launching pyblish-qml. Put another way, pyblish-qml would be launched from within Maya. It's due to the fact that Maya is launching it that Maya is able to communicate with it, and no other host. That's why every host would have to launch their own copy, like with pyblish-lite.

Member

mottosso commented Feb 16, 2017

Yes, that's right. :)

Maya would be launching pyblish-qml. Put another way, pyblish-qml would be launched from within Maya. It's due to the fact that Maya is launching it that Maya is able to communicate with it, and no other host. That's why every host would have to launch their own copy, like with pyblish-lite.

@BigRoy

This comment has been minimized.

Show comment
Hide comment
@BigRoy

BigRoy Feb 16, 2017

Member

Also, to be complete with this.


Would this solve the "No connection could be made because the target machine actively refused it" issue?

I'm assuming so since we're not dealing with sockets anymore.


How would we ensure the child process is killed if the host crashes? (Would that actually work?)

Member

BigRoy commented Feb 16, 2017

Also, to be complete with this.


Would this solve the "No connection could be made because the target machine actively refused it" issue?

I'm assuming so since we're not dealing with sockets anymore.


How would we ensure the child process is killed if the host crashes? (Would that actually work?)

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Feb 16, 2017

Member

Would this solve the "No connection could be made because the target machine actively refused it" issue?

Indeed it would, no more sockets involved.

How would we ensure the child process is killed if the host crashes? (Would that actually work?)

I thought about this, but then it struck me that the process would die with the graphical GUI. Once the user hits the X, the process is dead. So when Maya crashes, the GUI would remain alive, but I'd expect the user would simply close it himself.

More practically, I'd imagine the odds of a Maya crash happening whilst having the GUI open to be rather slim. Maya is more likely to crash at another time, and if QML isn't visible, it isn't running. (Unless we implement what we just talked about above.)

Member

mottosso commented Feb 16, 2017

Would this solve the "No connection could be made because the target machine actively refused it" issue?

Indeed it would, no more sockets involved.

How would we ensure the child process is killed if the host crashes? (Would that actually work?)

I thought about this, but then it struck me that the process would die with the graphical GUI. Once the user hits the X, the process is dead. So when Maya crashes, the GUI would remain alive, but I'd expect the user would simply close it himself.

More practically, I'd imagine the odds of a Maya crash happening whilst having the GUI open to be rather slim. Maya is more likely to crash at another time, and if QML isn't visible, it isn't running. (Unless we implement what we just talked about above.)

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Feb 16, 2017

Member

On performance, here's a wild thought.

The problem is that when showing QML, we need to (1) run python.exe which pulls in its own dependencies (~50mb) followed by (2) PyQt5 which pulls in its dependencies (~50mb) totalling about 100 mb being read each time QML is shown.

The second time, QML is bound to appear faster due to filesystem caching, it happens natively and under the hood from within the OS.

The question is, can we somehow manually "cache" this 100 mb, in preparation for it being used?

Member

mottosso commented Feb 16, 2017

On performance, here's a wild thought.

The problem is that when showing QML, we need to (1) run python.exe which pulls in its own dependencies (~50mb) followed by (2) PyQt5 which pulls in its dependencies (~50mb) totalling about 100 mb being read each time QML is shown.

The second time, QML is bound to appear faster due to filesystem caching, it happens natively and under the hood from within the OS.

The question is, can we somehow manually "cache" this 100 mb, in preparation for it being used?

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Feb 16, 2017

Member

If you guys could let me know the time it takes for this to boot, that would be most helpful. Both the first time, and second time. Doesn't need to be exact, but in seconds would be great.

Member

mottosso commented Feb 16, 2017

If you guys could let me know the time it takes for this to boot, that would be most helpful. Both the first time, and second time. Doesn't need to be exact, but in seconds would be great.

@BigRoy

This comment has been minimized.

Show comment
Hide comment
@BigRoy

BigRoy Feb 17, 2017

Member

Here are some quick numbers (was unable to get a fully functional UI; it remained blank but did show up.):

With a local copy of Python.exe and local PyQt5: 3-4 seconds
With a local copy of Python.exe and PyQt5 on new server: 4-5 seconds
With a local copy of Python.exe and PyQt5 on old server (slow): 5-6 seconds
With a copy of Python.exe on new server and PyQt5 locally: +/- 4 seconds
With a copy of Python.exe on new server and PyQt5 on new server: +/- 6 seconds
With a copy of Python.exe on new server and PyQt5 on old server: +/- 7 seconds

Additional runs (after first one) seemed to be similar speeds.
These weren't super accurately timed, but hopefully it's of any use.

Additionally, I'm getting this in the script editor:

Setting up Pyblish QML in Maya
Starting Pyblish..
Launching via popen
Listening on 127.0.0.1:9090
Entering state: "initialising"
Entering state: "visible"
Awaited statemachine for 7.00 ms
Entering state: "alive"
Entering state: "clean"
Entering state: "ready"
Entering state: "hidden"
Spent 1204.00 ms creating the application
Spent 26.00 ms resetting
Made 0 requests during publish.
Made 0 requests during reset.
Spent 215.00 ms resetting
Entering state: "ready"

And this in the Output Window:

file:///D:/git_pipeline/pyblish/pyblish-qml/pyblish_qml/qml/app.qml:9:1: module "QtQuick.Controls" version 1.3 is not installed

I was likely testing with an older version of python-qt5?

Member

BigRoy commented Feb 17, 2017

Here are some quick numbers (was unable to get a fully functional UI; it remained blank but did show up.):

With a local copy of Python.exe and local PyQt5: 3-4 seconds
With a local copy of Python.exe and PyQt5 on new server: 4-5 seconds
With a local copy of Python.exe and PyQt5 on old server (slow): 5-6 seconds
With a copy of Python.exe on new server and PyQt5 locally: +/- 4 seconds
With a copy of Python.exe on new server and PyQt5 on new server: +/- 6 seconds
With a copy of Python.exe on new server and PyQt5 on old server: +/- 7 seconds

Additional runs (after first one) seemed to be similar speeds.
These weren't super accurately timed, but hopefully it's of any use.

Additionally, I'm getting this in the script editor:

Setting up Pyblish QML in Maya
Starting Pyblish..
Launching via popen
Listening on 127.0.0.1:9090
Entering state: "initialising"
Entering state: "visible"
Awaited statemachine for 7.00 ms
Entering state: "alive"
Entering state: "clean"
Entering state: "ready"
Entering state: "hidden"
Spent 1204.00 ms creating the application
Spent 26.00 ms resetting
Made 0 requests during publish.
Made 0 requests during reset.
Spent 215.00 ms resetting
Entering state: "ready"

And this in the Output Window:

file:///D:/git_pipeline/pyblish/pyblish-qml/pyblish_qml/qml/app.qml:9:1: module "QtQuick.Controls" version 1.3 is not installed

I was likely testing with an older version of python-qt5?

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Feb 17, 2017

Member

Thanks Roy, this is most useful.

With a local copy of Python.exe and local PyQt5: 3-4 seconds

Wow, how is that possible? :O On my machine, this scenario takes less than a second.

With a copy of Python.exe on new server and PyQt5 on old server: +/- 7 seconds

That is terrible.

We must find a way to optimise this.

Member

mottosso commented Feb 17, 2017

Thanks Roy, this is most useful.

With a local copy of Python.exe and local PyQt5: 3-4 seconds

Wow, how is that possible? :O On my machine, this scenario takes less than a second.

With a copy of Python.exe on new server and PyQt5 on old server: +/- 7 seconds

That is terrible.

We must find a way to optimise this.

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Feb 17, 2017

Member

Here are a few new highlights.

  1. python -m pyblish_qml now launches QML like before, but using subprocess instead of RPC
  2. pyblish_qml.show() now launches QML with a splash screen to keep you company
  3. pyblish_qml.api now features the API previously available directly via pyblish_qml
  4. api.register_python_executable() takes an absolute path to your Python executable
  5. api.register_pyqt5() is optional and takes an absolute path to where you have PyQt5 available.
  6. If neither Python nor PyQt5 is provided or registered, it will attempt to search for it. But don't expect this to work, there simply isn't enough information available to something like Maya to figure this one our for itself unless you have taken care of the environment first.
  7. For the environment, you need python on your PATH and PyQt5 be readily available to that Python. Keep in mind that this isn't an option for Nuke, which has it's own executable on the PATH also called python.

Usage is still as before, except you can now also register paths instead. To facilitate running QML from a menu item.

from pyblish_qml import api
api.install()
api.register_python_executable("C:/Python27/python.exe")
api.register_pyqt5("C:/modules/python-qt5")
api.show()

Let me know how that works for you, next step is looking at improving startup performance.

Member

mottosso commented Feb 17, 2017

Here are a few new highlights.

  1. python -m pyblish_qml now launches QML like before, but using subprocess instead of RPC
  2. pyblish_qml.show() now launches QML with a splash screen to keep you company
  3. pyblish_qml.api now features the API previously available directly via pyblish_qml
  4. api.register_python_executable() takes an absolute path to your Python executable
  5. api.register_pyqt5() is optional and takes an absolute path to where you have PyQt5 available.
  6. If neither Python nor PyQt5 is provided or registered, it will attempt to search for it. But don't expect this to work, there simply isn't enough information available to something like Maya to figure this one our for itself unless you have taken care of the environment first.
  7. For the environment, you need python on your PATH and PyQt5 be readily available to that Python. Keep in mind that this isn't an option for Nuke, which has it's own executable on the PATH also called python.

Usage is still as before, except you can now also register paths instead. To facilitate running QML from a menu item.

from pyblish_qml import api
api.install()
api.register_python_executable("C:/Python27/python.exe")
api.register_pyqt5("C:/modules/python-qt5")
api.show()

Let me know how that works for you, next step is looking at improving startup performance.

mottosso added some commits Feb 17, 2017

Fix PyQt5 on Ubuntu 14
PyQt5 on Python 3 under Ubuntu 14 doesn't have QtXml, which causes Qt.py to consider it broken. Qt.py depends on QtXml.
Reimplement --demo flag
Support running pyblish_qml standalone.
@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Feb 17, 2017

Member

Ok, here's the performance on my machine. A moderate Alienware laptop from about 4 years ago.

untitled

Member

mottosso commented Feb 17, 2017

Ok, here's the performance on my machine. A moderate Alienware laptop from about 4 years ago.

untitled

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Feb 18, 2017

Member

Python and PyQt5 can now be registered up-front via the environment.

$ set PYBLISH_QML_PYTHON_EXECUTABLE=c:/python27/python.exe
$ set PYBLISH_QML_PYQT5=c:/modules/python-qt5
$ maya

Reducing the steps necessary at run-time.

from pyblish_qml import api
api.install()
api.show()
Member

mottosso commented Feb 18, 2017

Python and PyQt5 can now be registered up-front via the environment.

$ set PYBLISH_QML_PYTHON_EXECUTABLE=c:/python27/python.exe
$ set PYBLISH_QML_PYQT5=c:/modules/python-qt5
$ maya

Reducing the steps necessary at run-time.

from pyblish_qml import api
api.install()
api.show()
@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Feb 18, 2017

Member

It now also works with pyblish.api.register_gui("pyblish_qml") and thereby the standard menu item in all hosts.

Member

mottosso commented Feb 18, 2017

It now also works with pyblish.api.register_gui("pyblish_qml") and thereby the standard menu item in all hosts.

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Feb 18, 2017

Member

For future selves and those interested, here's the skinny on how it works currently.

 _____________                _______________
|             |    spawns    |               |
|    Maya     |~~~~~~~~~~~~~>|  pyblish-qml  | 1
|             |              |               |
|             |              |               |
|             |              |               |
|      listen o<-------------o stdout        | 2
|             |    command   |               |
|             |              |               |
|       speak o------------->o stdin         | 3
|             |    result    |               |
|_____________|              |_______________|

Starting from the top, when you run show(), pyblish-qml is spawned. This means a new Python process is run that in turn runs pyblish-qml and its graphical interface.

Once run, pyblish-qml will start making requests to the parent process, such as pyblish.api.discover() in order to fetch available plug-ins. Here is a short example of how communication happens between the two processes.

import subprocess

# Maya spawns QML
child = subprocess.Popen(["python", "-u", "-c", "print(raw_input())"],
                         stdout=subprocess.PIPE, stdin=subprocess.PIPE)

# Maya sends message to QML
message = "Hello World\n"
child.stdin.write(message)

# Maya receives message from QML
result = child.stdout.readline()  # Blocks until child prints "Hello World"

assert result == message, "Failed, '%s' != '%s'" % (result, message)
print("Success!")

Things to note.

  1. Writing to stdout is a matter of calling print, alternatively sys.stdout.write()
  2. Reading from stdin is a matter of calling raw_input(), alternatively sys.stdin.write()
  3. Anything written to child.stdin is received in raw_input() which blocks until something is given
  4. Anything read from child.stdout is coming from sys.stdout.write() from within the child.

And that's it really. But how are commands made? Clearly we can't simply print pyblish.api.discover() from within the child and expect a result from the parent?

Technically, yes we could, but we must somehow still distinguish between what is meant to be a command where a result is expected, and what is meant as just a print statement. If you pay attention to the launching of pyblish-qml, there are plenty of messages being printed that are not commands, such as Entering state: "alive".

So we introduce a format for the printed messages.

  1. If a printed message can be parsed with json, it may be a request.
  2. If the decoded json message contains a dictionary and a key header with the value pyblish-qml:popen.request then it is definitely a request and will be treated as such
  3. If not, pass it along to the parent stdout. (I.e. just print it)

Let's look at a more elaborate example of this in action.

import sys
import json
import subprocess

child_process = """
import sys
import json

message = json.dumps({
    "header": "pyblish-qml:popen.request",
    "payload": {
        "name": "discover",
        "args": []
    }
})

print(message)
line = sys.stdin.readline()

try:
    result = json.loads(line)
except ValueError:
    pass
else:
    if result.get("header") == "pyblish-qml:popen.response":
        payload = result["payload"]
        assert payload == ["PluginA"], "Failed"
        print("Success!")
"""

child = subprocess.Popen(["python", "-u", "-c", child_process],
                         stdout=subprocess.PIPE, stdin=subprocess.PIPE)

# Listen for requests
for line in iter(child.stdout.readline, b""):

    try:
        request = json.loads(line)

    except ValueError:
        # If it isn't JSON, it must be a regular message
        sys.stdout.write(line)

    else:
        if request.get("header") == "pyblish-qml:popen.request":
            response = json.dumps({
                "header": "pyblish-qml:popen.response",
                "payload": ["PluginA"]
            })

            # If it carries this header, the child process
            # expects to hear back from the parent.
            message = child.stdin.write(response + "\n")

The output of this script is this.

Success!

And if you pay attention to the script, you'll see that this message is coming from the child. Once the child has made its request to the parent, received the correct response it produces a second message to the parent. This second message - being neither JSON nor a pyblish-qml:popen.request - is simply forwarded by the parent to the parent's own stdout which is how you see it in your terminal.

And that's how communication happens between the two parent - e.g. Maya - and child pyblish-qml process. Now take a second gander at the initial illustration of the two processes and hopefully it makes more sense to you now!

Member

mottosso commented Feb 18, 2017

For future selves and those interested, here's the skinny on how it works currently.

 _____________                _______________
|             |    spawns    |               |
|    Maya     |~~~~~~~~~~~~~>|  pyblish-qml  | 1
|             |              |               |
|             |              |               |
|             |              |               |
|      listen o<-------------o stdout        | 2
|             |    command   |               |
|             |              |               |
|       speak o------------->o stdin         | 3
|             |    result    |               |
|_____________|              |_______________|

Starting from the top, when you run show(), pyblish-qml is spawned. This means a new Python process is run that in turn runs pyblish-qml and its graphical interface.

Once run, pyblish-qml will start making requests to the parent process, such as pyblish.api.discover() in order to fetch available plug-ins. Here is a short example of how communication happens between the two processes.

import subprocess

# Maya spawns QML
child = subprocess.Popen(["python", "-u", "-c", "print(raw_input())"],
                         stdout=subprocess.PIPE, stdin=subprocess.PIPE)

# Maya sends message to QML
message = "Hello World\n"
child.stdin.write(message)

# Maya receives message from QML
result = child.stdout.readline()  # Blocks until child prints "Hello World"

assert result == message, "Failed, '%s' != '%s'" % (result, message)
print("Success!")

Things to note.

  1. Writing to stdout is a matter of calling print, alternatively sys.stdout.write()
  2. Reading from stdin is a matter of calling raw_input(), alternatively sys.stdin.write()
  3. Anything written to child.stdin is received in raw_input() which blocks until something is given
  4. Anything read from child.stdout is coming from sys.stdout.write() from within the child.

And that's it really. But how are commands made? Clearly we can't simply print pyblish.api.discover() from within the child and expect a result from the parent?

Technically, yes we could, but we must somehow still distinguish between what is meant to be a command where a result is expected, and what is meant as just a print statement. If you pay attention to the launching of pyblish-qml, there are plenty of messages being printed that are not commands, such as Entering state: "alive".

So we introduce a format for the printed messages.

  1. If a printed message can be parsed with json, it may be a request.
  2. If the decoded json message contains a dictionary and a key header with the value pyblish-qml:popen.request then it is definitely a request and will be treated as such
  3. If not, pass it along to the parent stdout. (I.e. just print it)

Let's look at a more elaborate example of this in action.

import sys
import json
import subprocess

child_process = """
import sys
import json

message = json.dumps({
    "header": "pyblish-qml:popen.request",
    "payload": {
        "name": "discover",
        "args": []
    }
})

print(message)
line = sys.stdin.readline()

try:
    result = json.loads(line)
except ValueError:
    pass
else:
    if result.get("header") == "pyblish-qml:popen.response":
        payload = result["payload"]
        assert payload == ["PluginA"], "Failed"
        print("Success!")
"""

child = subprocess.Popen(["python", "-u", "-c", child_process],
                         stdout=subprocess.PIPE, stdin=subprocess.PIPE)

# Listen for requests
for line in iter(child.stdout.readline, b""):

    try:
        request = json.loads(line)

    except ValueError:
        # If it isn't JSON, it must be a regular message
        sys.stdout.write(line)

    else:
        if request.get("header") == "pyblish-qml:popen.request":
            response = json.dumps({
                "header": "pyblish-qml:popen.response",
                "payload": ["PluginA"]
            })

            # If it carries this header, the child process
            # expects to hear back from the parent.
            message = child.stdin.write(response + "\n")

The output of this script is this.

Success!

And if you pay attention to the script, you'll see that this message is coming from the child. Once the child has made its request to the parent, received the correct response it produces a second message to the parent. This second message - being neither JSON nor a pyblish-qml:popen.request - is simply forwarded by the parent to the parent's own stdout which is how you see it in your terminal.

And that's how communication happens between the two parent - e.g. Maya - and child pyblish-qml process. Now take a second gander at the initial illustration of the two processes and hopefully it makes more sense to you now!

mottosso added some commits Feb 19, 2017

Support registering multiple Python executables.
Have the most favorable Python first, fallbacks second. For example, the ideal candidate would be a local version of Python, where a network version may be available to those who do not have it installed locally.
@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Feb 19, 2017

Member

Each process in an OS supports 3 channels of communication called "streams".

  1. stdout for printed messages
  2. stderr, like stdout but reserved for error messages
  3. stdin for incoming messages

At the moment, requests are made to a host via stdout whereas stdin is being listened to for responses to these requests . For example, when QML asks the hosts to produce a list of available plug-ins, it expects to receive the fruits of that request via stdin. But when the hosts decides to make a request on its own, such as asking QML to show or appear above other windows, there is nowhere for this request to go as QML would interpret the request as a response to a request that it has not yet made.


Channels

To solve this, I implemented "channels". That is, stdin is listened to as per usual, but instead of handling messages directly, messages are distributed into queues based on the kind of message it is.

channels = {
  "request": Queue(),
  "parent": Queue()
}

if message["header"] == "pyblish-qml:popen.request":
  channels["request"].put(message)

if message["header"] == "pyblish-qml:popen.parent":
  channels["parent"].put(message)

From here, it's a matter of having two threads listening for entries into each corresponding Queue.

while True:
  message = self.channels["request"].get()
  # handle requests

The end result is bi-directional communication, where QML is able to make requests to a host, and vice versa.

data = json.dumps(
    {
        "header": "pyblish-qml:popen.parent",
        "payload": {
            "name": "show",
            "args": [],
        }
    }
)

# Ask GUI to show
popen.stdin.write(data + "\n")

Messages passed to QML can be made via the Proxy.

from pyblish_qml import show, ipc
server = show()

# Minimise or otherwise hide the GUI

proxy = ipc.server.Proxy(server.popen)
proxy.show()  # Show it again

Autoclose

What I'd like to add, is for the child to die when the parent dies, but I am unsuccessful.

Here is the current attempt, in short.

  1. Give the child the process id of the currently running process, i.e. the parent
  2. Have the child query the id for liveliness, and kill itself when the parent is seemingly dead
def _autoclose():
    import ctypes
    PROCESS_QUERY_INFROMATION = 0x1000

    def is_alive(pid):
        handle = ctypes.windll.kernel32.OpenProcess(
            PROCESS_QUERY_INFROMATION, 0, pid)

        if handle == 0:
            return False

        else:
            ctypes.windll.kernel32.CloseHandle(handle)

        return True

    while True:
        time.sleep(1)

        if not is_alive(pid):
            self.exit()

if sys.platform == "win32":
    print("Autoclosing when '%i' dies.." % pid)
    thread = threading.Thread(target=_autoclose)
    thread.daemon = True
    thread.start()

What happens is that the parent, though dead, appears alive so long as the child is querying it. As though the querying somehow kept the parent alive. I can confirm this by querying liveliness from a third process once both child and parent are dead.

Member

mottosso commented Feb 19, 2017

Each process in an OS supports 3 channels of communication called "streams".

  1. stdout for printed messages
  2. stderr, like stdout but reserved for error messages
  3. stdin for incoming messages

At the moment, requests are made to a host via stdout whereas stdin is being listened to for responses to these requests . For example, when QML asks the hosts to produce a list of available plug-ins, it expects to receive the fruits of that request via stdin. But when the hosts decides to make a request on its own, such as asking QML to show or appear above other windows, there is nowhere for this request to go as QML would interpret the request as a response to a request that it has not yet made.


Channels

To solve this, I implemented "channels". That is, stdin is listened to as per usual, but instead of handling messages directly, messages are distributed into queues based on the kind of message it is.

channels = {
  "request": Queue(),
  "parent": Queue()
}

if message["header"] == "pyblish-qml:popen.request":
  channels["request"].put(message)

if message["header"] == "pyblish-qml:popen.parent":
  channels["parent"].put(message)

From here, it's a matter of having two threads listening for entries into each corresponding Queue.

while True:
  message = self.channels["request"].get()
  # handle requests

The end result is bi-directional communication, where QML is able to make requests to a host, and vice versa.

data = json.dumps(
    {
        "header": "pyblish-qml:popen.parent",
        "payload": {
            "name": "show",
            "args": [],
        }
    }
)

# Ask GUI to show
popen.stdin.write(data + "\n")

Messages passed to QML can be made via the Proxy.

from pyblish_qml import show, ipc
server = show()

# Minimise or otherwise hide the GUI

proxy = ipc.server.Proxy(server.popen)
proxy.show()  # Show it again

Autoclose

What I'd like to add, is for the child to die when the parent dies, but I am unsuccessful.

Here is the current attempt, in short.

  1. Give the child the process id of the currently running process, i.e. the parent
  2. Have the child query the id for liveliness, and kill itself when the parent is seemingly dead
def _autoclose():
    import ctypes
    PROCESS_QUERY_INFROMATION = 0x1000

    def is_alive(pid):
        handle = ctypes.windll.kernel32.OpenProcess(
            PROCESS_QUERY_INFROMATION, 0, pid)

        if handle == 0:
            return False

        else:
            ctypes.windll.kernel32.CloseHandle(handle)

        return True

    while True:
        time.sleep(1)

        if not is_alive(pid):
            self.exit()

if sys.platform == "win32":
    print("Autoclosing when '%i' dies.." % pid)
    thread = threading.Thread(target=_autoclose)
    thread.daemon = True
    thread.start()

What happens is that the parent, though dead, appears alive so long as the child is querying it. As though the querying somehow kept the parent alive. I can confirm this by querying liveliness from a third process once both child and parent are dead.

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Feb 20, 2017

Member

Ok, on performance, here's what I've found.

The first time the GUI is opened, it takes between 1-7 seconds. If the user keeps the window open, he can simply re-show the GUI and it will pop to the foreground instantly. In practice, I'd imagine this could become a workflow thing.

I'm close to merging this, any additional thoughts, let me know!

Member

mottosso commented Feb 20, 2017

Ok, on performance, here's what I've found.

The first time the GUI is opened, it takes between 1-7 seconds. If the user keeps the window open, he can simply re-show the GUI and it will pop to the foreground instantly. In practice, I'd imagine this could become a workflow thing.

I'm close to merging this, any additional thoughts, let me know!

mottosso added some commits Feb 21, 2017

Enable console on Windows when not embedded.
Running with CREATE_NO_WINDOW under Windows chokes QML from outputting to stdout or stderr. I accept this cost when running from within a host - calling this "embedded" - but enable output when running from a terminal. That way, one can still develop and debug, but won't get error messages printed when software is used in production.
@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Feb 28, 2017

Member

Python 3 compatibility restored, auto-closing with Maya implemented.

Now QML also runs with Python 3.6 and Qt 5.7!

image

Member

mottosso commented Feb 28, 2017

Python 3 compatibility restored, auto-closing with Maya implemented.

Now QML also runs with Python 3.6 and Qt 5.7!

image

@mottosso mottosso merged commit b61c760 into pyblish:master Feb 28, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@xtvjxk123456

This comment has been minimized.

Show comment
Hide comment
@xtvjxk123456

xtvjxk123456 Jul 27, 2017

image

not working in maya 2015,
(env has set alreadly)
the code:

from pyblish_qml import api
api.install()
api.show()

will make maya2015 crash (or hang on)
please help.

xtvjxk123456 commented Jul 27, 2017

image

not working in maya 2015,
(env has set alreadly)
the code:

from pyblish_qml import api
api.install()
api.show()

will make maya2015 crash (or hang on)
please help.

@mottosso

This comment has been minimized.

Show comment
Hide comment
@mottosso

mottosso Jul 27, 2017

Member

@xtvjxk123456 Please open a new issue about this, and we'll talk about it there.

Member

mottosso commented Jul 27, 2017

@xtvjxk123456 Please open a new issue about this, and we'll talk about it there.

@pyblish pyblish locked and limited conversation to collaborators Jul 27, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.