Multiprocessing opens unwanted windows (on windows only, pyinstaller)

[TimSC]

I am trying to using multiprocessing with kivy. It works fine on linux, but on windows I had to put the kivy imports within the "name == 'main'" section, otherwise it tends to open more blank windows, which I don't want. Is this the best approach?

My problem is that when I package this using pyinstaller, the above work around is not effective and the program keeps spawning new windows. (I had to use a trunk build of pyinstaller due to https://stackoverflow.com/questions/36400111/pyinstaller-doesnt-import-queue/40706162 )

In python 2.7:

import multiprocessing, time
import future
from queue import Empty

def MyFunc(queue, queue2):
	count = 0
	running = True
	while running:
		try:
			msg = queue2.get(True, 1.0)
			print (msg)
			if msg == "stop":
				running = False
		except Empty:
			pass
		count += 1
		queue.put(count)

if __name__ == '__main__':
	import kivy
	kivy.require('1.9.0') # replace with your current kivy version !

	from kivy.app import App
	from kivy.uix.label import Label
	from kivy.clock import Clock

	class MyApp(App):

		def build(self):
			self.queue = multiprocessing.Queue()
			self.queue2 = multiprocessing.Queue()
			self.p = multiprocessing.Process(target=MyFunc, args=(self.queue, self.queue2))
			self.p.start()
	
			Clock.schedule_interval(self.idle, 0.1)
	
			self.lbl = Label(text='Hello world')
			return self.lbl

		def idle(self, dt):
			try:
				while True:
					msg = self.queue.get_nowait()
					self.lbl.text = str(msg)
			except Empty:
				pass
	
	myApp = MyApp()
	myApp.run()
	
	myApp.queue2.put("stop")
	myApp.p.join()

Basic problem and partial work around discussed back in 2014: https://groups.google.com/d/msg/kivy-users/mkW2deHqZGI/pfc_4tusWtwJ

[Deerenaros]

I think that this behaviour is due to multiprocessing architecture. While you "create thread" you do not create thread, you starting second python interpreter which connected to first with some pipe, socket or shared memory magic. Becouse GIL you have no better way to fully load cpu in python, so only way to fix is to check thread and pass creating second window.

And, i think, that it could be slightly better.

import multiprocessing, time
from Queue import Empty

def MyFunc(queue, queue2):
    count = 0
    running = True
    while running:
        try:
            msg = queue2.get(True, 1.0)
            print (msg)
            if msg == "stop":
                running = False
        except Empty:
            pass
        count += 1
        queue.put(count)

import kivy
kivy.require('1.9.0') # replace with your current kivy version !

from kivy.app import App
from kivy.uix.label import Label
from kivy.clock import Clock

class MyApp(App):

    def build(self):
        self.queue = multiprocessing.Queue()
        self.queue2 = multiprocessing.Queue()
        self.p = multiprocessing.Process(target=MyFunc, args=(self.queue, self.queue2))
        self.p.start()

        Clock.schedule_interval(self.idle, 0.1)

        self.lbl = Label(text='Hello world')
        return self.lbl

    def idle(self, dt):
        try:
            while True:
                msg = self.queue.get_nowait()
                self.lbl.text = str(msg)
        except Empty:
            pass

        
if __name__ == '__main__':
    myApp = MyApp()
    myApp.run() # you won't create second window right now i guess
    
    myApp.queue2.put("stop")
    myApp.p.join()

[Zen-CODE]

This is not so much an issue, but a documented feature.

https://kivy.org/docs/api-kivy.core.window.html

Each kivy app has their own window, and you are starting another app. Just make it pure Python (not a kivy app) if you don't want the window? Note that multiprocessing is not available in iOS,

[TimSC]

@Deerenaros your code looks good. I'm not sure why I didn't use that approach. How do I skip creating the window? I notice there is a configuration option in graphics called window_state. In any case, pyinstaller seems to have a different behaviour than running it directly in python.

@Zen-CODE Is it possible not to start another app in the second process? I only need kivy in the main process. I tried removing all kivy imports from the relevant source files but to no avail. :)

UPDATE: One issue might be the need to call multiprocessing.freeze_support() if pyinstaller is to be used on windows. However, it didn't fix the problem.

[Zen-CODE]

Hmm. If you don't import anything from kivy, not sure why it's creating a new one. Could you post a runnable example? The one above will still import kivy for each process....

[Deerenaros]

@TimSC Yeah, may be pyinstaller using more magic but i can not imagine how importing or class definition may create second window.

[Zen-CODE]

@Deerenaros. Importing executes the code in a python file. Some of the kivy widgets and libraries import the Window for metric calculations and binding which then needs to create an actual window to get accurate information. It's built as a GUI framework after all. Thus each process that imports kivy modules will create a window.

If you're still curious, you can try importing widget subclass and step into the code.

[KeyWeeUsr]

@Zen-CODE if only imports are the problem, he can "fix" them like we fix documentation for unwanted imports:

import os
if 'something' not in os.environ:
    import window here

therefore it might be possible to skip that part. Yet I don't see this thing fixing the issue (more work to do) + as said, some platforms might not support multiprocessing.

[Deerenaros]

@Zen-CODE This is bad idea to open window by import cause hard to control. If kivy follow this flow that mean no option except drop great framework due to architecture issues.

[Zen-CODE]

@Deerenaros. It's very easy to make judgements out of ignorance. It is done for a reason. Do you know why? How about understanding why it's done, then suggesting better approaches? We are open to pull pull requests and suggestions :-)

ps. Kivy is GUI provider that runs in a window by definition. Seems strange to suggest creating a window indicates an architectural shortcoming.

[TimSC]

@Zen-CODE if kivy does not work with common python modules due to import restrictions, they I think its fair to say its an architectural limitation. I'd expect it would be done in the App constructor, but that would just be my naive guess.

I suspect some of these design decisions were based on the desire for multiple windows in kivy, which I don't know much about.

[Zen-CODE]

@TimsSC. It does work with common python modules, and you can definitely use some parts of Kivy without creating a window, but importing certain things does create a window.

But look, I can't pretend to be an expert here and am just speaking off the top of my head. I just know an App requires a Window and I've seen other imports initialize a window, but that was not a problem for us so never investigated. As to which ones do, I don't know. Try and see. If you encounter an import that creates a window without reason, post here. But I don't think it should be considered a problem before we know it is one. It's reasonable, when creating GUI component, to create Windows to hold GUI components.

If someone thinks it creates one unnecessarily, we need specific examples. When and where? Until then it's empty speculation no? :-)

[TimSC]

@Zen-CODE I am having problems using kivy with multiprocessing within pyinstaller. This arguably a pyinstaller problem but it could be a kivy architectural issue. I hope that specific enough to dig a little deeper. I will try to explore the import behavior some more...

[Zen-CODE]

Is this specifically with the windows package? Is it a kivy window that opens or a terminal (command prompt)? Remember, on windows, Python has two executables: python.exe (command prompt) and pythonw.exe (windowless). Perhaps the PyInstaller builds the command prompt window? I recall you can specify that somewhere.

What you could do is use process explorer to inspect the window properties. The classname/properties could could probably confirm or eliminate this explanation?

[TimSC]

As far as I can tell, they are opengl windows rather than command terminals.

I am running in virtualbox, so I get some opengl weirdness sometimes.

[Zen-CODE]

Yes, those are errors because you do not have OpenGL 2 working. You need to get that working before Kivy will work. Have you installed the VirtualBox additions?

[TimSC]

Kivy works in my Windows 7 virtualbox, even when run inside of pyinstaller (but without multiprocessing).

[TimSC]

I have a working version in windows/pyinstaller. I found calling freeze_support() must be before the kivy imports but after the second process's entry point (MyFunc in this case). This work around is good enough for me but it is rather hard to discover!

import multiprocessing, time
import future
from queue import Empty

def MyFunc(queue, queue2):
	count = 0
	running = True
	while running:
		try:
			msg = queue2.get(True, 1.0)
			print (msg)
			if msg == "stop":
				running = False
		except Empty:
			pass
		count += 1
		queue.put(count)

if __name__ == '__main__':
	multiprocessing.freeze_support()		
	
import kivy
kivy.require('1.9.0') # replace with your current kivy version !
from kivy.app import App
from kivy.uix.label import Label
from kivy.clock import Clock

class MyApp(App):

	def build(self):
		self.queue = multiprocessing.Queue()
		self.queue2 = multiprocessing.Queue()
		self.p = multiprocessing.Process(target=MyFunc, args=(self.queue, self.queue2))
		self.p.start()

		Clock.schedule_interval(self.idle, 0.1)

		self.lbl = Label(text='Hello world')
		return self.lbl

	def idle(self, dt):
		try:
			while True:
				msg = self.queue.get_nowait()
				self.lbl.text = str(msg)
		except Empty:
			pass

if __name__ == '__main__':	
	myApp = MyApp()
	myApp.run() # you won't create second window right now i guess

	myApp.queue2.put("stop")
	myApp.p.join()

[Zen-CODE]

Okay, so it's a known issue with Windows.

http://stackoverflow.com/questions/13922597/multiprocessing-freeze-support#18195951

That would seem to imply that it's not really a Kivy issue? Can we close the ticket then?

It's perhaps something we should add to the docs, but then again, it seems that's more a Python side concern than Kivy one. Where could we add that? Will look into that and see if there is an appropriate place.

Thanks, and glad you got there :-)

[TimSC]

I still think that kivy is acting in a non-standard fashion, requiring special measures to be taken to operate with common python components (multiprocessing and pyinstaller). The order of code in the working example should not be that senstive. You can close the bug though. :)

[Zen-CODE]

I think one would need to understand/demonstrate the underlying cause before attributing blame. As you mention, it works fine on linux and the freeze thing is a known Windows issue. It happens with all/many python apps. So why blame Kivy?

In what way does kivy "behave in a non-standard" fashion? We can't really deduce that from the above issues. The order of imports is known to be important because imports ere executed, just like any other python function. The circular import issues would never occur if import orders were not important.

And I'm not trying to be difficult. It's just that I'm a sucker for science and I believe in evidence before making assertions. What evidence do we have the Kivy does anything "non-standard"?

Thanks. I'll close the issue, but feel free to carry on posting here. If there is something we can improve in Kivy, I'm keen :-)

[TimSC]

@Zen-CODE The work around violates the PEP8 style guide: "Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants." This is necessary because kivy window initialization happens on an import (I think). Therefore it is a kivy issue. While I admit there could be reasons for doing it this way, I can only base my opinion on what I know already. Question: is there a reason for doing it this way? :)

Python generally should use the "Principle of Least Astonishment" http://lucumr.pocoo.org/2011/7/9/python-and-pola/ and not do weird things during an import. Ideally by working example should be reorderable within the usual constraints of python code.

I don't expect this will be changed any time soon, but I just want to put the case forward in case this part of the architecture is discussed in future.

[Zen-CODE]

To be clear, the problem is not caused by kivy creating a window. If you read the reply in the link below, it because of the way windows creates processes. This is beyond Kivy's control.

http://stackoverflow.com/questions/13922597/multiprocessing-freeze-support#18195951
So let's carry on the discussion (as it's quite interesting and important to address), but understand it;s more for the sake of academics than application to this issue posted here.

"The work around violates the PEP8 style guide".

Correct, but required to handle multi-processing in Windows. I guess it's not a perfect world. But remember, it's a "guide", not n "truth" or "absolute". In some situations, you have to make things work when you cannot fix the core problem.

"This is necessary because kivy window initialization happens on an import (I think)."

That would not appear so. This would (probably?/almost certainly?) happen anyway.

"Therefore it is a kivy issue. While I admit there could be reasons for doing it this way, I can only base my opinion on what I know already. Question: is there a reason for doing it this way? :)"

Assuming this is based on your alarm that kivy creates a Window before you expect one, one has to consider the context and goals of kivy. Yes. some parts of kivy create objects on import and for very good reasons.

Consider the fact that widgets and graphics functions need to know many things about the current window is order to size, fire events, render textures. If you had to leave all of these calculations and bindings until the last minute, it would be a incredibly clumsy. You would have to add guards such as::

if Window is None:
    Window.create()

(or suchlike) all over the place. Then there are issues of preventing multiple instantiations, cyclical references, garbage collection etc. All for the sake of enforcing of ???

The Window is such a core part of Kivy's functionality, it makes no sense to do this. It's much easier, cleaner and simpler to just ensure a fully constructed Window is always available to your code. We want it to be easy and predictable.

This could have been achieved by simply making a static class/class definition handle this. But kivy does it in a very elegant way. I'd really suggest you look at the code. I had never seen this approach before and it really 'wowed' me. Making use of inheritance before your even try to construct anything. :-))

class WindowBase():

class Window(WindowBase):

Instead, what kivy does, on importing the Window module, is instantiate a Window class as set that as a Singleton instance which you access as if it was a module variable. Much simpler than caring about instances/initialization, yet it can still fully utilize inheritance because it normal, instantiated class.

Consider also that the goals of kivy are speed and simplicity. As kivy guarantees singleton instances of many important classes, reference are all direct (no "if" checks or property lookups) thus faster. Code is simpler and more robust. As a GUI has to have a window anyway, it makes not sense to delay it's creation.

It's about making practical and programmatic decisions so solve a problem or achieve a goal. That is always the "most correct" choice, not blindly applying abstract principles without context.

"I don't expect this will be changed any time soon, but I just want to put the case forward in case this part of the architecture is discussed in future."

I think I've made the case for Window being always available. Simplicity, speed, efficiency. Any suggested improvements and pull requests are always appreciated.

Peace out :-)

[TimSC]

I see that there is a case for simplicity. However, neither pygame or PySDL2 takes this approach. Both have a distinct init function ("sdl2.ext.init()" or "pygame.init()").

An automatically initialized singleton is basically a global variable. Perhaps this approach was influenced by OpenGL usually requiring global variables in some form or another.

[Zen-CODE]

Why should everyone have the same approach? :-) Different goals, different use case, different choices. I think in the case of kivy it makes more sense (as is arguably more pythonic) to have some instantiation implicit (not all, of course). Having to call an init() somewhere only make it more opaque, error prone and open to abuse. Why would you want to care if it can just work?

It's also probably also related to the fact that kivy is full-stack framework. Your examples are more libraries which may need finer grained control is different situations and need to integrate into other systems. Kivy is the system and aims to hide magic you don't need to know about.

And you say "global variables" as if they are wrong? Without any consideration of how they are used? All design patterns are there for a reason.

"Only a sith thinks in absolutes".
:-)

[TimSC]

@Zen-CODE I'm afraid you are just being obtuse now. If I point out how this contradicts what you said above, you are just going to come back with more quibbles and pedantic objections. Being defensive about architecture is just going to put off improving it.

[Zen-CODE]

:-) I'm try to point out the that implying bad design is meaningless with a concrete example. Asserting principles without considering context is a simplistic approach to solving problems. Which is what we want to do.

So, what we need is a concrete example exhibiting a design flaw. What exact import do you object to? It's also important to me that kivy does the 'correct' thing, so show us where we could do better?

[haveityourwa]

Not sure if I should bring to life a dead thread, however I am also having issues when trying to use multiprocessing to call a function. My issue is that each time I call an asycn function it launches a window. I tried placing

if __name__ == '__main__':
	multiprocessing.freeze_support()

above the rest of my application like so.

import multiprocessing
import time


def psse_run():
    time.sleep(1)


if __name__ == '__main__':
    multiprocessing.freeze_support()

import kivy

However it will still launch a blank window when called like so.

pool = Pool(processes=1)
pool.apply_async(psse_run, ())

In general I want to be able to use the Kivy application to work on files in the background however this is posing more of an issue then I was expecting.

[Zen-CODE]

@haveityourwa. Multi-processing in itself should not pose a problem. Our kivy app currently uses that approach (on desktop - iOS/Android are different) without problems. The issue only arises when using kivy imports the affect the UI, as Kivy is a single window app by definitions.

If you are having problems, I would post on the forums first (with runnable code), then open a new ticket if you don't resolve the issue...

[haveityourwa]

@Zen-CODE Ok will do, thanks for your reply. (Sort of just assumed this was a forum post but I can see that it is a 'bug' post area)

[ShootingStarDragon]

@haveityourwa I also have the same problem but this is the solution, written for posterity. If you want kivy to work with multiprocessing these are the steps:

Under the if __name__ == '__main__': guard:
import multiprocessing
call multiprocessing.freeze_support (this NEEDS to be before import kivy)
THEN
import kivy and other kivy imports

make sure all your kivy imports are under the if __name__ == '__main__': guard so that when you run multiple instances of your code through multiprocessing they have 0 kivy code to accidentally run if they're a spawned process/not main.