-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BF-MT: Fixes and enhancements #445
Conversation
Hi Steve, could you please try again? |
Hi @detlefarend , I still got the same errors and also while running the howto 01 and 02. Previously I ran them in Ubuntu using a virtual machine and they were fine. I will continue checking this on Monday. Regarding #446, PZoo just updated their version to 1.22.0 (less than a day). I also found this and have solved this issue in the MPC branch. I have rebased this branch and it is looking fine. They also introduce their own space, called gymnasium, which means that they can not recognize gym spaces anymore. |
Hi @steveyuwono, I added issue #449 to this PR and I moved the call of freeze_support() to the main sections of both howtos and removed it from bf.mt. In the best case both howtos are running even under Windows and nothing else is left to do... |
Hi @detlefarend, thank you for the updates. I have tried this again but unfortunately, the same issue occurs. I tested on my laptop, and the error message does not appear, but the run is always stuck in the same process as where the error message appears on my PC. Have you tested this file under Windows? Does it work in yours? |
Hi @steveyuwono, I didn't test it under Windows. It runs properly on my Linux PC... Just to exclude some error sources:
Which Python version are you running? |
3.9 for my PC and 3.7 for my laptop. Shall I try on Marlon's or someone else's PC to check whether it is a device-specific issue? |
Yes please |
Also not working on Marlon's PC |
@detlefarend Tried the following on branch bf_mt_windows_freeze
|
Thanks for checking. Calling freeze_support() is obviously not enough. I'll take a look... |
Hi guys, please temporarily add the following code directly under if/main before the call of freeze_support():
|
works on Mac, without calling the freeze_support(). And I would suggest to put the mp.set_start_method("fork") on mt.py instead on Howto. |
maybe some reference, windows doesnt support fork. |
Another reference, that fork is not safe in MacOS: python/cpython#77906 Maybe we need to find another way instead of manually set to fork. |
Hi guys, I'm one step further now. I installed vscode and mlpro on my business pc and I can reproduce the problem now. The problem occurs when creating the shared object (not when creating a new process). I'll keep on track and let you know... |
Hi guys, both howtos are running now under Windows as well. I solved two problems: 1) Multiprocessing is to be started in "main" context only. 2) Python standard package multiprocessing uses pickle for serialization. I replaced it by a fork multiprocess which uses dill. Must be installed explicitely. Under Linux the performance is as expected. Under Windows (my underpowered business laptop) the performance of the multiprocessing parts is not satisfying. Could you please try both howtos under Windows/Mac and let me know the speed factors of howto mt.001? Thx! Here are some related links: |
Additional info:
|
I might found where causing the problem. It is on the BaseManager socket listener. By default, it can hold up to 16 backlog of connection. I changed this value manually on the multiprocess package. It might be that my Mac is spamming too fast. After the changes, I have no error. As a result for speed factor:
|
Further checks, I think on my Mac, the BaseManager is bottlenecking. If I increase the task and the backlog number. The speed factor stays the same for Multiprocessing. Too many backlog connection to the shared data. Probably this will happen also if the system has more cores. I will check this on our server. |
Hi Rizky, I reworked both howtos: Howto 001 (async algo)
Howto 002 (workflow with task-hierarchy)
Could you please run Howto 001 under Linux, Mac, Windows once again? |
Here is the result with no error:
I have a question, how did you calculate this speed factor? Because as I increase the task, I gain more speed factor. |
That looks good!
self._num_tasks -> number of tasks (20) |
Shall we merge the stuffs? |
Running properly. We can merge this |
Yes this can be merge. I have only backlog problem from BaseManager on Mac. But, I think for now is enough, we are currently only using Windows and Linux. I will do more checks later on for Macs. I am still no sure whether this is a problem woth macs or due to fast computing. |
Description
Background
Checklists: