BF-MT: Fixes and enhancements #445

detlefarend · 2022-10-08T09:30:39Z

Description

Background

Checklists:

Read the CONTRIBUTION guide (required).
Update the history on the source code (required).

detlefarend · 2022-10-08T09:31:47Z

Hi Steve, could you please try again?

steveyuwono · 2022-10-08T15:16:49Z

Hi @detlefarend , I still got the same errors and also while running the howto 01 and 02. Previously I ran them in Ubuntu using a virtual machine and they were fine. I will continue checking this on Monday.

Regarding #446, PZoo just updated their version to 1.22.0 (less than a day). I also found this and have solved this issue in the MPC branch. I have rebased this branch and it is looking fine. They also introduce their own space, called gymnasium, which means that they can not recognize gym spaces anymore.

…ows_freeze

detlefarend · 2022-10-09T20:39:35Z

Hi @steveyuwono, I added issue #449 to this PR and I moved the call of freeze_support() to the main sections of both howtos and removed it from bf.mt. In the best case both howtos are running even under Windows and nothing else is left to do...

steveyuwono · 2022-10-10T07:36:32Z

Hi @detlefarend, thank you for the updates. I have tried this again but unfortunately, the same issue occurs. I tested on my laptop, and the error message does not appear, but the run is always stuck in the same process as where the error message appears on my PC. Have you tested this file under Windows? Does it work in yours?

detlefarend · 2022-10-10T07:44:15Z

Hi @steveyuwono, I didn't test it under Windows. It runs properly on my Linux PC... Just to exclude some error sources:

You tried it in branch bf_mt_windows_freeze
Under Linux it runs properly
The problem occurs under Windows only

Which Python version are you running?

steveyuwono · 2022-10-10T07:59:36Z

Hi @steveyuwono, I didn't test it under Windows. It runs properly on my Linux PC... Just to exclude some error sources:

You tried it in branch bf_mt_windows_freeze

Under Linux it runs properly

The problem occurs under Windows only

Which Python version are you running?

3.9 for my PC and 3.7 for my laptop. Shall I try on Marlon's or someone else's PC to check whether it is a device-specific issue?

detlefarend · 2022-10-10T08:05:55Z

Hi @steveyuwono, I didn't test it under Windows. It runs properly on my Linux PC... Just to exclude some error sources:

You tried it in branch bf_mt_windows_freeze

Under Linux it runs properly

The problem occurs under Windows only

Which Python version are you running?

3.9 for my PC and 3.7 for my laptop. Shall I try on Marlon's or someone else's PC to check whether it is a device-specific issue?

Yes please

steveyuwono · 2022-10-11T09:14:28Z

Hi @steveyuwono, I didn't test it under Windows. It runs properly on my Linux PC... Just to exclude some error sources:

You tried it in branch bf_mt_windows_freeze

Under Linux it runs properly

The problem occurs under Windows only

Which Python version are you running?

3.9 for my PC and 3.7 for my laptop. Shall I try on Marlon's or someone else's PC to check whether it is a device-specific issue?

Yes please

Also not working on Marlon's PC

rizkydiprasetya · 2022-10-11T09:46:59Z

@detlefarend Tried the following on branch bf_mt_windows_freeze

Mac, doesnt work
Linux, works
Windows, doesnt work

detlefarend · 2022-10-11T10:53:34Z

Thanks for checking. Calling freeze_support() is obviously not enough. I'll take a look...

detlefarend · 2022-10-11T11:15:41Z

Hi guys, please temporarily add the following code directly under if/main before the call of freeze_support():

start_method = mp.get_start_method()
print(start_method)
if start_method != 'fork': mp.set_start_method(method='fork', force=True)

steveyuwono · 2022-10-11T11:34:13Z

Hi guys, please temporarily add the following code directly under if/main before the call of freeze_support():
start_method = mp.get_start_method()
print(start_method)
if start_method != 'fork': mp.set_start_method(method='fork', force=True)

The method of 'fork' can not be processed.

rizkydiprasetya · 2022-10-11T11:38:06Z

works on Mac, without calling the freeze_support(). And I would suggest to put the mp.set_start_method("fork") on mt.py instead on Howto.

rizkydiprasetya · 2022-10-11T11:41:03Z

maybe some reference, windows doesnt support fork.

https://superfastpython.com/multiprocessing-start-method/

rizkydiprasetya · 2022-10-11T12:28:49Z

Another reference, that fork is not safe in MacOS: python/cpython#77906
As I check again, I get unstable result by setting up the mp to 'fork' on mac.

Maybe we need to find another way instead of manually set to fork.

detlefarend · 2022-10-11T18:17:48Z

Hi guys, I'm one step further now. I installed vscode and mlpro on my business pc and I can reproduce the problem now. The problem occurs when creating the shared object (not when creating a new process). I'll keep on track and let you know...

detlefarend · 2022-10-12T17:13:01Z

Hi guys, both howtos are running now under Windows as well. I solved two problems: 1) Multiprocessing is to be started in "main" context only. 2) Python standard package multiprocessing uses pickle for serialization. I replaced it by a fork multiprocess which uses dill. Must be installed explicitely.

Under Linux the performance is as expected. Under Windows (my underpowered business laptop) the performance of the multiprocessing parts is not satisfying.

Could you please try both howtos under Windows/Mac and let me know the speed factors of howto mt.001? Thx!

Here are some related links:
https://pypi.org/project/multiprocess/
https://stackoverflow.com/questions/40234771/replace-pickle-in-python-multiprocessing-lib

rizkydiprasetya · 2022-10-12T17:51:53Z

I ran 3 times on mac, showed the following.

rizkydiprasetya · 2022-10-12T21:01:28Z

Additional info:

If I decrease the number of task, then it runs fine. Above 30 tasks, it becomes unstable on native Mac. The chance of getting this error is like 3/10. If I put 50 tasks, then the chance of getting this error is 10/10.
I tried with more than 30 tasks in dockerized Ubuntu on Mac, works fine.

rizkydiprasetya · 2022-10-12T22:31:41Z

I might found where causing the problem. It is on the BaseManager socket listener. By default, it can hold up to 16 backlog of connection. I changed this value manually on the multiprocess package. It might be that my Mac is spamming too fast. After the changes, I have no error. As a result for speed factor:

Windows on Ryzen 7, 8 Cores 16 Threads
- Multithread : 2.86
- Multiprocessing : 2.55
Dockerized Ubuntu, 8 Cores 8 Threads
- Multithread : 5.18
- Multiprocessing : 16.42
Mac M1 Pro, 10 Cores
- Multithread : 5.57
- Multiprocessing : 25.84

rizkydiprasetya · 2022-10-13T06:03:56Z

Further checks, I think on my Mac, the BaseManager is bottlenecking. If I increase the task and the backlog number. The speed factor stays the same for Multiprocessing. Too many backlog connection to the shared data.

Probably this will happen also if the system has more cores. I will check this on our server.

detlefarend · 2022-10-13T06:57:41Z

Hi Rizky, I reworked both howtos:

Howto 001 (async algo)

Entire demo code is now running in main context to avoid side effects in multiprocessing
Number of tasks: 20
Duration/task: 1 sec
Call of freeze_support() added to stabilize the demo under Windows (but I didn't notice any effect)

Howto 002 (workflow with task-hierarchy)

Multiprocessing part removed

Could you please run Howto 001 under Linux, Mac, Windows once again?

rizkydiprasetya · 2022-10-13T08:45:04Z

Here is the result with no error:

Ubuntu (Dockerized):
- Multithread : 6.6
- Multiprocessing : 15.31
Windows (Lab PC):
- Multithread : 4.05
- Multiprocessing : 7.35
Mac:
- Multithread : 7.01
- Multiprocessing : 18.63

I have a question, how did you calculate this speed factor? Because as I increase the task, I gain more speed factor.

detlefarend · 2022-10-13T10:03:13Z

Here is the result with no error:

* Ubuntu (Dockerized):
  
  * Multithread : 6.6
  * Multiprocessing : 15.31

* Windows (Lab PC):
  
  * Multithread : 4.05
  * Multiprocessing : 7.35

* Mac:
  
  * Multithread : 7.01
  * Multiprocessing : 18.63

That looks good!

I have a question, how did you calculate this speed factor? Because as I increase the task, I gain more speed factor.
It's calculated in line 102:

self._num_tasks -> number of tasks (20)
self._duration_sec -> fixed runtime of a single task (1 sec)
self._duration_real_sec -> measured runtime for the entire run of 20 tasks (sync/async)

detlefarend · 2022-10-13T18:03:32Z

Shall we merge the stuffs?

steveyuwono · 2022-10-14T07:11:35Z

Shall we merge the stuffs?

Running properly. We can merge this

rizkydiprasetya · 2022-10-14T07:24:04Z

Yes this can be merge. I have only backlog problem from BaseManager on Mac. But, I think for now is enough, we are currently only using Windows and Linux. I will do more checks later on for Macs. I am still no sure whether this is a problem woth macs or due to fast computing.

Tried to fix the Windows freeze problem...

c326c09

detlefarend added bug Something isn't working BF Basic Functions/Infrastructure next release labels Oct 8, 2022

detlefarend added this to the Sprint 10/2022 milestone Oct 8, 2022

detlefarend requested a review from steveyuwono October 8, 2022 09:30

detlefarend self-assigned this Oct 8, 2022

detlefarend linked an issue Oct 8, 2022 that may be closed by this pull request

Bug: Failed to generate the rtd documenation due to howto.bf.mt.001 #443

Closed

3 tasks

steveyuwono added 2 commits October 8, 2022 17:18

Merge branch 'main' of https://github.com/fhswf/MLPro into bf_mt_wind…

5f417a9

…ows_freeze

Merge branch 'main' of https://github.com/fhswf/MLPro into bf_mt_wind…

554225a

…ows_freeze

detlefarend linked an issue Oct 9, 2022 that may be closed by this pull request

BF-MT: Class Shared - standardized access to results #449

Closed

3 tasks

detlefarend changed the title ~~BF-MT: Fixed the Windows freeze problem~~ BF-MT: Fixes and enhancements Oct 9, 2022

detlefarend added the enhancement New feature or request label Oct 9, 2022

Standardized result management in class Shared

5e387ea

detlefarend requested a review from rizkydiprasetya October 11, 2022 10:53

Arend and others added 4 commits October 12, 2022 15:36

Merge remote-tracking branch 'origin/main' into bf_mt_windows_freeze

4cc8422

Merge remote-tracking branch 'origin/main' into bf_mt_windows_freeze

48b2663

replaced multiprocessing by multiprocess

208e4b5

Fixed the multiprocessing bug for Windows

89e0cbe

reworked both howtos

52c46c6

Updated rtd docu of both howtos

59e1b59

little beautifying

f4e9629

steveyuwono approved these changes Oct 14, 2022

View reviewed changes

detlefarend merged commit bd3b5ec into main Oct 14, 2022

detlefarend deleted the bf_mt_windows_freeze branch October 14, 2022 07:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BF-MT: Fixes and enhancements #445

BF-MT: Fixes and enhancements #445

detlefarend commented Oct 8, 2022 •

edited

Loading

detlefarend commented Oct 8, 2022

steveyuwono commented Oct 8, 2022

detlefarend commented Oct 9, 2022

steveyuwono commented Oct 10, 2022

detlefarend commented Oct 10, 2022

steveyuwono commented Oct 10, 2022 •

edited by detlefarend

Loading

detlefarend commented Oct 10, 2022

steveyuwono commented Oct 11, 2022

rizkydiprasetya commented Oct 11, 2022 •

edited

Loading

detlefarend commented Oct 11, 2022

detlefarend commented Oct 11, 2022

steveyuwono commented Oct 11, 2022

rizkydiprasetya commented Oct 11, 2022

rizkydiprasetya commented Oct 11, 2022

rizkydiprasetya commented Oct 11, 2022

detlefarend commented Oct 11, 2022

detlefarend commented Oct 12, 2022

rizkydiprasetya commented Oct 12, 2022 •

edited

Loading

rizkydiprasetya commented Oct 12, 2022 •

edited

Loading

rizkydiprasetya commented Oct 12, 2022 •

edited

Loading

rizkydiprasetya commented Oct 13, 2022

detlefarend commented Oct 13, 2022

rizkydiprasetya commented Oct 13, 2022 •

edited

Loading

detlefarend commented Oct 13, 2022

detlefarend commented Oct 13, 2022

steveyuwono commented Oct 14, 2022

rizkydiprasetya commented Oct 14, 2022

BF-MT: Fixes and enhancements #445

BF-MT: Fixes and enhancements #445

Conversation

detlefarend commented Oct 8, 2022 • edited Loading

Description

Background

Checklists:

detlefarend commented Oct 8, 2022

steveyuwono commented Oct 8, 2022

detlefarend commented Oct 9, 2022

steveyuwono commented Oct 10, 2022

detlefarend commented Oct 10, 2022

steveyuwono commented Oct 10, 2022 • edited by detlefarend Loading

detlefarend commented Oct 10, 2022

steveyuwono commented Oct 11, 2022

rizkydiprasetya commented Oct 11, 2022 • edited Loading

detlefarend commented Oct 11, 2022

detlefarend commented Oct 11, 2022

steveyuwono commented Oct 11, 2022

rizkydiprasetya commented Oct 11, 2022

rizkydiprasetya commented Oct 11, 2022

rizkydiprasetya commented Oct 11, 2022

detlefarend commented Oct 11, 2022

detlefarend commented Oct 12, 2022

rizkydiprasetya commented Oct 12, 2022 • edited Loading

rizkydiprasetya commented Oct 12, 2022 • edited Loading

rizkydiprasetya commented Oct 12, 2022 • edited Loading

rizkydiprasetya commented Oct 13, 2022

detlefarend commented Oct 13, 2022

rizkydiprasetya commented Oct 13, 2022 • edited Loading

detlefarend commented Oct 13, 2022

detlefarend commented Oct 13, 2022

steveyuwono commented Oct 14, 2022

rizkydiprasetya commented Oct 14, 2022

detlefarend commented Oct 8, 2022 •

edited

Loading

steveyuwono commented Oct 10, 2022 •

edited by detlefarend

Loading

rizkydiprasetya commented Oct 11, 2022 •

edited

Loading

rizkydiprasetya commented Oct 12, 2022 •

edited

Loading

rizkydiprasetya commented Oct 12, 2022 •

edited

Loading

rizkydiprasetya commented Oct 12, 2022 •

edited

Loading

rizkydiprasetya commented Oct 13, 2022 •

edited

Loading