Process management enhancements

[Manu343726]

Hello Tobias, I'm here again :)

I'm trying to use your process management system to speed up biicode Boost builds (I'm having timeouts on my travis ci builds due to Boost.Log build). Since Travis expects output, my idea was to get your process_start_script() and run each library build job as a separate process, then wait for all. If that works as expected, there will be no timeout since your process_wait_all() prints some fancy progress output.

It works like a charm, I have it on a testing branch but I'm sure it will be part of the biicode Boost setup.

I have a couple of suggestions/questions:

• Custom progress information: I will be glad if some form of progress customization is made. The ....1/3... is great, but I think a bit of customization could improve a lot UX. In my case, I prompt Starting [boost lib] build job... before starting each script, one per line, will be great to print something similar when the process finishes [boost lib] build job finished. Maybe a callback on_success()/on_fail() approach?
• Portability: As far I know the implementation uses system dependent scripting to achieve process management. To be able to use this features on biicode, I need support on Windows, linux, and mac. I remember a conversation where you pointed out that you are using PS on windows and bash on linux. Will this work on Darwin/Mac OSX too?

[Manu343726]

I'm currently testing it on Windows. I noticed that the main console is closed and a new one is spawn for each process. It's a bit odd, I cannot ship Boost with that behavior :(

[toeb]

Hi
so custom progress information would be achievable very simply. I am thinking about promises/ on_success, on_failure callbacks with cmakepp. If you want to do it for yourself for now just copy the wait for all function and modify it however you like. (its just a while loop which checks if the process is still running and breaks as soon as none is running anymore...) - Or if you want to wait I will change it in the future (The process management needs a rework because I want to combine synchroneous and asynchroneous processes into one function)

The windows thing is a bit stupid and I will fix that. The big problem I still have is that I want to have the possiblity to hide console output (because I do not want git spamming my console) but also want the possibility to show it.. That does not work yet. Also spawning a process is pretty hard in windows..(if you want it to be async). If you ever chose to help the functions which you would have to look at are process_info_Windows process_isrunning_Windows process_kill_Windows process_list_Windowsandprocess_start_Windows`

[toeb]

Ah! and Mac OS might be supported.... (I did not test it yet) It only relies on bash functions so if macos has them (which it should) then it should not make a difference.

You can look at the procees/linux functions if you want to check

[toeb]

So I fixed the windows process management (now no window is opened, its faster and it does not close the parent window... :) ) therfore I am mixing cmd.exe powershell.exe and cmake... so thats just a bit behind the head through the legs into the heart.

I'll upload it soon - I am just adding the other part - callbacks for status indications

[toeb]

so you can try it now (in the developement branch). also tell me if it works under mac if you get chance :)

[Manu343726]

Cool. Can you please publish this to the biicode block as STABLE, at least as a development track? Also, Boost will be released today or tomorrow morning, get a chance to update your biicode badge, I expect some traffic to your repo since I will reference your project as part of the Boost hooks :)

The official biicode badges are available at the settings page of the block (Press the top-right gear at the block page):

[toeb]

Well I wouldn't call it stable yet :). But I'll try to publish it now.

I tried using the badges but when I add [![Build Status](https://webapi.biicode.com/v1/badges/toeb/toeb/cmakepp/master)](https://www.biicode.com/toeb/cmakepp) the following happens:

[Manu343726]

The issues with the badge is something related to how gihub catches images. We are investigating. Check the raw badge url, works perfectly: https://webapi.biicode.com/v1/badges/toeb/toeb/cmakepp/master

It's a bit odd, since you are the only user who noticed this, the rest of the badges are working perfectly. Check for example my biicode-boost README: https://github.com/Manu343726/boost-biicode

There's an issue on the github/markup repo about the topic: github/markup#224

[Manu343726]

I fetched your changes an integrated them in an opened version of your cmakepp block. Works like a charm on Windows :)

[Manu343726]

We deployed a little hotfix to solve this, let's try it:

Mmm github still catches the badge: Let's toogle the URL a bit ([![Build Status](https://webapi.biicode.com/v1/badges/toeb/toeb/cmakepp/master?pleasegithubstopcachingmycoolbadge)](https://www.biicode.com/toeb/cmakepp)):

[toeb]

ok that seems to work. its in my next push. A question: would it be possible for biicode to display documentation (readmes) the same way that github does? (relative file and readme links?)

[toeb]

I changed quite a few things now. so you might need to change your code. execute() now allows you to run async or sync. you can specify a error and success callback. the caveat is that execute has become slow - 300 to 400 ms to call. however that will be optimized.

process_start does not work the same way anymore - use execute instead

[Manu343726]

"process_start does not work the same way anymore - use execute instead" ok, I will get it a chance.

[Manu343726]

I ended-up using process_start_script(), it worked for me and I didn't found enough examples/docs to use execute().

THIS IS SO GREAT

It will be on the master biicode-boost branch in a couple of days. I like it a lot, seriously.

[Manu343726]

Note I patched your biicode block, it has a little bug in the CMakeLists.txt (It's INIT_BIICODE_BLOCK() not INIT_BIICODE_BLOCKS()) and some errors related to the translation from oo-cmake to cmakepp.

The track I'm using for Boost is here.

[toeb]

great! if youd like you could update my bii block config/deploy settings (as I am not very proficient in that aspect)
I have a couple of tips for you regarding how to use the processes. (see at your pull request)

[Manu343726]

Hello Tobias.
I think we should reopen this. I'm having multiple issues to make the system work:

• execute() WORKING_DIRECTORY does not work: I have confirmed it on linux only, but I was having issues when launching commands on mac and windows too. Test a simple case: execute(pwd WORKING_DIRECTORY foo) does not give foo for me. It returns the path where cmake is being run, biiproject/cmake/ in my case. I have been reading your tests, and the execute_tests.cmake only checks for the process info working_directory entry, which is ok. Please add a test to check if the program is being run in the correct directory too.

There are more issues, but I'm not that sure so I will add them tomorrow when I get time to test on more platforms.

I have the Boost update with cmakepp working, but until it works correctly on all platforms it cannot be released. That's why I'm writing you everyday ;)

As a side note, congratulations for your library. Seriously. For me the hottest thing is assign()

[Manu343726]

Let's continue:

• execute() and process_start() are ok on windows.

• The same for linux after the fix above.

• process_start_linux() is not working on mac osx, always fails to run the process i.e. enters this if:

  if(error)
    message(FATAL_ERROR "could not start process '${command_string}'")
  endif()

  (Line 49 of process_start_linux.cmake)

[toeb]

ok. I have mac installed - i'll try to find out the problem.

[toeb]

I have merged your pr into my execute_rework branch and added the working directory test (works under windows) - I am currently working on getting travis to build after some changes (shouldnt take too long)

[Manu343726]

Hi Tobias, I'm expectant of your feedback on Mac OSX. I want to release the cmakepp-based improvements for biicode boost hooks! :)

I tested it again. Seems like the problem is that cmake is not able to find bash. That's why the call to the "forker" script fails.

[toeb]

Hi Manuel,

I have tested it under mac osx yosemite and have not found an error. the only important thing is that bash is inside the PATH for the script to work. I guess that would have to be made a prerequisite.

alternatively it would be possbile to use find_program to find the bash executable.

I have increased the performance of process management in my execute_rework branch and will push the result momentarily.

[toeb]

sorry it took so long for the test - I had a super situation: my mac is running windows and I made the mac partition as small as possible (32GB) then I needed to update to yosemite (because of xcode/dev tools) and I downloaded it - then my partition was full (2GB left 7GB required) and I coudl not pull the image on a stick (fat32) then I was demotivated and did nothing (for osx testing) for a week - and everything because of the mac osx sleepfile (16GB of memory mirroring) which I finally managed to deactivate :) now I have yosemite installed and 20GB free diskspace - so thanks for kicking me everything turned out better than expecte ;)

[Manu343726]

Ok, were issues about the shell and PATH. It's working. I will write a couple of final improvements and the scripts will be out for everyone using boost with biicode.

[toeb]

Hi Manu,

I've made my tests run in parallel. It might interest you how I did it because I use a couple of new features. process_wait_n mainly you can see documented it in test_execute_glob_parallel

also note status_line the function which helps you printing status (clears and restores the status message before resp. after a normal output)

here is an image of what it looks like

[Manu343726]

Hello Tobias.

That looks great, as usually it's a pleasure to see how my feedback is translated into features in just a couple of days. You are the true CMake master.

Maybe I should take some of my free time to play more with cmakepp. My current projects (Both bii related and personal ones) are not very CMake intensive, but I feel I'm always reinventing the wheel with my own tricks for things that should be available in your lib. Things like maps, assign(), portable commands, command wrappers, etc. Things that make CMake a true programming (building) language. But I'm missing docs a lot (I'm sure you are aware of this problem and it's only related to the time you have for cmakepp). Learning from tests is a bit slow, but worth it.

I'm so sorry for taking up to two weeks to answer you, this was a very busy time on the hive :(

[toeb]

Hi Manu,

thank you for the complements - I appreciate them alot. You don't have to be sorry if you don't answer.

If you want to work more with cmakepp I would greatly appreciate it because I noticed that after your issues I really improved cmakepp alot so it really was a great experience for me.

I am concerned about the documentation as well and plan to spend more time on it (if its not a requirement from a client and "just for me" it is harder to motivate myself) But I have already made some good progress - I am trying to document every function with examples which are then dynamically run during documentation generation (very nifty) and put into the readmes in the corresponding folders. (see my template execution engine).

I hope that I will get some traction with my project but I still need consolidate it here and there.

Cheers
Tobi