refactoring : removed explicits os.fork(), exceptions are propagated from child process to parent

[alalazo]

Modifications

• removed explicit os.fork() in favor of multiprocessing.Process
• log_output logic has been modified and does not require calls to sys.settrace anymore
• we can throw from Package.do_install.build_process and the exceptions are propagated to parent process

Changes in log_ouput

In develop :

• on __enter__ the parent process forks , set his frame explicitly to avoid executing twice the same code (quite fragile to re-positioning of the with statement) , reads from a pipe and writes to file
• on __enter__ the child does the necessary to write to a pipe and then returns for execution

In the PR:

• a "daemonic" child is spawned in the acquire method and is set to read from the pipe
• the context manager just changes where the parent writes when needed
• everything is cleaned in the release method

Notes

In #1186 I wanted to be able to forward exceptions to parent to have a neater workflow in do_install. As I saw a couple of TODOs referring to the same issue, I extracted this PR. This has also the great advantage of not tweaking with sys.settrace and frames (which is a major pain for the debugger sometimes).

[alalazo]

Ready to be reviewed and tested

[tgamblin]

@alalazo: this is awesome! I haven't had a chance to look at it in detail but will. This also removes one of the obstacles to getting Spack working on Windows.

@goxberry Does this work for you on Windows?

[alalazo]

@tgamblin I would really like this to be merged, but to be fair I have to tell you that in #1186 spurious fails in test/install.py appeared possibly due to this change-set. I was not able to reproduce them outside Travis and I have the impression that the tests themselves may be flawed somehow, but it is worth a double check before merging.

[alalazo]

@tgamblin I got the point failing spuriously here : I'll write a fix a ping you when done.

[alalazo]

@tgamblin Done : now it should be 99,9% safe 😄

[alalazo]

@tgamblin I pushed some cosmetic changes to this in #1186 , let me know if you want them ported here

[goxberry]

@tgamblin Will check with my co-worker who uses Windows (he was the original reason for the request) and get back to you

[tgamblin]

@goxberry: I looked into this. There are a couple more things that probably need to happen:

1. Set use_tmp_stage to False in __init__.py.
2. In lib/spack/env, there is really only one file with code in it: cc. The other files in lib/spack/env and its subdirectories are symlinks to that. You probably need to overwrite every file that is not cc with the contents of cc. Apparently by default if you clone on windows, a symlink comes down as a file with the path to its target in it, which is not really what we want. Not sure how best to get around that right now, but this could be a potential workaround.

[alalazo]

Closing as it will be difficult to maintain this alongside #1186

[tgamblin]

If it were merged would it be easy or is it a fundamental incompatibility?

[alalazo]

@tgamblin No fundamental incompatibility : all the features here are also in #1186 except that maintaining both will create conflicts (due to synchronization with develop) that needs to be managed.

In the end I think it will be easier to re-create something like #1228 if you need just that rather than continue with this PR.

[tgamblin]

@alalazo: ok -- sounds good. Sorry for being slow!

[alalazo]

@tgamblin No worries. I closed it as leaving as it was was just adding noise to the list of PRs