Broken IPython parallel interface?

[twiecki]

Seems like a lot of API has changed but the IPython parallel client wasn't updated. Are there plans to update that, it's so much easier to work with than the mongodb based backend.

[jaberg]

Hey Thomas,

I have no plans to update that interface at the moment, but I agree that
it's a really nice one to have. In a way, it would be nice to make that the
primary interface for parallel experiments. If you (or anyone else) want to
take a crack at it I'd be happy to help you out.

On Wed, Aug 20, 2014 at 9:44 AM, Thomas Wiecki notifications@github.com
wrote:

Seems like a lot of API has changed but the IPython parallel client wasn't
updated. Are there plans to update that, it's so much easier to work with
than the mongodb based backend.

—
Reply to this email directly or view it on GitHub
#215.

[twiecki]

Hi James,

Thanks for the quick reply. I started adapting and at least got it to start but then nothing happened. This might become more relevant so I might have some cycles to work on this.

Also, are there any plans to add e.g. the GP optimization?

[willgroves]

Hey James,

I was able to write a working version of the ipython parallel interface. I think it's not quite ready for a merge though.

At first by following the existing design, I had an almost working version. But the existing placement of the apply_async call in ipy.py was causing large portions of the code to be pickled and sent to the engines (including the domain, the trials, and the pyll evalutator). In the end, this attempt failed because it was trying to pickle a <type '_hashlib.HASH'>.

I refactored the apply_async call to be in the evaluation() part inside of Domain (in base.py). This limits the pickled data sent to the engines to just the objective function and its arguments. The version works and passes the unit tests. It is in this branch: https://github.com/willgroves/hyperopt/tree/ipyworking

There are a few pieces of functionality that this version does not yet do that the mongo version does. But this could be a good start.

[jaberg]

Thanks so much for following up on this! Please feel welcome to submit this
work as a pull request to master, even if you'd prefer to mark it
work-in-progress. It sounds strictly better than what's currently in
master, and if that's true I would like to merge this even if it's not as
finished as you'd like.

• JB

On Fri, Mar 20, 2015 at 10:39 PM, willgroves notifications@github.com
wrote:

Hey James,

I was able to write a working version of the ipython parallel interface. I
think it's not quite ready for a merge though.

At first by following the existing design, I had an almost working
version. But the existing placement of the apply_async call in ipy.py was
causing large portions of the code to be pickled and sent to the engines
(including the domain, the trials, and the pyll evalutator). In the end,
this attempt failed because it was trying to pickle a .

I refactored the apply_async call to be in the evaluation() part inside of
Domain (in base.py). This limits the pickled data sent to the engines to
just the objective function and its arguments. The version works and passes
the unit tests. It is in this branch:
https://github.com/willgroves/hyperopt/tree/ipyworking

There are a few pieces of functionality that this version does not yet do
that the mongo version does. But this could be a good start.

—
Reply to this email directly or view it on GitHub
#215 (comment).