Breaking tests with Complex input

[thisrod]

This adds two tests, that currently throw errors for the following reasons:

1. (I think) The show or repr method for solutions tries to @printf complex numbers.

2. The default solver attempts to sort complex numbers.

[pkofod]

Interesting :) Good catach... I will think about a better way of printing complexs (or just not print these)

[antoine-levitt]

Can this be merged?

[thisrod]

I've fixed the first one, I think.

[thisrod]

I'm looking at point 2. The error occurs at line 156. This assumes that the type of the minimum, returned by the cost function, is the same as the eltype of the minimizer. The error is caught at line 167, which sorts an array that is supposed to hold real costs, but has inherited the complex eltype of the minimizer.

Optim.jl/src/multivariate/solvers/zeroth_order/nelder_mead.jl

Lines 151 to 167 in 56dabf7

	function initial_state(method::NelderMead, options, d, initial_x)
	T = eltype(initial_x)
	n = length(initial_x)
	m = n + 1
	simplex = simplexer(method.initial_simplex, initial_x)
	f_simplex = zeros(T, m)
	value!!(d, first(simplex))
	@inbounds for i in 1:length(simplex)
	f_simplex[i] = value(d, simplex[i])
	end
	# Get the indices that correspond to the ordering of the f values
	# at the vertices. i_order[1] is the index in the simplex of the vertex
	# with the lowest function value, and i_order[end] is the index in the
	# simplex of the vertex with the highest function value
	i_order = sortperm(f_simplex)

I presume that the code is supposed to handle a cost function that returns any subtype of Real, though I don't fully understand the design intent here. Is there a reliable way to determine the type of the minimum? I'd guess that a reliable and extensible implementation of Optim will need one. Should be a method to return that type for a given problem?

A quick fix would be add, after line 152, something like if T <: Complex; T = type_parameter_of(T); end. I don't know the idiomatic way to do that in Julia. It can be done by defining a function that dispatches on Complex{T} and T <: Real, but that seems a bit clunky.

[thisrod]

At a higher level, I think there's a choice to be made between the following designs:

1. Every gradient-free solver is implemented to handle complex minimizers, or

2. Gradient-free solvers are only implemented for real minimizers, and there is a generic constructor method for complex minimizers that splits the problem into its real and imaginary parts then converts the solution back to complex.

[pkofod]

Yeah, I think maybe gradient free solvers will have to use the splitting and or special cases for the complex case.The problem with nelder mead (if I fixed the type stuff you found above) for example is that the simplex will be of order two times too low, and it will not be able to disentangle the real and imaginary part the way it's written right now.

[thisrod]

I think maybe gradient free solvers will have to use the splitting and or special cases for the complex case.

Then the next step might be to merge the display fix, and raise the gradient free solvers as an issue that will take longer to fix.

What's the best way to rebase this PR onto the master branch? Should I just do it and force push to my github? I'll add an @test_broken to make Travis happy.

[thisrod]

I have a plan for zeroth order complex. Please tell me if there is an obvious problem or better way.

1. In NLSolversBase, implement ComplexNonDifferentiable. This wraps the NonDifferentiable methods with Complex, to combine real and imaginary parts.

2. In Optim, implement this pseudo code:

   function optimize(
   d::AbstractObjective,
   initial_x::AbstractArray{<:Complex},
   method::ZerothOrderOptimizer,
   options
   )
   c = ComplexObjective(d)
   # split initial_x into real and imaginary parts
   results = optimize(c, initial_parts, method, options)
   # combine real and imaginary results into complex
   results
   end