Callback function design

[awvwgk]

The callbacks for this bindings are implemented in an ad hoc fashion, with the primary aim to closely match the NLopt API in a Fortran-friendly way. However, this might not be the best/only way to define a callback. A better target for designing the callback for the objective function should be to provide an uniform experience compared to other optimization libraries.

The callback is for this project is defined in src/nlopt_callback.f90. Note that this callback design creates an issue with object lifetimes (see #13).

Related:

• User-defined callback routines fortran-lang/stdlib#117
• Optimization, Root finding, and Equation Solvers fortran-lang/stdlib#87
• Use nopass to simplify the interface for the user-defined function jacobwilliams/roots-fortran#1
• Pass data as pointer fortran-lang/minpack#4
• Allow passing of user data for callback procedure fortran-lang/minpack#30

[ivan-pi]

Generally speaking, these are the options I'm aware of:

• subroutine or function (procedural)
• derived type with procedure pointer callback (object-oriented using composition)
• derived type with deferred procedure callback (object-oriented using inheritance)
• reverse communication interface (???)

(It would be nice to add some bullet points with their respective strengths and weaknesses.)

The second part of the problem is how to pass any parameters of the function. A few approaches are outlined on fortran90.org in the section Type-casting in callbacks.

Roughly speaking, if we stay within Fortran (no raw C pointers), the options I can recall are:

• global variables (either in the main program, module, or containing scope); in this case the callback must be located in the same scope as the parameters
• parameters as work arrays rpar, ipar (this is common in the classic codes from Netlib and NAG or IMSL libraries)
• parameters encapsulated directly in the callback-containing derived type via inheritance (a variation of this could be to have parameter arrays within the derived type)
• parameters encapsulated in derived type which is passed externally to callback or via composition
• unlimited polymorphic object

The last two of these will require the consumer to use a select type construct.

Edit 1: are Fortran procedures only distinguished by TKR, or can it also specialize based upon the callbacks with different interfaces?

Edit 2: ironically, I think this issue was less of a problem in the old days of punch-cards and external subprograms. At that time you would always need to compile your code anyway. Switching to a different callback function or algorithm, just amounted to replacing a deck of cards. The name of the callback procedure would be hard-coded in the algorithm code.

[awvwgk]

I think we should be able to implement several callback mechanism in nlopt-f. Using generic to overload type-bound procedure interfaces should hide most of the logic from our users. This would give us some insight on each mechanism how good they work implementation-wise (especially when round-tripping through C) and how the impact the user-experience.

A few notes:

• using global/module variables or internal procedures is always a possibility to pass data, however it shouldn't be the only possibility, i.e. this is strictly a decision for the user side
• object-oriented callbacks with inheritance require another/third layer of wrapping, because we can't pass class-polymorphic objects as type(c_ptr), this is an internal detail for nlopt-f
• personally I don't mind select type constructs, but if this is an annoyance one can easily implement a pointer based cast procedure

[ivan-pi]

I'm on the same page. Generally, I find that composition (i.e. procedure pointer in a derived type) involves a bit less effort compared to inheritance. For most serious problems, the overhead from the indirect referencing should be negligible compared to the function evaluation. I also don't mind having a single select type (a small detail I would like to clarify is whether a default section would be a good practice or not; personally I think its not needed, but perhaps would be needed for debugging a deep a deep hierarchy of calls - maybe better if the compiler had an option to do this automatically).

In my old NLopt wrapper, instead of an unlimited polymorphic object for parameters, I expected consumers to extend an abstract derived type:

type, abstract :: nlopt_func_data
end type

But the unlimited polymorphic class involves slightly less effort and is also less restrictive, i.e. the same derived type that encapsulates problem parameters can be easily reused across various problems (optimization, differential equations, etc.).

[ivan-pi]

I noticed C23 is planning to introduce a pointer type for pairing code and data (https://www.open-std.org/jtc1/sc22/wg14/www/docs/n2787.pdf). It looks very similar to the type you are using in nlopt-f:

nlopt-f/src/nlopt_callback.f90

Line 75 in 83d21e7

type :: nlopt_func

Admittedly, I don't yet have a full understanding of the proposed C enhancement, but I'm wondering if Fortran should have a similar built-in (language-level) callback+context abstraction.

[ivan-pi]

Instead of having to use host association to adapt the callback interface, maybe something like this

procedure(minpack_func_plus_context), delegate(context) :: f 
   ! context is a named dummy argument of the new procedure callback, 
   ! of type minpack_context
procedure(minpack_func), closure :: fold  
   ! the old interface

type(minpack_context) :: mydata

fold => delegate(f,mydata) 

! or maybe a spin on the old statement function syntax:
fold(x) => f(x,mydata)

call minpack_hybr(fold, ...)

This would be equivalent to

call minpack_hybr(fold, ...)
contains
real function fold(x)
  real, intent(in) :: x
  fold = f(x,mydata) ! f and mydata available through host association
end function

The language is pretty close already when you look at the derived type, and the use of associate to capture the context. A built-in language feature could also solve the problem of lifetime mentioned in #13