Prevent the Highs instance from being called by a callback that it calls.

[jeffreydeankelly2]

I recently upgraded to HiGHS 1.6.0 using JuMP supplied Windows non-static binaries and added the MIP logging and improving solution callbacks - thank you for adding these.

I interfaced the HighsCallbackDataOut structure from C to Intel Fortran using the Fortran derived types.

The callback to retrieve the variable-length double 1D-array mip_solution(*) works well except that I noticed an arbitrary five (5) element offset in order to properly align the C mip_solution() vector into the Fortran mip_solution() i.e., mip_solution(1+5:n+5) where "n" is the known length of variables. Otherwise, the variable solution data for each integer-feasible solution is received properly including the mix of reals and integers placed before mip_solution().

However I noticed that when CRTL-C is typed, HiGHS returns immediately with the text "Press any key to continue . . .". All other key-strokes such as CRTL-A, -B, -D,..,-Z have no effect.

Therefore, it seems that HiGHS has some CRTL-C signal handler invoked inherently?

In our industrial software we use CRTL-C consistently across all 3rd-party LP, QP, MIP and NLP solvers in order for the user to terminate the MIP solving process gracefully but unfortunately with HiGHS, CRTL-C is preempted before our signal handler can be invoked.

[jajhall]

Oops, yes, I was trying to put a CRTL-C handler into HiGHS but gave up, and failed to remove the development code (Lines 52 onwards in lp_data/Highs.cpp)

void highsSignalHandler(int signum) {
// std::cout << "Interrupt signal (" << signum << ") received.\n";
exit(signum);
}
Highs::Highs() { signal(SIGINT, highsSignalHandler); }

Since there is not (yet) anything in the HiGHS Fortran API to handle callbacks, do I infer from the following that you had to wrote something yourself and had to get around the fact that in HighsCallbackDataOut (see lp_data/HighsCallbackStruct.h) the pointer to the values of the solution is not on an 8-byte boundary - since an int, two HighsInt (which can be 4 or 8 bytes) and an int64_t constitute an odd number of 4-byte words. I can get around this by making double* mip_solution the first member of the HighsCallbackDataOut struct. That scalar doubles won't be on 8-byte boundaries presumably has little overhead.

Assuming that you wrote your own Fortran API to handle callbacks, could you share it with me since I wouldn't know how to write it.

The callback to retrieve the variable-length double 1D-array mip_solution(*) works well except that I noticed an arbitrary five (5) element offset in order to properly align the C mip_solution() vector into the Fortran mip_solution() i.e., mip_solution(1+5:n+5) where "n" is the known length of variables. Otherwise, the variable solution data for each integer-feasible solution is received properly including the mix of reals and integers placed before mip_solution().

[jeffreydeankelly2]

Hi Julian; Thanks for letting me know about the CRTL-C signal handler - I look forward to it being removed in your next release :-) Technically speaking, all C to Fortran API's seem to be redundant given standard Fortran's C Interoperability capability. Hence I never use these API's supplied by 3rd C/C++ based solvers as they are usually out-of-date and incomplete as well. Here is the interface for the HiGHS MIP improving solution callback and the first few lines of the user written HIGHSintsolcb() routine. Notice that the bind(c) attribute is removed from Fortran derived type HighsCallbackDataOut_Struct as the mip_solution(:) vector is declared as allocatable which is not allowed with bind(c) (cf. the module ISO_C_BINDING). In addition, the message character string is declared with an assumed fixed-length of 132 characters using the user written HIGHSintsolcb() routine. *I can get around this by making double* mip_solution the first member of the HighsCallbackDataOut struct. That scalar doubles won't be on 8-byte boundaries presumably has little overhead.* That may work but I am not sure if the other appended integers and doubles will be affected with offsets in a similar way when receiving their values in Fortran. The way all of the other MIP callbacks are implemented is to allow the user to call for example your Highs_getSolution() routine (or the like) to retrieve the MIP improving solutions inside the callback routine. If you need anything else, do not hesitate to ask. All the best - Jeff interface * integer(4) function Highs_setCallback(highs,user_callback,user_callback_data)* cDEC$ ATTRIBUTES DLLIMPORT, STDCALL, ALIAS : "Highs_setCallback" :: Highs_setCallback integer(8) :: highs cDEC$ ATTRIBUTES VALUE :: highs external :: user_callback cDEC$ ATTRIBUTES REFERENCE :: user_callback integer(8) :: user_callback_data cDEC$ ATTRIBUTES REFERENCE :: user_callback_data end function *integer(4) function Highs_startCallback(highs,callback_type)* cDEC$ ATTRIBUTES DLLIMPORT, STDCALL, ALIAS : "Highs_startCallback" :: Highs_startCallback integer(8) :: highs cDEC$ ATTRIBUTES VALUE :: highs integer(4) :: callback_type cDEC$ ATTRIBUTES VALUE :: callback_type end function *integer(4) function Highs_stopCallback(highs,callback_type)* cDEC$ ATTRIBUTES DLLIMPORT, STDCALL, ALIAS : "Highs_stopCallback" :: Highs_stopCallback integer(8) :: highs cDEC$ ATTRIBUTES VALUE :: highs integer(4) :: callback_type cDEC$ ATTRIBUTES VALUE :: callback_type end function * integer(4) function HIGHSintsolcb(callback_type,message,HighsCallbackDataOut,HighsCallbackDataIn,user_callback_data) * cDEC$ ATTRIBUTES DLLEXPORT, STDCALL, REFERENCE, ALIAS : "HIGHSINTSOLCB" :: HIGHSINTSOLCB use ISO_C_BINDING integer(4), intent(in) :: callback_type cDEC$ ATTRIBUTES VALUE :: callback_type character(*), intent(in) :: message cDEC$ ATTRIBUTES REFERENCE :: message c * Note that we have removed the bind(c) attribute below as the C binding only supports non-allocatable-arrays. However, since c mip_solution(:) is a variable-length 1D-array we need to still declare it to be allocatable inside the Fortran derived type. type :: HighsCallbackDataOut_Struct integer (c_int) :: log_type real (c_double) :: running_time integer (c_int) :: simplex_iteration_count integer (c_int) :: ipm_iteration_count real (c_double) :: objective_function_value integer (c_long_long) :: mip_node_count real (c_double) :: mip_primal_bound real (c_double) :: mip_dual_bound real (c_double) :: mip_gap c * Note that we are required to declare the mip_solution(:) 1D-array as an allocatable-array as it is variable-length and fixed-length. real (c_double), allocatable, dimension(:) :: mip_solution end type HighsCallbackDataOut_Struct type, bind(c) :: HighsCallbackDataIn_Struct integer (c_int) :: user_interrupt end type HighsCallbackDataIn_Struct type(HighsCallbackDataOut_Struct) :: HighsCallbackDataOut cDEC$ ATTRIBUTES REFERENCE :: HighsCallbackDataOut type(HighsCallbackDataIn_Struct) :: HighsCallbackDataIn cDEC$ ATTRIBUTES REFERENCE :: HighsCallbackDataIn integer(8), optional, intent(in) :: user_callback_data cDEC$ ATTRIBUTES REFERENCE :: user_callback_data end function end interface cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc c HIGHS integer-feasible solution callback function. cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc * integer(4) function HIGHSintsolcb(callback_type,message,HighsCallbackDataOut,HighsCallbackDataIn,user_callback_data) * cDEC$ ATTRIBUTES DLLEXPORT, STDCALL, REFERENCE, ALIAS : "HIGHSINTSOLCB" :: HIGHSINTSOLCB use ISO_C_BINDING use IMPLserver implicit none c * Note that we have removed the bind(c) attribute below as the C binding only supports non-allocatable-arrays. However, since c mip_solution(:) is a variable-length 1D-array we need to still declare it to be allocatable inside the Fortran derived type. type :: HighsCallbackDataOut_Struct integer (c_int) :: log_type real (c_double) :: running_time integer (c_int) :: simplex_iteration_count integer (c_int) :: ipm_iteration_count real (c_double) :: objective_function_value integer (c_long_long) :: mip_node_count real (c_double) :: mip_primal_bound real (c_double) :: mip_dual_bound real (c_double) :: mip_gap c * Note that we are required to declare the mip_solution() 1D-array as an allocatable-array as it is variable length and fixed-length. real (c_double), allocatable, dimension(:) :: mip_solution end type HighsCallbackDataOut_Struct type, bind(c) :: HighsCallbackDataIn_Struct integer (c_int) :: user_interrupt end type HighsCallbackDataIn_Struct integer(4), intent(in) :: callback_type cDEC$ ATTRIBUTES VALUE :: callback_type character(132), intent(in) :: message cDEC$ ATTRIBUTES REFERENCE :: message type(HighsCallbackDataOut_Struct) :: HighsCallbackDataOut cDEC$ ATTRIBUTES REFERENCE :: HighsCallbackDataOut type(HighsCallbackDataIn_Struct) :: HighsCallbackDataIn cDEC$ ATTRIBUTES REFERENCE :: HighsCallbackDataIn integer(8), optional, intent(in) :: user_callback_data cDEC$ ATTRIBUTES REFERENCE :: user_callback_data

…

On Wed, Nov 22, 2023 at 7:30 AM Julian Hall ***@***.***> wrote: Oops, yes, I was trying to put a CRTL-C handler into HiGHS but gave up, and failed to remove the development code (Lines 52 onwards in lp_data/Highs.cpp) void highsSignalHandler(int signum) { // std::cout << "Interrupt signal (" << signum << ") received.\n"; exit(signum); } Highs::Highs() { signal(SIGINT, highsSignalHandler); } Since there is not (yet) anything in the HiGHS Fortran API to handle callbacks, do I infer from the following that you had to wrote something yourself and had to get around the fact that in HighsCallbackDataOut (see lp_data/HighsCallbackStruct.h) the pointer to the values of the solution is not on an 8-byte boundary - since an int, two HighsInt (which can be 4 or 8 bytes) and an int64_t constitute an odd number of 4-byte words. I can get around this by making double* mip_solution the first member of the HighsCallbackDataOut struct. That scalar doubles won't be on 8-byte boundaries presumably has little overhead. Assuming that you wrote your own Fortran API to handle callbacks, could you share it with me since I wouldn't know how to write it. The callback to retrieve the variable-length double 1D-array mip_solution(*) works well except that I noticed an arbitrary five (5) element offset in order to properly align the C mip_solution() vector into the Fortran mip_solution() i.e., mip_solution(1+5:n+5) where "n" is the known length of variables. Otherwise, the variable solution data for each integer-feasible solution is received properly including the mix of reals and integers placed before mip_solution(). — Reply to this email directly, view it on GitHub <#1523 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALHONEEPDTYOXLCRJIH5A2TYFXV5ZAVCNFSM6AAAAAA7WDMXKSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRSGY4DIMJWGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- ********************************************************** *I M P L - " M a k i n g O p t i m i z a t i o n a n d E s t i m i z a t i o n S m a r t e r"* Jeffrey D. Kelly Industrial Algorithms Limited - i n d u s t r i @ l g o r i t h m s Email: ***@***.*** Phone: (647) 917-4675 (IMPL) *Making Industrial AI (Algorithms & Integration) Real!* ********************************************************** This email and any files transmitted with it are confidential, proprietary and intended solely for the individual or entity to whom they are addressed. If you have received this email in error please delete it immediately.

[jajhall]

Thanks for letting me know about the CRTL-C signal handler - I look forward
to it being removed in your next release :-)

I've asked the two other folk with whom I've communicated regarding the CRTL-C signal handler, just out of politeness, as removing it would, technically, break the API

Technically speaking, all C to Fortran API's seem to be redundant given
standard Fortran's C Interoperability capability. Hence I never use these
API's supplied by 3rd C/C++ based solvers as they are usually out-of-date
and incomplete as well.

That's interesting to know. Our Fortran API can certainly be classed as "out-of-date and incomplete". I'll still keep it, as it does no harm, but add a comment to the effect that it's out-of-date and incomplete, and that folk should exploit standard Fortran's C Interoperability capability.

The way all of the other MIP callbacks are implemented is to allow the user
to call for example your Highs_getSolution() routine (or the like) to
retrieve the MIP improving solutions inside the callback routine.

This wouldn't work in HiGHS, as no MIP solver data can be reached via the Highs C++ API. Indeed, if a user's callback function were to start calling methods in the instance of the Highs class (by passing a pointer to it via the void* user_data parameter) then all sorts of chaos could ensue! I should add code to the Highs C++ API to ensure that all methods give an immediate error return!

Thanks for the Fortran callback function. I'll add it to examples/call_highs_from_fortran.f90.

I can get around this by making double* mip_solution the first member of
the HighsCallbackDataOut struct. That scalar doubles won't be on 8-byte
boundaries presumably has little overhead.

That may work but I am not sure if the other appended integers and doubles
will be affected with offsets in a similar way when receiving their values
in Fortran.

Ah, so they may be wrong, not just slower to access. I'm forgetting my Fortran implications!

[jeffreydeankelly2]

Thanks for the responses. FYI - SCIP has a built-in CRTL-C interrupt but it still returns control back to the user in order for the user to gracefully terminate their processing as well. All the best - Jeff

…

On Wed, Nov 22, 2023 at 8:40 AM Julian Hall ***@***.***> wrote: Thanks for letting me know about the CRTL-C signal handler - I look forward to it being removed in your next release :-) I've asked the two other folk with whom I've communicated regarding the CRTL-C signal handler, just out of politeness, as removing it would, technically, break the API Technically speaking, all C to Fortran API's seem to be redundant given standard Fortran's C Interoperability capability. Hence I never use these API's supplied by 3rd C/C++ based solvers as they are usually out-of-date and incomplete as well. That's interesting to know. Our Fortran API can certainly be classed as "out-of-date and incomplete". I'll still keep it, as it does no harm, but add a comment to the effect that it's out-of-date and incomplete, and that folk should exploit standard Fortran's C Interoperability capability. The way all of the other MIP callbacks are implemented is to allow the user to call for example your Highs_getSolution() routine (or the like) to retrieve the MIP improving solutions inside the callback routine. This wouldn't work in HiGHS, as no MIP solver data can be reached via the Highs C++ API. Indeed, if a user's callback function were to start calling methods in the instance of the Highs class (by passing a pointer to it via the void* user_data parameter) then all sorts of chaos could ensue! I should add code to the Highs C++ API to ensure that all methods give an immediate error return! Thanks for the Fortran callback function. I'll add it to examples/call_highs_from_fortran.f90. I can get around this by making double* mip_solution the first member of the HighsCallbackDataOut struct. That scalar doubles won't be on 8-byte boundaries presumably has little overhead. That may work but I am not sure if the other appended integers and doubles will be affected with offsets in a similar way when receiving their values in Fortran. Ah, so they may be wrong, not just slower to access. I'm forgetting my Fortran implications! — Reply to this email directly, view it on GitHub <#1523 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALHONEAGK2RYHKQ42CSXLPDYFX6E5AVCNFSM6AAAAAA7WDMXKSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRSG44TEMJQHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- ********************************************************** *I M P L - " M a k i n g O p t i m i z a t i o n a n d E s t i m i z a t i o n S m a r t e r"* Jeffrey D. Kelly Industrial Algorithms Limited - i n d u s t r i @ l g o r i t h m s Email: ***@***.*** Phone: (647) 917-4675 (IMPL) *Making Industrial AI (Algorithms & Integration) Real!* ********************************************************** This email and any files transmitted with it are confidential, proprietary and intended solely for the individual or entity to whom they are addressed. If you have received this email in error please delete it immediately.

[jajhall]

I'm trying to call the Highs instance from a call-back in check/TestCallbacks.cpp, but don't see how to pass and use the Highs instance via the void* pointer in highs-callback-no-highs-call in check/TestCallbacks.cpp

See branch fix-1523-latest