Highs solver interface does not work with latest version

[loongmxbt]

Hi, I made a default model, using Highs 1.2.1 on both Win10 and Mac

import linopy
import pandas as pd
import xarray as xr

m = linopy.Model()
time = pd.Index(range(10), name='time')

x = m.add_variables(lower=0, coords=[time], name='x', )
y = m.add_variables(lower=0, coords=[time], name='y')

factor = pd.Series(time, index=time)

con1 = m.add_constraints(3*x + 7*y >= 10*factor, name='con1')
con2 = m.add_constraints(5*x + 2*y >= 3*factor, name='con2')

m.add_objective(x + 2*y)
m.solve('highs')

m.solution.to_dataframe().plot(grid=True, ylabel='Optimal Value')

On mac the error is:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/var/folders/nj/hhq5qdcx2hd8h_5kpp15lscw0000gn/T/ipykernel_82794/4115417935.py in <module>
     15 
     16 m.add_objective(x + 2*y)
---> 17 m.solve('highs')
     18 
     19 m.solution.to_dataframe().plot(grid=True, ylabel='Optimal Value')

~/miniconda3/lib/python3.9/site-packages/linopy/model.py in solve(self, solver_name, io_api, problem_fn, solution_fn, log_fn, basis_fn, warmstart_fn, keep_files, remote, **solver_options)
   1042         try:
   1043             func = getattr(solvers, f"run_{solver_name}")
-> 1044             res = func(
   1045                 self,
   1046                 io_api,

~/miniconda3/lib/python3.9/site-packages/linopy/solvers.py in run_highs(Model, io_api, problem_fn, solution_fn, log_fn, warmstart_fn, basis_fn, keep_files, **solver_options)
    369 
    370     dual = pd.read_fwf(io.BytesIO(dual))["Dual"]
--> 371     dual.index = Model.constraints.ravel("labels", filter_missings=True)
    372 
    373     return dict(

~/miniconda3/lib/python3.9/site-packages/pandas/core/generic.py in __setattr__(self, name, value)
   5498         try:
   5499             object.__getattribute__(self, name)
-> 5500             return object.__setattr__(self, name, value)
   5501         except AttributeError:
   5502             pass

~/miniconda3/lib/python3.9/site-packages/pandas/_libs/properties.pyx in pandas._libs.properties.AxisProperty.__set__()

~/miniconda3/lib/python3.9/site-packages/pandas/core/series.py in _set_axis(self, axis, labels, fastpath)
    557         if not fastpath:
    558             # The ensure_index call above ensures we have an Index object
--> 559             self._mgr.set_axis(axis, labels)
    560 
    561     # ndarray compatibility

~/miniconda3/lib/python3.9/site-packages/pandas/core/internals/managers.py in set_axis(self, axis, new_labels)
    214     def set_axis(self, axis: int, new_labels: Index) -> None:
    215         # Caller is responsible for ensuring we have an Index object.
--> 216         self._validate_set_axis(axis, new_labels)
    217         self.axes[axis] = new_labels
    218 

~/miniconda3/lib/python3.9/site-packages/pandas/core/internals/base.py in _validate_set_axis(self, axis, new_labels)
     55 
     56         elif new_len != old_len:
---> 57             raise ValueError(
     58                 f"Length mismatch: Expected axis has {old_len} elements, new "
     59                 f"values have {new_len} elements"

ValueError: Length mismatch: Expected axis has 22 elements, new values have 20 elements

On win10 the error is:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
D:\Tools\Miniconda3\envs\py37\lib\site-packages\pandas\core\indexes\base.py in astype(self, dtype, copy)
    912         try:
--> 913             casted = self._values.astype(dtype, copy=copy)
    914         except (TypeError, ValueError) as err:

ValueError: cannot convert float NaN to integer

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_3220\3173557130.py in <module>
     15 
     16 m.add_objective(x + 2*y)
---> 17 m.solve('highs')
     18 
     19 m.solution.to_dataframe().plot(grid=True, ylabel='Optimal Value')

D:\Tools\Miniconda3\envs\py37\lib\site-packages\linopy\model.py in solve(self, solver_name, io_api, problem_fn, solution_fn, log_fn, basis_fn, warmstart_fn, keep_files, remote, **solver_options)
   1051                 basis_fn,
   1052                 keep_files,
-> 1053                 **solver_options,
   1054             )
   1055         finally:

D:\Tools\Miniconda3\envs\py37\lib\site-packages\linopy\solvers.py in run_highs(Model, io_api, problem_fn, solution_fn, log_fn, warmstart_fn, basis_fn, keep_files, **solver_options)
    366 
    367     sol = pd.read_fwf(io.BytesIO(sol))
--> 368     sol = sol.set_index("Name")["Primal"].pipe(set_int_index)
    369 
    370     dual = pd.read_fwf(io.BytesIO(dual))["Dual"]

D:\Tools\Miniconda3\envs\py37\lib\site-packages\pandas\core\generic.py in pipe(self, func, *args, **kwargs)
   5428         ...  )  # doctest: +SKIP
   5429         """
-> 5430         return com.pipe(self, func, *args, **kwargs)
   5431 
   5432     # ----------------------------------------------------------------------

D:\Tools\Miniconda3\envs\py37\lib\site-packages\pandas\core\common.py in pipe(obj, func, *args, **kwargs)
    469         return func(*args, **kwargs)
    470     else:
--> 471         return func(obj, *args, **kwargs)
    472 
    473 

D:\Tools\Miniconda3\envs\py37\lib\site-packages\linopy\solvers.py in set_int_index(series)
     65     Convert string index to int index.
     66     """
---> 67     series.index = series.index.str[1:].astype(int)
     68     return series
     69 

D:\Tools\Miniconda3\envs\py37\lib\site-packages\pandas\core\indexes\base.py in astype(self, dtype, copy)
    915             raise TypeError(
    916                 f"Cannot cast {type(self).__name__} to dtype {dtype}"
--> 917             ) from err
    918         return Index(casted, name=self.name, dtype=dtype)
    919 

TypeError: Cannot cast Index to dtype int32

In both environment I tested cbc, cplex no problem. And Pypsa with nomopyomo and highs no problem. So I think may be the linopy's highs interface problem.

[FabianHofmann]

I cannot reproduce the bug on Linux for both the current master branch of linopy and the latest version v0.0.10 (with xarray v2022.03).
So you say for both CBC and cplex the example works, right? I am a bit confused because the first error appears to be before the solving process and thus should be solver independent.
As for the second error, could you run a debugger and print out the series.index on the second lowest level? I would be interested how that looks like.

[loongmxbt]

I tried Mac + Python 3.9.5 + xarray 2022.3.0 + linopy 0.0.10 still the same issue. For cbc works fine, for highs comes to an error.
On my windows version Win10 + Python3.8 + xarray 2022.3.0 + linopy 0.0.10 highs solves the problem and also got same Objective value and then get TypeError: Cannot cast Index to dtype int32. (Different with mac)

Seems highs solves the problem and something went wrong in result reading process. So I think maybe the highs version update causes the problem?

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/var/folders/nj/hhq5qdcx2hd8h_5kpp15lscw0000gn/T/ipykernel_53892/3173557130.py in <module>
     15 
     16 m.add_objective(x + 2*y)
---> 17 m.solve('highs')
     18 
     19 m.solution.to_dataframe().plot(grid=True, ylabel='Optimal Value')

~/miniconda3/lib/python3.9/site-packages/linopy/model.py in solve(self, solver_name, io_api, problem_fn, solution_fn, log_fn, basis_fn, warmstart_fn, keep_files, remote, **solver_options)
   1042         try:
   1043             func = getattr(solvers, f"run_{solver_name}")
-> 1044             res = func(
   1045                 self,
   1046                 io_api,

~/miniconda3/lib/python3.9/site-packages/linopy/solvers.py in run_highs(Model, io_api, problem_fn, solution_fn, log_fn, warmstart_fn, basis_fn, keep_files, **solver_options)
    369 
    370     dual = pd.read_fwf(io.BytesIO(dual))["Dual"]
--> 371     dual.index = Model.constraints.ravel("labels", filter_missings=True)
    372 
    373     return dict(

~/miniconda3/lib/python3.9/site-packages/pandas/core/generic.py in __setattr__(self, name, value)
   5498         try:
   5499             object.__getattribute__(self, name)
-> 5500             return object.__setattr__(self, name, value)
   5501         except AttributeError:
   5502             pass

~/miniconda3/lib/python3.9/site-packages/pandas/_libs/properties.pyx in pandas._libs.properties.AxisProperty.__set__()

~/miniconda3/lib/python3.9/site-packages/pandas/core/series.py in _set_axis(self, axis, labels, fastpath)
    557         if not fastpath:
    558             # The ensure_index call above ensures we have an Index object
--> 559             self._mgr.set_axis(axis, labels)
    560 
    561     # ndarray compatibility

~/miniconda3/lib/python3.9/site-packages/pandas/core/internals/managers.py in set_axis(self, axis, new_labels)
    214     def set_axis(self, axis: int, new_labels: Index) -> None:
    215         # Caller is responsible for ensuring we have an Index object.
--> 216         self._validate_set_axis(axis, new_labels)
    217         self.axes[axis] = new_labels
    218 

~/miniconda3/lib/python3.9/site-packages/pandas/core/internals/base.py in _validate_set_axis(self, axis, new_labels)
     55 
     56         elif new_len != old_len:
---> 57             raise ValueError(
     58                 f"Length mismatch: Expected axis has {old_len} elements, new "
     59                 f"values have {new_len} elements"

ValueError: Length mismatch: Expected axis has 22 elements, new values have 20 elements

[FabianHofmann]

@loongmxbt FYI, I don't think windows machine are compatible with the latest version and hishs, as we are now using highspy and this seems to be working on windows yet.

[jonasghini]

Hi,

I'll revive this thread with some new info. Due to work restrictions I have to use Windows, and due to problem size and financial constraints, I have to use Highs. From some brief contact with the Highs team, there does not seem to be any plans to port highspy to Windows any time soon. As such, when preparing to solve my models, I go through and make several changes to the master branch code of Linopy (and technically also PyPSA, though those changes are less critical for the actual solving itself).

First, the check for availability of the Highs solver must be changed to look like the check for most of the other solvers (calling a sub, and checking for a response). Then the run_highs-function needs to be changed to use the sub-method again (like in some older version of Linopy), including some changes to the interface since Highs made a breaking change to their output format some time back. Then the solution extraction needs to be different than the master branch.

Going through my list of changes leads to a reliable implementation for using the Highs solver on Windows, so I have code I can share. Not currently doing any compatibility checks related to platform, though, since I know I only need it to work on Windows, and don't need to make sure it still works on Mac and Linux, though, so my code is definitely not "production ready".

We do have a few options, though. One very simple solution is that I just fork Linopy and PyPSA, include my changes, and leave it at that, on a public repo. Another is to make a pull request with several changes, though that would need work by someone else too, since I cannot easily test other environments than Windows.

Any thoughts?

[FabianHofmann]

Hey @jonasghini, I am always a bit confused about the status with highspy and windows. In our current CI, we are testing highspy in linopy on windows machines and it seems to working. Have you tried to install linopy like

pip install linopy[solvers]

?

[jonasghini]

Oh, OK! I'll give that a try!

[jonasghini]

Yup! Seems to work! This definitely makes my workflow easier! Thanks for the heads up. I'd written it off as a lost cause, but I guess the Highs team snuck in Windows support for highspy when I wasn't looking.

[FabianHofmann]

Okay perfect, good to hear. I'll close this issue, feel free to reopen if problems reappear.