Reading MPS Files #459
Comments
|
For reference, here is the Downloadable .mps: https://drive.google.com/file/d/1rpOx4GomiPzSry733hIXtCW1yccmG1iD/view?usp=sharing Simple .txt (can view in browser): https://drive.google.com/file/d/1RKYo63FYPvAyFM-R8Au48TLQLC3v9zsx/view?usp=sharing |
|
I just downloaded the MPS file, read it into Cbc without errors and solved the instance in a few seconds. So the problem is at least not with the MPS file. I'll leave it to others to determine whether the problem is with PuLP itself. Depending on what you're actually trying to do, there are other ways of reading an MPS file in Python, such as CyLP. |
|
Hello everyone!
The problem probably is with pulp. The functionality is quite recent and
although we have tested it with some mps files we probably have not done an
exhaustive test.
I'll try to check what's wrong at the end of this week and get back to you.
…On Wed, Jul 7, 2021, 04:27 Ted Ralphs ***@***.***> wrote:
I just downloaded the MPS file, read it into Cbc without errors and solved
the instance in a few seconds. So the problem is at least not with the MPS
file. I'll leave it to others to determine whether the problem is with PuLP
itself. Depending on what you're actually trying to do, there are other
ways of reading an MPS file in Python, such as CyLP.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#459 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABJUZ454RC7UKO2N4TUW4UTTWO3PPANCNFSM475DGCWQ>
.
|
@tkralphs could you expand on these other ways please? Would they enable me to initialise a pulp LpProblem, or would I need to use another api such as mip? I was thinking of having a look at pysmps and seeing how difficult it would be to read in and initialise a pulp LpProblem myself |
|
I actually adapted the pysmps code to pulp (as can be seen in
https://github.com/coin-or/pulp/blob/d273841cd70fdacfa00eb8d68213334e997973ce/pulp/mps_lp.py#L33
).
This may imply that whatever is wrong in pulp, it's also in pysmps.
…On Wed, Jul 7, 2021, 07:15 Christopher Parsonson ***@***.***> wrote:
I just downloaded the MPS file, read it into Cbc without errors and solved
the instance in a few seconds. So the problem is at least not with the MPS
file. I'll leave it to others to determine whether the problem is with PuLP
itself. Depending on what you're actually trying to do, there are other
ways of reading an MPS file in Python, such as CyLP.
@tkralphs <https://github.com/tkralphs> could you expand on these other
ways please? Would they enable me to initialise a pulp LpProblem, or would
I need to use another api such as mip?
I was thinking of having a look at pysmps and seeing how difficult it
would be to read in and initialise a pulp LpProblem myself
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#459 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABJUZ42MXCOS7TMCCK77JFDTWPPH5ANCNFSM475DGCWQ>
.
|
|
Aside: @pchtsp Technically, @cwfparsonson With CyLP, you could read the data from the MPS file into numpy objects and then do with that data as you wish. CyLP and PuLP can easily co-exist and share data. If you are working with MPS files, though, CyLP could be a better modeling environment overall. See here. |
|
Thanks @tkralphs, I'll check out CyLP although if possible I would like to stick with pulp since the syntax and flexibility is so nice. I had a crack at trying to use pysmps to generate a pulp LpProblem but was unsuccessful, so for last few hours I've been trying to replace pulp with mip but mip doesn't have the flexibility I'm finding. If anyone is able to help out with updating the |
|
Hi, I've been trying to interface pulp with the numpy arrays read in by CyLP. I cannot work out why pulp is still saying that the problem is infeasible. The instance initialises fine and looks correct (correct number of variables and constraints, mix of binary and continuous variable categories etc.). Is anyone more familiar with pulp able to spot something obvious which I am doing wrong? Here is how I am reading in the mps file: import pulp
import numpy as np
from cylp.cy import CyCoinMpsIO
class MPSLoader:
def __init__(self):
pass
def load_mps(self, path):
mps = CyCoinMpsIO()
mps.readMps(path)
return mps
def conv_sparse_to_full_matrix(self, c):
A = []
for row_idx in range(c.majorDim):
coeffs = [] # have coeff for each var
lhs_nonzero_coeffs = c.elements[c.vectorStarts[row_idx]:c.vectorStarts[row_idx+1]].tolist()
lhs_nonzero_vars = c.indices[c.vectorStarts[row_idx]:c.vectorStarts[row_idx+1]].tolist()
i = 0
coeff, var_idx = lhs_nonzero_coeffs[i], lhs_nonzero_vars[i]
for v_idx in range(c.minorDim):
if i < len(lhs_nonzero_coeffs):
# still have coeffs to add
if v_idx == lhs_nonzero_vars[i]:
# this coeff is applied to this variable
coeffs.append(lhs_nonzero_coeffs[i])
i += 1
else:
# this coeff not applied to this variable
coeffs.append(0)
else:
# this coeff not applied to this variable
coeffs.append(0)
A.append(coeffs)
return A
def load_mps_as_dict(self, path):
reader = self.load_mps(path)
attrs = ['name', 'objective_name', 'row_names', 'col_names', 'cats', 'types', 'c', 'A',
'b', 'LO', 'UP']
mps_dict = {attr: None for attr in attrs}
mps_dict['c'] = reader.objCoefficients
mps_dict['b'] = reader.rightHandSide.tolist()
mps_dict['LO'] = reader.variableLower.tolist()
mps_dict['UP'] = reader.variableUpper.tolist()
mps_dict['types'] = [chr(sign) for sign in reader.constraintSigns.tolist()]
mps_dict['A'] = self.conv_sparse_to_full_matrix(reader.matrixByRow)
mps_dict['cats'] = []
for i in reader.integerColumns:
if i == 0:
# continuous
mps_dict['cats'].append('Continuous')
elif i == 1:
# check if integer or binary
if mps_dict['LO'][i] == 0 and mps_dict['UP'][i] == 1:
# binary
mps_dict['cats'].append('Binary')
else:
# integer
mps_dict['cats'].append('Integer')
else:
raise Exception('Unrecognised integer indicator {}'.format(i))
return mps_dict
# load mps file into dict
path = '../milp/datasets/instances/1_item_placement/train/item_placement_1.mps'
mps_loader = MPSLoader()
mps_dict = mps_loader.load_mps_as_dict(path)And this is how I then use thie # initialise instance
instance = pulp.LpProblem(sense=1)
# initialise variables
variables = {f'x{i}': pulp.LpVariable(name=f'x{i}', lowBound=lb, upBound=ub, cat=cat) for i, lb, ub, cat in zip([x for x in range(1, len(mps_dict['LO']))], mps_dict['LO'], mps_dict['UP'], mps_dict['cats'])}
# initialise constraints
# collect lhs constraint values
constrs = []
A = np.array(mps_dict['A'])
for row_idx in range(len(A[:, 0])):
constr = []
for coeff, var in zip(A[row_idx, :], list(variables.values())):
constr.append(float(coeff) * var)
constrs.append(constr)
# add constraints
for i in range(len(constrs)):
if mps_dict['types'][i] == 'E':
instance += pulp.lpSum(constrs[i]) == mps_dict['b'][i]
elif mps_dict['types'][i] == 'L':
instance += pulp.lpSum(constrs[i]) <= mps_dict['b'][i]
elif mps_dict['types'][i] == 'G':
instance += pulp.lpSum(constrs[i]) >= mps_dict['b'][i]
else:
raise Exception('Unrecognised constraint type {}'.format(mps_dict['types'][i]))
# register objective function
instance += pulp.lpSum([float(coeff) * var for coeff, var in zip(mps_dict['c'], list(variables.values()))])
# solve
status = instance.solve()
print(status)
-1Any help would be much appreciated! |
|
Can you try generating the mps file from the pulp "instance" object in your
code and compare it with the original file ? So you can see if something is
lost /modified at some point during the translation ?
…On Thu, Jul 8, 2021, 12:38 Christopher Parsonson ***@***.***> wrote:
Hi,
I've been trying to interface pulp with the numpy arrays read in by CyLP.
I cannot work out why pulp is still saying that the problem is infeasible.
The instance initialises fine and looks correct (correct number of
variables and constraints, mix of binary and continuous variable categories
etc.). Is anyone more familiar with pulp able to spot something obvious
which I am doing wrong?
Here is how I am reading in the mps file:
import pulpimport numpy as npfrom cylp.cy import CyCoinMpsIO
class MPSLoader:
def __init__(self):
pass
def load_mps(self, path):
mps = CyCoinMpsIO()
mps.readMps(path)
return mps
def conv_sparse_to_full_matrix(self, c):
A = []
for row_idx in range(c.majorDim):
coeffs = [] # have coeff for each var
lhs_nonzero_coeffs = c.elements[c.vectorStarts[row_idx]:c.vectorStarts[row_idx+1]].tolist()
lhs_nonzero_vars = c.indices[c.vectorStarts[row_idx]:c.vectorStarts[row_idx+1]].tolist()
i = 0
coeff, var_idx = lhs_nonzero_coeffs[i], lhs_nonzero_vars[i]
for v_idx in range(c.minorDim):
if i < len(lhs_nonzero_coeffs):
# still have coeffs to add
if v_idx == lhs_nonzero_vars[i]:
# this coeff is applied to this variable
coeffs.append(lhs_nonzero_coeffs[i])
i += 1
else:
# this coeff not applied to this variable
coeffs.append(0)
else:
# this coeff not applied to this variable
coeffs.append(0)
A.append(coeffs)
return A
def load_mps_as_dict(self, path):
reader = self.load_mps(path)
attrs = ['name', 'objective_name', 'row_names', 'col_names', 'cats', 'types', 'c', 'A',
'b', 'LO', 'UP']
mps_dict = {attr: None for attr in attrs}
mps_dict['c'] = reader.objCoefficients
mps_dict['b'] = reader.rightHandSide.tolist()
mps_dict['LO'] = reader.variableLower.tolist()
mps_dict['UP'] = reader.variableUpper.tolist()
mps_dict['types'] = [chr(sign) for sign in reader.constraintSigns.tolist()]
mps_dict['A'] = self.conv_sparse_to_full_matrix(reader.matrixByRow)
mps_dict['cats'] = []
for i in reader.integerColumns:
if i == 0:
# continuous
mps_dict['cats'].append('Continuous')
elif i == 1:
# check if integer or binary
if mps_dict['LO'][i] == 0 and mps_dict['UP'][i] == 1:
# binary
mps_dict['cats'].append('Binary')
else:
# integer
mps_dict['cats'].append('Integer')
else:
raise Exception('Unrecognised integer indicator {}'.format(i))
return mps_dict
# load mps file into dictpath = '../milp/datasets/instances/1_item_placement/train/item_placement_1.mps'mps_loader = MPSLoader()mps_dict = mps_loader.load_mps_as_dict(path)
And this is how I then use thie mps_dict to initialise pulp:
# initialise instanceinstance = pulp.LpProblem(sense=1)
# initialise variablesvariables = {f'x{i}': pulp.LpVariable(name=f'x{i}', lowBound=lb, upBound=ub, cat=cat) for i, lb, ub, cat in zip([x for x in range(1, len(mps_dict['LO']))], mps_dict['LO'], mps_dict['UP'], mps_dict['cats'])}
# initialise constraints# collect lhs constraint valuesconstrs = []A = np.array(mps_dict['A'])for row_idx in range(len(A[:, 0])):
constr = []
for coeff, var in zip(A[row_idx, :], list(variables.values())):
constr.append(float(coeff) * var)
constrs.append(constr)# add constraintsfor i in range(len(constrs)):
if mps_dict['types'][i] == 'E':
instance += pulp.lpSum(constrs[i]) == mps_dict['b'][i]
elif mps_dict['types'][i] == 'L':
instance += pulp.lpSum(constrs[i]) <= mps_dict['b'][i]
elif mps_dict['types'][i] == 'G':
instance += pulp.lpSum(constrs[i]) >= mps_dict['b'][i]
else:
raise Exception('Unrecognised constraint type {}'.format(mps_dict['types'][i]))
# register objective functioninstance += pulp.lpSum([float(coeff) * var for coeff, var in zip(mps_dict['c'], list(variables.values()))])
# solvestatus = instance.solve()print(status)-1
Any help would be much appreciated!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#459 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABJUZ47T7Z7RA4SK5X62UWDTWV5YXANCNFSM475DGCWQ>
.
|
|
@tkralphs , regarding adding pysmps as a dependancy: I agree it would
definitely be a better option. I don't remember why I didn't do it at the
time but I suspect I have a technical reason. I'll see if I can make a PR
to pysmps with a refactored function that we could then use from inside
pulp...
…On Wed, Jul 7, 2021, 16:19 Ted Ralphs ***@***.***> wrote:
Aside: @pchtsp <https://github.com/pchtsp> Technically, pysmps is under
the [MIT license]( and you should reproduce the copyright notice and the
license if you re-use it. Since pysmps is on Pypi, though, it would seem
better to fork it, improve it, and submit a pull request back to the
original project. Then it can be a dependency of PuLP. It would be nice to
have a stand-alone MPS reader in pure Python that all could build on, so as
to avoid re-inventing the wheel. (Making a compliant and robust MPS reader
is probably more difficult than it seems).
@cwfparsonson <https://github.com/cwfparsonson> With CyLP, you could read
the data from the MPS file into numpy objects and then do with that data as
you wish. CyLP and PuLP can easily co-exist and share data. If you are
working with MPS files, though, CyLP could be a better modeling environment
overall. See here <http://coin-or.github.io/CyLP/modules/CyCoinMpsIO.html>
.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#459 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABJUZ4ZU5VJ5MME74ANH6ULTWRO6NANCNFSM475DGCWQ>
.
|
In my Original .mps file: https://drive.google.com/file/d/1rpOx4GomiPzSry733hIXtCW1yccmG1iD/view?usp=sharing Code to reproduce: import pulp
path = '../milp/datasets/instances/1_item_placement/train/item_placement_1.mps'
variables, instance = pulp.LpProblem.fromMPS(path, sense=1)
path = '../milp/datasets/instances/1_item_placement/train/pulp_item_placement_1.mps'
instance.writeMPS(path) |
|
Hi, I'm wondering if the problem is more to do with Trying to use To solve this, I've implemented my own from pysmps import smps_loader as smps
class MPSLoader:
def __init__(self):
pass
def load_mps(self, path):
return smps.load_mps(path)
def conv_mps_to_dict(self, path):
reader = self.load_mps(path)
# init attrs
attrs = ['name', 'objective_name', 'row_names', 'col_names', 'cats', 'types', 'c', 'A', 'rhs_names',
'rhs', 'bnd_names', 'bnd']
idxs = [i for i in range(len(attrs))]
idx_to_attr = {idx: attr for idx, attr in zip(idxs, attrs)}
mps_dict = {}
for idx, attr in idx_to_attr.items():
mps_dict[attr] = reader[idx]
# reconfigure attrs for pulp
_lbs = [mps_dict['bnd'][bnd_name]['LO'] for bnd_name in mps_dict['bnd_names']]
mps_dict['LO'] = [float(item) for sublist in _lbs for item in sublist]
_ubs = [mps_dict['bnd'][bnd_name]['UP'] for bnd_name in mps_dict['bnd_names']]
mps_dict['UP'] = [float(item) for sublist in _ubs for item in sublist]
cats = []
for cat in mps_dict['cats']:
if cat == 'integral':
cats.append('Binary')
elif cat == 'continuous':
cats.append('Continuous')
elif cat == 'binary':
cats.append('Binary')
else:
raise Exception('Unrecognised variable category {}'.format(cat))
mps_dict['cats'] = cats
# collect rhs constraint values
b = []
for rhs_name in mps_dict['rhs_names']:
for coeff in mps_dict['rhs'][rhs_name]:
b.append(float(coeff))
mps_dict['b'] = b
return mps_dictLoading this mps data into a pulp instance: path = 'mip_data_set_1.mps'
mps_loader = MPSLoader()
mps_dict = mps_loader.conv_mps_to_dict(path)
# init problem instance
instance = pulp.LpProblem(name=mps_dict['name'], sense=1)
# init vars
variables = {name: pulp.LpVariable(name=name, lowBound=lb, upBound=ub, cat=cat) for name, lb, ub, cat in zip(mps_dict['col_names'], mps_dict['LO'], mps_dict['UP'], mps_dict['cats'])}
# init constraints
# collect lhs constraint values
constrs = []
for row_idx in range(len(mps_dict['A'][:, 0])):
constr = []
for coeff, var in zip(mps_dict['A'][row_idx, :], list(variables.values())):
constr.append(float(coeff) * var)
constrs.append(constr)
# add constraints
for i in range(len(constrs)):
if mps_dict['types'][i] == 'E':
instance += pulp.lpSum(constrs[i]) == mps_dict['b'][i]
elif mps_dict['types'][i] == 'L':
instance += pulp.lpSum(constrs[i]) <= mps_dict['b'][i]
elif mps_dict['types'][i] == 'G':
instance += pulp.lpSum(constrs[i]) >= mps_dict['b'][i]
else:
raise Exception('Unrecognised constraint type {}'.format(mps_dict['types'][i]))
# register objective function
instance += pulp.lpSum([float(coeff) * var for coeff, var in zip(mps_dict['c'], list(variables.values()))])Calling However, when I call The mps is certainly feasible, as shown by solving it with mip: import mip
path = 'mip_data_set_1.mps'
instance = mip.Model()
instance.read(path)
status = instance.optimize(max_seconds=300)
if status == mip.OptimizationStatus.OPTIMAL:
print('optimal solution cost {} found'.format(instance.objective_value))
elif status == mip.OptimizationStatus.FEASIBLE:
print('sol.cost {} found, best possible: {}'.format(instance.objective_value, instance.objective_bound))
elif status == mip.OptimizationStatus.NO_SOLUTION_FOUND:
print('no feasible solution found, lower bound is: {}'.format(instance.objective_bound))
if status == mip.OptimizationStatus.OPTIMAL or status == mip.OptimizationStatus.FEASIBLE:
print('solution:')
for v in instance.vars:
if abs(v.x) > 1e-6: # only printing non-zeros
print('{} : {}'.format(v.name, v.x))Output: Is this |
|
Here comes my analysis so far. As always, it's a mix of things. Load an mps with minimization PuLP's mps reader crashing PuLP's PulpSolveError but if I generate an mps with the instance.writeMPS("generated_by_pulp.mps")and then give it to cbc it throws an error: Next time, if you want to get more information from the solver, you have to solve the problem like this: |
|
@tkralphs I checked the pysmps project and remembered the main reason why I did not add it as dependency: they have |
|
@tkralphs I've opened an issue at |
Hi,
I am trying to read an MPS file as part of an optimisation competition (https://github.com/ds4dm/ml4co-competition). The dataset is located in instances.tar.gz (https://drive.google.com/file/d/1MytdY3IwX_aFRWdoc0mMfDN9Xg1EKUuq/view). This is quite a big data set, so here's a single 300kB .mps file from the data set which I am trying to read in as a pulp LpProblem: https://drive.google.com/file/d/1rpOx4GomiPzSry733hIXtCW1yccmG1iD/view?usp=sharing
Once downloaded, I am using the below to load the .mps instance:
This appears to read the .mps file without crashing, however it seems to load the problem as a maximisation problem when I believe it is meant to be a minimisation problem. Furthermore, when I run:
I get the following error:
Setting
sense=1seems to prevent the above crash, however it results inpulpsaying the problem is infeasible:I do not think the problem is with the .mps file, because the following code appears to work with the
miplibrary:Do you know if there might be any bugs with
pulp.LpProblem.fromMPS()?The text was updated successfully, but these errors were encountered: