doubts on correctness of sol.h:WriteSolFile

[svigerske]

I tried to adapt the code

// Writes a solution to a .sol file.
template <typename Solution>
void WriteSolFile(fmt::CStringRef filename, const Solution &sol) {
  fmt::BufferedFile file(filename, "w");
  internal::WriteMessage(file, sol.message());
  // Write options.
  file.print("Options\n");
  if (int num_options = sol.num_options()) {
    file.print("{}\n", num_options);
    for (int i = 0; i < num_options; ++i)
      file.print("{}\n", sol.option(i));
  }
  // TODO: check precision
  int num_values = sol.num_values(), num_dual_values = sol.num_dual_values();
  file.print("{0}\n{0}\n{1}\n{1}\n", num_dual_values, num_values);
  for (int i = 0; i < num_values; ++i)
    file.print("{}\n", sol.value(i));
  for (int i = 0, n = num_dual_values; i < n; ++i)
    file.print("{}\n", sol.dual_value(i));
  file.print("objno 0 {}\n", sol.status());
  suf::Kind kinds[] = {suf::VAR, suf::CON, suf::OBJ, suf::PROBLEM};
  for (std::size_t i = 0, n = sizeof(kinds) / sizeof(*kinds); i < n; ++i)
    internal::WriteSuffixes(file, sol.suffixes(kinds[i]));
  // TODO: test
}

for an AMPL interface I was writing, but had problems to get the solution accepted by AMPL.

I don't have options, so the lines

  file.print("Options\n");
  if (int num_options = sol.num_options()) {
    file.print("{}\n", num_options);
    for (int i = 0; i < num_options; ++i)
      file.print("{}\n", sol.option(i));
  }

would only print the line Options. For my tiny test problem (2 vars, 2 constraints), I then get this from AMPL:

Error executing "solve" command:
expected nOpts between 3 and 9; got 2: Error executing "solve" command:
Bad ASCII solution file /tmp/at1341615.sol

If I change my code to do essentially

  file.print("Options\n");
  file.print("{}\n", num_options);
  if (int num_options = sol.num_options()) {
    for (int i = 0; i < num_options; ++i)
      file.print("{}\n", sol.option(i));
  }

thus print Options\n0\n, then, as expected, I get this from AMPL:

Error executing "solve" command:
expected nOpts between 3 and 9; got 0: Error executing "solve" command:
Bad ASCII solution file /tmp/at1341615.sol

(@andreaslundell does this really work for you?)

What worked from me (and I copied that from a .sol file I had around somewhere) was to print Options\n3\n2\n1\n0\n, but I have no idea what the 2, 1, 0, actually mean.

Next, there is

  file.print("{0}\n{0}\n{1}\n{1}\n", num_dual_values, num_values);

but when I look at writesol.c from ASL, then my impression is that this should actually be

• number of constraints
• number of constraint dual values to be written
• number of variables
• number of variable primal values to be written

And after this, we have

  for (int i = 0; i < num_values; ++i)
    file.print("{}\n", sol.value(i));
  for (int i = 0, n = num_dual_values; i < n; ++i)
    file.print("{}\n", sol.dual_value(i));

but it seems that AMPL expects to see the constraint dual values first and the variable primal values afterwards (https://github.com/ampl/asl/blob/64919f75fa7a438f4b41bce892dcbe2ae38343ee/src/solvers/writesol.c#L434-L443).

[andreaslundell]

@svigerske I have actually never tried calling SHOT from AMPL, only Pyomo. Perhaps Pyomo is more forgiving?

[svigerske]

@svigerske I have actually never tried calling SHOT from AMPL, only Pyomo. Perhaps Pyomo is more forgiving?

I could imagine that Pyomo doesn't treat it as error if SHOT says that it specifies only 0 of these mysterious options instead of between 3 and 9. Given the response of the AMPL folks here, it doesn't seem that they intend to explain (or remember) the full specification of that file.

But you might want to try running SHOT under the free version of AMPL and see if that works (doesn't this need to accept -AMPL (one dash), too?).

[fdabrandao]

Hi @svigerske, what is the header of your .nl file? The options that need to be written in .sol file originate from the first line of the .nl file.

[svigerske]

I don't know what header AMPL was writing when creating a .nl file.
One of the .nl files I usually try starts with

b3 0 1 0        # problem alan
 8 7 1 0 2      # vars, constraints, objectives, ranges, eqns
 0 1    # nonlinear constraints, objectives
 0 0    # network constraints: nonlinear, linear
 0 3 0  # nonlinear vars in constraints, objectives, both
 0 0 1 1        # linear network variables; functions; arith, flags
 4 0 0 0 0      # discrete variables: binary, integer, nonlinear (b,c,o)
 20 3   # nonzeros in Jacobian, gradients
 3 2    # max name lengths: constraints, variables
 0 0 0 0 0      # common exprs: b,c,o,c1,o1

So I should write 3\n0\n1\n0\n ?

But I use mp::ReadNLFile() to read the NL file.
So do one need to reproduce the entries from NLHeader::ampl_options there? Or are this some other kind of options?

[fdabrandao]

The options are from the first line of the nl file. In this case "3 0 1 0". These options are used internally by AMPL and are mostly related with the presolver to know some settings that were set during the nl file generation. One of our drivers using the mp interface is ilogcp and and it seems to be passing the options correctly from the nl file to the sol file. Is mp::ReadNLFile() not reading this information correctly? If it is reading this information correctly, then you should only need to write the options again in the sol file ("3\n0\n1\n0\n" in this case).

[svigerske]

I tried to follow WriteSolFile() and understand where sol.option() is filled in the code. I ended up at the constructor of SolutionWriter, which initializes options to an empty array, https://github.com/ampl/mp/blob/master/include/mp/solver.h#L1011.
So if you say I need to replicate NLHeader::ampl_options, then I can do this. What about everything that follows options? Is it really file.print("{0}\n{0}\n{1}\n{1}\n", num_dual_values, num_values) and is it really variable primal values before constraint dual values or the other way around?

[fdabrandao]

Looking at an example may be easier to understand the content of a sol file:

ampl: model diet.mod;
ampl: data diet.dat;
ampl: write gdiet; # write diet.nl
ampl: solve; # write diet.sol
MINOS 5.51: optimal solution found.
6 iterations, objective 88.2
ampl: display Buy;
Buy [*] :=
BEEF   0
 CHK   0
FISH   0
 HAM   0
 MCH  46.6667
 MTL   1.57618e-15
 SPG   8.42982e-15
 TUR   0
;

ampl: display Diet.dual;
Diet.dual [*] :=
 A   0.00181818
B1   0.00818182
B2   0.116
 C  -8.16933e-18
;

The header of diet.nl is "g3 0 1 0 # problem diet;" and the content of diet.sol is:

MINOS 5.51: optimal solution found.
6 iterations, objective 88.19999999999997

Options
3
0
1
0
4
4
8
8
0.0018181818181818108
0.008181818181818278
0.11599999999999991
-8.169327588411304e-18
0
0
0
0
46.66666666666666
1.5761812194954111e-15
8.429823983987501e-15
0
objno 0 0
suffix 0 8 8 0 0
sstatus
0 3
1 3
2 3
3 3
4 1
5 1
6 1
7 3
suffix 1 4 8 0 0
sstatus
0 3
1 3
2 3
3 1

The first part is related to the header "3 0 1 0":

Options
3
0
1
0

Then we have the dimensions (number of constraints, number of values for constraints in the solution, number of variables, number of values for variables in the solution):

4
4
8
8

Dual values for Diet.dual:

0.0018181818181818108
0.008181818181818278
0.11599999999999991
-8.169327588411304e-18

Primal values for variable Buy:

0
0
0
0
46.66666666666666
1.5761812194954111e-15
8.429823983987501e-15
0

After this we have the objective number and status and then some suffixes are included as well.

So we have dual values first and primal values after. In https://github.com/ampl/asl/blob/64919f75fa7a438f4b41bce892dcbe2ae38343ee/src/solvers/writesol.c#L434-L443 the same happens with y1 = y; first followed by a loop while(--m >= 0) where m is the number of values for constraints, and then y1 = x; followed by a loop while(--n >= 0) where n is the number of values for variables.

In https://github.com/ampl/asl/blob/64919f75fa7a438f4b41bce892dcbe2ae38343ee/src/solvers/writesol.c, the array z is used to hold the 4 values related to dimensions:

z[0] = m = asl->i.n_con0;
if (!y)
        m = 0;
z[1] = m;
z[2] = n = asl->i.n_var0;
if (!x)
        n = 0;
z[3] = n;
fprintf(f,"%ld\n%ld\n%ld\n%ld\n", (long)z[0],(long)z[1],(long)z[2],(long)z[3]);

[glebbelov]

Concerning the Options issue: ilogcp correctly transfers "3 0 1 0" from alan.nl into alan.sol, so if you use standard mp setup it should work.

Concerning the order of dual/primal values: the code in sol.h is indeed buggy:

  for (int i = 0; i < num_values; ++i)
    file.print("{}\n", sol.value(i));
  for (int i = 0, n = num_dual_values; i < n; ++i)
    file.print("{}\n", sol.dual_value(i));

I fixed it on master. mp drivers have never done much with continuous variables (ilogcp's option 'optimizer=cplex' is non-default), so this is just not enough tested (but we are getting there).

[svigerske]

So the issue that remains then in WriteSolFile is that

file.print("{0}\n{0}\n{1}\n{1}\n", num_dual_values, num_values);

is wrong if there are no dual values or there are no primal values.
The first and third entry should be the number of constraints and number of variables, respectively.

Examples are nice, but they cover only one case each. I had hoped you could point to a specification of .sol files.