highs-js internal exception with >= 32 variables ?

[lequant40]

Hello !

Trying to play with the package, it seems there is an error when I try to increase a very simple problem from 31 variables (working OK) to 32 variables (exception).

Would it be possible you have a look ?

My reproducer is:

31 variables => OK

var pb = `Maximize
 obj: 0.007285383333 x1 0.0088992 x2 0.01417630833 x3 0.009235308333 x4 0.01318765 x5 0.01406648333 x6 0.01360380833 x7 0.01448148333 x8 0.01661586667 x9 0.004312425 x10 0.008158858333 x11 0.006681016667 x12 0.0005192 x13 0.006546558333 x14 0.005799741667 x15 0.009362716667 x16 0.005585858333 x17 0.01105469167 x18 0.01514394167 x19 0.0008603833333 x20 0.005390983333 x21 0.004285958333 x22 0.002965683333 x23 0.00882215 x24 0.01205829167 x25 0.007865941667 x26 0.009536483333 x27 0.01536361667 x28 0.01206999167 x29 0.001890775 x30 0.001219958333 x31
Subject To
 c1: 1 x1 1 x2 1 x3 1 x4 1 x5 1 x6 1 x7 1 x8 1 x9 1 x10 1 x11 1 x12 1 x13 1 x14 1 x15 1 x16 1 x17 1 x18 1 x19 1 x20 1 x21 1 x22 1 x23 1 x24 1 x25 1 x26 1 x27 1 x28 1 x29 1 x30 1 x31 = 1
Bounds
 0 <= x1 <= 1
 0 <= x2 <= 1
 0 <= x3 <= 1
 0 <= x4 <= 1
 0 <= x5 <= 1
 0 <= x6 <= 1
 0 <= x7 <= 1
 0 <= x8 <= 1
 0 <= x9 <= 1
 0 <= x10 <= 1
 0 <= x11 <= 1
 0 <= x12 <= 1
 0 <= x13 <= 1
 0 <= x14 <= 1
 0 <= x15 <= 1
 0 <= x16 <= 1
 0 <= x17 <= 1
 0 <= x18 <= 1
 0 <= x19 <= 1
 0 <= x20 <= 1
 0 <= x21 <= 1
 0 <= x22 <= 1
 0 <= x23 <= 1
 0 <= x24 <= 1
 0 <= x25 <= 1
 0 <= x26 <= 1
 0 <= x27 <= 1
 0 <= x28 <= 1
 0 <= x29 <= 1
 0 <= x30 <= 1
 0 <= x31 <= 1
End`

highs.solve(pb)

> {
  Status: 'Optimal',
  Columns: {
    x1: {
      Index: 0,
      Status: 'LB',
      Lower: 0,
      Upper: 1,
      Primal: 0,
      Dual: -0.00807823,
      Name: 'x1'
...

32 variables => KO

var pb = `Maximize
 obj: 0.007285383333 x1 0.0088992 x2 0.01417630833 x3 0.009235308333 x4 0.01318765 x5 0.01406648333 x6 0.01360380833 x7 0.01448148333 x8 0.01661586667 x9 0.004312425 x10 0.008158858333 x11 0.006681016667 x12 0.0005192 x13 0.006546558333 x14 0.005799741667 x15 0.009362716667 x16 0.005585858333 x17 0.01105469167 x18 0.01514394167 x19 0.0008603833333 x20 0.005390983333 x21 0.004285958333 x22 0.002965683333 x23 0.00882215 x24 0.01205829167 x25 0.007865941667 x26 0.009536483333 x27 0.01536361667 x28 0.01206999167 x29 0.001890775 x30 0.001219958333 x31 0.005129775 x32
Subject To
 c1: 1 x1 1 x2 1 x3 1 x4 1 x5 1 x6 1 x7 1 x8 1 x9 1 x10 1 x11 1 x12 1 x13 1 x14 1 x15 1 x16 1 x17 1 x18 1 x19 1 x20 1 x21 1 x22 1 x23 1 x24 1 x25 1 x26 1 x27 1 x28 1 x29 1 x30 1 x31 1 x32 = 1
Bounds
 0 <= x1 <= 1
 0 <= x2 <= 1
 0 <= x3 <= 1
 0 <= x4 <= 1
 0 <= x5 <= 1
 0 <= x6 <= 1
 0 <= x7 <= 1
 0 <= x8 <= 1
 0 <= x9 <= 1
 0 <= x10 <= 1
 0 <= x11 <= 1
 0 <= x12 <= 1
 0 <= x13 <= 1
 0 <= x14 <= 1
 0 <= x15 <= 1
 0 <= x16 <= 1
 0 <= x17 <= 1
 0 <= x18 <= 1
 0 <= x19 <= 1
 0 <= x20 <= 1
 0 <= x21 <= 1
 0 <= x22 <= 1
 0 <= x23 <= 1
 0 <= x24 <= 1
 0 <= x25 <= 1
 0 <= x26 <= 1
 0 <= x27 <= 1
 0 <= x28 <= 1
 0 <= x29 <= 1
 0 <= x30 <= 1
 0 <= x31 <= 1
 0 <= x32 <= 1
End`

highs.solve(pb)

> /node_modules/highs/build/highs.js:12
if(x)y=v?require("path").dirname(y)+"/":__dirname+"/",ia=function(a,b){ka||(ka=require("fs"));la||(la=require("path"));a=la.normalize(a);return ka.readFileSync(a,b?null:"utf8")},B=function(a){a=ia(a,!0);a.buffer||(a=new Uint8Array(a));assert(a.buffer);return a},1<process.argv.length&&(da=process.argv[1].replace(/\\/g,"/")),process.argv.slice(2),process.on("uncaughtException",function(a){if(!(a instanceof ma))throw a;}),process.on("unhandledRejection",C),ea=function(a){process.exit(a)},e.inspect=function(){return"[Emscripten Module object]"};
                                                                                                                                                                                                                                                                                                                                                                                                                            ^
RuntimeError: abort(5406152). Build with -s ASSERTIONS=1 for more info.
    at process.C (/node_modules/highs/build/highs.js:20:54)
    at process.emit (events.js:314:20)
    at process.EventEmitter.emit (domain.js:483:12)
    at processPromiseRejections (internal/process/promises.js:209:33)
    at processTicksAndRejections (internal/process/task_queues.js:98:32)

Seems an exception is being issued by either the underlying Node.js glue to the C program, but with no more information.

Cheers,

Roman

[lequant40]

Hello !

I comment on myself here in case someone else has the same issue: the LP problem must be provided in CPLEX LP format, which comes with some nasty restrictions on the length of the lines => http://web.mit.edu/lpsolve/doc/CPLEX-format.htm#:~:text=The%20CPLEX%20LP%20file%20format,format%20that%20lpsolve%20can%20read.

So, this is working instead (notice the new line in the objective function definition):

pb = `Maximize
 obj: 0.007285383333 x1 0.0088992 x2 0.01417630833 x3 0.009235308333 x4 0.01318765 x5 0.01406648333 x6 0.01360380833 x7 0.01448148333 x8 0.01661586667 x9 0.004312425 x10 0.008158858333 x11 0.006681016667 x12 0.0005192 x13 0.006546558333 x14 0.005799741667 x15 0.009362716667 x16 0.005585858333 x17 0.01105469167 x18 0.01514394167 x19 0.0008603833333 x20 0.005390983333 x21 0.004285958333 x22 0.002965683333 x23 0.00882215 x24 0.01205829167 x25 0.007865941667 x26 0.009536483333 x27 0.01536361667 x28 0.01206999167 x29 0.001890775 x30 0.001219958333 x31
       0.005129775 x32
...
End`

Still, I think this issue is still valid in order to improve the output to the user in case of error.

Here, I suspected a problem of length from the behaviour, but for others potential issues (memory errors, model file writting or reading to/from disk impossible, ...), you might want to provide the full output from the C program to Node.js.

Cheers,

Roman

[jajhall]

@lovasoa Is the LP file is being read by highs-js, or passed to HiGHS itself? The HiGHS LP file reader only handles the default maximum line length - as exposed by another user a few months ago. Whatever, I'm prompted to see what error return is given by HiGHS if the LP file line length is exceeded.

[lovasoa]

The LP file is parsed by highs, not by this library. I'm closing this since it's a bug in HiGHS.

[lovasoa]

The error reporting issue is also a bug in HiGHS. HiGHS aborts instead of returning a clean error when calling Highs_readModel.

[jajhall]

What's the githash of the version of HiGHS that you're using. I'm looking at what happens in highs/master, but it doesn't abort. This suggests that the fix that was put in a few months ago isn't in the version you're using.

More generally, we're working towards a first formal release of HiGHS, so we'll have proper version numbers to refer to!

[lovasoa]

HiGHS is a git submodule of this repo. You can see the exact version used, and open a pr to update it.

[lovasoa]

@jajhall : Since you are a maintainer of HiGHS, can I add you as a maintainer on this project too ?

[lovasoa]

Oh, I see you already are ^^

[lovasoa]

I'm looking at what happens in highs/master, but it doesn't abort

You you share your test ?

[jajhall]

I think that this is why I got the initial notification!

I've looked through the behaviour of Highs/master, and Highs_readModel should return a value 2 (Error). Are you interpreting this?

Just one thing makes me cautious. I'm testing on Linux, and the LP file reader handles the excessive line length by throwing an exception that is then caught, allowing HiGHS to return the error. This seems to be a standard coding approach, so I'd expect it to be OK on Windows/MacOS

[jajhall]

I'm looking at what happens in highs/master, but it doesn't abort

You you share your test ?

I've created a .lp file from the text above and am just running the HiGHS executable.
The issue about LP file line length was raised in mid-February, so the fix that was put in would have been in master around then.

[jajhall]

HiGHS is a git submodule of this repo. You can see the exact version used, and open a pr to update it.

Thanks. New ground for me on GitHub, but I see that you appear to be using HiGHS @ 91e72a0. Specifically
https://github.com/ERGO-Code/HiGHS/tree/91e72a00ef215fbee49af10efe213087490efa1e
I've checked out this version of HiGHS and it runs the same way as the current master:
I've also tested it via the C interface, and Highs_readModel does return a value 2

A useful exercise in several ways!

[lovasoa]

I've created a .lp file from the text above and am just running the HiGHS executable.

highs-js uses the C interface, not the HiGHS binary directly.

The issue about LP file line length was raised in mid-February, so the fix that was put in would have been in master around then.

This repo uses https://github.com/ERGO-Code/HiGHS/tree/91e72a00ef215fbee49af10efe213087490efa1e from March 18.
There doesn't seem to be any .lp related commit since then.

[jajhall]

So, given my comment made just before your last one, there's little more that HiGHS can do in this case - other than setting the model_status to correspond to a load error - rather than leaving it "unset".

The line limit for .lp files is very unfortunate, particularly as I think I'm right in saying that CPLEX itself doesn't apply the standard, and reads files with any line length!

Of course, for toy LP problems, it's not an issue - and then folk do what your used has done and solvers stop working!

[lovasoa]

Is there a reason to keep this limit in highs ?

[lovasoa]

there's little more that HiGHS can do in this case - other than setting the model_status to correspond to a load error - rather than leaving it "unset"

HiGHS could return a meaningful error code and add a function to the C api associating error code to error messages.

[jajhall]

The limit is part of the file format standard, so we'll keep it for now. We've not had much interaction with .lp file users so it hasn't been much if a priority

As for error return codes, a more sophisticated system would be in order - I don't know what Gurobi offers - but, again, it's not a priority.

There is, of course, an error code returned that you don't seem to have interpreted 🙂

[lovasoa]

The current code does raise a proper js error when readModel does not return 0. The problem is on HiGHS side, it doesn't return a proper error and aborts instead.

[lovasoa]

I improved error handling further in #4, but it will still require Highs_readModel to return a proper error instead of crashing.

[jajhall]

The current code does raise a proper js error when readModel does not return 0. The problem is on HiGHS side, it doesn't return a proper error and aborts instead.

Ah, I see now! HiGHS returns an error with my Linux and Windows builds, but crashes on your architecture. I realise that it's embedded within js, but how (and on what architecture) is HiGHS compiled?

I can create a branch to introduce print statements so that you can help me to identify where HiGHS crashes. Is this feasible?

That said, my instinct is that it's something to do with HiGHS throwing an exception that is then caught. Eliminating this standard method of error-handling is non-trivial.

[lovasoa]

Yes, wasm doesn't support exceptions. Emscripten has an option to add a js wrapper around each exception, but with a high performance impact:
https://emscripten.org/docs/porting/exceptions.html

wasm isn't the only target that doesn't support exceptions, especially in the embedded world.

[lequant40]

To prevent any exceptions raised in HiGHS from completely crashing the browser/node.js process, would there be any possibility to catch the wasm errors (RuntimeError: abort(5406152)) inside the .solve() function ?

[jajhall]

Thanks. I see that you've been doing things to highs-js to work around this.
I've searched the source code of HiGHS and, other than in the LP file reader, "throw" is only used in the interior point method (IPM) solver . So, as long as you don't request that HiGHS uses the IMP solver, there's no other source of this kind of exception when solving LP problems.
I'll share our experience with the other HiGHS developers.

[lovasoa]

@lequant40 Yes, I added that and published a new version.

[lovasoa]

@jajhall Thanks !

[jajhall]

You're welcome @lovasoa . This has exposed the need for us to document the (limited) use of exceptions in HiGHS, and to ensure that all internal calls to the IPM solver are protected by "try {}" and exceptions are handled properly.

This is how we all win in the open source world!