Please provide more examples for the C++ NL reader

[baharev]

Please provide more examples for the C++ nl reader. I have already found:

http://zverovich.net/2014/09/19/reading-nl-files.html

https://ampl.github.io/nl.html

The example on the blog seems weird. The function body of ExprCounter.OnBinary() returns nothing. The other example, the VarCounter.OnVariableRef(int) returns Reference(). How am I supposed to know what to return?

Please provide at least the following self-contained examples.

1. A standalone executable that prints the number of constraints, number of variables, number of nonzeros to the standard output.
   Note that it is trivial to get these pieces of information from the nl file header. The nl file reader API should give us a similarly trivial way to get these data.
2. A standalone executable the prints the structural sparsity pattern of the g, J and H. It is different from the gjh "solver" in that we only need the structural sparsity pattern, hence we avoid the need for evaluating the g, J and H where the evaluation can fail (we may not easily find a suitable point for that).
   Note that it is fairly easy to get the structural sparsity pattern of g and J from the corresponding segment of the nl file, without having to evaluate them.
   Also note that if we are willing to, and we are able to evaluate g, J and H, it is trivial to parse the textual output of the gjh solver. See also gjh fails with a strange error message #73
3. A standalone executable that pretty prints the expression graph of a selected constraint / objective in SSA form. See for example proposed standard for nonlinear expression graphs JuliaOpt/MathProgBase.jl#42 (comment) to understand what I mean.
   Note that it is moderately easy to write a toy stack machine to do this, and in fact, I implemented one back in 2009 exactly for this purpose. (I gave up reverse engineering the nl file format when I ran into undocumented cases with V blocks referring to other V blocks, and with the linear and nonlinear parts split.)

The complexity of the nl reader C++ API is terrifying. As I understand, Miles and Jack already gave up, see JuliaSmoothOptimizers/AmplNLReader.jl#11 and as it currently stands, I would rather roll my own „ad hoc, informally-specified, bug-ridden, slow implementation of half of“ the nl reader by parsing the nl file as text and based on the Writing .nl files report.

I apologize if that was too harsh but I guess you do not realize the difficulty those people face who are not so familiar with the ASL and the nl file format as you are as an employee of the company.

Any help is greatly appreciated.

CC @mlubin @JackDunnNZ

[vitaut]

Thanks for the feedback.

The function body of ExprCounter.OnBinary() returns nothing.

Good catch, fixed. ExprCounter.OnBinary() should return an expression object which, in this case, is simply an enum. In general you return an expression of the type you passed as a template argument to NLHandler.

I've added examples you suggested (in a single executable to avoid proliferation of build targets): https://github.com/ampl/mp/blob/master/src/nl-example.cc. It consists of the following NL handlers:

• ExprCounter: example from the blog post.

• DimensionPrinter: prints problem dimensions: the number of constraints, variables and nonzeros (in constraints). For example:

  > ./nl-example dimen lib1_etamac.nl
  Variables:   97
  Constraints: 70
  Nonzeros:    216
  

• SparsityPrinter: prints sparsity pattern of the objective and constraint gradients:

  viz@dione ~/w/m/b/bin> ./nl-example sparsity lib1_etamac.nl
  Constraint 0:
  9
  79
  Constraint 1:
  10
  80
  ...
  Objective 0:
  70
  71
  72
  73
  74
  75
  76
  77
  78
  

• ExprPrinter: print objective or constraint expression in SSA-like form:

  > ./nl-example obj-expr lib1_etamac.nl 0
  e0 = -0.815373
  e1 = log(x70)
  e2 = e0 * e1
  e3 = -0.664833
  e4 = log(x71)
  e5 = e3 * e4
  e6 = -0.542086
  e7 = log(x72)
  e8 = e6 * e7
  e9 = -0.442002
  e10 = log(x73)
  e11 = e9 * e10
  e12 = -0.360397
  e13 = log(x74)
  e14 = e12 * e13
  e15 = -0.293858
  e16 = log(x75)
  e17 = e15 * e16
  e18 = -0.239603
  e19 = log(x76)
  e20 = e18 * e19
  e21 = -0.195366
  e22 = log(x77)
  e23 = e21 * e22
  e24 = -3.98241
  e25 = log(x78)
  e26 = e24 * e25
  e27 = sum(e2, e5, e8, e11, e14, e17, e20, e23, e26)
  o0 = e27
  > ./nl-example con-expr lib1_etamac.nl 10
  e0 = 0.632762
  e1 = -0.342222
  e2 = x11 ^ e1
  e3 = e0 * e2
  e4 = 0.306708
  e5 = -0.427778
  e6 = x46 ^ e5
  e7 = e4 * e6
  e8 = -0.794444
  e9 = x64 ^ e8
  e10 = e7 * e9
  e11 = e3 + e10
  e12 = -0.818182
  e13 = e11 ^ e12
  e14 = -e13
  e15 = e14 + x28 * 1
  c10 = e15
  

  (Note that ExprPrinter doesn't handle all types of expressions, but it should be straightforward to extend it.)

The complexity of the nl reader C++ API is terrifying. As I understand, Miles and Jack already gave up, see JuliaSmoothOptimizers/AmplNLReader.jl#11 and as it currently stands, I would rather roll my own „ad hoc, informally-specified, bug-ridden, slow implementation of half of“ the nl reader by parsing the nl file as text and based on the Writing .nl files report.

I thought the problem there was not the complexity of the API but the fact that Julia code cannot consume C++ interfaces. The API is pretty standard if you are familiar with modern parser interfaces like SAX or Clang parser API. The main difference is that the C++ NL reader uses static instead of dynamic polymorphism to keep performance on the level (in fact, better) of the ASL reader implemented in C.

Hopefully the new examples will make the new API more approachable. If you have any problems with using it or ideas on how to improve the API or the documentation, please let me know. I'll be happy to help.

[baharev]

Great! Unfortunately, it is late night here, one hour past my bedtime. I will play with this example code tomorrow. Many thanks for the quick response!

[baharev]

I have taken a quick look at this new example code; I do not have questions at the moment. Unfortunately, I have deadlines to meet and I cannot spend more time on this but I will definitely will in the not too distant future. Thanks again for your help!