benchmark improvement #60
Conversation
Codecov Report
@@ Coverage Diff @@
## master #60 +/- ##
==========================================
- Coverage 87.37% 87.34% -0.03%
==========================================
Files 28 28
Lines 3018 3019 +1
==========================================
Hits 2637 2637
- Misses 381 382 +1
Continue to review full report at Codecov.
|
| @@ -1,10 +1,9 @@ | |||
| To run benchmarks, run the following script. | |||
| Running benchmark requires pacakge ArgParse. First install ArgParse with: | |||
There was a problem hiding this comment.
Nice that we depends on ArgParse now 👍
There was a problem hiding this comment.
I think there is a typo: pacakge -> package
benchmark/benchmark-cutest.jl
Outdated
| @@ -67,13 +78,17 @@ exclude = [ | |||
| ] | |||
|
|
|||
|
|
|||
| probs = CUTEst.select() | |||
| if QUICK | |||
| probs = ["PRIMALC1", "DIXMAANI", "HIER13", "LUKVLI7", "GAUSS2", "LUKSAN13LS", "CHARDIS1", "A5NSDSIL", "QPCBOEI1", "POLAK4", "DUAL2", "EXPFITA", "VAREIGVL", "MPC2", "BLOWEYA", "DECONVB", "MSS1", "POWELLBC", "ACOPP57", "WALL50", "FBRAIN2", "ACOPP300", "AUG2D", "HS106", "GMNCASE2", "LUKVLE8", "READING2", "MAXLIKA", "CHEBYQAD", "HYDROELM", "GULFNE", "CLEUVEN4", "HAIFAL", "JUDGENE", "DITTERT", "TRIGON1NE", "OBSTCLAE", "READING6", "SBRYBND", "ARGLINC", "CVXQP2", "TABLE8", "NINENEW", "STEENBRA", "BA-L1SP", "EXPFITNE", "LUKSAN17", "DUALC5", "STCQP1", "DEGENQP", "DEGDIAG", "LEUVEN7", "DALLASM", "READING8", "HS101", "GENROSEBNE", "EIGENALS", "READING7", "OET3", "CHANDHEQ", "YATP1LS", "OSCIPATH", "SEMICN2U", "MODBEALENE", "JANNSON4", "DTOC1NC", "KSS", "TABLE1", "DRCAV2LQ", "MGH17SLS", "BRAINPC2", "PROBPENL", "MGH17S", "DIAGPQB", "DEMBO7", "HS119", "PORTSNQP", "YATP1CNE", "THURBER", "VESUVIOU", "TAX213322", "NGONE", "MRIBASIS", "EXPFITC", "FBRAINNE", "LINVERSENE", "HYDCAR6LS", "GMNCASE4", "ZAMB2-11", "ALJAZZAF", "HIMMELBK", "WOODSNE", "LUKVLI10", "CHANNEL", "ORBIT2", "EIGENA2", "ACOPP118", "CHNRSNBM", "LHAIFAM", "NASH", "CYCLIC3LS", "CYCLIC3", "BA-L49LS", "HS99", "CATENA", "CHWIRUT1", "OPTPRLOC", "HYDROELL", "BIGBANK", "OSORIO", "SPINLS", "MNISTS0", "ANTWERP", "PORTFL4", "PDE1", "CURLY20", "DEGENLPA", "LUBRIFC", "MANCINONE", "DALE", "HATFLDC", "INTEGREQ", "NET1", "LUKSAN12", "UBH5", "AGG", "WATSONNE", "TAX13322", "PRIMAL1", "10FOLDTR", "QPCBLEND", "CYCLOOCT", "FIVE20B", "HUESTIS", "TWIRISM1", "DMN37142", "LIARWHDNE", "COOLHANS", "MSS3", "BDRY2", "TRO5X5", "MSS2", "TAX53322", "CORE1", "LINSPANH", "ZAMB2-9", "KSIP", "CHAINWOONE", "DEGTRIDL", "LINCONT", "TWIRIBG1", "POWER", "DMN37143", "PRIMAL3", "EIGENBCO", "TRIMLOSS", "SPANHYD", "OPTCNTRL", "ROSEPETAL", "SANTALS", "PRIMALC8", "SPECANNE", "READING5", "EXPQUAD", "ARGLCLE", "CHNRSBNE", "MODBEALE", "EIGENC", "ARGTRIG", "STATIC3", "CRESC132", "CHANDHEU", "KISSING2", "EXPLIN", "GILBERT", "GPP", "LUKVLI9", "RES", "LCH", "MUONSINE", "BA-L73", "TRO21X5", "SSEBNLN", "ELATTAR", "TWOD", "PRIMAL4", "COATING", "WALL100", "MSQRTA", "PRIMAL2", "ODNAMUR", "GENROSENE", "TRIGON2NE", "CHARDIS0", "SPMSQRT", "QING", "SMMPSF", "NUFFIELD", "GOFFIN", "ELEC", "BA-L16LS", "SYNPOP24", "ZIGZAG", "SSEBLIN", "BA-L1", "FCCU", "CONT5-QP", "QPNBAND", "AIRPORT", "FEEDLOC", "KISSING", "FERRISDC", "MAKELA4", "VANDANIUMS", "AVION2", "BROWNALE", "ROSEPETAL2", "DEGENQPC", "DRUGDISE", "QINGNE", "BA-L52LS", "JANNSON3", "NONMSQRTNE", "DECONVC", "BA-L52", "WALL10", "MODEL", "OPTMASS", "ORTHREGF"] | |||
There was a problem hiding this comment.
could we define this list in a separate text file (like cutest_instances_quick.txt)? I think that will be easier to edit.
|
|
||
| writedlm("name-cutest.csv",probs,',') | ||
| writedlm("time-cutest-$(SOLVER).csv",time,',') | ||
| writedlm("status-cutest-$(SOLVER).csv",status),',' | ||
| writedlm("mem-cutest-$(SOLVER).csv",mem,',') |
There was a problem hiding this comment.
Can we return the number of iterations as well? Might be interesting to compare the number of iterations with Ipopt
| DCPPowerModel, DCMPPowerModel, NFAPowerModel, | ||
| DCPLLPowerModel,LPACCPowerModel, SOCWRPowerModel, | ||
| QCRMPowerModel,QCLSPowerModel] | ||
| if QUICK |
There was a problem hiding this comment.
Very nice that we restrict to ACP and ACR formulation in this case!
| for solver in SOLVERS | ||
| launch_script = joinpath(PROJECT_PATH, "benchmark-$class.jl") | ||
| run(`julia --project=$PROJECT_PATH $launch_script $NP $solver $VERBOSE`) | ||
| try |
There was a problem hiding this comment.
Maybe this is a bit hacky. Why not specified ArgParse directly in the Project.toml file?
There was a problem hiding this comment.
I left it like this because Project.toml for this directory is reserved for the later benchmarking script. We can set up a separate project directory, but I thought that would be introducing too much complication. Is there any reason I shouldn't do this?
| function parse_commandline() | ||
| s = ArgParseSettings() | ||
|
|
||
| @add_arg_table s begin |
|
@frapac Thanks for the fast review. Do you even sleep? 😂 |
|
Ahah, you are welcome :) |
sshin23
force-pushed
the
ss/benchmark-improve
branch
from
ca4f9d0
to
2c71b53
Compare
sshin23
force-pushed
the
ss/benchmark-improve
branch
from
2c71b53
to
8009b96
Compare
* Add AbstractKKTSystem structure * implement SparseReducedKKTSystem and SparseAugmentedKKTSystem * refactor Solver * Avoid unecessary allocations by forcing specialization * Deactive logs in benchmark scripts (#59) * deactive logs in benchmark scripts * benchmark: add verbose option to main script * barrier iterations (#61) * benchmark improvement (#60) * allocation issue fixed * added option buffered for NLPModels.jl * added option buffered for NLPModels.jl * added option buffered for NLPModels.jl * ma27 fix Co-authored-by: Sungho Shin <sshin@anl.gov>
With this change, now we have