-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
benchmark improvement #60
Conversation
Codecov Report
@@ Coverage Diff @@
## master #60 +/- ##
==========================================
- Coverage 87.37% 87.34% -0.03%
==========================================
Files 28 28
Lines 3018 3019 +1
==========================================
Hits 2637 2637
- Misses 381 382 +1
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Apart of a few minor points, it overall looks good to me! Looking forward to test this new version of the benchmark script :)
@@ -1,10 +1,9 @@ | |||
To run benchmarks, run the following script. | |||
Running benchmark requires pacakge ArgParse. First install ArgParse with: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice that we depends on ArgParse now 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think there is a typo: pacakge
-> package
benchmark/benchmark-cutest.jl
Outdated
@@ -67,13 +78,17 @@ exclude = [ | |||
] | |||
|
|||
|
|||
probs = CUTEst.select() | |||
if QUICK | |||
probs = ["PRIMALC1", "DIXMAANI", "HIER13", "LUKVLI7", "GAUSS2", "LUKSAN13LS", "CHARDIS1", "A5NSDSIL", "QPCBOEI1", "POLAK4", "DUAL2", "EXPFITA", "VAREIGVL", "MPC2", "BLOWEYA", "DECONVB", "MSS1", "POWELLBC", "ACOPP57", "WALL50", "FBRAIN2", "ACOPP300", "AUG2D", "HS106", "GMNCASE2", "LUKVLE8", "READING2", "MAXLIKA", "CHEBYQAD", "HYDROELM", "GULFNE", "CLEUVEN4", "HAIFAL", "JUDGENE", "DITTERT", "TRIGON1NE", "OBSTCLAE", "READING6", "SBRYBND", "ARGLINC", "CVXQP2", "TABLE8", "NINENEW", "STEENBRA", "BA-L1SP", "EXPFITNE", "LUKSAN17", "DUALC5", "STCQP1", "DEGENQP", "DEGDIAG", "LEUVEN7", "DALLASM", "READING8", "HS101", "GENROSEBNE", "EIGENALS", "READING7", "OET3", "CHANDHEQ", "YATP1LS", "OSCIPATH", "SEMICN2U", "MODBEALENE", "JANNSON4", "DTOC1NC", "KSS", "TABLE1", "DRCAV2LQ", "MGH17SLS", "BRAINPC2", "PROBPENL", "MGH17S", "DIAGPQB", "DEMBO7", "HS119", "PORTSNQP", "YATP1CNE", "THURBER", "VESUVIOU", "TAX213322", "NGONE", "MRIBASIS", "EXPFITC", "FBRAINNE", "LINVERSENE", "HYDCAR6LS", "GMNCASE4", "ZAMB2-11", "ALJAZZAF", "HIMMELBK", "WOODSNE", "LUKVLI10", "CHANNEL", "ORBIT2", "EIGENA2", "ACOPP118", "CHNRSNBM", "LHAIFAM", "NASH", "CYCLIC3LS", "CYCLIC3", "BA-L49LS", "HS99", "CATENA", "CHWIRUT1", "OPTPRLOC", "HYDROELL", "BIGBANK", "OSORIO", "SPINLS", "MNISTS0", "ANTWERP", "PORTFL4", "PDE1", "CURLY20", "DEGENLPA", "LUBRIFC", "MANCINONE", "DALE", "HATFLDC", "INTEGREQ", "NET1", "LUKSAN12", "UBH5", "AGG", "WATSONNE", "TAX13322", "PRIMAL1", "10FOLDTR", "QPCBLEND", "CYCLOOCT", "FIVE20B", "HUESTIS", "TWIRISM1", "DMN37142", "LIARWHDNE", "COOLHANS", "MSS3", "BDRY2", "TRO5X5", "MSS2", "TAX53322", "CORE1", "LINSPANH", "ZAMB2-9", "KSIP", "CHAINWOONE", "DEGTRIDL", "LINCONT", "TWIRIBG1", "POWER", "DMN37143", "PRIMAL3", "EIGENBCO", "TRIMLOSS", "SPANHYD", "OPTCNTRL", "ROSEPETAL", "SANTALS", "PRIMALC8", "SPECANNE", "READING5", "EXPQUAD", "ARGLCLE", "CHNRSBNE", "MODBEALE", "EIGENC", "ARGTRIG", "STATIC3", "CRESC132", "CHANDHEU", "KISSING2", "EXPLIN", "GILBERT", "GPP", "LUKVLI9", "RES", "LCH", "MUONSINE", "BA-L73", "TRO21X5", "SSEBNLN", "ELATTAR", "TWOD", "PRIMAL4", "COATING", "WALL100", "MSQRTA", "PRIMAL2", "ODNAMUR", "GENROSENE", "TRIGON2NE", "CHARDIS0", "SPMSQRT", "QING", "SMMPSF", "NUFFIELD", "GOFFIN", "ELEC", "BA-L16LS", "SYNPOP24", "ZIGZAG", "SSEBLIN", "BA-L1", "FCCU", "CONT5-QP", "QPNBAND", "AIRPORT", "FEEDLOC", "KISSING", "FERRISDC", "MAKELA4", "VANDANIUMS", "AVION2", "BROWNALE", "ROSEPETAL2", "DEGENQPC", "DRUGDISE", "QINGNE", "BA-L52LS", "JANNSON3", "NONMSQRTNE", "DECONVC", "BA-L52", "WALL10", "MODEL", "OPTMASS", "ORTHREGF"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
could we define this list in a separate text file (like cutest_instances_quick.txt
)? I think that will be easier to edit.
|
||
writedlm("name-cutest.csv",probs,',') | ||
writedlm("time-cutest-$(SOLVER).csv",time,',') | ||
writedlm("status-cutest-$(SOLVER).csv",status),',' | ||
writedlm("mem-cutest-$(SOLVER).csv",mem,',') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we return the number of iterations as well? Might be interesting to compare the number of iterations with Ipopt
DCPPowerModel, DCMPPowerModel, NFAPowerModel, | ||
DCPLLPowerModel,LPACCPowerModel, SOCWRPowerModel, | ||
QCRMPowerModel,QCLSPowerModel] | ||
if QUICK |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very nice that we restrict to ACP and ACR formulation in this case!
for solver in SOLVERS | ||
launch_script = joinpath(PROJECT_PATH, "benchmark-$class.jl") | ||
run(`julia --project=$PROJECT_PATH $launch_script $NP $solver $VERBOSE`) | ||
try |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe this is a bit hacky. Why not specified ArgParse directly in the Project.toml file?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I left it like this because Project.toml for this directory is reserved for the later benchmarking script. We can set up a separate project directory, but I thought that would be introducing too much complication. Is there any reason I shouldn't do this?
function parse_commandline() | ||
s = ArgParseSettings() | ||
|
||
@add_arg_table s begin |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice!
@frapac Thanks for the fast review. Do you even sleep? 😂 |
Ahah, you are welcome :) |
ca4f9d0
to
2c71b53
Compare
2c71b53
to
8009b96
Compare
* Add AbstractKKTSystem structure * implement SparseReducedKKTSystem and SparseAugmentedKKTSystem * refactor Solver * Avoid unecessary allocations by forcing specialization * Deactive logs in benchmark scripts (#59) * deactive logs in benchmark scripts * benchmark: add verbose option to main script * barrier iterations (#61) * benchmark improvement (#60) * allocation issue fixed * added option buffered for NLPModels.jl * added option buffered for NLPModels.jl * added option buffered for NLPModels.jl * ma27 fix Co-authored-by: Sungho Shin <sshin@anl.gov>
With this change, now we have