/
solver_benchmark.html
623 lines (591 loc) · 79.7 KB
/
solver_benchmark.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
<!DOCTYPE html>
<!-- Generated by pkgdown: do not edit by hand --><html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Solver Benchmarks • prioritizr</title>
<!-- favicons --><link rel="icon" type="image/png" sizes="16x16" href="../favicon-16x16.png">
<link rel="icon" type="image/png" sizes="32x32" href="../favicon-32x32.png">
<link rel="apple-touch-icon" type="image/png" sizes="180x180" href="../apple-touch-icon.png">
<link rel="apple-touch-icon" type="image/png" sizes="120x120" href="../apple-touch-icon-120x120.png">
<link rel="apple-touch-icon" type="image/png" sizes="76x76" href="../apple-touch-icon-76x76.png">
<link rel="apple-touch-icon" type="image/png" sizes="60x60" href="../apple-touch-icon-60x60.png">
<!-- jquery --><script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.4.1/jquery.min.js" integrity="sha256-CSXorXvZcTkaix6Yvo6HppcZGetbYMGWSFlBw8HfCJo=" crossorigin="anonymous"></script><!-- Bootstrap --><link href="https://cdnjs.cloudflare.com/ajax/libs/bootswatch/3.4.0/flatly/bootstrap.min.css" rel="stylesheet" crossorigin="anonymous">
<script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.4.1/js/bootstrap.min.js" integrity="sha256-nuL8/2cJ5NDSSwnKD8VqreErSWHtnEP9E7AySL+1ev4=" crossorigin="anonymous"></script><!-- bootstrap-toc --><link rel="stylesheet" href="../bootstrap-toc.css">
<script src="../bootstrap-toc.js"></script><!-- Font Awesome icons --><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.1/css/all.min.css" integrity="sha256-mmgLkCYLUQbXn0B1SRqzHar6dCnv9oZFPEC1g1cwlkk=" crossorigin="anonymous">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.12.1/css/v4-shims.min.css" integrity="sha256-wZjR52fzng1pJHwx4aV2AO3yyTOXrcDW7jBpJtTwVxw=" crossorigin="anonymous">
<!-- clipboard.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/clipboard.js/2.0.6/clipboard.min.js" integrity="sha256-inc5kl9MA1hkeYUt+EC3BhlIgyp/2jDIyBLS6k3UxPI=" crossorigin="anonymous"></script><!-- headroom.js --><script src="https://cdnjs.cloudflare.com/ajax/libs/headroom/0.11.0/headroom.min.js" integrity="sha256-AsUX4SJE1+yuDu5+mAVzJbuYNPHj/WroHuZ8Ir/CkE0=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/headroom/0.11.0/jQuery.headroom.min.js" integrity="sha256-ZX/yNShbjqsohH1k95liqY9Gd8uOiE1S4vZc+9KQ1K4=" crossorigin="anonymous"></script><!-- pkgdown --><link href="../pkgdown.css" rel="stylesheet">
<script src="../pkgdown.js"></script><!-- docsearch --><script src="../docsearch.js"></script><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/docsearch.js/2.6.3/docsearch.min.css" integrity="sha256-QOSRU/ra9ActyXkIBbiIB144aDBdtvXBcNc3OTNuX/Q=" crossorigin="anonymous">
<link href="../docsearch.css" rel="stylesheet">
<script src="https://cdnjs.cloudflare.com/ajax/libs/mark.js/8.11.1/jquery.mark.min.js" integrity="sha256-4HLtjeVgH0eIB3aZ9mLYF6E8oU5chNdjU6p6rrXpl9U=" crossorigin="anonymous"></script><link href="../extra.css" rel="stylesheet">
<meta property="og:title" content="Solver Benchmarks">
<meta property="og:description" content="prioritizr">
<meta property="og:image" content="https://prioritizr.net/logo.png">
<!-- mathjax --><script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js" integrity="sha256-nvJJv9wWKEm88qvoQl9ekL2J+k/RWIsaSScxxlsrv8k=" crossorigin="anonymous"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/config/TeX-AMS-MML_HTMLorMML.js" integrity="sha256-84DKXVJXs0/F8OTMzX4UR909+jtl4G7SPypPavF+GfA=" crossorigin="anonymous"></script><!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.3/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]--><!-- Global site tag (gtag.js) - Google Analytics --><script async src="https://www.googletagmanager.com/gtag/js?id=UA-70466840-4"></script><script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'UA-70466840-4');
</script>
</head>
<body data-spy="scroll" data-target="#toc">
<div class="container template-article">
<header><div class="navbar navbar-default navbar-fixed-top" role="navigation">
<div class="container">
<div class="navbar-header">
<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<span class="navbar-brand">
<a class="navbar-link" href="../index.html">prioritizr</a>
<span class="version label label-default" data-toggle="tooltip" data-placement="bottom" title="Released version">7.1.1.3</span>
</span>
</div>
<div id="navbar" class="navbar-collapse collapse">
<ul class="nav navbar-nav">
<li>
<a href="../articles/prioritizr.html">Get started</a>
</li>
<li>
<a href="../reference/index.html">Reference</a>
</li>
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false">
Articles
<span class="caret"></span>
</a>
<ul class="dropdown-menu" role="menu">
<li>
<a href="../articles/gurobi_installation.html">Gurobi Installation Guide</a>
</li>
<li>
<a href="../articles/publication_record.html">Publication Record</a>
</li>
<li>
<a href="../articles/saltspring.html">Salt Spring Island Tutorial</a>
</li>
<li>
<a href="../articles/solver_benchmark.html">Solver Benchmarks</a>
</li>
<li>
<a href="../articles/tasmania.html">Tasmania Tutorial</a>
</li>
<li>
<a href="../articles/zones.html">Management Zones</a>
</li>
</ul>
</li>
<li>
<a href="../news/index.html">Changelog</a>
</li>
</ul>
<ul class="nav navbar-nav navbar-right">
<li>
<a href="https://github.com/prioritizr/prioritizr/" class="external-link">
<span class="fab fa-github fa-lg"></span>
</a>
</li>
</ul>
<form class="navbar-form navbar-right hidden-xs hidden-sm" role="search">
<div class="form-group">
<input type="search" class="form-control" name="search-input" id="search-input" placeholder="Search..." aria-label="Search for..." autocomplete="off">
</div>
</form>
</div>
<!--/.nav-collapse -->
</div>
<!--/.container -->
</div>
<!--/.navbar -->
</header><script src="solver_benchmark_files/header-attrs-2.11/header-attrs.js"></script><div class="row">
<div class="col-md-9 contents">
<div class="page-header toc-ignore">
<h1 data-toc-skip>Solver Benchmarks</h1>
<small class="dont-index">Source: <a href="https://github.com/prioritizr/prioritizr/blob/master/vignettes/solver_benchmark.Rmd" class="external-link"><code>vignettes/solver_benchmark.Rmd</code></a></small>
<div class="hidden name"><code>solver_benchmark.Rmd</code></div>
</div>
<div id="introduction" class="section level1">
<h1 class="hasAnchor">
<a href="#introduction" class="anchor" aria-hidden="true"></a>Introduction</h1>
<p>The <em>prioritizr R</em> package supports a variety of optimization solvers for generating prioritizations. Specifically, the following functions can be used: <a href="https://prioritizr.net/reference/add_gurobi_solver.html"><code>add_gurobi_solver()</code></a> (interfaces with the <a href="https://www.gurobi.com/" class="external-link"><em>Gurobi</em> software</a>), <a href="https://prioritizr.net/reference/add_cplex_solver.html"><code>add_cplex_solver()</code></a> (interfaces with the <a href="https://www.ibm.com/analytics/cplex-optimizer" class="external-link"><em>IBM CPLEX</em> software</a>), <a href="https://prioritizr.net/reference/add_cbc_solver.html"><code>add_cbc_solver()</code></a> (interfaces with the <a href="https://projects.coin-or.org/Cbc" class="external-link"><em>CBC</em> software</a> using the <a href="https://dirkschumacher.github.io/rcbc/" class="external-link"><em>rcbc R</em> package</a>), <a href="https://prioritizr.net/reference/add_rsymphony_solver.html"><code>add_rsymphony_solver()</code></a> (interfaces with the <a href="https://projects.coin-or.org/SYMPHONY" class="external-link"><em>SYMPHONY</em> software</a> using the <a href="https://CRAN.R-project.org/package=Rsymphony" class="external-link"><em>Rsymphony R</em> package</a>), and the <a href="https://prioritizr.net/reference/add_lsymphony_solver.html"><code>add_lpsymphony_solver()</code></a> function (interfaces with the <a href="https://projects.coin-or.org/SYMPHONY" class="external-link"><em>SYMPHONY</em> software</a> using the <a href="https://www.bioconductor.org/packages/release/bioc/html/lpsymphony.html" class="external-link"><em>lpsymphony R</em> package</a>). Broadly speaking, <em>IBM CPLEX</em> and <em>Gurobi</em> tend to be the fastest among the supported solvers. Although they are both commercial software, special academic licenses are available at no cost.</p>
<p>In this vignette, we will explore the performance of different solvers. Using a benchmark analysis, we will see how well they can tackle problems of varying size (e.g. number of planning units) and complexity (e.g. adding boundary length penalties to reduce spatial fragmentation). Since users working in governmental and non-governmental organizations may need to purchase licenses for <em>IBM CPLEX</em> or <em>Gurobi</em> to use them, this vignette also aims to provide insight into whether the potential benefits of purchasing such licenses is worthwhile. Indeed – depending on the size and complexity of a given conservation planning problem – solvers based on open source software may only take slightly longer than commercial solvers.</p>
</div>
<div id="methods" class="section level1">
<h1 class="hasAnchor">
<a href="#methods" class="anchor" aria-hidden="true"></a>Methods</h1>
<p>This vignette will report the results of a benchmark analysis. To reduce computational burden, we previously completed the benchmark analysis and <a href="https://github.com/prioritizr/benchmark/releases" class="external-link">uploaded the results to an online repository</a> (<a href="https://github.com/prioritizr/benchmark" class="external-link">code available online</a>). This analysis involved generating prioritizations using different solvers and recording how long it took for the solvers to finish. To help understand the factors that influence how long it takes for solvers to generate a prioritization, we examined a suite of conservation planning problems with varying size (i.e. number of planning units), complexity (i.e. varying penalties to reduce spatial fragmentation), and with different objective functions (i.e. <a href="https://prioritizr.net/reference/objectives.html">metric used to evaluate competing solutions</a>). In this section, we will download the results for the previously completed benchmark analysis and examine the parameters used to conduct it.</p>
<div id="set-up" class="section level2">
<h2 class="hasAnchor">
<a href="#set-up" class="anchor" aria-hidden="true"></a>Set up</h2>
<p>To start off, we will the load packages used in this vignette.</p>
<div class="sourceCode" id="cb1"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="co"># load packages</span>
<span class="kw"><a href="https://rdrr.io/r/base/library.html" class="external-link">library</a></span><span class="op">(</span><span class="va"><a href="https://prioritizr.net">prioritizr</a></span><span class="op">)</span>
<span class="kw"><a href="https://rdrr.io/r/base/library.html" class="external-link">library</a></span><span class="op">(</span><span class="va"><a href="https://docs.ropensci.org/piggyback" class="external-link">piggyback</a></span><span class="op">)</span>
<span class="kw"><a href="https://rdrr.io/r/base/library.html" class="external-link">library</a></span><span class="op">(</span><span class="va"><a href="https://ggplot2.tidyverse.org" class="external-link">ggplot2</a></span><span class="op">)</span>
<span class="kw"><a href="https://rdrr.io/r/base/library.html" class="external-link">library</a></span><span class="op">(</span><span class="va"><a href="https://github.com/r-quantities/units/" class="external-link">units</a></span><span class="op">)</span>
<span class="kw"><a href="https://rdrr.io/r/base/library.html" class="external-link">library</a></span><span class="op">(</span><span class="va"><a href="https://dplyr.tidyverse.org" class="external-link">dplyr</a></span><span class="op">)</span></code></pre></div>
</div>
<div id="download-benchmark-results" class="section level2">
<h2 class="hasAnchor">
<a href="#download-benchmark-results" class="anchor" aria-hidden="true"></a>Download benchmark results</h2>
<p>Let’s download the results of the benchmark analysis. This code will save the results to a temporary folder on your computer. Please note that downloading the results might take several minutes to complete depending on your Internet connection. If you are unable to download the results onto your computer, you can view the graphs shown in this vignette. You only need to run the code in this vignette if you wish to explore certain aspects of the results yourself.</p>
<div class="sourceCode" id="cb2"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="co"># download data to temporary folder</span>
<span class="fu"><a href="https://docs.ropensci.org/piggyback/reference/pb_download.html" class="external-link">pb_download</a></span><span class="op">(</span>
file <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span><span class="st">"solutions.zip"</span>, <span class="st">"results.rda"</span><span class="op">)</span>,
repo <span class="op">=</span> <span class="st">"prioritizr/benchmark"</span>, dest <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/tempfile.html" class="external-link">tempdir</a></span><span class="op">(</span><span class="op">)</span>, tag <span class="op">=</span> <span class="st">"latest"</span>,
show_progress <span class="op">=</span> <span class="cn">FALSE</span><span class="op">)</span>
<span class="co"># load benchmark results</span>
<span class="fu"><a href="https://rdrr.io/r/base/load.html" class="external-link">load</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/file.path.html" class="external-link">file.path</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/tempfile.html" class="external-link">tempdir</a></span><span class="op">(</span><span class="op">)</span>, <span class="st">"results.rda"</span><span class="op">)</span><span class="op">)</span></code></pre></div>
</div>
<div id="benchmark-parameters" class="section level2">
<h2 class="hasAnchor">
<a href="#benchmark-parameters" class="anchor" aria-hidden="true"></a>Benchmark parameters</h2>
<p>After downloading the benchmark results, let’s have a look at the parameters that were used to conduct it. Note that all benchmark scenarios have 72 features.</p>
<div class="sourceCode" id="cb3"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="co"># numbers of planning units examined the benchmark analysis</span>
<span class="va">n_planning_units</span> <span class="op"><-</span> <span class="fu"><a href="https://rdrr.io/pkg/raster/man/unique.html" class="external-link">unique</a></span><span class="op">(</span><span class="va">benchmark_results</span><span class="op">$</span><span class="va">number_of_planning_units</span><span class="op">)</span>
<span class="fu"><a href="../reference/print.html">print</a></span><span class="op">(</span><span class="va">n_planning_units</span><span class="op">)</span></code></pre></div>
<pre><code>## [1] 1478 12902 102242 606180</code></pre>
<div class="sourceCode" id="cb5"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="co"># number of features (e.g. number of different species examined)</span>
<span class="fu"><a href="https://rdrr.io/pkg/raster/man/unique.html" class="external-link">unique</a></span><span class="op">(</span><span class="va">benchmark_results</span><span class="op">$</span><span class="va">number_features</span><span class="op">)</span></code></pre></div>
<pre><code>## [1] 72</code></pre>
<div class="sourceCode" id="cb7"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="co"># representation targets,</span>
<span class="co"># units are proportion of the total amount of each feature (e.g. 0.1 = 10%)</span>
<span class="fu"><a href="https://rdrr.io/pkg/raster/man/unique.html" class="external-link">unique</a></span><span class="op">(</span><span class="va">benchmark_results</span><span class="op">$</span><span class="va">relative_target</span><span class="op">)</span></code></pre></div>
<pre><code>## [1] 0.10 0.15 0.20 0.30</code></pre>
<div class="sourceCode" id="cb9"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="co"># number of planning units</span>
<span class="fu"><a href="https://rdrr.io/pkg/raster/man/unique.html" class="external-link">unique</a></span><span class="op">(</span><span class="va">benchmark_results</span><span class="op">$</span><span class="va">number_of_planning_units</span><span class="op">)</span></code></pre></div>
<pre><code>## [1] 1478 12902 102242 606180</code></pre>
<div class="sourceCode" id="cb11"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="co"># objective functions</span>
<span class="fu"><a href="https://rdrr.io/pkg/raster/man/unique.html" class="external-link">unique</a></span><span class="op">(</span><span class="va">benchmark_results</span><span class="op">$</span><span class="va">objective</span><span class="op">)</span></code></pre></div>
<pre><code>## [1] "add_min_set_objective" "add_min_shortfall_objective"</code></pre>
<div class="sourceCode" id="cb13"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="co"># extract boundary penalty values,</span>
<span class="co"># note that different values were examined for different objective functions</span>
<span class="va">boundary_penalty_values</span> <span class="op"><-</span>
<span class="va">benchmark_results</span> <span class="op"><a href="../reference/pipe.html">%>%</a></span>
<span class="fu">plyr</span><span class="fu">::</span><span class="fu"><a href="https://rdrr.io/pkg/plyr/man/dlply.html" class="external-link">dlply</a></span><span class="op">(</span><span class="st">"objective"</span>, <span class="kw">function</span><span class="op">(</span><span class="va">x</span><span class="op">)</span> <span class="fu"><a href="https://rdrr.io/pkg/raster/man/unique.html" class="external-link">unique</a></span><span class="op">(</span><span class="va">x</span><span class="op">$</span><span class="va">boundary_penalty</span><span class="op">)</span><span class="op">)</span>
<span class="co">## boundary penalty values for min set objective function</span>
<span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_set_objective</span></code></pre></div>
<pre><code>## [1] 0e+00 1e-05 1e-03</code></pre>
<div class="sourceCode" id="cb15"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="co">## boundary penalty values for min shortfall objective function</span>
<span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_shortfall_objective</span></code></pre></div>
<pre><code>## [1] 0e+00 1e-14 1e-13</code></pre>
<div class="sourceCode" id="cb17"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="co"># budgets examined for budget-limited objectives (e.g. 0.1 = 10% of total cost)</span>
<span class="co">## note that the min set objective function does not use a budget,</span>
<span class="co">## and thus it has a NA value</span>
<span class="fu"><a href="https://tibble.tidyverse.org/reference/tibble.html" class="external-link">tibble</a></span><span class="op">(</span>objective <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/pkg/raster/man/unique.html" class="external-link">unique</a></span><span class="op">(</span><span class="va">benchmark_results</span><span class="op">$</span><span class="va">objective</span><span class="op">)</span>,
budget <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/pkg/raster/man/unique.html" class="external-link">unique</a></span><span class="op">(</span><span class="va">benchmark_results</span><span class="op">$</span><span class="va">budget</span><span class="op">)</span><span class="op">)</span></code></pre></div>
<pre><code>## # A tibble: 2 × 2
## objective budget
## <chr> <dbl>
## 1 add_min_set_objective NA
## 2 add_min_shortfall_objective 0.1</code></pre>
</div>
<div id="helper-function" class="section level2">
<h2 class="hasAnchor">
<a href="#helper-function" class="anchor" aria-hidden="true"></a>Helper function</h2>
<p>Now we will define a helper function to quickly plot the results from the benchmark analysis. This will be helpful for interpreting the results of the benchmark analysis in the following section.</p>
<div class="sourceCode" id="cb19"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="co"># define helper function to create plots</span>
<span class="va">plot_benchmark</span> <span class="op"><-</span> <span class="kw">function</span><span class="op">(</span>
<span class="va">objective</span>, <span class="va">n_pu</span>, <span class="va">boundary_penalty</span>, <span class="va">solver</span> <span class="op">=</span> <span class="cn">NULL</span><span class="op">)</span><span class="op">{</span>
<span class="co"># assert arguments are valid</span>
<span class="co">## verify parameters with no default arguments</span>
<span class="fu">assertthat</span><span class="fu">::</span><span class="fu"><a href="https://rdrr.io/pkg/assertthat/man/assert_that.html" class="external-link">assert_that</a></span><span class="op">(</span>
<span class="fu">assertthat</span><span class="fu">::</span><span class="fu"><a href="https://rdrr.io/pkg/assertthat/man/scalar.html" class="external-link">is.count</a></span><span class="op">(</span><span class="va">n_pu</span><span class="op">)</span>, <span class="fu">assertthat</span><span class="fu">::</span><span class="fu"><a href="https://rdrr.io/pkg/assertthat/man/noNA.html" class="external-link">noNA</a></span><span class="op">(</span><span class="va">n_pu</span><span class="op">)</span>,
<span class="va">n_pu</span> <span class="op"><a href="https://rdrr.io/pkg/raster/man/match.html" class="external-link">%in%</a></span> <span class="fu"><a href="https://rdrr.io/pkg/raster/man/unique.html" class="external-link">unique</a></span><span class="op">(</span><span class="va">benchmark_results</span><span class="op">$</span><span class="va">number_of_planning_units</span><span class="op">)</span>,
<span class="fu">assertthat</span><span class="fu">::</span><span class="fu"><a href="https://rdrr.io/pkg/assertthat/man/scalar.html" class="external-link">is.number</a></span><span class="op">(</span><span class="va">boundary_penalty</span><span class="op">)</span>, <span class="fu">assertthat</span><span class="fu">::</span><span class="fu"><a href="https://rdrr.io/pkg/assertthat/man/noNA.html" class="external-link">noNA</a></span><span class="op">(</span><span class="va">boundary_penalty</span><span class="op">)</span>,
<span class="fu">assertthat</span><span class="fu">::</span><span class="fu"><a href="https://rdrr.io/pkg/assertthat/man/scalar.html" class="external-link">is.string</a></span><span class="op">(</span><span class="va">objective</span><span class="op">)</span>, <span class="fu">assertthat</span><span class="fu">::</span><span class="fu"><a href="https://rdrr.io/pkg/assertthat/man/noNA.html" class="external-link">noNA</a></span><span class="op">(</span><span class="va">objective</span><span class="op">)</span>,
<span class="va">objective</span> <span class="op"><a href="https://rdrr.io/pkg/raster/man/match.html" class="external-link">%in%</a></span> <span class="fu"><a href="https://rdrr.io/pkg/raster/man/unique.html" class="external-link">unique</a></span><span class="op">(</span><span class="va">benchmark_results</span><span class="op">$</span><span class="va">objective</span><span class="op">)</span><span class="op">)</span>
<span class="co">## set default argument for solver if needed</span>
<span class="kw">if</span> <span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/NULL.html" class="external-link">is.null</a></span><span class="op">(</span><span class="va">solver</span><span class="op">)</span><span class="op">)</span> <span class="op">{</span>
<span class="va">solver</span> <span class="op"><-</span> <span class="fu"><a href="https://rdrr.io/pkg/raster/man/unique.html" class="external-link">unique</a></span><span class="op">(</span><span class="va">benchmark_results</span><span class="op">$</span><span class="va">solver</span><span class="op">)</span>
<span class="op">}</span>
<span class="co">## verify solver argument</span>
<span class="fu">assertthat</span><span class="fu">::</span><span class="fu"><a href="https://rdrr.io/pkg/assertthat/man/assert_that.html" class="external-link">assert_that</a></span><span class="op">(</span>
<span class="fu"><a href="https://rdrr.io/r/base/character.html" class="external-link">is.character</a></span><span class="op">(</span><span class="va">solver</span><span class="op">)</span>, <span class="fu"><a href="https://rdrr.io/r/base/all.html" class="external-link">all</a></span><span class="op">(</span><span class="va">solver</span> <span class="op"><a href="https://rdrr.io/pkg/raster/man/match.html" class="external-link">%in%</a></span> <span class="va">benchmark_results</span><span class="op">$</span><span class="va">solver</span><span class="op">)</span><span class="op">)</span>
<span class="co">## verify that only a single set of features was used</span>
<span class="fu">assertthat</span><span class="fu">::</span><span class="fu"><a href="https://rdrr.io/pkg/assertthat/man/assert_that.html" class="external-link">assert_that</a></span><span class="op">(</span>
<span class="fu">dplyr</span><span class="fu">::</span><span class="fu"><a href="https://dplyr.tidyverse.org/reference/n_distinct.html" class="external-link">n_distinct</a></span><span class="op">(</span><span class="va">benchmark_results</span><span class="op">$</span><span class="va">number_features</span><span class="op">)</span> <span class="op">==</span> <span class="fl">1</span><span class="op">)</span>
<span class="co"># prepare data for plotting</span>
<span class="co">## rename variables to avoid scoping issues</span>
<span class="va">sol</span> <span class="op"><-</span> <span class="va">solver</span>
<span class="va">obj</span> <span class="op"><-</span> <span class="va">objective</span>
<span class="va">bp</span> <span class="op"><-</span> <span class="va">boundary_penalty</span>
<span class="co">## subset data relevant for plotting</span>
<span class="va">plot_data</span> <span class="op"><-</span>
<span class="va">benchmark_results</span> <span class="op"><a href="../reference/pipe.html">%>%</a></span>
<span class="fu"><a href="https://dplyr.tidyverse.org/reference/filter.html" class="external-link">filter</a></span><span class="op">(</span><span class="va">.</span><span class="op">$</span><span class="va">objective</span> <span class="op">==</span> <span class="va">obj</span>, <span class="va">.</span><span class="op">$</span><span class="va">solver</span> <span class="op"><a href="https://rdrr.io/pkg/raster/man/match.html" class="external-link">%in%</a></span> <span class="va">sol</span>,
<span class="va">.</span><span class="op">$</span><span class="va">number_of_planning_units</span> <span class="op">==</span> <span class="va">n_pu</span>,
<span class="va">.</span><span class="op">$</span><span class="va">boundary_penalty</span> <span class="op">==</span> <span class="va">bp</span><span class="op">)</span>
<span class="co">## scale run time to helpful units for plotting</span>
<span class="va">plot_units</span> <span class="op"><-</span>
<span class="fu">dplyr</span><span class="fu">::</span><span class="fu"><a href="https://dplyr.tidyverse.org/reference/case_when.html" class="external-link">case_when</a></span><span class="op">(</span>
<span class="co"># show hours if max(run_time) > 3 h</span>
<span class="fu"><a href="https://rdrr.io/r/base/Extremes.html" class="external-link">max</a></span><span class="op">(</span><span class="va">plot_data</span><span class="op">$</span><span class="va">run_time</span><span class="op">)</span> <span class="op">></span> <span class="fl">60</span> <span class="op">*</span> <span class="fl">60</span> <span class="op">*</span> <span class="fl">3</span> <span class="op">~</span> <span class="st">"hours"</span>,
<span class="co"># show minutes if max(run_time) > 3 M</span>
<span class="fu"><a href="https://rdrr.io/r/base/Extremes.html" class="external-link">max</a></span><span class="op">(</span><span class="va">plot_data</span><span class="op">$</span><span class="va">run_time</span><span class="op">)</span> <span class="op">></span> <span class="fl">60</span> <span class="op">*</span> <span class="fl">3</span> <span class="op">~</span> <span class="st">"minutes"</span>,
<span class="co"># else show seconds</span>
<span class="cn">TRUE</span> <span class="op">~</span> <span class="st">"seconds"</span><span class="op">)</span>
<span class="va">plot_data</span><span class="op">$</span><span class="va">min_set_scaled</span> <span class="op"><-</span>
<span class="va">plot_data</span><span class="op">$</span><span class="va">run_time</span> <span class="op"><a href="../reference/pipe.html">%>%</a></span>
<span class="fu">units</span><span class="fu">::</span><span class="fu"><a href="https://rdrr.io/pkg/units/man/units.html" class="external-link">set_units</a></span><span class="op">(</span><span class="va">s</span><span class="op">)</span> <span class="op"><a href="../reference/pipe.html">%>%</a></span>
<span class="fu">units</span><span class="fu">::</span><span class="fu"><a href="https://rdrr.io/pkg/units/man/units.html" class="external-link">set_units</a></span><span class="op">(</span><span class="va">plot_units</span>, mode <span class="op">=</span> <span class="st">"standard"</span><span class="op">)</span> <span class="op"><a href="../reference/pipe.html">%>%</a></span>
<span class="fu"><a href="https://rdrr.io/r/base/numeric.html" class="external-link">as.numeric</a></span><span class="op">(</span><span class="op">)</span>
<span class="co">## plot labels</span>
<span class="va">n_f</span> <span class="op"><-</span> <span class="fu"><a href="https://rdrr.io/pkg/raster/man/unique.html" class="external-link">unique</a></span><span class="op">(</span><span class="va">benchmark_results</span><span class="op">$</span><span class="va">number_features</span><span class="op">)</span><span class="op">[</span><span class="fl">1</span><span class="op">]</span>
<span class="va">plot_title</span> <span class="op">=</span>
<span class="fu"><a href="https://rdrr.io/r/base/paste.html" class="external-link">paste0</a></span><span class="op">(</span>
<span class="fu">dplyr</span><span class="fu">::</span><span class="fu"><a href="https://dplyr.tidyverse.org/reference/case_when.html" class="external-link">case_when</a></span><span class="op">(</span>
<span class="va">objective</span> <span class="op">==</span> <span class="st">"add_min_set_objective"</span> <span class="op">~</span> <span class="st">"Min. set"</span>,
<span class="va">objective</span> <span class="op">==</span> <span class="st">"add_min_shortfall_objective"</span> <span class="op">~</span> <span class="st">"Min. shortfall"</span>,
<span class="cn">TRUE</span> <span class="op">~</span> <span class="va">objective</span><span class="op">)</span>,
<span class="st">": "</span>,
<span class="fu"><a href="https://rdrr.io/r/base/formatc.html" class="external-link">formatC</a></span><span class="op">(</span>
<span class="va">n_f</span>, big.mark <span class="op">=</span> <span class="st">","</span>, digits <span class="op">=</span> <span class="fl">2</span>, format <span class="op">=</span> <span class="st">"f"</span>,
drop0trailing <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span>,
<span class="st">" features, "</span>,
<span class="fu"><a href="https://rdrr.io/r/base/formatc.html" class="external-link">formatC</a></span><span class="op">(</span>
<span class="va">n_pu</span>, big.mark <span class="op">=</span> <span class="st">","</span>, digits <span class="op">=</span> <span class="fl">2</span>, format <span class="op">=</span> <span class="st">"f"</span>,
drop0trailing <span class="op">=</span> <span class="cn">TRUE</span><span class="op">)</span>,
<span class="st">" planning units"</span><span class="op">)</span>
<span class="kw">if</span> <span class="op">(</span><span class="va">bp</span> <span class="op">></span> <span class="fl">1e-15</span><span class="op">)</span> <span class="op">{</span>
<span class="va">plot_title</span> <span class="op"><-</span> <span class="fu"><a href="https://rdrr.io/r/base/paste.html" class="external-link">paste0</a></span><span class="op">(</span><span class="va">plot_title</span>, <span class="st">", "</span>, <span class="va">bp</span>, <span class="st">" boundary penalty"</span><span class="op">)</span>
<span class="op">}</span>
<span class="co">## determine colors for solvers (so that solvers always have same color</span>
<span class="va">solver_names</span> <span class="op"><-</span> <span class="fu"><a href="https://rdrr.io/pkg/raster/man/unique.html" class="external-link">unique</a></span><span class="op">(</span><span class="va">benchmark_results</span><span class="op">$</span><span class="va">solver</span><span class="op">)</span>
<span class="va">solver_colors</span> <span class="op"><-</span> <span class="fu">scales</span><span class="fu">::</span><span class="fu"><a href="https://scales.r-lib.org/reference/hue_pal.html" class="external-link">hue_pal</a></span><span class="op">(</span><span class="op">)</span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/r/base/length.html" class="external-link">length</a></span><span class="op">(</span><span class="va">solver_names</span><span class="op">)</span><span class="op">)</span>
<span class="fu"><a href="https://rdrr.io/pkg/raster/man/names.html" class="external-link">names</a></span><span class="op">(</span><span class="va">solver_colors</span><span class="op">)</span> <span class="op"><-</span> <span class="va">solver_names</span>
<span class="co"># return plot for selected benchmark runs</span>
<span class="fu"><a href="https://ggplot2.tidyverse.org/reference/ggplot.html" class="external-link">ggplot</a></span><span class="op">(</span>
data <span class="op">=</span> <span class="va">plot_data</span>,
mapping <span class="op">=</span> <span class="fu"><a href="https://ggplot2.tidyverse.org/reference/aes.html" class="external-link">aes</a></span><span class="op">(</span>x <span class="op">=</span> <span class="va">relative_target</span>, y <span class="op">=</span> <span class="va">min_set_scaled</span>, color <span class="op">=</span> <span class="va">solver</span><span class="op">)</span><span class="op">)</span> <span class="op">+</span>
<span class="fu"><a href="https://ggplot2.tidyverse.org/reference/scale_continuous.html" class="external-link">scale_y_continuous</a></span><span class="op">(</span>limits <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span><span class="fl">0</span>, <span class="cn">NA_real_</span><span class="op">)</span><span class="op">)</span> <span class="op">+</span>
<span class="fu"><a href="https://ggplot2.tidyverse.org/reference/geom_path.html" class="external-link">geom_line</a></span><span class="op">(</span><span class="op">)</span> <span class="op">+</span>
<span class="fu"><a href="https://ggplot2.tidyverse.org/reference/geom_point.html" class="external-link">geom_point</a></span><span class="op">(</span><span class="op">)</span> <span class="op">+</span>
<span class="fu"><a href="https://ggplot2.tidyverse.org/reference/scale_continuous.html" class="external-link">scale_x_continuous</a></span><span class="op">(</span>labels <span class="op">=</span> <span class="kw">function</span><span class="op">(</span><span class="va">x</span><span class="op">)</span> <span class="va">x</span> <span class="op">*</span> <span class="fl">100</span><span class="op">)</span> <span class="op">+</span>
<span class="fu"><a href="https://ggplot2.tidyverse.org/reference/scale_manual.html" class="external-link">scale_color_manual</a></span><span class="op">(</span>values <span class="op">=</span> <span class="va">solver_colors</span><span class="op">)</span> <span class="op">+</span>
<span class="fu"><a href="https://ggplot2.tidyverse.org/reference/labs.html" class="external-link">labs</a></span><span class="op">(</span>
title <span class="op">=</span> <span class="va">plot_title</span>,
x <span class="op">=</span> <span class="st">"Representation targets (%)"</span>,
y <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/paste.html" class="external-link">paste0</a></span><span class="op">(</span><span class="st">"Run time ("</span>, <span class="va">plot_units</span>, <span class="st">")"</span><span class="op">)</span><span class="op">)</span>
<span class="op">}</span></code></pre></div>
</div>
</div>
<div id="results" class="section level1">
<h1 class="hasAnchor">
<a href="#results" class="anchor" aria-hidden="true"></a>Results</h1>
<p>We will now inspect the results of the benchmark analysis. The <code>benchmark_results</code> object is a table (i.e. <code><a href="https://tibble.tidyverse.org/reference/tibble.html" class="external-link">tibble()</a></code>) containing information for each benchmark run (e.g. run time), and the <code>solution_raster_data</code> object (i.e. <code>RasterStack</code>) contains the prioritizations generated for each benchmark run.</p>
<div class="sourceCode" id="cb20"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="co"># preview results</span>
<span class="fu"><a href="../reference/print.html">print</a></span><span class="op">(</span><span class="va">benchmark_results</span><span class="op">)</span></code></pre></div>
<pre><code>## # A tibble: 480 × 21
## id pu_data number_of_plannin… number_features objective budget
## <chr> <int> <int> <int> <chr> <dbl>
## 1 faa32c2f85231… 1 1478 72 add_min_set… NA
## 2 d45739ff528cd… 2 12902 72 add_min_set… NA
## 3 dace50564e47d… 3 102242 72 add_min_set… NA
## 4 36e6f35f017be… 4 606180 72 add_min_set… NA
## 5 fb66b4160dac8… 1 1478 72 add_min_set… NA
## 6 2bb12ee18e2c8… 2 12902 72 add_min_set… NA
## 7 21916c745740a… 3 102242 72 add_min_set… NA
## 8 44995e0a5a1e4… 4 606180 72 add_min_set… NA
## 9 2e848142afa12… 1 1478 72 add_min_set… NA
## 10 641e377558217… 2 12902 72 add_min_set… NA
## # … with 470 more rows, and 15 more variables: relative_target <dbl>,
## # boundary_penalty <dbl>, solver <chr>, threads <int>, time_limit <dbl>,
## # gap <dbl>, replicate <int>, id2 <int>, objective_value <dbl>, status <chr>,
## # total_time <dbl>, run_time <dbl>, exceeded_run_time <lgl>, solution <chr>,
## # cache <lgl></code></pre>
<p>Let’s examine the average run time for each solver across all benchmark runs. To achieve this, we will create a box plot. Since we prefer solvers that generate prioritizations in a short period of time, solvers that have shorter (lower) run times are considered better. When looking at this box plot, we can see that average run times for the <em>IBM CPLEX</em> and <em>Gurobi</em> solvers are consistently low and don’t differ much from each other. Additionally, the run times for the open source <em>CBC</em> solver are generally low too. However, there are some cases where the run times for the <em>CBC</em> solver are much higher than those for the <em>IBM CPLEX</em> and <em>Gurobi</em> solvers. We can also see that the <em>lpsymphony</em> and <em>Rsymphony</em> solvers – which both use the open source <em>SYMPHONY</em> software – can take much longer to generate prioritizations than the other solvers.</p>
<div class="sourceCode" id="cb22"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="co"># plot overall summary of solver performance</span>
<span class="fu"><a href="https://ggplot2.tidyverse.org/reference/ggplot.html" class="external-link">ggplot</a></span><span class="op">(</span>
data <span class="op">=</span>
<span class="va">benchmark_results</span> <span class="op"><a href="../reference/pipe.html">%>%</a></span>
<span class="fu"><a href="https://dplyr.tidyverse.org/reference/mutate.html" class="external-link">mutate</a></span><span class="op">(</span>
min_set_scaled <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/numeric.html" class="external-link">as.numeric</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/pkg/units/man/units.html" class="external-link">set_units</a></span><span class="op">(</span><span class="fu"><a href="https://rdrr.io/pkg/units/man/units.html" class="external-link">set_units</a></span><span class="op">(</span>
<span class="va">run_time</span>, <span class="st">"seconds"</span><span class="op">)</span>, <span class="st">"hours"</span><span class="op">)</span><span class="op">)</span><span class="op">)</span>,
<span class="fu"><a href="https://ggplot2.tidyverse.org/reference/aes.html" class="external-link">aes</a></span><span class="op">(</span>x <span class="op">=</span> <span class="va">solver</span>, y <span class="op">=</span> <span class="va">min_set_scaled</span><span class="op">)</span><span class="op">)</span> <span class="op">+</span>
<span class="fu"><a href="https://ggplot2.tidyverse.org/reference/geom_boxplot.html" class="external-link">geom_boxplot</a></span><span class="op">(</span><span class="op">)</span> <span class="op">+</span>
<span class="fu"><a href="https://ggplot2.tidyverse.org/reference/theme.html" class="external-link">theme</a></span><span class="op">(</span>axis.text.x <span class="op">=</span> <span class="fu"><a href="https://ggplot2.tidyverse.org/reference/element.html" class="external-link">element_text</a></span><span class="op">(</span>size <span class="op">=</span> <span class="fl">7</span><span class="op">)</span><span class="op">)</span> <span class="op">+</span>
<span class="fu"><a href="https://ggplot2.tidyverse.org/reference/labs.html" class="external-link">labs</a></span><span class="op">(</span>x <span class="op">=</span> <span class="st">"Solver"</span>, y <span class="op">=</span> <span class="st">"Run time (hours)"</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/overall_average_run_times-1.png" width="672" style="display: block; margin: auto;"></p>
<div id="minimum-set-results-no-boundary-penalty" class="section level2">
<h2 class="hasAnchor">
<a href="#minimum-set-results-no-boundary-penalty" class="anchor" aria-hidden="true"></a>Minimum set results (no boundary penalty)</h2>
<p>Now, let’s investigate the solver behavior in more detail. Specifically, we will examine the benchmark results generated for the minimum set objective function. This is because the minimum set objective function is the most commonly used objective function in systematic conservation, due to the fact that it is used by the <a href="https://marxansolutions.org/" class="external-link"><em>Marxan</em> decision support software</a>. To begin with, let’s examine the results for the smallest and simplest conservation planning problems examined in the benchmark analysis. Here, all prioritizations were generated using problems that involved 1,478 planning units and did not contain any boundary length penalties. Since all benchmark scenarios have 72 features – as mentioned earlier – all of these prioritizations were generated with 72 features. When looking at the results, we can see that all solvers solve the problem in a comparable amount of time across all targets investigated.</p>
<div class="sourceCode" id="cb23"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_set_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">1</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="fl">0</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_set_pu_n_1-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Next, let’s look at the results for a more realistic problem involving 12,902 planning units and see how the timing of the different solvers used compares. Note that all other factors (e.g. absence of boundary length penalties) are the same as for the previous graph.</p>
<div class="sourceCode" id="cb24"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_set_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="fl">0</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_set_pu_n_2-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Next, we will look at a medium sized problem with 102,242 planning units. Now we really start to see the difference between the <em>Rsymphony</em> and <em>lpsymphony</em> solvers – which use the <em>SYMPHONY</em> software – and the other solvers.</p>
<div class="sourceCode" id="cb25"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_set_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">3</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="fl">0</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_set_pu_n_3-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Finally, let’s look at timing comparisons for a large problem with 606,180 planning units. We can see that both the <em>Rsymphony</em> and <em>lpsymphony</em> solvers are really taking a much longer time to generate prioritizations now.</p>
<div class="sourceCode" id="cb26"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_set_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">4</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="fl">0</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_set_pu_n_4-1.png" width="672" style="display: block; margin: auto;"></p>
<p>To get a better sense of how the faster solvers (i.e. based on <em>CBC</em>, <em>IBM CPLEX</em>, <em>Gurobi</em>) compare for such a large problem, let’s take a closer look at these three solvers. Interestingly, we can see that the solver based on the open source <em>CBC</em> software is slightly faster – by a few minutes – than the other solvers.</p>
<div class="sourceCode" id="cb27"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_set_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">4</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="fl">0</span>,
solver <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span><span class="st">"add_cbc_solver"</span>, <span class="st">"add_cplex_solver"</span>, <span class="st">"add_gurobi_solver"</span><span class="op">)</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_set_pu_n_4_fast_only-1.png" width="672" style="display: block; margin: auto;"></p>
</div>
<div id="minimum-set-results-with-low-boundary-penalty" class="section level2">
<h2 class="hasAnchor">
<a href="#minimum-set-results-with-low-boundary-penalty" class="anchor" aria-hidden="true"></a>Minimum set results with low boundary penalty</h2>
<p>Now let’s at the same problem types, but this time with a low boundary length penalty value added to the problem formulation. To start with, we will look at scenarios with a low <code>boundary_penalty</code> value (i.e. <span class="math inline">\(10^{-5}\)</span>). Let’s start again with the smallest problem size we’ve benchmarked. This problem has only 1,478 planning units.</p>
<div class="sourceCode" id="cb28"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_set_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">1</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_set_objective</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_set_pu_n_1_with_low_boundary_penalty-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Next, let’s look at the results for a more realistic problem with 12,902 planning units and see how the timing of the different solvers used compares.</p>
<div class="sourceCode" id="cb29"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_set_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_set_objective</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_set_pu_n_2_with_low_boundary_penalty-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Next, we will look at a medium sized problem with 102,242 planning units. Now we really start to see the difference between <em>lpsymphony</em> and <em>Rsymphony</em> solvers and the other solvers.</p>
<div class="sourceCode" id="cb30"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_set_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">3</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_set_objective</span><span class="op">[</span><span class="fl">3</span><span class="op">]</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_set_pu_n_3_with_low_boundary_penalty-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Finally, let’s look at timing comparisons for a large problem with 606,180 planning units. As with the scenario without boundary penalties, <em>lpsymphony</em> and <em>Rsymphony</em> solvers take a lot longer to generate prioritizations than the other three solvers.</p>
<div class="sourceCode" id="cb31"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_set_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">4</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_set_objective</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_set_pu_n_4_with_low_boundary_penalty-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Similar to earlier, let’s take a look at just the <em>CBC</em>, <em>IBM CPLEX</em>, and <em>Gurobi</em> solvers. We can see that the <em>Gurobi</em> solver has the best performance, generating prioritizations in under half an hour in all cases. This result shows that the commercial solvers can massively outperform the open source solvers for large-scale problems with boundary length penalties.</p>
<div class="sourceCode" id="cb32"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_set_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">4</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_set_objective</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span>,
solver <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span><span class="st">"add_cbc_solver"</span>, <span class="st">"add_cplex_solver"</span>, <span class="st">"add_gurobi_solver"</span><span class="op">)</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_set_pu_n_4_with_low_boundary_penalty_and_fast_solvers-1.png" width="672" style="display: block; margin: auto;"></p>
</div>
<div id="minimum-set-results-with-high-boundary-penalty" class="section level2">
<h2 class="hasAnchor">
<a href="#minimum-set-results-with-high-boundary-penalty" class="anchor" aria-hidden="true"></a>Minimum set results with high boundary penalty</h2>
<p>Now let’s look at the same problem types, but this time with a high boundary length penalty parameter added to the problem formulation (i.e. <span class="math inline">\(0.001\)</span>). Let’s start again with the smallest problem size we’ve benchmarked. This problem has only 1,478 planning units. Although there are some differences between the solvers, they all have very similar run times (i.e. all less than one second).</p>
<div class="sourceCode" id="cb33"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_set_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">1</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_set_objective</span><span class="op">[</span><span class="fl">3</span><span class="op">]</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_set_pu_n_1_with_high_boundary_penalty-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Next, let’s look at the results for a more realistic problem with 12,902 planning units and see how the timing of the different solvers used compares.</p>
<div class="sourceCode" id="cb34"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_set_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_set_objective</span><span class="op">[</span><span class="fl">3</span><span class="op">]</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_set_pu_n_2_with_high_boundary_penalty-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Next, we will look at a medium sized problem with 102,242 planning units. We again see the difference between <em>Rsymphony</em> and <em>lpsymphony</em> solvers and the other solvers.</p>
<div class="sourceCode" id="cb35"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_set_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">3</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_set_objective</span><span class="op">[</span><span class="fl">3</span><span class="op">]</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_set_pu_n_3_with_high_boundary_penalty-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Finally, let’s look at timing comparisons for a large problem with 606,180 planning units. As with the previous scenarios, <em>Rsymphony</em> and <em>lpsymphony</em> solvers take a lot longer to generate prioritizations than the other solvers.</p>
<div class="sourceCode" id="cb36"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_set_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">4</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_set_objective</span><span class="op">[</span><span class="fl">3</span><span class="op">]</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_set_pu_n_4_with_high_boundary_penalty-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Similar to before, let’s take a look at just the <em>CBC</em>, <em>IBM CPLEX</em>, and <em>Gurobi</em> solvers. We can see that <em>Gurobi</em> has the best performance. This result further emphasizes the potential speed gains of commercial solvers for large-scale conservation planning problems with boundary penalties.</p>
<div class="sourceCode" id="cb37"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_set_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">4</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_set_objective</span><span class="op">[</span><span class="fl">3</span><span class="op">]</span>,
solver <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span><span class="st">"add_cbc_solver"</span>, <span class="st">"add_cplex_solver"</span>, <span class="st">"add_gurobi_solver"</span><span class="op">)</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_set_pu_n_4_with_high_boundary_penalty_and_fast_solvers-1.png" width="672" style="display: block; margin: auto;"></p>
</div>
<div id="minimize-shortfall-results-no-boundary-penalty" class="section level2">
<h2 class="hasAnchor">
<a href="#minimize-shortfall-results-no-boundary-penalty" class="anchor" aria-hidden="true"></a>Minimize shortfall results (no boundary penalty)</h2>
<p>Now, let’s investigate the solver behavior for the minimum shortfall objective function. Let’s start with the smallest problem size examined. All benchmark scenarios have 72 features. This problem has only 1,478 planning units. We can see that all solvers solve the problem in a comparable amount of time across all targets investigated.</p>
<div class="sourceCode" id="cb38"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_shortfall_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">1</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="fl">0</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_short_pu_n_1-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Next, let’s look at the results for a more realistic problem with 12,902 planning units and see how the timing of the different solvers used compares. Here, the <em>CBC</em> solver takes slightly longer than the other solvers.</p>
<div class="sourceCode" id="cb39"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_shortfall_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="fl">0</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_short_pu_n_2-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Next, we will look at a medium sized problem with 102,242 planning units. Now we see a larger difference between the <em>CBC</em> solver and the other solvers, with the <em>CBC</em> solver taking several minutes longer to complete.</p>
<div class="sourceCode" id="cb40"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_shortfall_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">3</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="fl">0</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_short_pu_n_3-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Finally, let’s look at timing comparisons for a large problem with 606,180 planning units. We can see that all the open source solvers (i.e. <em>CBC</em>, <em>Rsymphony</em> and <em>lpsymphony</em>) take a lot longer than the commercial solvers (i.e. <em>IBM CPLEX</em> and <em>Guorbi</em>).</p>
<div class="sourceCode" id="cb41"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_shortfall_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">4</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="fl">0</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_short_pu_n_4-1.png" width="672" style="display: block; margin: auto;"></p>
</div>
<div id="minimize-shortfall-results-with-low-boundary-penalty" class="section level2">
<h2 class="hasAnchor">
<a href="#minimize-shortfall-results-with-low-boundary-penalty" class="anchor" aria-hidden="true"></a>Minimize shortfall results with low boundary penalty</h2>
<p>Now let’s look at the same problem type, but this time with a low <code>boundary_penalty</code> parameter added to the problem formulation (i.e. <span class="math inline">\(10^{-14}\)</span>). Let’s start again with the smallest problem size examined. This problem has only 1,478 planning units.</p>
<div class="sourceCode" id="cb42"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_shortfall_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">1</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_shortfall_objective</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_short_pu_n_1_with_low_boundary_penalty-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Next, let’s look at the results for a more realistic problem with 12,902 planning units and see how the timing of the different solvers compare. Similar to before, we can see the <em>CBC</em> solver takes slightly longer than the other solvers.</p>
<div class="sourceCode" id="cb43"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_shortfall_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_shortfall_objective</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_short_pu_n_2_with_low_boundary_penalty-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Next, we will look at a medium sized problem with 102,242 planning units. Now we really start to see the difference between the <em>CBC</em> solver and the other solvers. Additionally, the <em>Rsymphony</em> and <em>lpsymphony</em> solvers also take considerably longer than the commercial solvers too.</p>
<div class="sourceCode" id="cb44"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_shortfall_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">3</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_shortfall_objective</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_short_pu_n_3_with_low_boundary_penalty-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Finally, let’s look at timing comparisons for a large problem with 606,180 planning units. The <em>CBC</em> solver now performs better than the <em>Rsymphony</em> and <em>lpsymphony</em> solvers, which can take over a whole day to complete.</p>
<div class="sourceCode" id="cb45"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_shortfall_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">4</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_shortfall_objective</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_short_pu_n_4_with_low_boundary_penalty-1.png" width="672" style="display: block; margin: auto;"></p>
<p>To get a better sense of how the faster solvers compare (i.e. <em>CBC</em>, <em>IBM CPLEX</em>, and <em>Gurobi</em>), let’s take a closer look at these three solvers. We can see that the <em>CBC</em> solver takes a lot longer to generate prioritizations than the <em>IBM CPLEX</em> and <em>Gurobi</em> solvers. This result suggests that <em>IBM CPLEX</em> and <em>Gurobi</em> could be really beneficial for large-scale conservation planning problems with boundary length penalties and the minimum shortfall objective function.</p>
<div class="sourceCode" id="cb46"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_shortfall_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">4</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_shortfall_objective</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span>,
solver <span class="op">=</span> <span class="fu"><a href="https://rdrr.io/r/base/c.html" class="external-link">c</a></span><span class="op">(</span><span class="st">"add_cbc_solver"</span>, <span class="st">"add_cplex_solver"</span>, <span class="st">"add_gurobi_solver"</span><span class="op">)</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_short_pu_n_4_and_boundary_penalty_and_fast_solvers-1.png" width="672" style="display: block; margin: auto;"></p>
</div>
<div id="minimize-shortfall-results-with-high-boundary-penalty" class="section level2">
<h2 class="hasAnchor">
<a href="#minimize-shortfall-results-with-high-boundary-penalty" class="anchor" aria-hidden="true"></a>Minimize shortfall results with high boundary penalty</h2>
<p>Now let’s look at the same problem types, but this time with a higher <code>boundary_penalty</code> parameter added to the problem formulation (<span class="math inline">\(10^{-13}\)</span>). Let’s start again with the smallest problem size examined. This problem has only 1,478 planning units.</p>
<div class="sourceCode" id="cb47"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_shortfall_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">1</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_shortfall_objective</span><span class="op">[</span><span class="fl">3</span><span class="op">]</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_short_pu_n_1_with_high_boundary_penalty-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Next, let’s look at the results for a greater number of planning units (i.e. 12,902 planning units) and see how the timings compare. All the solvers have a similar run time now. Interestingly, the <em>Gurobi</em> solver is the slowest – but only by a couple of minutes – for these benchmark parameters.</p>
<div class="sourceCode" id="cb48"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_shortfall_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">2</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_shortfall_objective</span><span class="op">[</span><span class="fl">3</span><span class="op">]</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_short_pu_n_2%20with%20boundary%20penalty-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Next, we will look at a medium sized problem with 102,242 planning units. Similar to some of the previous results for the minimum shortfall objective, the <em>CBC</em> solver is the slowest and the <em>IBM CPLEX</em> and <em>Gurobi</em> commercial solvers are fastest.</p>
<div class="sourceCode" id="cb49"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_shortfall_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">3</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_shortfall_objective</span><span class="op">[</span><span class="fl">3</span><span class="op">]</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_short_pu_n_3_with_high_boundary_penalty-1.png" width="672" style="display: block; margin: auto;"></p>
<p>Finally, let’s look at timing comparisons for a large problem with 606,180 planning units. We can see that the benchmark times vary greatly for the <em>Rsymphony</em> and <em>lpsymphony</em> solvers, with run times ranging from one to two whole days. Additionally, although the <em>CBC</em> solver can still require several hours to generate a prioritization, it now outperforms the <em>Rsymphony</em> and <em>lpsymphony</em> solvers. Finally, the <em>IBM CPLEX</em> and <em>Gurobi</em> solvers perform much, much better than all the other solvers.</p>
<div class="sourceCode" id="cb50"><pre class="downlit sourceCode r">
<code class="sourceCode R"><span class="fu">plot_benchmark</span><span class="op">(</span>
objective <span class="op">=</span> <span class="st">"add_min_shortfall_objective"</span>,
n_pu <span class="op">=</span> <span class="va">n_planning_units</span><span class="op">[</span><span class="fl">4</span><span class="op">]</span>,
boundary_penalty <span class="op">=</span> <span class="va">boundary_penalty_values</span><span class="op">$</span><span class="va">add_min_shortfall_objective</span><span class="op">[</span><span class="fl">3</span><span class="op">]</span><span class="op">)</span></code></pre></div>
<p><img src="solver_benchmark_files/figure-html/min_short_pu_n_4_with_high_boundary_penalty-1.png" width="672" style="display: block; margin: auto;"></p>
</div>
</div>
<div id="conclusion" class="section level1">
<h1 class="hasAnchor">
<a href="#conclusion" class="anchor" aria-hidden="true"></a>Conclusion</h1>
<p>The benchmark results demonstrate that the time required to solve a conservation planning problem can vary considerably depending on the size and complexity of the problem, and also the solver used to generate the prioritization. Indeed, some solvers (e.g. <em>Rsymphony</em> solver) may require many hours to solve a problem that other solvers (e.g. <em>CBC</em> or <em>Gurobi</em> solvers) can solve within minutes. Broadly speaking, we recommend using the <em>Gurobi</em> and <em>IBM CPLEX</em> solvers where possible. This is because they often have the best performance. Although academics can obtain a special license to use these solvers at no cost, conservation planners working in governmental or non-governmental organizations may not have access to these solvers. In such cases, we recommend using the <em>CBC</em> solver because it generally has better performance than the other open source solvers (i.e. the <em>Rsymphony</em> and <em>lpsymphony</em> solvers). Since the <em>CBC</em> solver did not always have better performance than the other open source solvers, we recommend trying the <em>lpsymphony</em> solver if the <em>CBC</em> solver is taking a long time to solve a particular problem.</p>
</div>
</div>
<div class="col-md-3 hidden-xs hidden-sm" id="pkgdown-sidebar">
<nav id="toc" data-toggle="toc"><h2 data-toc-skip>Contents</h2>
</nav>
</div>
</div>
<footer><div class="copyright">
<p></p>
<p>Developed by <a href="https://jeffrey-hanson.com" class="external-link external-link">Jeffrey O Hanson</a>, <a href="http://www.richard-schuster.com" class="external-link external-link">Richard Schuster</a>, Nina Morrell, <a href="http://strimas.com" class="external-link external-link">Matthew Strimas-Mackey</a>, Brandon P M Edwards, Matthew E Watts, <a href="https://arcese.forestry.ubc.ca" class="external-link external-link">Peter Arcese</a>, <a href="https://josephrbennett.wordpress.com" class="external-link external-link">Joseph Bennett</a>, <a href="http://www.possinghamlab.org" class="external-link external-link">Hugh P Possingham</a>.</p>
</div>
<div class="pkgdown">
<p></p>
<p>Site built with <a href="https://pkgdown.r-lib.org/" class="external-link external-link">pkgdown</a> 1.6.1.9001.</p>
</div>
</footer>
</div>
<script src="https://cdnjs.cloudflare.com/ajax/libs/docsearch.js/2.6.1/docsearch.min.js" integrity="sha256-GKvGqXDznoRYHCwKXGnuchvKSwmx9SRMrZOTh2g4Sb0=" crossorigin="anonymous"></script><script>
docsearch({
apiKey: '486efa122ea6783724263412c5f28ab3',
indexName: 'prioritizr',
inputSelector: 'input#search-input.form-control',
transformData: function(hits) {
return hits.map(function (hit) {
hit.url = updateHitURL(hit);
return hit;
});
}
});
</script>
</body>
</html>