RFC: general low effort performance tweaks

[KristofferC]

This PR fixes all the low hanging performance tweaks I could find.

This includes:

• @simd on pertinent loops
• Using generic functions that dispatches to optimized versions if possible, for example, using scale! instead of for loops.
• Avoids creating copies in many places that includes an operation with a column of a matrix. Uses a slice instead of a copy.
• Preallocates temporary vectors in some places.

I currently only have access to a laptop where I run Ubuntu on a VM so the following benchmarks are not too interesting: https://docs.google.com/spreadsheets/d/1oHa7DcNFRy7s176yTWcNuLcuAsMvZ6qDlWzlOKiZpRY/edit?usp=sharing

Either someone can rerun them or I will do it in a few days when I get access to my beefy machine.

[mlubin]

Thanks!

[KristofferC]

Image below created with:

library("ggplot2")

results_m <- read.csv("benchmarks/results_master.tsv", sep = "\t")
results_o <- read.csv("benchmarks/results_optim.tsv", sep = "\t")
results_diff = (results_o - results_m) /results_m
results_diff$Problem = results_m$Problem
results_diff$Algorithm = results_m$Algorithm
n <- length(unique(results$Problem))

ggplot(results_diff, aes(x = Algorithm, y = AverageRunTimeInMilliseconds, fill = Algorithm)) +
  geom_bar() +
  coord_flip() +
  theme(legend.position = "none") +
  xlab("") +
  ylab("Relative difference in runtime") +
  ggtitle("Speed of Optimization Algorithms") +
  facet_grid(Problem ~ .)
ggsave("benchmarks/graphs/run_times.png")

I'm not surprised some of the small problems run slower because from my experience @simd can be slower on small vectors. Question is how much we care if we lose some percentage on problems with 2 variables.

[johnmyleswhite]

This looks good by me. Thanks for doing it!

[KristofferC]

Thanks for merge.

[pkofod]

Nice improvements, especially for Nelder-Mead!

[multidis]

I keep getting WARNING: could not attach metadata for @simd loop. This is on Linux with Julia version

Julia Version 0.4.6
Commit 2e358ce (2016-06-19 17:16 UTC)
Platform Info:
System: Linux (x86_64-linux-gnu)
CPU: Intel(R) Core(TM) i3 CPU M 380 @ 2.53GHz
WORD_SIZE: 64
BLAS: libopenblas (NO_LAPACK NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Nehalem)
LAPACK: liblapack.so.3
LIBM: libopenlibm
LLVM: libLLVM-3.3

The same happens on a different CPU architecture (AMD) as well. Is this warning something to care about performance-wise?

[blakejohnson]

@KristofferC I'm obviously late to the party here, but did you do any benchmarking of the levenberg-marquardt changes?

[KristofferC]

I don't think so no, but it should be easy to check this commit vs the one before it to get an estimate.