Repository navigation
For data sets with many segments (>10000, say), model fitting can take a very very long time. Waiting for these models to fit (or fail to fit) can be a time-consuming (and perhaps frustrating process). In dsm we can use the bam engine, based on routines from mgcv.
bam uses efficient numerical methods to reduce computational time, these methods can be paired with parallelisation (fitting the model using more than one processor, or even computer), along with other "tricks" in order to reduce frustration. These speed-ups do not come for free, and bam is certainly not suitable for all modelling scenarios.
This page proceeds to first give some situations in which this approach will be useful (and some when it's not), then gives code for how to use bam, finally some technical details on how the routine works.
Parallelising computation using bam is useful in some situations but not in others. Here are some "plus" points to consider:
- parallelisation -- most modern computers have two or more cores, R usually only uses one of them. Using more than one core can improve performance massively.
- reduced memory footprint -- the technique used to speed up the computation reduces the memory usage, making models that might not ordinarily fit into memory feasible to compute
- fit large models -- a (representative) subset of the data can be used to find starting values for the rest of the data
There are, of course, some disadvantages of this approach:
-
only works with "information rich" models --
bamdoesn't work well if there is not much information in your model. For example, if you have a lot of zeros in your model, then this approach might not work. -
no estimation of negative binomial parameters -- only fixed
thetamodels can be estimated. -
no extra shrinkage -- not possible to use the
select=TRUEoption. - no GAMMs -- no models based on GAMMs (structured random effects, correlation structures).
With the above in mind, I'll show an example of how to use bam and a few different options that might be useful.
This section shows some examples of bam usage, going from the simplest to more complex situations. For simplicity's sake we use a pseudo-example, rather than actual data. The base (pseudo-)model we will work from is:
b <- dsm(N~s(x,y)+s(depth),ddf.obj,segment.data, observation.data,
engine="gam",method="REML")
I include the engine="gam" above (even though it is unnecessary, to illustrate changes in the model specification.
To simply use the algorithms which make bam faster (and perhaps to see if your data is suitable) once can use the bam engine
b.bam <- dsm(N~s(x,y)+s(depth),ddf.obj,segment.data, observation.data,
engine="bam",method="fREML")
All we have done here is replaced the engine argument and changed the method (used to estimate the degree of smoothness in the model) to "fREML" (fast REML).
The above should run faster in most situations. See below for why.
To have the model fit using more than one processor (or computer) we can harness the parallel package.
library(parallel)
One can then have R work out how many cores there are on the machine you are using:
detectCores()
Depending on what else you (and others) want to do on your computer while your model fits you can use anywhere from 2 to all of the cores. Note that on some machines this information may be reported inaccurately (in terms of how R will use the cores) so it may take some experimentation to get this right. Starting with using 2 cores seems sensible.
Before asking bam to use the core we need to first set them up for use, this is done via makeCluster, for example
cl <- makeCluster(2)
allocates 2 cores for use and assigns the information needed to use them to the object cl. To get dsm to use these clusters we modify the above to:
b.bam <- dsm(N~s(x,y)+s(depth),ddf.obj,segment.data, observation.data,
engine="bam",method="fREML",cluster=cl)
One can check this is working (and the cores are being used) by checking top in Linux, Activity Monitor in Mac OS or Task Manager in Windows, where you will see more than one process called "R".