Add location parameters for coefficient specific priors

[egillax]

Hi, @msuchard,

Me and @lhjohn are working on transfer learning at ErasmusMC. One of the methods we are interested in evaluation is for linear models and is called the prior method. It involves first training a model on a large source database as usual. Then for a target database which would not have as much data we train a regularized regression model, but instead of pushing the coefficients toward zero we push them towards the coefficients of the source model. I believe this is equivalent to having coefficient specific priors with the location parameter as the coefficient from the source model.

I can already deduce from the Cyclops source code (correct me if I'm wrong) that it's already possible to have coefficient specific priors with different variances. My question is, do you think it would be complicated to add location/scale information for the priors as well? This is something me and @lhjohn are interested in implementing, for example for the laplace prior, if it doesn't require extensive changes to the code and if we can get some guidance.

Although we are mostly thinking about prediction I think this can benefit the community everywhere where we are working in the small data regime.

Regards,
Egill Fridgeirsson

[msuchard]

Hi @egillax and @lhjohn,

This should be possible without modification to the code-base, although I have yet to check it.

Let's say you want to regularize g(Y) = X \beta + ||\beta - \mu||^p where X is your covariate with regression coefficient \beta and \mu is the fixed location. You should be able to re-parameterize this as g(Y) = X \mu + X \beta + ||\beta||^p. In Cyclops terms, this means adding an extra covariate (a copy of X) to your regression model, setting its regression coefficient to \mu and fixing that coefficient during model fitting. The "setting" and "fixing" parts are possible via options to fitCyclopsModel().

Please give this a try. If you run into issues, I can carve out some time to put together a small example.

[lhjohn]

Hi @msuchard,

@egillax and I put together a script following your method.

• Could you have a look at the script to verify, if it follows your method accurately?
• What is the default value we provide for startingCoefficients, if we do not actually want to use a starting value for that coefficient?
• Below script only seems to give non-zero coefficients when variance = 0.1. For variance = 0.01, which is the default in the PLP package, it shrinks all coefficients almost always to 0.

Result of script below is:

> sourceCoef
(Intercept)           1           2           3           4           5           6           7           8           9          10 
-1.51838416 -0.03694814  0.00000000  0.00000000 -0.45813812 -0.11476308  0.00000000  0.55423546 -0.01051334  0.00000000  0.00000000 

> targetCoef
(Intercept)         -10          -9          -8          -7          -6          -5          -4          -3          -2          -1 
-0.94449366  0.00000000  0.00000000 -0.01051334  0.55423546  0.00000000 -0.11476308 -0.45813812  0.00000000  0.00000000 -0.03694814 
          1           2           3           4           5           6           7           8           9          10 
 0.29588724  0.00000000  0.00000000  0.33867037  0.25897893  0.58839854 -0.10966460 -0.17068622  0.00000000  0.00000000 

library(Cyclops)

###################
# source database #
###################
set.seed(1)

# number of covariates
nCovars <- 10

# simulate the larger source data
sourceSimulant <- simulateCyclopsData(nstrata = 1,
                                      nrows = 10000,
                                      ncovars = nCovars,
                                      model = "logistic")

sourceData <- convertToCyclopsData(sourceSimulant$outcomes,
                                   sourceSimulant$covariates,
                                   modelType = "lr")

# do not fix the intercept or any coefficients for the source fit
sourceFixedCoef <- c(F,
                     rep(F, nCovars))

# fit source model
sourceFit <- fitCyclopsModel(sourceData,
                             prior = createPrior("laplace", variance = 0.1),
                             forceNewObject = TRUE,
                             fixedCoefficients = sourceFixedCoef)

# store source coefficients
sourceCoef <- coef(sourceFit)

# print source coefficients
sourceCoef

###################
# target database #
###################
set.seed(2)

# simulate the smaller target data
targetSimulant <- simulateCyclopsData(nstrata = 1,
                                      nrows = 5000,
                                      ncovars = 10,
                                      model = "logistic")

# keep a copy of the target covariates
targetCovariates <- targetSimulant$covariates

# in this copy rename covariates to have unique names
targetCovariates$covariateId <- targetSimulant$covariates$covariateId*-1

# combine original covariates with renamed covariates
combinedCovariates <- rbind(targetCovariates,
                            targetSimulant$covariates)

# create the target data from the combined covariates and outcomes
targetData <- convertToCyclopsData(targetSimulant$outcomes,
                                   combinedCovariates,
                                   modelType = "lr")

# check on coefficient names
targetData$coefficientNames

# fix only the source covariates, which are the non-negative ones
# targetFixedCoef <- c(FALSE,
#                      rep(TRUE, nCovars),
#                      rep(FALSE, nCovars))

targetFixedCoef <- c(FALSE,
                     rep(FALSE, nCovars),
                     rep(TRUE, nCovars))

# initialize a vector for starting coefficients (to zero?)
targetStartingCoef <- rep(0, 21)

# assign source coefficients as target coefficients
# probably does not matter, but also revert source coefficient, since they were sorted above
# targetStartingCoef[2:11] <- rev(sourceCoef[2:11])
targetStartingCoef[12:21] <- sourceCoef[2:11]

targetStartingCoef# fit the target model
targetFit <- fitCyclopsModel(targetData,
                             prior = createPrior("laplace", variance = 0.1),
                             forceNewObject = TRUE,
                             fixedCoefficients = targetFixedCoef,
                             startingCoefficients = targetStartingCoef)

# store target coefficients
targetCoef <- coef(targetFit)

# print target coefficients
targetCoef

EDIT: Clean up script.