Adding a Link object #14
Conversation
|
An alternative would be to not use different link functions but define additional likelihood functions (i.e., one for Poisson with the |
|
I thought that it would be cooler to be able to input really any kind of link! One could even think about passing a constrained NN (paper idea here 😆 ) |
There was a problem hiding this comment.
Is it possible to keep links separated from the likelihoods? That is, not have it part of every likelihood? We could think of this as an optional step between the GP outputs and likelihood function.
We could also define LinkedLikelihood similar to transforms in KernelFunctions.jl such that it is just another likelihood.
struct LinkedLikelihood{T}
link::Link
lik::T
end
(ll::LinkedLikelihood)(f::Real) = ll.lik(ll.link(f))
(ll::LinkedLikelihood)(fs::AbstractVector{<:Real}) = ll.lik(ll.link.(fs))or something like
struct LinkedLikelihood{T1,T2}
link::T1
lik::T2
end
(ll::LinkedLikelihood)(f::Real) = ll.lik(ll.link(f))
(ll::LinkedLikelihood)(fs::AbstractVector{<:Real}) = ll.lik(ll.link.(fs))I am supportive of the second approach as both links and likelihoods are essentially functions. And the second approach provides the maximum amount of flexibility.
I don't think it is possible to separate links and likelihoods. Unlike in |
Okay that makes sense. 🙂 But we should always have a sensible default link in such cases. |
|
Actually one change that could be need is the naming. Should it be |
|
As an alternative (I feel I am rewriting existing code) we could use |
|
IMO we shouldn't use Bijectors.jl, mainly because it's a very heavy dependency and, more importantly, since my impression is that its design is very much focused on Turing and AdvancedHMC. In particular, all inputs and outputs have to be arrays and have to be of the same dimension and size. IMO this is quite limiting in more general applications where you want to map a submanifold of lower dimension (e.g. matrices with a special structure or the simplex - there are some open issues but I don't think the design will be changed in the near future since it would be very breaking for Turing). However, I also think it would be good to use existing functionality in other packages, if possible. Maybe TransformVariables could be helpful? IIRC it supports more flexible mappings and input and output types and would be a lighter dependency. |
|
I had a look at TransformVariables and it looks great indeed, only I think we would need to use the low level API the |
Depends on which way around you define the bijector. In the statistics community, the link function is well defined to be the object that satisfies link(y) = f, where y is your observations/likelihood parameters, and f is your latent (generally assumed linear) model. So y = invlink(f). In implementing likelihoods for our GP models we generally care about the f -> y direction only, so we define the invlink explicitly. |
|
@st-- Thanks for the explanation. I was indeed not familiar with the statistics nomenclature. The names have been changed accordingly already. One can switch from one representation to another (when possible) with |
|
Can I get a fresh review on this @devmotion @st-- @willtebbutt ? Happy to get it back from the deads |
Codecov Report
@@ Coverage Diff @@
## master #14 +/- ##
===========================================
+ Coverage 69.56% 82.53% +12.97%
===========================================
Files 6 7 +1
Lines 23 63 +40
===========================================
+ Hits 16 52 +36
- Misses 7 11 +4
Continue to review full report at Codecov.
|
|
@st-- @willtebbutt |
We need to support different type of mapping when a constraint is applied on the GP, e.g. for Poisson the given standard was
exp(f)but there are more existing options :f^2, ors * logistic(f)The convention is to call the connection between the parameter(s)
mneeded for the likelihood and the input(s)fthe inverse link. Cause traditionally the link would be the function to go frommtof. See for example : http://web.pdx.edu/~newsomj/mvclass/ho_link.pdfThis PR aims at introducing this with
AbstractLinktype.Some likelihoods need an inverse link, for example
BernoulliLikelihoodorPoissonLikelihood. So it makes sense to have these likelihoods contain the inverse linkinvlinkand parametrize it. This allows furthermore to differentiate different overloads later and for people to use their own transformations. For example my augmentation stuff only works for theLogitLinkbut not theProbitLink.For a small collection of
AbstractLinks I defined justapply(link, x)(which is called by(link::LinkType)(x). and alsoBase.invwhich returns the existing inverse link if it exists (similarly to Bijectors).I already did a basic implementation but I realised a lot of it is overlapping with
Bijectors.jland we could consider using them. However this is also a heavier dependency ? I thinkNNlibis pretty heavy right?