Truncated Poisson Distribution probs not adding up to one

[azev77]

using Distributions
λ = 2.0; #Poisson parameter
zp = 0.4; #zero prob. 
dpoi = Poisson(λ)
dzip = MixtureModel([Dirac(0.0), dpoi], [zp, 1.0-zp])  #ZeroInflated ZIP
dztp = Truncated(dpoi, 1, Inf)                         #ZeroTruncated ZTP 
dzmp = MixtureModel([Dirac(0.0), dztp], [zp,1.0-zp])   #ZeroModified ZMP
####
sum(pdf(dpoi, 0:1000)) #1.0
sum(pdf(dzip, 0:1000)) #1.0
sum(pdf(dzmp, 0:1000)) #1.2734073051133634
sum(pdf(dztp, 0:1000)) #1.4556788418556055

Shouldn’t the Truncated Poisson distribution be a distribution whose probs add up to one?

[andreasnoack]

It should. I'm wondering if the insupport function disagrees with the function that computes the probability mass outside the support.

[mschauer]

julia> dump(tpoi)
Truncated{Poisson{Float64}, Discrete, Float64}
  untruncated: Poisson{Float64}
    λ: Float64 2.0
  lower: Float64 1.0
  upper: Float64 Inf
  lcdf: Float64 0.4060058497098381
  ucdf: Float64 1.0
  tp: Float64 0.5939941502901619
  logtp: Float64 -0.5208858076643444

julia> 1-cdf(poi, 1)
0.5939941502901619

julia> 1-cdf(poi, 0)
0.8646647167633873

[mschauer]

Note that in https://en.wikipedia.org/wiki/Truncated_distribution a needs to be removed from the support for the denominator to be given by F(b) - F(a).

[azev77]

So this might be a discrete v continuous thing.
w/ discrete you need the smallest element in the support bigger than ‘a’

[mschauer]

It's cleaner to use cdf(poi, a) - pdf(poi, a) for discrete random variables (I suppose we want the user to provide [a, b] inclusive?). Just annoying that we cannot explicitly ask for P(X = b) with pdf without getting a wrong answer for continuous variables.

[azev77]

Quick recap:

using Distributions
λ = 2.0; #Poisson parameter
dpoi = Poisson(λ)
dztp = Truncated(dpoi, 1, Inf) #ZeroTruncated ZTP 
sum(pdf(dpoi, 0:1000)) # 1.0
sum(pdf(dztp, 0:1000)) # 1.4556788418556055

lcdf = cdf(dpoi, 1)     # 0.4060058497098381
ucdf = cdf(dpoi, 10000) # 1.0
tp = ucdf - lcdf        # 0.5939941502901619 
ztp_pdf(x;tp=tp) = 1 ≤ x ≤ Inf ?  pdf(dpoi, x) / tp : 0.0
sum(ztp_pdf, 0:1000) # 1.455678841855605
ztp_pdf(x;tp=tp) = 1 < x ≤ Inf ?  pdf(dpoi, x) / tp : 0.0
sum(ztp_pdf, 0:1000) # 1.0

As mentioned above, the problem is the current support of truncated pdf includes the lower bound [lb,ub]. For continuous RVs this is innocuous P(X=lb)=0. For discrete RVs this is a problem...

What if we rewrite:

Distributions.jl/src/truncate.jl

Line 95 in a746130

if d.lower <= x <= d.upper

to: if d.lower < x <= d.upper

[devmotion]

To me it seems more natural to perform the correction suggested in #1328 (comment): just use lcdf = cdf(d, lower) - pdf(d, lower) for discrete distributions. IMO it would be quite confusing to not include the lower bound in the support.

[azev77]

confusing to not include the lower bound in the support

Why confusing? The pdf for a truncated distribution is defined to be:
tpdf(d, lb, ub, x) := lb < x ≤ ub ? pdf(d, x) / (cdf(d, ub) - cdf(d, lb)) : 0.0

This works for continuous & discrete RV.

Currently they do:
tpdf(d, lb, ub, x) := lb ≤ x ≤ ub ? pdf(d, x) / (cdf(d, ub) - cdf(d, lb)) : 0.0
which is still correct for continuous RVs (P(X=lb)=0), but not for discrete RVs...

[devmotion]

It is only excluded if you choose this specific definition 🤷 One could also use tpdf(d, lb, ub, x) = lb <= x <= ub ? pdf(d, x) / \int_{[lb, ub]} pdf(d, x) mu0(dx) : 0.0 where mu0 is the reference measure (Lebesgue for continuous distributions and counting measure for discrete distributions). This works in all cases and both bounds are included in the support.

I would argue that it is a bit strange to write truncated(Poisson(), 3, 8) if you want a truncated Poisson distribution with support {4, 5, 6, 7, 8}. Also it feels more consistent with the continuous case which is correct and where it is included currently.

[azev77]

Btw, I really like the idea of using Lebesgue integrals w/r/t Lebesgue/counting/etc measures.
@cscherrer MeasureTheory.jl comes to mind.

[mschauer]

It's exactly the point of MeasureTheory getting that right.