HW5 MLE and OLS

[isabelalmazan]

Hi Jeremy,

I got up to Q6 on the MLE and OLS section of the homework, but the results I'm getting from the MLE are really far off from the estimates we started off with:

[isabelalmazan]

Ok so I don't really know what I did because I changed multiple things, but I now have
mu_MLE= 11.0 sig_MLE= 0.5 as my output, so disregard the above issue.
Thanks

[jmbejara]

Hi @isabelalmazan . It looks like you are getting a negative sigma. sigma should be positive---however, the optimizer doesn't know this. Add in a constraint via the bounds argument to keep sigma greater than zero.

[isabelalmazan]

Thanks, I set bounds so that neither mu nor sigma can be negative.

On Q7, how can i show the actual hessian matrix rather than just getting <2x2 LbfgsInvHessProduct with dtype=float64>?

And should I be worried that the standard error of the sigma estimate is Nan?

[jmbejara]

When you use the constrained optimizer, it gives you the hessian in a "sparse" format. You need to write results.hess_inv.todense() to see the matrix.

However, when I looked at this on mine, it looks like using the unconstrained optimizer gives a better result. (Graphically the fit looks better.) One way to get the unconstrained optimizer to work is to put an absolute value sign on sigma. I did so like this:

def loglik(params, data):
    """Compute log-likelihood of parameters
    
    data should be provided in `args`. 
    The data should be a pandas.Series or a numpy array
    """
    mu, sigma = params
    f = lambda x: (1/(x * np.abs(sigma) * np.sqrt(2 * np.pi)) * 
                   np.exp(-(np.log(x) - mu)**2/(2 * sigma**2)))
#     f = lambda x: (1/(x * sigma * np.sqrt(2 * np.pi)) * 
#                    np.exp(-(np.log(x) - mu)**2/(2 * sigma**2)))
    f_array = f(data)
    ll = np.sum(np.log(f_array))
    return ll

[isabelalmazan]

I added the absolute value, but I'm still getting nan for the standard error of my sigma estimate.
I don't understand where I'm going wrong with my function that is making these red messages pop up even when the code is going through. I compared my function with yours, and everything is the same; I checked that none of the x are 0 and that the sigma in the denominator is positive. (Removing the constraints made the standard error for mu also nan so I put them back.)

Here is my code from Q3 and Q4 (m and s stand for mu and sigma):

def PDF(x, m, s):
    return (1/(x * np.abs(s) * (math.sqrt(2 * np.pi)))) * math.exp(-((math.log(x) - m)**2)/(2 * (s**2)))

def log_like(x, m, s):
    pdf_vals = PDF(x, m, s)
    ln_pdf_vals = np.log(pdf_vals)
    log_lik_val = ln_pdf_vals.sum()
    return log_lik_val

Thanks for your help!

[jmbejara]

I don't know why you're getting a negative value for vcv_mle[1,1]. That don't think that that should be happening.

What happens if you remove the constraints? This is what I get. In the first method, I don't use constraints---but I have np.abs on sigma. In the second, I have np.abs, but I also have the constraints.

Here are my results for Q7:

[bariscangoc]

Hi Jeremy,

My numbers are pretty off. What can be the possible error? Or is it normal?

[jmbejara]

It seems just a little off to me. I'm not sure what could be the cause. Does the estimate for mu and sigma seem ok?

[bariscangoc]

My estimates are these values mu_MLE= 11.359793608832375 sig_MLE= 0.2040956953111885.
And this is the fit.

It looks like a good fit. I just check my pdf. It also integrates to 1.

[jmbejara]

Though, I understand that there might be some small difference due to possibly different optimizers. I'll make sure @PhilipCaoChicago knows

[jmbejara]

@bariscangoc . Looks close, but not the same as mine. It's probably close enough that I wouldn't worry about it. What are your starting values for the optimizer?

[bariscangoc]

I used
mu_init = 11
sig_init = 0.5
Should I use a different value?

[jmbejara]

Looks good to me. I just wanted to make sure that you weren't using something like mu=11.3 and sigma = 0.2

[bariscangoc]

This is our MLE estimation. I think the problem is here. When I add constraint, I get weird estimates such as mu = 150.

[jmbejara]

I've noticed as well that the constraints make my results a little weird.

BTW, I see log_lik_trunclognorm in your code. Are you using a truncated lognormal? You should be using a standard, un-truncated lognormal distribution.

[mcatalay]

aren't we truncating the the lognormal at 150k so that the pdf to 150k is equal to 1?

[jmbejara]

No. However, I see the confusion. In Q3 I said

Plot the PDF of the log-normal distribution f(x|μ,σ)f(x|μ,σ) for values μ=11.0μ=11.0 and σ=0.5σ=0.5 over the range x∈(0,150,000)x∈(0,150,000) .

However, all I meant is that the limits of the plot should be 0 to 150,000. I meant for the distribution to be a standard lognormal. Sorry for the confusing wording.

[mcatalay]

Thanks for the clarification!

[bariscangoc]

Thanks, I fixed it!

[isabelalmazan]

When I remove the constraints, I get a negative sigma estimator again:

But both with and without the constraints, my Hessian matrix has very different values than yours...

[jmbejara]

@isabelalmazan . I don't think it will change anything, but you might change math.sqrt, math.exp, and math.log to np.sqrt, np.exp and np.log.

[jmbejara]

@isabelalmazan What if you use my code directly? (See #53 (comment))

You can then see the rest of the code here: #53 (comment)

The function nloglik just changes the sign of loglik

(I just realized that I had basically posted all of it here anyway. :) )