Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

L1regularized rebased #465

Closed
wants to merge 140 commits into from

2 participants

@langmore

No description provided.

langmore and others added some commits
@langmore langmore Made some test changes 9cc156a
@langmore langmore Did not use -a in last commit 97b74a5
@langmore langmore things work...in the middle of changes 41d8d78
@langmore langmore just added trim_params cc56030
@langmore langmore L1 regularization works. Erratic results when we use small regulariza…
…tion parameter and have variables that dont really affect the target
d6cfc4a
@langmore langmore Got rid of xxx_fit_lasso 2efa42f
@langmore langmore Added l1.get_bic 8468e3c
@langmore langmore Added utility bic/aic functions 6f6ac06
@langmore langmore added l1_test, which used to be statsmodels_test 4d1a72b
@langmore langmore Got demo ready and removed epsilon as a parameter ee4bf5c
@langmore langmore added README_l1.txt 948096a
@langmore langmore Combined READMEs 77f4172
@langmore langmore Added note on l1_demo/demo.py to the README.txt 54b04b4
@langmore langmore Slight change to README.txt 43393f5
@langmore langmore Added explanation of the conversion to constrained problem 71eb0f1
@langmore langmore Modified explanation of the conversion to constrained problem c8d3c80
@langmore langmore See for the bse problem: Started work on the l1ModelResults 3685176
@langmore langmore Added a new class: discrete_model.py:L1MultinomialResults(Multinomial…
…Results). This gives special formatting to results for the MNLogit model (when trained with method==l1)
658edc2
@langmore langmore Removed unnecessary functions from l1.py and updated demo.py 31b7e6e
@langmore langmore alpha can now be passed in as a vector. Updated demo ce663cc
@langmore langmore fixed a small typo in fprime_ieqcons. This typo had no effect c81ed70
@langmore langmore Added in cvxopt_cp fitting. Not fully integrated yet 268bcef
@langmore langmore Broke up l1 optimization functions. Fixed bug in cvxopt and re-wrote it 7c9958f
@langmore langmore Re-wrote l1_slsqp to look like l1_cvxopt a51286e
@langmore langmore Re-wrote demo.py to allow for use of noise 39bd2ca
@langmore langmore Added docstring to demo.py f62747f
@langmore langmore added more docstring to demo.py d07117e
@langmore langmore Updated README.txt 5f0e87c
@langmore langmore Updated demo. Realized long correlations made the singular KKT matrix…
… error happen
4037397
@langmore langmore Erased a TODO in demo.py b7397b1
@langmore langmore Revised docstring in demo 56e6e8e
@langmore langmore Made demo much nicer 28f0382
@langmore langmore Changed the method name for the slsqp fit. Updated and improved the demo 490d683
@langmore langmore Updated docstring in the demo 264f782
@langmore langmore Corrected demo to say RMS rather than MS f7479ca
@langmore langmore Protected the import of cvxopt 86c2452
@langmore langmore Fixed demo: Previously demo used wrong number of targets 06be00e
@langmore langmore demo now has a CLI 2587a64
@langmore langmore Unraveled params to allow for different shapes (n,1) or (n,) to be used cf9931a
@langmore langmore We now return x.ravel() since this was expected bac32ec
@langmore langmore Added probit to demo.py 9e30846
@langmore langmore Updated README.txt ff6c37f
@langmore langmore Bugfix in demo. Now probit gets its endog variables correctly bf71bcf
@langmore langmore Cleaned up demo.py 564bdfd
@langmore langmore Updated docs in demo cc2b581
@langmore langmore Added L1BinaryResults. Now probit and logit results are handled like …
…mnlogit, viz. nan retured for parameters that are zero (hence more than just covariance is important to statistics)
a9c17af
@langmore langmore Got rid of unused arguments in model.py:LikelihoodModel.fit() a77065c
@langmore langmore Got rid of bad line in demo 5bcf400
@langmore langmore Former code assumed correct input of alpha...now convert to numpy arr…
…ay of correct size
e3050e5
@langmore langmore added short_demo.py, which would work well in the html docs 3d2ef28
@langmore langmore Corrected docstring 0bfe60b
@langmore langmore Modified short_demo to properly calculate and display un-regularized …
…results
e2ac2f5
@langmore langmore Added sklearn_compare to compare results with sklearn 70c4aff
@langmore langmore Got sklearn_compare.py working with the Spector dataset ecc7120
@langmore langmore Added capability to handle anes96 datasets 10eaddd
@langmore langmore Made sklearn_compare work from command line by putting plt.show() in c754820
@langmore langmore Updated README.txt fef68a8
@langmore langmore Made code PEP8 compliant (did not finish the demo code...but who cares) ec6cb16
@langmore langmore Started unit tests 4a8ff0e
@langmore langmore Protected import of cvxopt and subclassed test a031eca
@langmore langmore Changed method of protection for cvxopt 8417ee1
@langmore langmore Added MNLogit example to short_demo.py 83edfdd
@langmore langmore Made trim_params=True in short_demo.py 74ecd51
@langmore langmore Made trim_params=True the default f79cb0a
@langmore langmore added test to check if ML and l1 results match. Corrected cvxopt test 6b936d6
@josef-pkt josef-pkt compatibility numpy 1.5 (ravel order) 9951581
@langmore langmore Merged ravel call change and changed ravel call in l1_cvxopt.py aa3ee4a
@langmore langmore Fixed typo in discrete_model 1f6bedf
@langmore langmore Added auto capability to do_trim_params and made this the default f4217a6
@langmore langmore Derivative should be less than or equal to alpha or else something is…
… wrong
bc3d8cd
@langmore langmore Return trimmed as a boolean array now 9cf21d3
@langmore langmore Now trimmed is used to get cov_nz_params, the covariance of the nonze…
…ro params
4520b1c
@langmore langmore Changed default magic_tol 6105eaa
@langmore langmore Now correct cov_params implementation works with MNLogit too 2efa6f4
@langmore langmore Extended auto param trimming to l1_cvxopt_cp 7e85c5e
@langmore langmore Now alpha is coverted to array. This should have been done earlier 05f9d29
@langmore langmore Added more tests for L1 Logit 5aae374
@langmore langmore If all parames are zerod, then do not store the Hessian or attempt to…
… invert it
c4a70b7
@langmore langmore Added comments 26fe0b4
@langmore langmore Fixed an issue that happened when trim_params==False (previously we d…
…idn't return trimmed)
f49a9e7
@langmore langmore Added capability to use nm (scipy.optimize.fmin) to do things. It see…
…ms to suck
2685d0e
@langmore langmore Made some options explicit in tests aa44c81
@langmore langmore Removed references to l1_nm 91cf3db
@langmore langmore Refactored to address comments on WIP pull request d22d3cd
@langmore langmore Implemented the fit_regularized paradigm a0c792a
@langmore langmore Updated demo.py and README.txt for fit_regularized ba2339a
@langmore langmore moved files to base 7737f97
@langmore langmore Increased default accuracy in slsqp and removed a comment a766a6e
@langmore langmore Added demo showing regularization path 8d28f2d
@langmore langmore Added QC_verbose option that prints out a more detailed QC report 3545288
@langmore langmore Added QC_verbose option to slsqp 7982a11
@langmore langmore Modified short_demo to work without warnings and to use QC_verbose 112bce3
@langmore langmore Replaced the reduced cov_params matrix with a full one containing nan 3c69f0e
@langmore langmore Added nan_dot function 35223d6
@langmore langmore to save model.t_test_check_valid 0e02648
@langmore langmore Changed t_test to account for nan values in cov_params 53c94ad
@josef-pkt josef-pkt fix errors from merge conflict during rebase cf0017b
@langmore langmore checked out short_demo from master 1abc28b
@langmore langmore Uncommented lines in short_demo cfd8ad8
@langmore langmore Bugfix: Added auto generation of correct start_params for MNLogit.fit…
…_regularized
d254565
@langmore langmore Updated the demos 733107d
@langmore langmore Updated cvxopt and moved something around in slsqp a421d7a
@langmore langmore Added new tests dcb6011
@langmore langmore Cleaned up code aebbfa2
@langmore langmore Added test_tests to the alpha 0 test 7e5c5fb
@langmore langmore Added TODO_ian.txt 1554b0d
@langmore

TODO:

Decide whether to detach fit_regularized from fit, or use "callback idea", or leave extra arguments as is.

Docs

DONE:

Tests to add to CheckLikelihoodModel (here we test new L1 results versus stored L1 results, Logit, MNLogit, Probit):
t_tests (done)
f_tests (done)
aic (done)
bic (done)
cov_params (done)

Make TestLogitL1AlphaZero extend CheckBinaryResults. (I made it mimic much of CheckBinaryResults)

Understand and check get_margeff() (done)

@josef-pkt
Owner

the margins, get_margeff() are not very easy reading material.

The main thing is to try out some examples to see whether the zeros and nans create problems.
For the rest, nonzero parameters with valid bse and cov_params, I don't see any theoretical difference.

@langmore

get_margeff().summary() prints with no problem. We get lots of nan and zeros.

Logit Marginal Effects

Dep. Variable: y
Method: dydx
At: overall

dy/dx std err z P>|z| [95.0% Conf. Int.]

x1 0 nan nan 0.000 nan nan
x2 -0.0084 nan nan 0.000 nan nan
x3 0.1495 nan nan 0.000 nan nan

@langmore
@josef-pkt
Owner

I think Skipper used mostly Stata as reference.

http://www.ats.ucla.edu/stat/stata/faq/margins_mlogcatcon.htm
http://www.nd.edu/~rwilliam/stats/Margins01.pdf
http://www.stata.com/help.cgi?margins

Since I have Stata I also relied mostly on their manual.

The main point:
In linear models the beta/params coefficient tell you (almost) all about the effect of of an explanatory variable on the dependent variable, derivative: d endog / d exog_i = beta_i

In nonlinear models (and discrete models that have a nonlinear relationship between exog and endog), there is no simple relationship. As a consequence we get all these different derivatives dydx, ..., and it depends on where ("at") we evaluate them.

(For example, if y is choosing a brand, and exog is a price, what is the marginal effect of an increase in relative price on the shares/choice of consumers to buy one specific brand.)

Details are all Skippers

@langmore
@josef-pkt
Owner

I don't see a reason why we shouldn't get standard errors on the margins for non-zero parameters. They are standard errors for only the non-zero, non-nan parameters. It might be a similar problem as the t_test.

But as we get nans and not wrong number, I would consider this as a not so urgent enhancement, which might require some rethinking of the structure of this.

I'm working on getting some documentation improvements and pull requests into master, then I will come back to L1.

@langmore
@langmore
@josef-pkt
Owner

Don't merge master into your branch.

What would be possible is in your branch (or better a new branch)
git rebase origin/master
assuming origin points to statsmodels/statsmodels.

However, I don't think it's necessary. I think that all changes in master are unrelated to this, and we only need a rebase just before merging.

@josef-pkt josef-pkt commented on the diff
statsmodels/base/model.py
@@ -1165,15 +1188,24 @@ def t_test(self, r_matrix, q_matrix=None, cov_p=None, scale=None):
_t = _sd = None
_effect = np.dot(r_matrix, self.params)
@josef-pkt Owner

doesn't self.params have nans here? in that case it would require nan_dot

self.params contains zeros...it's only self.cov_params that has nan

@josef-pkt Owner

good, I forgot about that

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
statsmodels/base/model.py
@@ -1165,15 +1188,24 @@ def t_test(self, r_matrix, q_matrix=None, cov_p=None, scale=None):
_t = _sd = None
_effect = np.dot(r_matrix, self.params)
+ # If self.cov_params contains nan, we have to make sure this contrast
+ # is valid.
+ if np.isnan(self.cov_params().diagonal()).any():
@josef-pkt Owner

I don't know if it matters here (in terms of extra work)

In general, I would prefer setting an attribute on the results instance (in L1 fit) with self.use_nan_dot=True to automatic checking for an "exceptional" case.

TODO

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
statsmodels/base/model.py
@@ -189,6 +192,11 @@ def fit(self, start_params=None, method='newton', maxiter=100,
retall : bool
Set to True to return list of solutions at each iteration.
Available in Results object's mle_retvals attribute.
+ extra_fit_funcs : Dictionary
+ Keys are methods and values are fit functions to use. Populating
+ this dictionary has the effect of adding new methods
@josef-pkt Owner

which kind of new methods? from the docstring it's not clear what these extra_fit_funcs should do. Some information in the Notes section will be helpful if the docstring cannot explain it well enough

TODO once discussion below is resolved

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
statsmodels/base/model.py
@@ -189,6 +192,11 @@ def fit(self, start_params=None, method='newton', maxiter=100,
retall : bool
Set to True to return list of solutions at each iteration.
Available in Results object's mle_retvals attribute.
+ extra_fit_funcs : Dictionary
+ Keys are methods and values are fit functions to use. Populating
+ this dictionary has the effect of adding new methods
+ cov_params_func : Function
+ Used to compute the cov_params
@josef-pkt Owner

extra_fit_funcs and cov_params_func are similar to my callback idea.

What I don't see right now is whether in this case it's better to have them as extra arguments or have them as extra methods with explicit hasattr check.

My guess is that this will be only relevant when we have more levels in the class hierarchy and we need to hand around extra function arguments.
One disadvantage is that this is mainly an internal argument, while the signature for the fit method is targeted towards users.

TODO

An alternative is to detach fit_regularized from fit. This makes sense, since fit_regularized could grow and we don't want a bunch of patches.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@josef-pkt josef-pkt commented on the diff
statsmodels/base/model.py
@@ -1061,7 +1084,7 @@ def cov_params(self, r_matrix=None, column=None, scale=None, cov_p=None,
other = r_matrix
else:
other = np.asarray(other)
- tmp = np.dot(r_matrix, np.dot(cov_p, np.transpose(other)))
+ tmp = dot_fun(r_matrix, dot_fun(cov_p, np.transpose(other)))
@josef-pkt Owner

if I understand correctly, then the outer dot_fun could be np.dot.
maybe not, depends on how nan_dot handles a*nan with a !=0

In nan_dot, if a != 0, then a * nan = nan. So both need to be nan_dot.

@josef-pkt Owner

good, I like that version better (than the initial valueerror IIRC)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
statsmodels/discrete/discrete_model.py
((94 lines not shown))
+ maxiter = 1000
+ elif method == 'l1_cvxopt_cp':
+ maxiter = 70
+
+ ## Parameters to pass to super(...).fit()
+ # For the 'extra' parameters, pass all that are available,
+ # even if we know (at this point) we will only use one.
+ extra_fit_funcs = {'l1': fit_l1_slsqp}
+ if have_cvxopt:
+ from statsmodels.base.l1_cvxopt import fit_l1_cvxopt_cp
+ extra_fit_funcs['l1_cvxopt_cp'] = fit_l1_cvxopt_cp
+ elif method.lower() == 'l1_cvxopt_cp':
+ message = """Attempt to use l1_cvxopt_cp failed since cvxopt
+ could not be imported"""
+
+ if method in ['l1', 'l1_cvxopt_cp']:
@josef-pkt Owner

these are the only two methods that are allowed, not necessary
we should raise a valueerror at the beginning if the method is invalid

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
statsmodels/discrete/discrete_model.py
((87 lines not shown))
+ kwargs['auto_trim_tol'] = auto_trim_tol
+ kwargs['qc_tol'] = qc_tol
+ kwargs['qc_verbose'] = qc_verbose
+
+ ### Define default keyword arguments to be passed to super(...).fit()
+ if maxiter == 'defined_by_method':
+ if method == 'l1':
+ maxiter = 1000
+ elif method == 'l1_cvxopt_cp':
+ maxiter = 70
+
+ ## Parameters to pass to super(...).fit()
+ # For the 'extra' parameters, pass all that are available,
+ # even if we know (at this point) we will only use one.
+ extra_fit_funcs = {'l1': fit_l1_slsqp}
+ if have_cvxopt:
@josef-pkt Owner

we should do this only if method is for cvxopt, to avoid unnecessary import

Done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
statsmodels/discrete/discrete_model.py
((123 lines not shown))
+ def cov_params_func_l1(self, likelihood_model, xopt, retvals):
+ """
+ Computes cov_params on a reduced parameter space
+ corresponding to the nonzero parameters resulting from the
+ l1 regularized fit.
+
+ Returns a full cov_params matrix, with entries corresponding
+ to zero'd values set to np.nan.
+ """
+ H = likelihood_model.hessian(xopt)
+ trimmed = retvals['trimmed']
+ nz_idx = np.nonzero(trimmed == False)[0]
+ nnz_params = (trimmed == False).sum()
+ if nnz_params > 0:
+ H_restricted = np.zeros((nnz_params, nnz_params))
+ for new_i, old_i in enumerate(nz_idx):
@josef-pkt Owner

isn't this for loop just a numpy indexing
H[nz_idx[:,None], nz_idx]

You're right. Change made.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
statsmodels/discrete/discrete_model.py
((130 lines not shown))
+ to zero'd values set to np.nan.
+ """
+ H = likelihood_model.hessian(xopt)
+ trimmed = retvals['trimmed']
+ nz_idx = np.nonzero(trimmed == False)[0]
+ nnz_params = (trimmed == False).sum()
+ if nnz_params > 0:
+ H_restricted = np.zeros((nnz_params, nnz_params))
+ for new_i, old_i in enumerate(nz_idx):
+ for new_j, old_j in enumerate(nz_idx):
+ H_restricted[new_i, new_j] = H[old_i, old_j]
+ # Covariance estimate for the nonzero params
+ H_restricted_inv = np.linalg.inv(-H_restricted)
+
+ cov_params = np.nan * np.ones(H.shape)
+ for new_i, old_i in enumerate(nz_idx):
@josef-pkt Owner

numpy indexing instead of for

cov_params[nz_idx[:,None], nz_idx] = H_restricted_inv

Done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@josef-pkt josef-pkt commented on the diff
statsmodels/discrete/discrete_model.py
@@ -147,6 +304,23 @@ def fit(self, start_params=None, method='newton', maxiter=35,
return BinaryResultsWrapper(discretefit)
fit.__doc__ = DiscreteModel.fit.__doc__
+ def fit_regularized(self, start_params=None, method='l1',
+ maxiter='defined_by_method', full_output=1, disp=1, callback=None,
+ alpha=0, trim_mode='auto', auto_trim_tol=0.01, size_trim_tol=1e-4,
+ qc_tol=0.03, **kwargs):
+ bnryfit = super(BinaryModel, self).fit_regularized(
+ start_params=start_params, method=method, maxiter=maxiter,
+ full_output=full_output, disp=disp, callback=callback,
+ alpha=alpha, trim_mode=trim_mode, auto_trim_tol=auto_trim_tol,
+ size_trim_tol=size_trim_tol, qc_tol=qc_tol, **kwargs)
+ if method in ['l1', 'l1_cvxopt_cp']:
@josef-pkt Owner

the check for valid method should be before the call to super. Or better, it can be dropped if the check is made in the super method

Moved check to super.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@josef-pkt josef-pkt commented on the diff
statsmodels/discrete/discrete_model.py
@@ -267,6 +441,29 @@ def fit(self, start_params=None, method='newton', maxiter=35,
return MultinomialResultsWrapper(mnfit)
fit.__doc__ = DiscreteModel.fit.__doc__
+ def fit_regularized(self, start_params=None, method='l1',
+ maxiter='defined_by_method', full_output=1, disp=1, callback=None,
+ alpha=0, trim_mode='auto', auto_trim_tol=0.01, size_trim_tol=1e-4,
+ qc_tol=0.03, **kwargs):
+ if start_params is None:
+ start_params = np.zeros((self.K * (self.J-1)))
@josef-pkt Owner

better start_params ?
open a new issue and check this later

I opened an issue for langmore/statsmodels and assigned myself to it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
statsmodels/discrete/discrete_model.py
((11 lines not shown))
+ nnz_params : Integer
+ The number of nonzero parameters in the model. Train with
+ trim_params==True or else numerical error will distort this.
+ trimmed : Boolean array
+ trimmed[i] == True if the ith parameter was trimmed from the model.
+ """
+ def __init__(self, model, bnryfit):
+ super(L1BinaryResults, self).__init__(model, bnryfit)
+ # self.trimmed is a boolean array with T/F telling whether or not that
+ # entry in params has been set zero'd out.
+ self.trimmed = bnryfit.mle_retvals['trimmed']
+ self.nnz_params = (self.trimmed == False).sum()
+
+ @cache_readonly
+ def aic(self):
+ return -2*(self.llf - self.nnz_params)
@josef-pkt Owner

my guess (not verified yet)
this should work generically if df_model = nnz_params

(or because of the current treatment of constant
df_model = nnz_params - 1
df_modelwc = nnz_params in GenericLikelihoodModel)

Is df_model set somewhere with fit_regularized?

So far I don't set df_model anywhere. You're right that it would work for aic/bic calculations. Will setting df_model = nnz_params work everywhere? Is there a clear definition for what df_model should be?

@josef-pkt Owner

for discrete models it is set in DiscreteModel initialize (which is called by init)

df_model is the number of parameters minus 1 (or minus number of endog levels J in MNLogit)
It assumes a constant and subtracts one for it.
(I don't like the -1 anymore but that needs a systematic refactoring)

here you can just reset df_model in fit(),
the inherited df_model is corrently incorrect if there are zeros.

I'm sure what you mean by "reset df_model in fit." I tried re-setting it all over the place. Nothing seems to work. The place that makes sense is inside L1BinaryResults and L1MultinomialResults. However, if I try in either place (using the @cache_readonly decorator) I get the warning "the attribute 'df_model' cannot be overwritten" (and then tests fail).

I see your comment "not now" above...so I'll just leave things unchanged.

@josef-pkt Owner

I put not now in the wrong class
"not now" refers to replacing aic and bic in MNLogit with the generic MultinomialResult version. I think the df_xx should be fixed.
Maybe then aic in MNLogit will also be correct, but I didn't go through the roundabout calculation (first subtract constants than add them again)

I can try df_model also tomorrow:
The main df_model is an attribute of the model instance. Initially I thought you have nnz_params in the model not in the result.
what should work right now, from my reading of the source, is to add after self.nnz_params = (self.trimmed == False).sum() in the __init__
self.model.df_model = self.nnz_params - 1
and
self.model.df_resid = self.model.nobs - self.model.df_model #I'm not sure where nobs is attached.
so the two df are correctly updated

the more appropriate place might be to calculate nnz_params in fit() of the model

did you try to attach df_model as a property (decorated def df_model)? I guess that doesn't work because we assign to df_model.

@josef-pkt Owner

a possible complication: df_model is an int or float, so when df_model is assigned in the results instance it will be a copy of the one in the model instance.
then we need to set df_model both in the model and the results instance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
statsmodels/discrete/discrete_model.py
((9 lines not shown))
+ New Attributes
+ --------------
+ nnz_params : Integer
+ The number of nonzero parameters in the model. Train with
+ trim_params==True or else numerical error will distort this.
+ """
+ def __init__(self, model, mlefit):
+ super(L1MultinomialResults, self).__init__(model, mlefit)
+ # self.trimmed is a boolean array with T/F telling whether or not that
+ # entry in params has been set zero'd out.
+ self.trimmed = mlefit.mle_retvals['trimmed']
+ self.nnz_params = (self.trimmed == False).sum()
+
+ @cache_readonly
+ def aic(self):
+ return -2*(self.llf - self.nnz_params)
@josef-pkt Owner

replacing this with df_model and super aic, is more difficult to read, not now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
statsmodels/discrete/tests/results/results_discrete.py
((53 lines not shown))
+ def probit(self):
+ """
+ Results generated with
+ data = sm.datasets.spector.load()
+ data.exog = sm.add_constant(data.exog, prepend=True)
+ alpha = np.array([0.1, 0.2, 0.3, 10])
+ res2 = sm.Probit(data.endog, data.exog).fit_regularized(
+ method="l1", alpha=alpha, disp=0, trim_mode='auto',
+ auto_trim_tol=0.02, acc=1e-10, maxiter=1000)
+ """
+ nan = np.nan
+ self.params = [-5.40476992, 1.25018458, 0.04744558, 0. ]
+ self.conf_int = [[-9.44077951, -1.36876033],
+ [ 0.03716721, 2.46320194],
+ [-0.09727571, 0.19216687],
+ [ np.nan, np.nan]]
@josef-pkt Owner

cosmetic: indent array to align vertically

Done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
statsmodels/discrete/tests/test_discrete.py
((105 lines not shown))
+ raise SkipTest("Skipped test_cvxopt since cvxopt is not available")
+
+
+class TestSweepAlphaL1(object):
+ @classmethod
+ def setupClass(cls):
+ data = sm.datasets.spector.load()
+ data.exog = sm.add_constant(data.exog, prepend=True)
+ cls.model = Logit(data.endog, data.exog)
+ cls.alphas = np.array(
+ [[0.1, 0.1, 0.1, 0.1],
+ [0.4, 0.4, 0.5, 0.5], [0.5, 0.5, 1, 1]])
+ cls.res1 = DiscreteL1()
+ cls.res1.sweep()
+
+ def sweep_alpha_and_test(self):
@josef-pkt Owner

I prefer test_xxx, e.g. test_sweep_alpha so it's easier/faster to see that this is a test method

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
statsmodels/discrete/tests/test_discrete.py
((125 lines not shown))
+ trim_mode='off', maxiter=1000)
+ assert_almost_equal(res2.params, self.res1.params[i], DECIMAL_4)
+
+
+class TestL1Compatability(object):
+ """
+ Tests compatability between l1 and unregularized by setting alpha such
+ that certain parameters should be effectively unregularized, and others
+ should be ignored by the model.
+ """
+ @classmethod
+ def setupClass(cls):
+ cls.data = sm.datasets.spector.load()
+ cls.data.exog = sm.add_constant(cls.data.exog, prepend=True)
+
+ def l1_compatability_logit(self):
@josef-pkt Owner

nose will not recognize these as tests

test_l1_.... as method name

(as check, put in temporarily a mistake and verify that nose raises a failure)

Done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@langmore

After this flurry of activity, one thing remains: Decide whether to detach fit_regularized from fit, use "callback idea", or leave extra arguments as is.

@josef-pkt josef-pkt commented on the diff
l1_demo/short_demo.py
@@ -0,0 +1,105 @@
+"""
@josef-pkt Owner

we can use this script as a top-level example, that can be rendered in html
needs editing to render to html
it's possible to keep the original as unchanged copy

I added l1 logit and MNLogit to examples/example_discrete.py. Should I add Probit? Will this file be converted to html or do I need to do that elsewhere?

@josef-pkt Owner

I don't think it's necessary. It follows the same pattern, and it's better if the examples are not too long.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
l1_demo/short_demo.py
@@ -0,0 +1,105 @@
+"""
+You can fit your LikelihoodModel using l1 regularization by changing
+ the method argument and adding an argument alpha. See code for
+ details.
+
+The Story
+---------
+The maximum likelihood (ML) solution works well when the number of data
+points is large and the noise is small. When the ML solution starts
+"breaking", the regularized solution should do better.
+
+The l1 Solvers
+--------------
+The solvers are slower than standard Newton, and sometimes have
+ convergence issues Nonetheless, the final solution makes sense and
@josef-pkt Owner

If there is a convergence failure, does "the final solution make sense"

I took those two lines out. We changed the default solver settings so that thigns work most of the time. No solver is perfect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
l1_demo/short_demo.py
((31 lines not shown))
+spector_data.exog = sm.add_constant(spector_data.exog, prepend=True)
+N = len(spector_data.endog)
+K = spector_data.exog.shape[1]
+
+### Logit Model
+logit_mod = sm.Logit(spector_data.endog, spector_data.exog)
+## Standard logistic regression
+logit_res = logit_mod.fit()
+## Regularized regression
+# Set the reularization parameter to something reasonable
+alpha = 0.05 * N * np.ones(K)
+# Use l1, which solves via a built-in (scipy.optimize) solver
+logit_l1_res = logit_mod.fit_regularized(method='l1', alpha=alpha, acc=1e-6)
+# Use l1_cvxopt_cp, which solves with a CVXOPT solver
+logit_l1_cvxopt_res = logit_mod.fit_regularized(
+ method='l1_cvxopt_cp', alpha=alpha)
@josef-pkt Owner

cosmetic/readability: I like empty lines to separate logical parts (setup, estimate, print and display)

Done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@josef-pkt
Owner
statsmodels/discrete/tests/test_discrete.py
((140 lines not shown))
+ def test_l1_compatability_logit(self):
+ # Do a regularized fit with alpha, effectively dropping the last column
+ alpha = np.array([0, 0, 0, 10])
+ res_reg = Logit(self.data.endog, self.data.exog).fit_regularized(
+ method="l1", alpha=alpha, disp=0, acc=1e-15, maxiter=2000,
+ trim_mode='off')
+ # Actually drop the last columnand do an unregularized fit
+ exog_no_PSI = self.data.exog[:, :3]
+ res_unreg = Logit(self.data.endog, exog_no_PSI).fit(disp=0, tol=1e-15)
+ ## Compare arrays
+ # The first three params should be equal
+ assert_almost_equal(res_unreg.params, res_reg.params[:3], DECIMAL_4)
+ # The last entry should be close to zero
+ assert_almost_equal(0, res_reg.params[3], DECIMAL_4)
+ # The restricted cov_params should be equal
+ assert_almost_equal(res_unreg.cov_params(), res_reg.cov_params()[:3, :3], DECIMAL_1)
@josef-pkt Owner

here we could add additional asserts,
for example t_test and f_test, which would also check that df_model agrees.

I didn't see any other tests that check that the nan_dot in t_test, f_test and cov_params works.

TODO

Inside test_l1_compatability_logit (and mnlogit) I put t_tests.

The f_tests are tricky since they require the inverse of cov_params. e.g. invcov = np.linalg.inv(self.cov_params).

I got f_test "working" by subclassing f_test in L1BinaryResults (and later L1MultinomialResults if we like this). The results are not correct yet since we need to adjust the degrees of freedom, which brings us back to the earlier question (I can't figure out out to change df_model, df_resid).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
README.txt
@@ -1,3 +1,48 @@
+What the l1 addition is
+=======================
@josef-pkt Owner

This part has to be removed from the toplevel README.txt.
README.txt is the main description for statsmodels for pypi.

I don't know where this should go. As a description of the files it's more oriented towards maintainer notes.
Maybe park it in a readme_L1.txt until we go over changes to the general documentation for the enhancements.

I made README_l1.txt and took l1 lines out of README.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@josef-pkt
Owner

Reading through this it looks mostly ready to be merged.

I need to play with some examples again.

@josef-pkt
Owner

The boiler plate for CountModel and an example for Poisson is missing.

(separate issue: there should be a way to avoid boiler plate classes and methods.)

@josef-pkt

being picky if it's a kwd, then add if not invcov is None: invcov = ...

not clear what will or should happen if users use cov_p

looks good to me, at least until f_test and similar itself are refactored

I made the change if not invcov is None: ...

@josef-pkt
Owner

I will look at fixing the df_model and df_resid tonight or tomorrow.

@langmore

The Poisson example gives a convergence error: "positive directional derivative in line search." My best guess is that this is an uncertain convergence issue. The Fortran code shows that one convergence criteria is met, but another is not. CVXOPT converges without errors, and gives the same result as slsqp.

@josef-pkt

(just a random thought while doing other things)

is inserting zeros the right way, or do we need nans?
I haven't checked the source yet.

@josef-pkt
Owner

I fixed the df_model in https://github.com/josef-pkt/statsmodels/commits/langmore_l1regularized_rebased
but there is something fishy with the MNLogit degrees of freedom. I need to check what they are supposed to be.

I changed the subclass aic bic names by adding an underline, to have a quick check if generic and specific implementation agree. will need cleanup

I saw the convergence statement of poisson, but didn't read yet what it means. The score at the optimum looks reasonable. With much smaller alpha, I get convergence return code 0 (alpha*0.0001)

@josef-pkt
Owner

There was an error in the example file, example_discrete.py
If we want to compare mlogit_res and mlogit_l1_res, then we need a new model instance before fit_regularized.

(plus I had one too many for the constants subtracted)

@josef-pkt
Owner

I have the t_tests passing, but there is something wrong with the f_test

example is from example_discrete

Note: f_test should only include rows for non-zero params

>>> logit_res.f_test(np.eye(4)[[1,3],:])
<class 'statsmodels.stats.contrast.ContrastResults'>
<F test: F=array([[ 3.75578691]]), p=[[ 0.03502189]], df_denom=30, df_num=2>

>>> logit_l1_res.f_test(np.eye(4)[[1,3],:])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new\statsmodels\statsmodels\discrete\discrete_model.py", line 2221, in f_test
    r_matrix, q_matrix, cov_p, scale, invcov)
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new\statsmodels\statsmodels\base\model.py", line 1318, in f_test
    F = nan_dot(nan_dot(Rbq.T, invcov), Rbq) / J
  File "e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new\statsmodels\statsmodels\tools\tools.py", line 538, in nan_dot
    should_be_nan_1 = np.dot(np.isnan(A), (B != 0))
ValueError: objects are not aligned

with square restriction matrix, I get an answer which should be nan

>>> logit_l1_res.f_test(np.eye(4))
<class 'statsmodels.stats.contrast.ContrastResults'>
<F test: F=array([[ 1.05848083]]), p=[[ 0.39409117]], df_denom=30, df_num=4>

>>> logit_l1_res.bse
array([        nan,  0.11201831,         nan,  2.56611327])
@josef-pkt
Owner

In the probit test, nnz_params is 4 instead of 3, shouldn't this be generic and identical across models?

E:\>nosetests -v --pdb-failure E:\...\statsmodels\statsmodels\discrete\tests\test_discrete.py:TestL1Compatability

> e:\josef\eclipsegworkspace\statsmodels-git\statsmodels-all-new\statsmodels\statsmodels\discrete\tests\test_discrete.py(583)test_l1_compatability_probit()
-> assert_equal(res_unreg.df_model, res_reg.df_model)
(Pdb) res_unreg.df_model, res_reg.df_model
(2.0, 3)
(Pdb) res_unreg.params, res_reg.params
(array([-6.03432654,  1.40957514,  0.05266746]), array([-6.03432654,  1.40957514,  0.05266746,  0.
     ]))
(Pdb) res_reg.nnz_params
4

need to include an assert for nnz_params in tests

@josef-pkt
Owner

with latest commit cfc2862
https://github.com/josef-pkt/statsmodels/commits/langmore_l1regularized_rebased
all tests pass
generic f_test works for Logit (I didn't look long enough to see what's wrong with the subclass f_test)

if r_matrix includes a zero param (np.eye(4) in test), then it raises: LinAlgError, 'Singular matrix'
we might need to catch it somewhere to raise a more informative exception

@josef-pkt
Owner

a thought to get f_test to work for MNLogit
define subclass f_test that temporarily reshapes params and then calls super. (needs a safe guard try ... finally ?)

Ian, you could just merge my branch into yours, should be without problems since I forked yours. I opened a PR

statsmodels/base/model.py
@@ -1294,7 +1314,10 @@ def f_test(self, r_matrix, q_matrix=None, cov_p=None, scale=1.0,
if invcov is None:
invcov = np.linalg.inv(self.cov_params(r_matrix=r_matrix,
cov_p=cov_p))
- F = np.dot(np.dot(Rbq.T, invcov), Rbq) / J
+ if self.mle_settings['optimizer'] in ['l1', 'l1_cvxopt_cp']:
@josef-pkt Owner

this doesn't work
the linear models (I checked), and I think GLM and RLM, do not define mle_settings.
raises an error when running the tests for regression

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
statsmodels/discrete/discrete_model.py
((24 lines not shown))
+ @cache_readonly
+ def aic(self):
+ return -2*(self.llf - self.nnz_params)
+
+ @cache_readonly
+ def bic(self):
+ return -2*self.llf + np.log(self.nobs)*self.nnz_params
+
+ def f_test(self, r_matrix, q_matrix=None, cov_p=None, scale=1.0,
+ invcov=None):
+ if invcov is None:
+ invcov = self.invcov()
+ return super(L1BinaryResults, self).f_test(
+ r_matrix, q_matrix, cov_p, scale, invcov)
+
+ def invcov(self):
@josef-pkt Owner

invcov is the inverse of covariance matrix of the linear transformation
inv of dot_fun(r_matrix, dot_fun(cov_p, np.transpose(other))) #from cov_params method

we don't need/use the inv of the cov of the params.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@langmore

I think I have a fundamental misunderstanding of what f_test is doing. I'll merge your pull request and try to figure things out.

@josef-pkt
Owner

first google search result http://jackman.stanford.edu/classes/350B/07/ftestforWeb.pdf
test of linear restriction
H0 : Rbeta = q
beta is normal distributed N(beta, sigma_b)
R
beta is normal distributed N(Rbeta, Rsigma_bR.T) different number of rows than beta if R is not square
W = (R
beta - q) * inv(Rsigma_bR) * (R*beta - q)

W is wald test statistic, either F under normal assumption, or chisquare under asymptotic normality of beta estimate.

@josef-pkt
Owner

We are mostly using Greene Econometric Analysis as general reference

statsmodels/discrete/tests/test_discrete.py
((125 lines not shown))
+ trim_mode='off', maxiter=1000)
+ assert_almost_equal(res2.params, self.res1.params[i], DECIMAL_4)
+
+
+class TestL1Compatability(object):
+ """
+ Tests compatability between l1 and unregularized by setting alpha such
+ that certain parameters should be effectively unregularized, and others
+ should be ignored by the model.
+ """
+ @classmethod
+ def setupClass(cls):
+ cls.data = sm.datasets.spector.load()
+ cls.data.exog = sm.add_constant(cls.data.exog, prepend=True)
+
+ def test_l1_compatability_logit(self):
@josef-pkt Owner

The compatibility tests could be more easily extended with a check superclass, and avoid code duplication when it works for all tested models

TODO

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@josef-pkt
Owner

I don't see any major problems or missing parts anymore. I think after a bit of cleanup and finishing some missing details, we can merge this.

(working on this, I saw some other changes that I would like to do in base/model, but I wait until this is merged to avoid merge conflicts.
one of the things: #50 since we should use chisquare and normal distributions instead of t and normal for the wald tests in discrete.)

@langmore

Although I still need to read up on F tests, I do agree with your logic here. I'll remove f_test_ and invcov

@langmore

I'll remove the aic_ and bic_

@langmore

Tests for Poisson are missing. Should I just add them in like Logit/Probit?

@josef-pkt
Owner

test for Poisson: unless there are specific issues, I think we don't need the full set of tests again, compatibility and one of the others should be enough.

(mostly offline until tonight)

@josef-pkt
Owner

another optimization improvement, based on convergence failure in poisson
rescale loglikelihood by 1/nobs in fit

last 2 commits in https://github.com/josef-pkt/statsmodels/commits/langmore_l1regularized_rebased_2
now alpha is relative to mean loglikelihood and shouldn't be scaled by nobs anymore. I did only the adjustments to the alphas to get the test suite to pass again.

@josef-pkt
Owner

Looks good.

Can you run the entire statsmodels test suite, just to make sure we are clean now?

Do you know of any missing/unclean details?

One rst page that explains the L1 penalized estimation to put in the docs next to discrete will be useful.

I don't see any problems anymore, my impression from the last examples that I ran is that it works well, but maybe requires the usual caution and twisting for extreme cases.

I can rebase and merge into master, but most likely not today.

@langmore

All tests pass. I ran nosetests from the base directory:
Ran 1724 tests in 111.822s
OK (SKIP=11)

I'll work on the rst page.

There was the issue of choosing good starting params...but the demo and example finish now with clean exits. Maybe we can wait till we have an example that stops working.

@langmore

Any hints for compiling the math in the rst? I can use rst2html to compile the normal rst, but to use math I assume I need to use some configuration file (e.g. conf.rst)...but I don't know how to use it.

Thanks!

@josef-pkt
Owner

The simplest is to just use the doc build

in statsmodels docs type "make html" and it will do the full html doc build (assumes you have sphinx, ipython and matplotlib installed)

then under docs/source you can add a .rst file and add the name of the file (without .rst extension) to index.rst

on repeated builds with make html, sphinx will only rebuild the changed files, so editing is reasonably fast after the first time.

It should also pick up the docstrings for the new fit_regularized automatically.

html is under docs/build/html

@josef-pkt
Owner

rebased again and merged in e49d7fc

I quickly merged to avoid another rebase with many merge conflicts.

all tests pass, except for one unrelated failure.

TODO: move example folder, add docs

Also, I forgot to run the test suite on master before the merge. It would be interesting to see the difference in the time it takes to run the test suite.

@josef-pkt josef-pkt closed this
@langmore

I'll work on the docs. When I have something I'll put in a pull WIP pull request.

Also, I noticed that statsmodels/discrete/l1_nm.py snuck its way in. This was deleted from my local copy...so I'm a bit confused. In any case it has not been maintained and should probably be removed.

@josef-pkt
Owner

I guess a rebase conflict mistake. Thanks for checking. I deleted it again.

Don't forget to start a new branch from master for further work.

Thank you for this contribution. After working with the examples for some time, I think it looks really good and brings one of the newer popular approaches to statsmodels.

(next up: cross-validation and L2)

@langmore

I'll be using the L1 code quite a bit over the next month, with about 6 other people, so having this as part of a major package helps me out quite a bit, thanks!

L2 should be easy...we re-define score and Hessian. No parameters dropped (that was the hardest part). I don't have any ideas for cross-validation (other than using the previous parameter values for the next step). I'm sure there are good ideas out there though.

We should talk to some R experts and see what they have done. I imagine there are fully Bayesian toolboxes that use hyperpriors to select alpha.

@langmore
@josef-pkt
Owner

(can you add further comments to issue #519, since that one is still open)

see #520
I had the same problem regenerating the docs. I don't know why sphinx doesn't see the changes.
I needed to delete the generated files (I guess deleting /doctrees is necessary), which makes it much slower to see make html and see a change.

@langmore
@josef-pkt
Owner

Yes, sphinx is importing statsmodels, so it's always using the version that is on the python path.
I usually have a development branch on the python path, (and installed versions in other python versions and virtualenvs)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Sep 14, 2012
  1. @langmore @josef-pkt

    Made some test changes

    langmore authored josef-pkt committed
  2. @langmore @josef-pkt

    Did not use -a in last commit

    langmore authored josef-pkt committed
  3. @langmore @josef-pkt

    things work...in the middle of changes

    langmore authored josef-pkt committed
  4. @langmore @josef-pkt

    just added trim_params

    langmore authored josef-pkt committed
  5. @langmore @josef-pkt

    L1 regularization works. Erratic results when we use small regulariza…

    langmore authored josef-pkt committed
    …tion parameter and have variables that dont really affect the target
  6. @langmore @josef-pkt

    Got rid of xxx_fit_lasso

    langmore authored josef-pkt committed
  7. @langmore @josef-pkt

    Added l1.get_bic

    langmore authored josef-pkt committed
  8. @langmore @josef-pkt

    Added utility bic/aic functions

    langmore authored josef-pkt committed
  9. @langmore @josef-pkt

    added l1_test, which used to be statsmodels_test

    langmore authored josef-pkt committed
  10. @langmore @josef-pkt

    Got demo ready and removed epsilon as a parameter

    langmore authored josef-pkt committed
  11. @langmore @josef-pkt

    added README_l1.txt

    langmore authored josef-pkt committed
  12. @langmore @josef-pkt

    Combined READMEs

    langmore authored josef-pkt committed
  13. @langmore @josef-pkt

    Added note on l1_demo/demo.py to the README.txt

    langmore authored josef-pkt committed
  14. @langmore @josef-pkt

    Slight change to README.txt

    langmore authored josef-pkt committed
  15. @langmore @josef-pkt
  16. @langmore @josef-pkt
  17. @langmore @josef-pkt
  18. @langmore @josef-pkt

    Added a new class: discrete_model.py:L1MultinomialResults(Multinomial…

    langmore authored josef-pkt committed
    …Results). This gives special formatting to results for the MNLogit model (when trained with method==l1)
  19. @langmore @josef-pkt
  20. @langmore @josef-pkt

    alpha can now be passed in as a vector. Updated demo

    langmore authored josef-pkt committed
  21. @langmore @josef-pkt
  22. @langmore @josef-pkt

    Added in cvxopt_cp fitting. Not fully integrated yet

    langmore authored josef-pkt committed
  23. @langmore @josef-pkt
  24. @langmore @josef-pkt

    Re-wrote l1_slsqp to look like l1_cvxopt

    langmore authored josef-pkt committed
  25. @langmore @josef-pkt

    Re-wrote demo.py to allow for use of noise

    langmore authored josef-pkt committed
  26. @langmore @josef-pkt

    Added docstring to demo.py

    langmore authored josef-pkt committed
  27. @langmore @josef-pkt

    added more docstring to demo.py

    langmore authored josef-pkt committed
  28. @langmore @josef-pkt

    Updated README.txt

    langmore authored josef-pkt committed
  29. @langmore @josef-pkt

    Updated demo. Realized long correlations made the singular KKT matrix…

    langmore authored josef-pkt committed
    … error happen
  30. @langmore @josef-pkt

    Erased a TODO in demo.py

    langmore authored josef-pkt committed
  31. @langmore @josef-pkt

    Revised docstring in demo

    langmore authored josef-pkt committed
  32. @langmore @josef-pkt

    Made demo much nicer

    langmore authored josef-pkt committed
  33. @langmore @josef-pkt
  34. @langmore @josef-pkt

    Updated docstring in the demo

    langmore authored josef-pkt committed
  35. @langmore @josef-pkt

    Corrected demo to say RMS rather than MS

    langmore authored josef-pkt committed
  36. @langmore @josef-pkt

    Protected the import of cvxopt

    langmore authored josef-pkt committed
  37. @langmore @josef-pkt

    Fixed demo: Previously demo used wrong number of targets

    langmore authored josef-pkt committed
  38. @langmore @josef-pkt

    demo now has a CLI

    langmore authored josef-pkt committed
  39. @langmore @josef-pkt
  40. @langmore @josef-pkt

    We now return x.ravel() since this was expected

    langmore authored josef-pkt committed
  41. @langmore @josef-pkt

    Added probit to demo.py

    langmore authored josef-pkt committed
  42. @langmore @josef-pkt

    Updated README.txt

    langmore authored josef-pkt committed
  43. @langmore @josef-pkt
  44. @langmore @josef-pkt

    Cleaned up demo.py

    langmore authored josef-pkt committed
  45. @langmore @josef-pkt

    Updated docs in demo

    langmore authored josef-pkt committed
  46. @langmore @josef-pkt

    Added L1BinaryResults. Now probit and logit results are handled like …

    langmore authored josef-pkt committed
    …mnlogit, viz. nan retured for parameters that are zero (hence more than just covariance is important to statistics)
  47. @langmore @josef-pkt
  48. @langmore @josef-pkt

    Got rid of bad line in demo

    langmore authored josef-pkt committed
  49. @langmore @josef-pkt

    Former code assumed correct input of alpha...now convert to numpy arr…

    langmore authored josef-pkt committed
    …ay of correct size
  50. @langmore @josef-pkt
  51. @langmore @josef-pkt

    Corrected docstring

    langmore authored josef-pkt committed
  52. @langmore @josef-pkt
  53. @langmore @josef-pkt

    Added sklearn_compare to compare results with sklearn

    langmore authored josef-pkt committed
  54. @langmore @josef-pkt

    Got sklearn_compare.py working with the Spector dataset

    langmore authored josef-pkt committed
  55. @langmore @josef-pkt

    Added capability to handle anes96 datasets

    langmore authored josef-pkt committed
  56. @langmore @josef-pkt
  57. @langmore @josef-pkt

    Updated README.txt

    langmore authored josef-pkt committed
  58. @langmore @josef-pkt
  59. @langmore @josef-pkt

    Started unit tests

    langmore authored josef-pkt committed
  60. @langmore @josef-pkt

    Protected import of cvxopt and subclassed test

    langmore authored josef-pkt committed
  61. @langmore @josef-pkt

    Changed method of protection for cvxopt

    langmore authored josef-pkt committed
  62. @langmore @josef-pkt

    Added MNLogit example to short_demo.py

    langmore authored josef-pkt committed
  63. @langmore @josef-pkt

    Made trim_params=True in short_demo.py

    langmore authored josef-pkt committed
  64. @langmore @josef-pkt

    Made trim_params=True the default

    langmore authored josef-pkt committed
  65. @langmore @josef-pkt
  66. @josef-pkt
  67. @langmore @josef-pkt
  68. @langmore @josef-pkt

    Fixed typo in discrete_model

    langmore authored josef-pkt committed
  69. @langmore @josef-pkt
  70. @langmore @josef-pkt
  71. @langmore @josef-pkt

    Return trimmed as a boolean array now

    langmore authored josef-pkt committed
  72. @langmore @josef-pkt

    Now trimmed is used to get cov_nz_params, the covariance of the nonze…

    langmore authored josef-pkt committed
    …ro params
  73. @langmore @josef-pkt

    Changed default magic_tol

    langmore authored josef-pkt committed
  74. @langmore @josef-pkt
  75. @langmore @josef-pkt

    Extended auto param trimming to l1_cvxopt_cp

    langmore authored josef-pkt committed
  76. @langmore @josef-pkt
  77. @langmore @josef-pkt

    Added more tests for L1 Logit

    langmore authored josef-pkt committed
  78. @langmore @josef-pkt

    If all parames are zerod, then do not store the Hessian or attempt to…

    langmore authored josef-pkt committed
    … invert it
  79. @langmore @josef-pkt

    Added comments

    langmore authored josef-pkt committed
  80. @langmore @josef-pkt

    Fixed an issue that happened when trim_params==False (previously we d…

    langmore authored josef-pkt committed
    …idn't return trimmed)
  81. @langmore @josef-pkt

    Added capability to use nm (scipy.optimize.fmin) to do things. It see…

    langmore authored josef-pkt committed
    …ms to suck
  82. @langmore @josef-pkt

    Made some options explicit in tests

    langmore authored josef-pkt committed
  83. @langmore @josef-pkt

    Removed references to l1_nm

    langmore authored josef-pkt committed
  84. @langmore @josef-pkt

    Refactored to address comments on WIP pull request

    langmore authored josef-pkt committed
  85. @langmore @josef-pkt

    Implemented the fit_regularized paradigm

    langmore authored josef-pkt committed
  86. @langmore @josef-pkt

    Updated demo.py and README.txt for fit_regularized

    langmore authored josef-pkt committed
  87. @langmore @josef-pkt

    moved files to base

    langmore authored josef-pkt committed
  88. @langmore @josef-pkt
  89. @langmore @josef-pkt

    Added demo showing regularization path

    langmore authored josef-pkt committed
  90. @langmore @josef-pkt
  91. @langmore @josef-pkt

    Added QC_verbose option to slsqp

    langmore authored josef-pkt committed
  92. @langmore @josef-pkt
  93. @langmore @josef-pkt
  94. @langmore @josef-pkt

    Added nan_dot function

    langmore authored josef-pkt committed
  95. @langmore @josef-pkt

    to save model.t_test_check_valid

    langmore authored josef-pkt committed
  96. @langmore @josef-pkt

    Changed t_test to account for nan values in cov_params

    langmore authored josef-pkt committed
  97. @josef-pkt
  98. @langmore
  99. @langmore
  100. @langmore
  101. @langmore

    Updated the demos

    langmore authored
  102. @langmore
  103. @langmore

    Added new tests

    langmore authored
Commits on Sep 16, 2012
  1. @langmore

    Cleaned up code

    langmore authored
Commits on Sep 17, 2012
  1. @langmore
  2. @langmore

    Added TODO_ian.txt

    langmore authored
Commits on Sep 18, 2012
  1. @langmore

    Added more tests

    langmore authored
Commits on Sep 23, 2012
  1. @langmore

    Refactored nan_dot for speed

    langmore authored
Commits on Sep 25, 2012
  1. @langmore
  2. @langmore

    Updated docs

    langmore authored
  3. @langmore
  4. @langmore
  5. @langmore
  6. @langmore

    Updated examples

    langmore authored
  7. @langmore

    Updated examples

    langmore authored
  8. @langmore
  9. @langmore

    Added Poisson example

    langmore authored
  10. @langmore

    Added Poisson example

    langmore authored
  11. @langmore

    Minor change to allow keyword

    langmore authored
Commits on Sep 26, 2012
  1. @langmore
  2. @josef-pkt
Commits on Sep 27, 2012
  1. @josef-pkt
  2. @josef-pkt
  3. @josef-pkt
  4. @josef-pkt
  5. @josef-pkt
  6. @langmore
  7. @langmore

    Removed f_test_ and invcov

    langmore authored
  8. @langmore

    Removed aic_ bic_

    langmore authored
  9. @langmore
Commits on Sep 29, 2012
  1. @josef-pkt
  2. @josef-pkt
Commits on Oct 4, 2012
  1. @josef-pkt

    REF: add alpha_rescaled

    josef-pkt authored
  2. @josef-pkt
  3. @josef-pkt
  4. @josef-pkt
Commits on Oct 5, 2012
  1. @langmore
  2. @josef-pkt
  3. @langmore
  4. @langmore
Something went wrong with that request. Please try again.