[MRG+2] LOF algorithm (Anomaly Detection) #5279

ngoix · Sep 16, 2015

Local Outlier Factor implementation.

Motivated by previous discussions
http://sourceforge.net/p/scikit-learn/mailman/message/32485020
and
#4163

benchmarks of LOF on:
https://github.com/ngoix/scikit-learn/blob/AD_benchmarks/benchmarks/bench_lof_vs_iforest.py

thanks for the early PR

let me know when you need a review ie when you addressed the standard things (tests, example, some basic doc)

agramfort · Sep 16, 2015

thanks for the early PR

let me know when you need a review ie when you addressed the standard things (tests, example, some basic doc)

I'd also be interested in reviewing this when you've moved past the WIP stage.

jmschrei · Sep 21, 2015

I'd also be interested in reviewing this when you've moved past the WIP stage.

I think it is ready for a first review @agramfort @jmschrei !

ngoix · Oct 9, 2015

I think it is ready for a first review @agramfort @jmschrei !

agramfort · Oct 9, 2015

agramfort reviewed Oct 9, 2015

View changes

agramfort · Oct 9, 2015

sklearn/neighbors/lof.py

+__all__ = ["LOF"]
+
+
+class LOFMixin(object):


why do you need a mixin here?

I would put all methods from Mixin into LOF class and make them private.

I agree. Mixins imply that multiple estimators will be using them,

agramfort · Oct 9, 2015

agramfort reviewed Oct 9, 2015

View changes

agramfort · Oct 9, 2015

sklearn/neighbors/lof.py

+    """
+    def __init__(self, n_neighbors=5, algorithm='auto', leaf_size=30,
+                 metric='minkowski', p=2, metric_params=None,
+                 n_jobs=1, **kwargs):


**kwargs in init are not ok. why do you need this?

Ok I will remove it.

agramfort · Oct 9, 2015

agramfort reviewed Oct 9, 2015

View changes

agramfort · Oct 9, 2015

sklearn/neighbors/lof.py

+                          metric_params=metric_params, n_jobs=n_jobs, **kwargs)
+
+    def predict(self, X=None, n_neighbors=None):
+        """Predict LOF score of X.


empty line missing

agramfort · Oct 9, 2015

agramfort reviewed Oct 9, 2015

View changes

agramfort · Oct 9, 2015

sklearn/neighbors/lof.py

+                          leaf_size=leaf_size, metric=metric, p=p,
+                          metric_params=metric_params, n_jobs=n_jobs, **kwargs)
+
+    def predict(self, X=None, n_neighbors=None):


predict signature should be

def predict(self, X):

that's it

@agramfort mentions this, but X must be provided in predict methods.

discussion below ? predict must not be able to handle X=None?

agramfort · Oct 9, 2015

agramfort reviewed Oct 9, 2015

View changes

agramfort · Oct 9, 2015

sklearn/neighbors/lof.py

+
+        Parameters
+        ----------
+


remove line

agramfort · Oct 9, 2015

agramfort reviewed Oct 9, 2015

View changes

agramfort · Oct 9, 2015

sklearn/neighbors/lof.py

+
+        X : array-like, last dimension same as that of fit data, optional
+        (default=None)
+        The querry sample or samples to compute the LOF wrt to the training


agramfort · Oct 9, 2015

agramfort reviewed Oct 9, 2015

View changes

agramfort · Oct 9, 2015

sklearn/neighbors/lof.py

+        Returns
+        -------
+        lof_scores : array of shape (n_samples,)
+        The LOF score of each input samples. The lower, the more normal.


jmschrei · Oct 9, 2015

jmschrei reviewed Oct 9, 2015

View changes

jmschrei · Oct 9, 2015

sklearn/neighbors/lof.py

+
+    def k_distance(self, X=None):
+        """
+        Compute the k_distance and the neighborhood of querry samples X wrt


This isn't a terribly informative docstring. It doesn't define what k_distance is, or what self._fit_X is. Maybe change it to describe things in terms of the algorithm, irrespective of the underlying implementation. Also, query not querry.

jmschrei · Oct 9, 2015

jmschrei reviewed Oct 9, 2015

View changes

jmschrei · Oct 9, 2015

sklearn/neighbors/lof.py

+
+        Parameters
+        ----------
+


remove line

jmschrei · Oct 9, 2015

jmschrei reviewed Oct 9, 2015

View changes

jmschrei · Oct 9, 2015

sklearn/neighbors/lof.py

+
+        Parameters
+        ----------
+


remove line

jmschrei · Oct 9, 2015

jmschrei reviewed Oct 9, 2015

View changes

jmschrei · Oct 9, 2015

sklearn/neighbors/tests/test_lof.py

+    # Test LOF
+    clf = neighbors.LOF()
+    clf.fit(X)
+    pred = clf.predict()


Predict must take in values to predict anomalies in.

jmschrei · Oct 9, 2015

jmschrei reviewed Oct 9, 2015

View changes

jmschrei · Oct 9, 2015

sklearn/neighbors/tests/test_lof.py

+    clf = neighbors.LOF()
+    clf.fit(X)
+    pred = clf.predict()
+    assert_array_equal(clf._fit_X, X)


The ranking of samples as to their anomaly status must be easily received by the user; usually returned by the predict method. Having to do in and get an attribute is not okay.

Sorry I don't see your point, the ranking is received directly in pred...
clf._fit_X is just the training samples (defined in the base estimator NeighborsBase), the user doesn't see or need it...

jmschrei · Oct 9, 2015

jmschrei reviewed Oct 9, 2015

View changes

jmschrei · Oct 9, 2015

sklearn/neighbors/lof.py

+        """
+        distances, neighbors_indices = self.kneighbors(
+            X=X, n_neighbors=self.n_neighbors)
+        neighbors_indices = neighbors_indices


What is the point of this line?

Right thanks!

jmschrei · Oct 9, 2015

jmschrei reviewed Oct 9, 2015

View changes

jmschrei · Oct 9, 2015

sklearn/neighbors/lof.py

+        The LRD of p.
+        """
+
+        p_0 = self._fit_X if p is None else p


You should have to explicitly pass in a dataset to this function.

discussion below

jmschrei · Oct 9, 2015

jmschrei reviewed Oct 9, 2015

View changes

jmschrei · Oct 9, 2015

sklearn/neighbors/lof.py

+
+        neighbors_indices = self.neighbors_indices_fit_X_ if p is None else self.k_distance(p)[1]
+
+        n_jobs = _get_n_jobs(self.n_jobs)


When you merge this into the LOF class, just call _get_n_jobs once in the __init__ function.

jmschrei · Oct 9, 2015

jmschrei reviewed Oct 9, 2015

View changes

jmschrei · Oct 9, 2015

sklearn/neighbors/lof.py

+        dist = pairwise_distances(p_0, self._fit_X,
+                                  self.effective_metric_,
+                                  n_jobs=n_jobs,
+                                  **self.effective_metric_params_)


What is **self.effective_metric_params_?

jmschrei · Oct 9, 2015

jmschrei reviewed Oct 9, 2015

View changes

jmschrei · Oct 9, 2015

sklearn/neighbors/lof.py

+        lrd = p_lrd if p is None else self.local_reachability_density(p=None)
+
+        for j in range(p_0.shape[0]):
+            cpt = -1


I don't understand what's going on with this cpt variable

I rename it neighbors_number.

jmschrei · Oct 9, 2015

jmschrei reviewed Oct 9, 2015

View changes

jmschrei · Oct 9, 2015

sklearn/neighbors/lof.py

+        Parameters
+        ----------
+
+        p : array-like of shape (n_samples, n_features)


p should be named X

jmschrei · Oct 9, 2015

jmschrei reviewed Oct 9, 2015

View changes

jmschrei · Oct 9, 2015

sklearn/neighbors/lof.py

+        p_lrd = self.local_reachability_density(p)
+        lrd_ratios_array = np.zeros((p_0.shape[0], self.n_neighbors))
+
+        # Avoid re-computing p_lrd if p is None:


p should never be None

jmschrei · Oct 9, 2015

jmschrei reviewed Oct 9, 2015

View changes

jmschrei · Oct 9, 2015

sklearn/neighbors/lof.py

+
+
+class LOF(NeighborsBase, KNeighborsMixin, LOFMixin, UnsupervisedMixin):
+    """Unsupervised Outlier Detection.


I like the documentation.

I would like to see some more extensive unit tests, particularly in cases where the algorithm should fail (wrong dimensions or other incorrect types of data passed in). I'll be able to look more at the performance of the code once you merge the mixin with the other class, and change the API to always take in an X matrix.

jmschrei · Oct 9, 2015

I would like to see some more extensive unit tests, particularly in cases where the algorithm should fail (wrong dimensions or other incorrect types of data passed in). I'll be able to look more at the performance of the code once you merge the mixin with the other class, and change the API to always take in an X matrix.

I'd also like to see an example of it performing against a/many current algorithm(s), so that it is clear it is a valuable contribution.

jmschrei · Oct 9, 2015

I'd also like to see an example of it performing against a/many current algorithm(s), so that it is clear it is a valuable contribution.

If you have a dataset X and want to remove outliers from it, you don't want to do

fit(X)
predict(X)

because then each sample is considered in its own neighbourhoud: in predict(X), X is considered as 'new observations'.

What the user wants is:

for each x in X,
fit(X-{x})
predict(x)

which is allowed by

fit(X)
predict()

It is like looking for k-nearest-neighbors of points in a dataset X: you can do:

neigh = NearestNeighbors()
neigh.fit(X)
neigh.kneighbors()

which is different from

neigh = NearestNeighbors()
neigh.fit(X)
neigh.kneighbors(X)

I can make predict() have as signature

def predict(self, X):

and allows taking X=None in argument... Is it allowed ?

ngoix · Oct 12, 2015

If you have a dataset X and want to remove outliers from it, you don't want to do

fit(X)
predict(X)

because then each sample is considered in its own neighbourhoud: in predict(X), X is considered as 'new observations'.

What the user wants is:

for each x in X,
fit(X-{x})
predict(x)

which is allowed by

fit(X)
predict()

It is like looking for k-nearest-neighbors of points in a dataset X: you can do:

neigh = NearestNeighbors()
neigh.fit(X)
neigh.kneighbors()

which is different from

neigh = NearestNeighbors()
neigh.fit(X)
neigh.kneighbors(X)

I can make predict() have as signature

def predict(self, X):

and allows taking X=None in argument... Is it allowed ?

If you have a dataset X and want to remove outliers from it, you don't want to do

fit(X)
predict(X)

implement a fit_predict(X) method is the way to go.

agramfort · Oct 12, 2015

If you have a dataset X and want to remove outliers from it, you don't want to do

fit(X)
predict(X)

implement a fit_predict(X) method is the way to go.

Ok thanks !

ngoix · Oct 12, 2015

Ok thanks !

I merged the mixin with LOF class, changed the API and added a comparison example.
@agramfort @jmschrei what do you think?

ngoix · Oct 12, 2015

I merged the mixin with LOF class, changed the API and added a comparison example.
@agramfort @jmschrei what do you think?

agramfort · Oct 12, 2015

agramfort reviewed Oct 12, 2015

View changes

agramfort · Oct 12, 2015

examples/covariance/plot_outlier_detection.py

-        clf.fit(X)
-        y_pred = clf.decision_function(X).ravel()
+
+        if clf_name=="Local Outlier Factor":


agramfort · Oct 12, 2015

agramfort reviewed Oct 12, 2015

View changes

agramfort · Oct 12, 2015

examples/covariance/plot_outlier_detection.py

+            y_pred = -clf.fit_predict(X).ravel()
+        else:
+            clf.fit(X)
+            y_pred = clf.decision_function(X).ravel()


this should be consistent between all estimators. LOF should not be a special case.

I agree but then how can we differentiate a new sample which has already been sampled in training, from a sample from training we want to score ?

agramfort · Oct 12, 2015

agramfort reviewed Oct 12, 2015

View changes

agramfort · Oct 12, 2015

examples/neighbors/plot_lof.py

+LOF example
+==========================================
+
+An example using LOF for anomaly detection.


You should be more explicit.

LOF is jargon. I vote to rename the object LocalOutlierFactor to be explicit.
Also never use the acronym without writing the full words.

you could add a ref to the doc of this example too.

LOF is jargon. I vote to rename the object LocalOutlierFactor to be explicit.

+1

Also never use the acronym without writing the full words.

+1

agramfort · Oct 12, 2015

agramfort reviewed Oct 12, 2015

View changes

agramfort · Oct 12, 2015

sklearn/neighbors/lof.py

+
+
+class LOF(NeighborsBase, KNeighborsMixin, UnsupervisedMixin):
+    """Unsupervised Outlier Detection.


Local Outlier Factor (LOF)

LOF is an unsupervised outlier detection method. It returns ...

agramfort · Oct 12, 2015

agramfort reviewed Oct 12, 2015

View changes

agramfort · Oct 12, 2015

sklearn/neighbors/lof.py

+                          leaf_size=leaf_size, metric=metric, p=p,
+                          metric_params=metric_params, n_jobs=n_jobs)
+
+        self.get_n_jobs_ = _get_n_jobs(self.n_jobs)


this should be in the fit method. init just sets attributes from params.

n_jobs is one of the params, though. I think self.n_jobs = _get_n_jobs(n_jobs) would be best. You're just correcting the number of jobs being passed in. Though this might be a correction for _init_params in NeighborsBase?

The base class does _get_n_jobs(self.n_jobs), isn't it a problem?

don't do it twice then

Yes but the result is not kept in memory!

You can remove the line self.get_n_jobs_ = _get_n_jobs(self.n_jobs) but you have to add n_jobs = _get_n_jobs(self.n_jobs) in the method _local_reachability_density (see other comment)

agramfort · Oct 12, 2015

agramfort reviewed Oct 12, 2015

View changes

agramfort · Oct 12, 2015

sklearn/neighbors/lof.py

+
+        Parameters
+        ----------
+        X : array-like, last dimension same as that of fit data, optional


it's usually implicit in all docstrings.

X : array-like, shape (n_samples, n_features)

it's not optional.

agramfort · Oct 12, 2015

agramfort reviewed Oct 12, 2015

View changes

agramfort · Oct 12, 2015

sklearn/neighbors/lof.py

+
+        reach_dist_array = np.zeros((p_0.shape[0], self.n_neighbors))
+
+        for j in range(p_0.shape[0]):


no way to vectorize this? a loop over samples is costly.

I tried but didn't succeed...

agramfort · Oct 12, 2015

agramfort reviewed Oct 12, 2015

View changes

agramfort · Oct 12, 2015

sklearn/neighbors/lof.py

+
+    Attributes
+    ----------
+


agramfort · Oct 12, 2015

agramfort reviewed Oct 12, 2015

View changes

agramfort · Oct 12, 2015

sklearn/neighbors/lof.py

+        The LOF of p. The lower, the more normal.
+
+        """
+        p_0 = self._fit_X if p is None else p


I agree with @jmschrei that keeping X as attribute is a bit brutal. Do you really need it just to skip first sample? can't you just check if min dist is 0?

We only keep X as attribute when p is None, ie when the querry samples are equal to the training ones (in this case indeed, the base class method KNeighborsMixin.kneighbors checks if min dist is 0 and removes the first sample in this case). I believe this case is optimal...

If the querry samples are different from the training ones, p is not None and p contains the querry samples. As commented the line below, this case is not really optimal for now, since only the neighbors of neighbors of p need to be considered (LOF is local).

Just to be sure we are on the same page: self._fit_X is an attribute of the base class, it is kept in memory anyway.

agramfort · Oct 12, 2015

agramfort reviewed Oct 12, 2015

View changes

agramfort · Oct 12, 2015

sklearn/neighbors/lof.py

+        # May be optimized (if p is not None) by only computing it for
+        # X = neighbors or neighbors of p
+
+        self.neighbors_indices_p_ = self.neighbors_indices_fit_X_ if p is None else self._k_distance(p)[1]


line too long ...

Yes I don't know how to write it on two lines...

do it like this

self.neighbors_indices_p_ = \ self.neighbors_indices_fit_X_ if p is None else self._k_distance(p)[1]

jmschrei · Oct 12, 2015

jmschrei reviewed Oct 12, 2015

View changes

jmschrei · Oct 12, 2015

sklearn/neighbors/__init__.py

@@ -26,4 +27,5 @@
           'kneighbors_graph',
           'radius_neighbors_graph',
           'KernelDensity',
-           'LSHForest']
+           'LSHForest',
+           'LOF']


Expand LOF out here too

Thanks @agramfort I've done the changes.

ngoix · Oct 18, 2016

Thanks @agramfort I've done the changes.

agramfort · Oct 18, 2016

agramfort approved these changes Oct 18, 2016

View changes

@ngoix please update what's new and let's merge this !

agramfort · Oct 18, 2016

@ngoix please update what's new and let's merge this !

done!

ngoix · Oct 19, 2016

done!

@amueller want to take a final look?

for me it's good enough to merge

agramfort · Oct 20, 2016

@amueller want to take a final look?

for me it's good enough to merge

amueller · Oct 20, 2016

amueller requested changes Oct 20, 2016

View changes

I think caching the LRD on the training set would be good (and actually make the code easier to follow). I think either predict and decision_function should both be private or neither. I kinda tend towards both, as making public is easier than hiding.
The rest is mostly minor, though how to tune n_neighbors seems pretty important.

amueller · Oct 20, 2016

sklearn/neighbors/lof.py

+        Returns
+        -------
+        lof_scores : array, shape (n_samples,)
+            The Local Outlier Factor of each input samples. The lower,


This seems to contradict the title of the docstring.

Yes it is -lof_scores

amueller · Oct 20, 2016

sklearn/neighbors/lof.py

+        return is_inlier
+
+    def decision_function(self, X):
+        """Opposite of the Local Outlier Factor of X (as bigger is better).


I think the docstring should be more explicit. Is low outlier or high outlier?
Actually, to be consistent with the other estimators, I think negative needs to be outlier.

I don't think so, for all the decision functions, bigger is better (large values correspond to inliers). For prediction, negative values (-1) correspond to outliers though. (It's true that this is a bit odd)

amueller · Oct 20, 2016

examples/covariance/plot_outlier_detection.py

@@ -18,6 +18,9 @@
  hence more adapted to large-dimensional settings, even if it performs
  quite well in the examples below.

+- using the Local Outlier Factor to measure the local deviation of a given


It's kinda odd that this example lives in this folder... but whatever..

Yes very weird! it is the folder of the first outlier detection algorithm in scikit-learn.

amueller · Oct 20, 2016

sklearn/neighbors/lof.py

+
+        # Avoid re-computing X_lrd if same parameters:
+        if not (np.all(distances_X == self._distances_fit_X_) *
+                np.all(self._neighbors_indices_fit_X_ == neighbors_indices_X)):


this == raises a deprecation warning

/home/andy/checkout/scikit-learn/sklearn/neighbors/lof.py:279: DeprecationWarning: elementwise == comparison failed; this will raise an error in the future.
np.all(self.neighbors_indices_fit_X == neighbors_indices_X)):

This means they have different size, I think. So I guess you should check the shape first?

Also happens for the line above.

amueller · Oct 20, 2016

doc/modules/outlier_detection.rst

+The question is not, how isolated the sample is, but how isolated it is
+with respect to the surrounding neighborhood.
+
+This strategy is illustrated below.


I don't feel that the example illustrates the point that was just made about the different densities. I'm fine to leave it as-is but I don't get a good idea of the global vs local. It would be nice to also illustrate a failure mode maybe?

No global vs local anymore!

amueller · Oct 20, 2016

sklearn/neighbors/lof.py

+        # Avoid re-computing X_lrd if same parameters:
+        if not (np.all(distances_X == self._distances_fit_X_) *
+                np.all(self._neighbors_indices_fit_X_ == neighbors_indices_X)):
+            lrd = self._local_reachability_density(


It seems that lrd is "small" compared to _distances_fit_X_ and _neighbors_indices_fit_X_. Why not compute it in fit and store it once and for all? You are currently recomputing it on every call to _local_outlier_factor.

amueller · Oct 20, 2016

sklearn/neighbors/lof.py

+        Parameters
+        ----------
+        distances_X : array, shape (n_query, self.n_neighbors)
+            Distances to the neighbors (in the training samples self._fit_X) of


I would put backticks around _fit_X to be save ;)

Do you mean replacing self._fit_X by self._fit_X or self._fit_X or just by _fit_X? I don't understand the purpose...

I meant putting backticks around self._fit_X. a) for nicer highlighting b) I'm not sure sphinx will render the current version correctly because of the underscore. But I might be paranoid.

amueller · Oct 20, 2016

sklearn/neighbors/tests/test_lof.py

+    score = clf.fit(X).outlier_factor_
+    assert_array_equal(clf._fit_X, X)
+
+    # Assert scores are good:


Assert smallest outlier score is is greater than largest inlier score

amueller · Oct 20, 2016

sklearn/neighbors/tests/test_lof.py

+    clf = neighbors.LocalOutlierFactor().fit(X_train)
+
+    # predict scores (the lower, the more normal)
+    y_pred = - clf.decision_function(X_test)


I would find it more natural to give the outliers the negative label. If you want to leave it like this, remove space after -

I agree but this is to be consistent with OneClassSVM, EllipticEnvelop and IsolationForest.

amueller · Oct 20, 2016

sklearn/neighbors/lof.py

+        distance between them. This works for Scipy's metrics, but is less
+        efficient than passing the metric name as a string.
+
+        Distance matrices are not supported.


I don't understand this comment.

amueller · Oct 24, 2016

amueller reviewed Oct 24, 2016

View changes

amueller · Oct 24, 2016

sklearn/neighbors/tests/test_lof.py

+    clf = neighbors.LocalOutlierFactor().fit(X_train)
+
+    # predict scores (the lower, the more normal)
+    y_pred = -clf.decision_function(X_test)


I meant changing y_test to be [0] * 20 + [-1] * 20 and then remove the -

amueller · Oct 24, 2016

sklearn/neighbors/lof.py

+
+        return self
+
+    def _predict(self, X=None):


I would really like to be consistent. I don't think there's a good argument to have one but not the other. Not sure if the example is a strong enough point to make them both public.

amueller · Oct 24, 2016

examples/covariance/plot_outlier_detection.py

    for i, (clf_name, clf) in enumerate(classifiers.items()):
        # fit the data and tag outliers
-        clf.fit(X)
-        scores_pred = clf.decision_function(X)
+        if clf_name == "Local Outlier Factor":


Wait, I don't understand this. Please elaborate.

amueller · Oct 24, 2016

sklearn/neighbors/lof.py

+    Attributes
+    ----------
+    outlier_factor_ : numpy array, shape (n_samples,)
+        The LOF of X. The lower, the more normal.


I don't know which comment of yours refers to which comment of mine.

For the first comment: Yes, I'd either do negative_outlier_factor or inlier_score or something generic?

For the second comment: The explanation of outlier_factor_ as an attribute says "The LOF of X" what is X? It's the training set this LocalOutlierFactor estimator was trained on, right?

ngoix added some commits Oct 24, 2016

amueller review + remove _local_outlier_factor method

Loading status checks…

f59d70a

add n_neighbors_ parameter the effective nb neighbors we use

Loading status checks…

6500640

make decision_function private and negative_outlier_factor attribute

Loading status checks…

bcab8ea

scikit-learn/scikit-learn

Join GitHub today

[MRG+2] LOF algorithm (Anomaly Detection) #5279

Conversation

ngoix commented Sep 16, 2015 • edited Edited 1 time ngoix edited Aug 19, 2016 (most recent)

This comment has been minimized.

agramfort commented Sep 16, 2015

This comment has been minimized.

jmschrei commented Sep 21, 2015

This comment has been minimized.

ngoix commented Oct 9, 2015

agramfort reviewed Oct 9, 2015 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

agramfort reviewed Oct 9, 2015 View changes

This comment has been minimized.

This comment has been minimized.

agramfort reviewed Oct 9, 2015 View changes

This comment has been minimized.

agramfort reviewed Oct 9, 2015 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

agramfort reviewed Oct 9, 2015 View changes

This comment has been minimized.

agramfort reviewed Oct 9, 2015 View changes

This comment has been minimized.

agramfort reviewed Oct 9, 2015 View changes

This comment has been minimized.

jmschrei reviewed Oct 9, 2015 View changes

This comment has been minimized.

jmschrei reviewed Oct 9, 2015 View changes

This comment has been minimized.

jmschrei reviewed Oct 9, 2015 View changes

This comment has been minimized.

jmschrei reviewed Oct 9, 2015 View changes

This comment has been minimized.

jmschrei reviewed Oct 9, 2015 View changes

This comment has been minimized.

This comment has been minimized.

jmschrei reviewed Oct 9, 2015 View changes

This comment has been minimized.

This comment has been minimized.

jmschrei reviewed Oct 9, 2015 View changes

This comment has been minimized.

This comment has been minimized.

jmschrei reviewed Oct 9, 2015 View changes

This comment has been minimized.

This comment has been minimized.

jmschrei reviewed Oct 9, 2015 View changes

This comment has been minimized.

jmschrei reviewed Oct 9, 2015 View changes

This comment has been minimized.

This comment has been minimized.

jmschrei reviewed Oct 9, 2015 View changes

This comment has been minimized.

jmschrei reviewed Oct 9, 2015 View changes

This comment has been minimized.

jmschrei reviewed Oct 9, 2015 View changes

This comment has been minimized.

This comment has been minimized.

jmschrei commented Oct 9, 2015

This comment has been minimized.

jmschrei commented Oct 9, 2015

This comment has been minimized.

ngoix commented Oct 12, 2015

This comment has been minimized.

agramfort commented Oct 12, 2015

This comment has been minimized.

ngoix commented Oct 12, 2015

This comment has been minimized.

ngoix commented Oct 12, 2015

agramfort reviewed Oct 12, 2015 View changes

This comment has been minimized.

agramfort reviewed Oct 12, 2015 View changes

This comment has been minimized.

ngoix commented Sep 16, 2015 •

edited

Edited 1 time

ngoix edited Aug 19, 2016 (most recent)

agramfort reviewed Oct 9, 2015

View changes

agramfort reviewed Oct 9, 2015

View changes

agramfort reviewed Oct 9, 2015

View changes

agramfort reviewed Oct 9, 2015

View changes

agramfort reviewed Oct 9, 2015

View changes

agramfort reviewed Oct 9, 2015

View changes

agramfort reviewed Oct 9, 2015

View changes

jmschrei reviewed Oct 9, 2015

View changes

jmschrei reviewed Oct 9, 2015

View changes

jmschrei reviewed Oct 9, 2015

View changes

jmschrei reviewed Oct 9, 2015

View changes

jmschrei reviewed Oct 9, 2015

View changes

jmschrei reviewed Oct 9, 2015

View changes

jmschrei reviewed Oct 9, 2015

View changes

jmschrei reviewed Oct 9, 2015

View changes

jmschrei reviewed Oct 9, 2015

View changes

jmschrei reviewed Oct 9, 2015

View changes

jmschrei reviewed Oct 9, 2015

View changes

jmschrei reviewed Oct 9, 2015

View changes

jmschrei reviewed Oct 9, 2015

View changes

agramfort reviewed Oct 12, 2015

View changes

agramfort reviewed Oct 12, 2015

View changes