[MRG+1] ransac is running too many iterations #8271

aivision2020 · 2017-02-02T11:53:54Z

Fixing number of iteration in ransac

lesteve · 2017-02-02T12:20:50Z

@ahojnnes you seem to be the one who implemented RANSAC, do you have an opinion on this change?

@aivision2020 a first step to convince me as someone who is not familiar at all with this code: run the examples using RANSAC and make sure that the output is similar to the one with master. You can find the latter on scikit-learn.org/dev/auto_examples.

jnothman · 2017-02-03T05:09:04Z

sklearn/linear_model/tests/ransac_iterations.py

+    yhat_ransac = clt_ransac.predict(X)
+    inlier_mask = clt_ransac.inlier_mask_
+    print '*************************'
+    print 'actual     trials:' , clt_ransac.n_trials_


use assert_equal(clt_ransac.n_trials_, _dynamic_max_trials(np.sum(inlier_mask), X.shape[0],
X.shape[1] + 1, clt_ransac.stop_probability))

It's a statistical algo so it can't be guaranteed that n_trials_ won't be higher, I'll check <= 21. That will give us a 1e-9 probability of failure

I see you have a test for this (test_ransac_max_trials). but there are only 3 outliers and you run the test once. if you up the outliers to 40 (~10%), don't set random_state to zero, it will run 4 iterations once every 100ish runs. but that's ok.

jnothman · 2017-02-03T05:09:05Z

sklearn/linear_model/tests/ransac_iterations.py

+Y[:80] += np.random.rand(80,1)
+
+for i in range(10):
+    clt_ransac = linear_model.RANSACRegressor(linear_model.LinearRegression(),max_trials=100)


we'd like to use fixed random state, but I realise this test is to ensure the property is invariant to randomness. So add random_state=i here.

I don't know what clt is here. maybe just call it reg

jnothman · 2017-02-03T05:09:07Z

sklearn/linear_model/tests/ransac_iterations.py

@@ -0,0 +1,20 @@
+from sklearn import linear_model


The test should be in sklearn/linear_model/tests/test_ransac.py and should be structured as a function with an assertion.

jnothman · 2017-02-03T05:09:11Z

sklearn/linear_model/ransac.py

@@ -415,13 +418,12 @@ def fit(self, X, y, sample_weight=None):
            X_inlier_best = X_inlier_subset
            y_inlier_best = y_inlier_subset

+            self.max_trials = min(self.max_trials, _dynamic_max_trials(n_inliers_best, n_samples,
+                                min_samples, self.stop_probability))


This should have aligned indentation as described by PEP8

aivision2020 · 2017-02-03T18:16:27Z

some pep8 stuff in ransac.py

lesteve · 2017-02-08T08:33:05Z

The CI errors seems genuine, please have a look at them.

some pep8 stuff in ransac.py fix data generation (some outliers were not)

aivision2020 · 2017-02-10T07:04:50Z

I've done my best.

jnothman · 2017-02-11T10:40:50Z

thank you. someone will take another look soon

…

On 10 Feb 2017 6:04 pm, "aivision2020" ***@***.***> wrote: I've done my best. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#8271 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz6_QT9VxzXVcAOJTYSxDRkQw-xjFCks5rbAwTgaJpZM4L1BNU> .

jnothman · 2017-02-13T13:39:03Z

The PEP8 failures:

./sklearn/linear_model/ransac.py:428:17: W503 line break before binary operator
                or score_best >= self.stop_score):
                ^

this is often a bit nasty, but you can change:

        if (...
            or score_best >= self.stop_score):
            # do stuff


to 
```python
            if ... \
               or score_best >= self.stop_score):
                # do stuff

for instance. Or:

            if (...
                    or score_best >= self.stop_score):
                # do stuff

./sklearn/linear_model/tests/test_ransac.py:95:80: E501 line too long (101 > 79 characters)
        max_trials = _dynamic_max_trials(len(X) - len(outliers), X.shape[0], X.shape[1] + 1, 1 - 1e9)

can be fixed as

        max_trials = _dynamic_max_trials(len(X) - len(outliers), X.shape[0],
                                         X.shape[1] + 1, 1 - 1e9)

jnothman · 2017-02-13T13:41:14Z

The AppVeyor issue may be a heisenbug, but is certainly not due to your code; the Circle issue isn't your problem either.

Sorry I've not yet reviewed the substance of the PR.

jnothman

Apart from these nitpicks and the PEP8 fixes, this LGTM

jnothman · 2017-02-15T12:27:57Z

sklearn/linear_model/tests/test_ransac.py

-data[outliers[1], :] = (-1000, -1000)
-data[outliers[2], :] = (-100, -50)
-
+np.random.seed(1)


global seeding isn't appropriate. You can use rng = np.random.RandomState(1)

OK, but it's important to seed with a different seed that I use in the test, or I will "randomly" sample only outliers. That was fun

I used rng = np.random.RandomState(1000)

jnothman · 2017-02-15T12:28:44Z

sklearn/linear_model/tests/test_ransac.py

-data[outliers[2], :] = (-100, -50)
-
+np.random.seed(1)
+outliers = (np.random.rand(80) * len(X)).astype(np.uint32)


this is np.random.randint(len(X), size=80), but more obfuscated?

jnothman · 2017-02-15T12:29:34Z

sklearn/linear_model/tests/test_ransac.py

-    assert getattr(ransac_estimator, 'n_trials_', None) is None
-    ransac_estimator.fit(X, y)
-    assert_equal(ransac_estimator.n_trials_, 2)
+    for i in range(400):


how long does this take to run? I think we'd rather a smaller number of runs.

it doesn't take long. I chose 400 based on the probability of the previous code to fail.

How long is not long?

1.6 seconds

I thought I had already said this is too long. Can you find some way to cut it back, using different probabilities, etc.

jnothman · 2017-02-15T12:31:11Z

sklearn/linear_model/tests/test_ransac.py

+                                           random_state=i)
+        assert getattr(ransac_estimator, 'n_trials_', None) is None
+        ransac_estimator.fit(X, y)
+        max_trials = _dynamic_max_trials(len(X) - len(outliers), X.shape[0], X.shape[1] + 1, 1 - 1e9)


for clarity, replace 1 - 1e9 with 1 - ransac_estimator.stop_probability

1e9 is a much smaller number than 1-ransac_estimator.stop_probability. Using the same stop_probability for estimation and varification would cause a non-trivial chance of failure even for well behaved code. I could use (1-ransac_estimator.stop_probability)*1e3. The point is that is should be much smaller

Okay. Please add a comment for what this is intending.

no global seed for np.random

aivision2020 · 2017-02-26T10:42:03Z

@jnothman so? Are you waiting for me?

jnothman

travis reports flake8 failures:


./sklearn/linear_model/ransac.py:427:55: E502 the backslash is redundant between brackets
            if (n_inliers_best >= self.stop_n_inliers \
                                                      ^
./sklearn/linear_model/ransac.py:428:25: E127 continuation line over-indented for visual indent
                        or score_best >= self.stop_score):
                        ^
./sklearn/linear_model/tests/test_ransac.py:96:9: E265 block comment should start with '# '
        #there is a 1e9 chance it will take thise many trials. No good reason
        ^
./sklearn/linear_model/tests/test_ransac.py:97:9: E265 block comment should start with '# '
        #1e2 isn't enough, can still happen
        ^
./sklearn/linear_model/tests/test_ransac.py:529:1: W391 blank line at end of file
^

Also note that once I've approved your PR, we will still await another reviewer.

jnothman · 2017-02-26T11:09:38Z

sklearn/linear_model/tests/test_ransac.py

-    assert getattr(ransac_estimator, 'n_trials_', None) is None
-    ransac_estimator.fit(X, y)
-    assert_equal(ransac_estimator.n_trials_, 2)
+    for i in range(400):


I thought I had already said this is too long. Can you find some way to cut it back, using different probabilities, etc.

jnothman · 2017-02-26T11:09:52Z

sklearn/linear_model/tests/test_ransac.py

+    for i in range(400):
+        ransac_estimator = RANSACRegressor(base_estimator, min_samples=2,
+                                           random_state=i)
+        assert getattr(ransac_estimator, 'n_trials_', None) is None


we avoid raw assert. Please use assert_true

codecov · 2017-02-27T16:12:53Z

Codecov Report

❗ No coverage uploaded for pull request base (master@53f8082). Click here to learn what that means.
The diff coverage is 100%.

@@            Coverage Diff            @@
##             master    #8271   +/-   ##
=========================================
  Coverage          ?   94.44%           
=========================================
  Files             ?      342           
  Lines             ?    61423           
  Branches          ?        0           
=========================================
  Hits              ?    58013           
  Misses            ?     3410           
  Partials          ?        0

Impacted Files	Coverage Δ
sklearn/linear_model/tests/test_ransac.py	`100% <100%> (ø)`
sklearn/linear_model/ransac.py	`98.67% <100%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 53f8082...8b63d25. Read the comment docs.

aivision2020 · 2017-03-13T05:16:17Z

@jnothman I made the pep8 changes. where does this stand?

jnothman

Where does this stand? In a large muddled mass of Pull Requests awaiting review, unfortunately. I'm happy to approve it, and would like you to add an entry in the changelog (whats_new.rst), but again, it will await another review.

jnothman · 2017-03-13T05:49:29Z

sklearn/linear_model/ransac.py

-                                              min_samples,
-                                              self.stop_probability)):
+            if n_inliers_best >= self.stop_n_inliers or \
+                            score_best >= self.stop_score:


This indentation is failing flake8. One option:

if n_inliers_best >= self.stop_n_inliers or \ score_best >= self.stop_score:

aivision2020 · 2017-03-13T06:27:06Z

For such a small fix this is a really arduous process. I hope it will make it in at some point.

…

On Mon, Mar 13, 2017 at 7:51 AM, Joel Nothman ***@***.***> wrote: ***@***.**** commented on this pull request. Where does this stand? In a large muddled mass of Pull Requests awaiting review, unfortunately. I'm happy to approve it, and would like you to add an entry in the changelog (whats_new.rst), but again, it will await another review. ------------------------------ In sklearn/linear_model/ransac.py <#8271 (comment)> : > # break if sufficient number of inliers or score is reached - if (n_inliers_best >= self.stop_n_inliers - or score_best >= self.stop_score - or self.n_trials_ - >= _dynamic_max_trials(n_inliers_best, n_samples, - min_samples, - self.stop_probability)): + if n_inliers_best >= self.stop_n_inliers or \ + score_best >= self.stop_score: This indentation is failing flake8. One option: if n_inliers_best >= self.stop_n_inliers or \ score_best >= self.stop_score: — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#8271 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANbZjw-qVuan-QJ42c5jp8Kex4Y3a2iTks5rlNlHgaJpZM4L1BNU> .

jnothman · 2017-03-13T07:07:24Z

I'm sorry that you feel the process is arduous, but a second reviewer often finds things the first didn't. Review also means that any individual is more likely to be familiar enough with some part of the code to maintain it. It's very possible that the pr will get merged without more input from you, but it may be slow on our part. we are a group of volunteers and lately have very little time between us to review.

…

On 13 Mar 2017 5:27 pm, "aivision2020" ***@***.***> wrote: For such a small fix this is a really arduous process. I hope it will make it in at some point. On Mon, Mar 13, 2017 at 7:51 AM, Joel Nothman ***@***.***> wrote: > ***@***.**** commented on this pull request. > > Where does this stand? In a large muddled mass of Pull Requests awaiting > review, unfortunately. I'm happy to approve it, and would like you to add > an entry in the changelog (whats_new.rst), but again, it will await > another review. > ------------------------------ > > In sklearn/linear_model/ransac.py > <#8271# discussion_r105593845> > : > > > # break if sufficient number of inliers or score is reached > - if (n_inliers_best >= self.stop_n_inliers > - or score_best >= self.stop_score > - or self.n_trials_ > - >= _dynamic_max_trials(n_inliers_best, n_samples, > - min_samples, > - self.stop_probability)): > + if n_inliers_best >= self.stop_n_inliers or \ > + score_best >= self.stop_score: > > This indentation is failing flake8. One option: > > if n_inliers_best >= self.stop_n_inliers or \ > score_best >= self.stop_score: > > — > You are receiving this because you modified the open/close state. > Reply to this email directly, view it on GitHub > <#8271# pullrequestreview-26479926>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ANbZjw-qVuan- QJ42c5jp8Kex4Y3a2iTks5rlNlHgaJpZM4L1BNU> > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8271 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz69uq9PiVAvQY1AtYIF3gJDhzhTTuks5rlOG8gaJpZM4L1BNU> .

MechCoder · 2017-04-03T19:34:49Z

sklearn/linear_model/tests/test_ransac.py

+        # there is a 1e9 chance it will take these many trials. No good reason
+        # 1e2 isn't enough, can still happen
+        max_trials = _dynamic_max_trials(
+            len(X) - len(outliers), X.shape[0], X.shape[1] + 1, 1 - 1e9)


Isn't this supposed to be 1 - 1e-9?!

Can you change X.shape[1] + 1 explicitly to 3. It is not obvious that the default number of samples are set to number of features + 1 just by looking at the test.

Also you can move this computation outside the for loop.

I like shape[1]+1 since that's what ransac uses. I don't really mind though. I'll put it in comment

and X.shape[1] + 1 = 2 (which is the min number of samples needed for a linear estimator in in 2d

MechCoder · 2017-04-03T19:36:56Z

sklearn/linear_model/tests/test_ransac.py

-    ransac_estimator.fit(X, y)
-    assert_equal(ransac_estimator.n_trials_, 2)
+    for i in range(50):
+        ransac_estimator = RANSACRegressor(base_estimator, min_samples=2,


nitpick: You can move this to the outside and use set_params to avoid creating many objects.

Actually can you please move this to a separate test?

This new test is in spirit of the original test except the extra iterations

MechCoder · 2017-04-03T19:51:28Z

Thanks for your contribution! Can you please document this change under whatsnew and fix the pending flake error?

jnothman · 2017-04-03T22:51:44Z

The 1 - 1e-9 is correct: 1-p for some small probability. I also have a preference for n_features+1 over constant 3 so that the code was more literate, but i don't mind. perhaps comments in the test are deserved.

…

On 4 Apr 2017 5:51 am, "Manoj Kumar" ***@***.***> wrote: Thanks for your contribution! Can you please document this change under whatsnew and fix the pending flake error? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8271 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz60AaBmtU1YT_TNNshaP0s3-rsJ9dks5rsU3CgaJpZM4L1BNU> .

MechCoder · 2017-04-03T23:35:04Z

yeah, I know that. But in the code it is (1 - 1e9) not (1 - 1e-9)

jnothman · 2017-04-03T23:47:42Z

oh. whoops.

…

On 4 April 2017 at 09:35, Manoj Kumar ***@***.***> wrote: yeah, I know that. But in the code it is (1 - 1e9) not (1 - 1e-9) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8271 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz6ytr-uoU1RbTKj2O0Z6IToyasayGks5rsYIpgaJpZM4L1BNU> .

Move computation out of loop, fix 1e-9 (from 1e9)...

aivision2020 · 2017-04-22T09:42:43Z

where do I edit this whatsnew?

…

On Mon, Apr 3, 2017 at 10:51 PM, Manoj Kumar ***@***.***> wrote: Thanks for your contribution! Can you please document this change under whatsnew and fix the pending flake error? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#8271 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANbZjyzQ9kKeRrVpNoiHioM_RTJ_pgzGks5rsU3fgaJpZM4L1BNU> .

jnothman · 2017-04-22T10:01:20Z

doc/whats_new.rst, under bug fixes

…

On 22 April 2017 at 19:42, aivision2020 ***@***.***> wrote: where do I edit this whatsnew? On Mon, Apr 3, 2017 at 10:51 PM, Manoj Kumar ***@***.***> wrote: > Thanks for your contribution! Can you please document this change under > whatsnew and fix the pending flake error? > > — > You are receiving this because you modified the open/close state. > Reply to this email directly, view it on GitHub > <#8271# issuecomment-291254042>, > or mute the thread > <https://github.com/notifications/unsubscribe- auth/ANbZjyzQ9kKeRrVpNoiHioM_RTJ_pgzGks5rsU3fgaJpZM4L1BNU> > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8271 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz6xLotKeIt8IWOqSUhxmQACnEIcr0ks5rycuVgaJpZM4L1BNU> .

MechCoder · 2017-04-24T00:29:36Z

sklearn/linear_model/tests/test_ransac.py

+    ransac_estimator = RANSACRegressor(base_estimator, min_samples=2)
+    for i in range(50):
+        ransac_estimator.set_params(min_samples=2,random_state=i)
+        ransac_estimator.n_trials_ = None


You can remove this line. It will be overwritten by fit.

MechCoder · 2017-04-24T00:30:01Z

Can you please fix the couple of pep8 errors? LGTM apart from that.

glemaitre · 2017-04-29T11:46:52Z

sklearn/linear_model/ransac.py

@@ -415,13 +418,14 @@ def fit(self, X, y, sample_weight=None):
            X_inlier_best = X_inlier_subset
            y_inlier_best = y_inlier_subset

+            max_trials = min(
+                max_trials,


max_trials = min(max_trials, _dynamic_max_trials(n_inliers_best, n_samples, min_samples, self.stop_probability))

IMO, this is more readable.

glemaitre · 2017-04-29T11:57:36Z

sklearn/linear_model/tests/test_ransac.py

+    # 1e-2 isn't enough, can still happen
+    # 2 is the what ransac defines  as min_samples = X.shape[1] + 1
+    max_trials = _dynamic_max_trials(
+        len(X) - len(outliers), X.shape[0], 2, 1 - 1e-9)


It would be easier to read without the need to refer to the docstring:

max_trials = _dynamic_max_trials(n_inliers=len(X) - len(outliers), n_samples=len(X), min_samples=2, probability=1 - 1e-9)

you guys are killing me

Isn't the only difference here that len(X) is used instead of X.shape[0]?

Anyway, yes, I agree we need to try less nitpicking on this one.

Oh, I see, visual indent stuff. No matter.

jnothman · 2017-04-29T21:55:26Z

Thanks, @aivision2020. Merging

jnothman · 2017-04-29T21:58:23Z

And thanks for raising the issue and persisting with it.

…earn#8271)

ransac is running too many iterations

9ea8d38

jnothman requested changes Feb 3, 2017

View reviewed changes

Moved ransac iteration test to test_ransac

89d0225

some pep8 stuff in ransac.py

aivision2020 closed this Feb 3, 2017

aivision2020 reopened this Feb 3, 2017

aivision2020 added 5 commits February 9, 2017 22:57

Moved ransac iteration test to test_ransac

97cc890

some pep8 stuff in ransac.py fix data generation (some outliers were not)

dont change self.max_trials

b60bf32

flake8 line too long

c77ea88

flake8 line too long

e2830b6

flake8 line too long

838ff5b

jnothman reviewed Feb 15, 2017

View reviewed changes

aivision2020 added 4 commits February 15, 2017 18:26

Some pep8

7b0fca4

no global seed for np.random

document

dce49e5

document

dc9d1dd

document

fbdc2ef

jnothman approved these changes Feb 26, 2017

View reviewed changes

jnothman reviewed Feb 26, 2017

View reviewed changes

jnothman reviewed Mar 13, 2017

View reviewed changes

jnothman changed the title ~~ransac is running too many iterations~~ [MRG+1] ransac is running too many iterations Mar 13, 2017

MechCoder reviewed Apr 3, 2017

View reviewed changes

refactor for @MechCoder

78b48a3

Move computation out of loop, fix 1e-9 (from 1e9)...

Add documentation to whats_new

ca7d95a

MechCoder reviewed Apr 24, 2017

View reviewed changes

aivision2020 added 2 commits April 28, 2017 20:47

Add documentation to whats_new

99d764d

Add documentation to whats_new

b62aeb7

glemaitre reviewed Apr 29, 2017

View reviewed changes

jnothman added 2 commits April 30, 2017 07:56

Fix typo in whatsnew

bc2fb64

more whats new formatting

8b63d25

jnothman merged commit 6e8ebda into scikit-learn:master Apr 29, 2017

Sundrique pushed a commit to Sundrique/scikit-learn that referenced this pull request Jun 14, 2017

[MRG+1] FIX bug where ransac is running too many iterations (scikit-l…

02a5024

…earn#8271)

dmohns pushed a commit to dmohns/scikit-learn that referenced this pull request Aug 7, 2017

[MRG+1] FIX bug where ransac is running too many iterations (scikit-l…

62f4b8a

…earn#8271)

dmohns pushed a commit to dmohns/scikit-learn that referenced this pull request Aug 7, 2017

[MRG+1] FIX bug where ransac is running too many iterations (scikit-l…

40cf8a8

…earn#8271)

NelleV pushed a commit to NelleV/scikit-learn that referenced this pull request Aug 11, 2017

[MRG+1] FIX bug where ransac is running too many iterations (scikit-l…

34a3bc9

…earn#8271)

paulha pushed a commit to paulha/scikit-learn that referenced this pull request Aug 19, 2017

[MRG+1] FIX bug where ransac is running too many iterations (scikit-l…

dbcfee8

…earn#8271)

AishwaryaRK pushed a commit to AishwaryaRK/scikit-learn that referenced this pull request Aug 29, 2017

[MRG+1] FIX bug where ransac is running too many iterations (scikit-l…

52aea47

…earn#8271)

maskani-moh pushed a commit to maskani-moh/scikit-learn that referenced this pull request Nov 15, 2017

[MRG+1] FIX bug where ransac is running too many iterations (scikit-l…

6c38d95

…earn#8271)

jwjohnson314 pushed a commit to jwjohnson314/scikit-learn that referenced this pull request Dec 18, 2017

[MRG+1] FIX bug where ransac is running too many iterations (scikit-l…

e6bb8e1

…earn#8271)

[MRG+1] ransac is running too many iterations #8271

[MRG+1] ransac is running too many iterations #8271

Conversation

aivision2020 commented Feb 2, 2017 • edited by jnothman Loading

lesteve commented Feb 2, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aivision2020 commented Feb 3, 2017

lesteve commented Feb 8, 2017

aivision2020 commented Feb 10, 2017

jnothman commented Feb 11, 2017 via email

jnothman commented Feb 13, 2017

jnothman commented Feb 13, 2017

jnothman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aivision2020 commented Feb 26, 2017

jnothman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Feb 27, 2017 • edited Loading

Codecov Report

aivision2020 commented Mar 13, 2017

jnothman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aivision2020 commented Mar 13, 2017 via email

jnothman commented Mar 13, 2017 via email

MechCoder Apr 3, 2017 • edited Loading

Choose a reason for hiding this comment

MechCoder Apr 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MechCoder commented Apr 3, 2017

jnothman commented Apr 3, 2017 via email

MechCoder commented Apr 3, 2017

jnothman commented Apr 3, 2017 via email

aivision2020 commented Apr 22, 2017 via email

jnothman commented Apr 22, 2017 via email

Choose a reason for hiding this comment

MechCoder commented Apr 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jnothman commented Apr 29, 2017

jnothman commented Apr 29, 2017

aivision2020 commented Feb 2, 2017 •

edited by jnothman

Loading

codecov bot commented Feb 27, 2017 •

edited

Loading

MechCoder Apr 3, 2017 •

edited

Loading

MechCoder Apr 3, 2017 •

edited

Loading