Code classifier frontend prediction API services. #3719

prasanna08 · 2017-08-07T11:00:21Z

This PR implements frontend prediction API for code classifier. It implements all necessary reprocessing and predict function. Also changed eslint ecmaVersion to 6 because this PR uses backtick strings (for raw strings).

This PR can only be merged after @pranavsid98's #3716 is merged. Until then we should review this PR.

prasanna08 · 2017-08-07T11:01:26Z

@anmolshkl @AllanYangZhou @seanlip can you check the overall structure and placement of different modules? Also I am currently working on frontend tests.

seanlip

Just commenting on code structure as requested. Leaving further review to @AllanYangZhou.

seanlip · 2017-08-08T00:46:30Z

core/templates/dev/head/services/classifier/CountVectorizerService.js

@@ -0,0 +1,45 @@
+// Copyright 2017 The Oppia Authors. All Rights Reserved.


Maybe put these in templates/dev/head/domain/classifier. They're not specific to any one interaction, right?

OK. No they are not specific to any interaction.

seanlip · 2017-08-08T00:50:58Z

extensions/interactions/CodeRepl/CodeClassifierPredictionService.js

+ * Oppia-ml.
+ */
+
+oppia.factory('CodeClassifierPredictionService', [


Maybe call this CodeReplPredictionService so we have a standard naming scheme for these things ("interaction id" + prediction service). Also some changes may be needed to the .py file to say whether such a service exists so that the relevant files can be imported only if necessary?

We can include these services directly in CodeRepl.html, can't we? Then they will be included automatically every time CodeRepl interaction is included. Or are you talking about something else here?

Yes -- look at how the interaction stuff gets loaded; there are files in the extensions/interaction/{{InteractionID}} directory and they are named in quite a specific way. So no, don't include things directly in CodeRepl.html. You probably need to:

add a bool attr to the interaction .py file definition that indicates whether the interaction supports a prediction service

for interactions that do support a prediction service, add that service to the {{InteractionId}}.js file

write a backend test to ensure that all interactions have that boolean defined in their .py file, and that all interactions with that boolean set to True have a service with the appropriate name defined in their JS file. You can just do string matching on the raw JS file, I think.

I think this should go into @pranavsid98's PR. Since he is going to manage how prediction services will be detected and loaded. Also classifiers may also require additional services (like code classifier requires SVMPredictionService, CountVectorizerService etc...), how should we keep track of those extra services? May be maintain a dependency list for each classifier.

Also I think it'd be better to not have tight coupling between interaction and classifier. This enables us to use multiple classifiers per interaction. If we name classifier service same as interaction id then we will be able to use only prediction algorithm at a time. Consider a scenario where we upgarde an existing classifier code. In that case, for backward compatibility, if exploration has old version of classifier data then we can still use old prediction API. We can't do this if we name each prediction service with its interaction id. In general, I think, it would be better to name prediction service same with its algorithm id rather than interaction id. What do you think?

AllanYangZhou · 2017-08-08T07:09:47Z

Overall structure seems fine, I'll do a more detailed review tomorrow.

Regarding the use of template strings/ES6, won't there be an issue of browser compatability?

prasanna08 · 2017-08-08T09:47:48Z

@AllanYangZhou, according to this most browsers now days support for template literals (except older IE11 browser). So this should be fine as long as user's browser is updated.

codecov-io · 2017-08-08T10:11:36Z

Codecov Report

Merging #3719 into develop will increase coverage by 0.59%.
The diff coverage is 70.39%.

@@             Coverage Diff             @@
##           develop    #3719      +/-   ##
===========================================
+ Coverage    45.04%   45.63%   +0.59%     
===========================================
  Files          265      275      +10     
  Lines        20510    20956     +446     
  Branches      3184     3259      +75     
===========================================
+ Hits          9238     9563     +325     
- Misses       11272    11393     +121

Impacted Files	Coverage Δ
...domain/classifier/WinnowingPreprocessingService.js	`100% <100%> (ø)`
...v/head/domain/classifier/CountVectorizerService.js	`100% <100%> (ø)`
...interactions/CodeRepl/CodeReplPredictionService.js	`53.48% <53.48%> (ø)`
...v/head/domain/classifier/PythonProgramTokenizer.js	`67.41% <67.41%> (ø)`
...dev/head/domain/classifier/SVMPredictionService.js	`98.21% <98.21%> (ø)`
...ead/domain/exploration/InteractionObjectFactory.js	`68.57% <0%> (-11.43%)`	⬇️
.../dev/head/domain/exploration/StateObjectFactory.js	`80% <0%> (-6.67%)`	⬇️
.../exploration_player/AnswerClassificationService.js	`95.65% <0%> (-1.79%)`	⬇️
...s/dev/head/pages/library/SearchResultsDirective.js	`5.88% <0%> (-0.37%)`	⬇️
...v/head/pages/learner_dashboard/LearnerDashboard.js	`2.76% <0%> (-0.03%)`	⬇️
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b2ccf54...6c08be6. Read the comment docs.

prasanna08 · 2017-08-08T10:12:01Z

I have added few frontend tests for different services. I will add rest of the tests later.

anmolshkl

@prasanna08 a few minor changes.

This looks a little complex and I can imagine the efforts had to put in! Really great! @seanlip Prasanna had to do a lot of duplication (of libraries/our python code) to get this in place. In the long term, does front end prediction looks maintainable to you? The scope for things to go wrong is just huge with front end prediction.

anmolshkl · 2017-08-08T12:18:45Z

core/templates/dev/head/domain/classifier/CountVectorizerService.js

+// limitations under the License.
+
+/**
+ * Vecotrize function for CountVectorizer feature extractor of sklearn.


Vecotrize -> Vectorizer
Perhaps we can say -> Vectorizer function which mirrors the CountVectorizer feature extractor of sklearn

anmolshkl · 2017-08-08T12:19:17Z

core/templates/dev/head/domain/classifier/CountVectorizerService.js

+/**
+ * Vecotrize function for CountVectorizer feature extractor of sklearn.
+ *
+ * IMPORTANT NOTE: The vectorize function is uses the vocabulary that was


Vectorizer*
is uses -> uses

anmolshkl · 2017-08-08T12:19:31Z

core/templates/dev/head/domain/classifier/CountVectorizerService.js

+ * Vecotrize function for CountVectorizer feature extractor of sklearn.
+ *
+ * IMPORTANT NOTE: The vectorize function is uses the vocabulary that was
+ * extracted during classifier's training. During training scikit's


During training,

anmolshkl · 2017-08-08T12:19:45Z

core/templates/dev/head/domain/classifier/CountVectorizerService.js

+ *
+ * IMPORTANT NOTE: The vectorize function is uses the vocabulary that was
+ * extracted during classifier's training. During training scikit's
+ * CountVectorize class is used for this purpose. If there are any changes


CountVectorizer*

seanlip · 2017-08-09T05:02:23Z

This looks a little complex and I can imagine the efforts had to put in! Really great! @seanlip Prasanna had to do a lot of duplication (of libraries/our python code) to get this in place. In the long term, does front end prediction looks maintainable to you? The scope for things to go wrong is just huge with front end prediction.

Hm, yeah -- the duplication is a pity, but how would you do ML offline or in poor-connectivity environments otherwise? I think this is probably the best we can do...

AllanYangZhou · 2017-08-09T07:08:45Z

core/templates/dev/head/domain/classifier/PythonProgramTokenizer.js

+              pos += 1;
+            }
+
+            if (pos == max) {


Preferable to use ===, here and elsewhere

AllanYangZhou · 2017-08-09T07:30:59Z

core/templates/dev/head/domain/classifier/PythonProgramTokenizer.js

+      return group(arguments) + '?';
+    };
+
+    var any = function() {


Maybe more descriptive names for these 3 functions?

Using groupOfRegEx, regExMayBePresent, repeatedRegEx for them respectively.

AllanYangZhou · 2017-08-09T07:32:33Z

core/templates/dev/head/domain/classifier/PythonProgramTokenizer.js

+    };
+
+    var Whitespace = String.raw`[ \f\t]*`;
+    var Comment = String.raw`#[^\r\n]*`;


Comment is keyword in JS. Also, is there a reason to uppercase these variables?

No specific reason, I just kept them as it is. I will change them to lowercase.

AllanYangZhou · 2017-08-09T07:35:40Z

@seanlip Do we still care about IE support? I think a lot of these ES6 features don't work on IE at all.

seanlip · 2017-08-09T07:40:38Z

It would be nice to support some version of IE, especially since this is the learner view and students do not always have control over the computers which are available to them. @prasanna08 could we do that?

anmolshkl · 2017-08-09T08:35:45Z

Just FYI, I tried using Oppia on IE 11 once and many things (like editor) completely broke on it. So, I'm not sure about supporting IE unless we are talking about Microsoft edge browser.

…

On 09-Aug-2017 1:10 PM, "Sean Lip" ***@***.***> wrote: It would be nice to support some version of IE, especially since this is the learner view and students do not always have control over the computers which are available to them. @prasanna08 <https://github.com/prasanna08> could we do that? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3719 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGC47wL9-zdsacUacPv35aaewZrks7MKks5sWWJ5gaJpZM4OvSXU> .

seanlip · 2017-08-09T08:38:14Z

Yeah, I think the editor view may have problems (and I'm not actually too worried about that). But what about the learner view? If we are to be accessible to learners then we need to try and meet them where they're at, and that includes their browser. I thought the learner view currently works OK in IE...

prasanna08 · 2017-08-09T12:34:10Z

@seanlip if its not very much helpful then I'd rather not do that. We need ES6 for raw strings of regular expression. And its difficult to convert them to escaped strings, that's why many prefer raw strings over escaped strings when it comes to regex. So if its not much helpful than it would be better to use raw strings. I don't even know that whether it will be possible to convert each raw string into escaped string. But I will give it a shot if this is important feature.

Aside from that, can't we fallback to default outcome in case of IE browser?

anmolshkl · 2017-08-09T12:48:39Z

Hmm the learner view seems ok but the other pages that lead you to the learner view seems to be broken. I'm not sure if a learner would actually play any exploration after having a bad experience unless they are directly going to play a particular exploration.

…

On 09-Aug-2017 2:08 PM, "Sean Lip" ***@***.***> wrote: Yeah, I think the editor view may have problems (and I'm not actually too worried about that). But what about the learner view? If we are to be accessible to learners then we need to try and meet them where they're at, and that includes their browser. I thought the learner view currently works OK in IE... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3719 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGC473_VSvWZr2iyNJ3FH4u2DgMUWK3Nks5sWW_7gaJpZM4OvSXU> .

seanlip · 2017-08-09T19:20:23Z

I'm torn. IE 11 is apparently the second most popular browser after Chrome: https://www.netmarketshare.com/browser-market-share.aspx?qprid=2&qpcustomd=0 . So, definitely non-negligible, and in fact we've generally tried to support later versions of IE in the past (with a cutoff of 8 or 9). If the pages leading up to the learner view are broken, then we really should fix that (@anmolshkl, would you mind filing issues for this please?) It's similar to how we currently do i18n for the learner-facing but not the editor-facing pages.

On the other hand I do see the issue here, and I agree that not using String.raw would make the code quite a bit less maintainable.

I think a reasonable thing to do is preserve the current impl, but make sure it fails gracefully on IE. So, e.g., explorations with no code classification should still work fine, but explorations with code classification should show an error message at the time the classification is triggered that says something like "Answer classification does not work in IE. Please use another browser." and return the default outcome. That way, there are no surprises, and the page doesn't just break completely under the user. Would that work?

AllanYangZhou · 2017-08-09T20:57:13Z

I think disabling classification on IE and then returning the warning message (without breaking completely) is a good compromise for now.

prasanna08 · 2017-08-10T10:49:20Z

Returning warning message sounds good to me. But I am not sure about how that message should be displayed to learner. Some sort of prompt or pop-up box?

seanlip · 2017-08-10T10:51:15Z

Actually, thinking about it, you probably don't need to surface this to the learner. Just do a $log.error(); that should suffice.

prasanna08 · 2017-08-10T10:52:22Z

Yep that's what I thought. Learner does not need to know about it. I will make that change.

Edit: I think there might be a way to use escape strings somehow. Look at this. I'll try this out and let you guys know if it works.

prasanna08 · 2017-08-10T11:41:45Z

I have removed use of template literals, so now there shouldn't be any compatibility issue. PR is ready for review now.

AllanYangZhou · 2017-08-10T19:00:05Z

core/templates/dev/head/domain/classifier/CountVectorizerService.js

+// limitations under the License.
+
+/**
+ * Vecotrizer function which mirrors the CountVectorizer feature


AllanYangZhou · 2017-08-10T19:02:04Z

core/templates/dev/head/domain/classifier/CountVectorizerService.js

+oppia.factory('CountVectorizerService', [function() {
+  return {
+    vectorize: function(tokens, vocabulary) {
+      var vectorLength = Math.max.apply(Math, Object.values(vocabulary)) + 1;


Object.values is also ES6 like template literals.

Oh I see. I will change it to Object.key because we only need to know length, so it does not matter whether we use keys or values. Is that fine?

Yeah, seems fine to use keys instead.

AllanYangZhou · 2017-08-10T19:05:46Z

core/templates/dev/head/domain/classifier/PythonProgramTokenizer.js

+              else if (namechars.indexOf(initial) !== -1) {
+                tokenizedProgram.push([PythonProgramTokenType.NAME, token]);
+              }
+              // continued stmt


Better to just spell this out.

AllanYangZhou · 2017-08-10T19:12:36Z

core/templates/dev/head/domain/classifier/SVMPredictionService.js

+      var maxVoteIdx = 0;
+      for(var i = 0; i < votes.length; i++) {
+        if (votes[i] > votes[maxVoteIdx]) {
+          maxVoteIdx = votes[i];


I'm a bit confused reading this. The name maxVoteIdx suggests we're trying to find an index, not the max number of votes, right? Unless you are actually trying to find the biggest number of votes, but then you should call it maxVotes.

Oh, the code is wrong. I want index and not votes. Thanks for spotting it.

Great! Btw, were any of the SVMPredictionService tests supposed to have caught this error?

AllanYangZhou · 2017-08-10T19:15:38Z

extensions/interactions/CodeRepl/CodeReplPredictionService.js

+    var TOKEN_NAME_UNK = 'UNK';
+
+    // List of python keywords.
+    var KWLIST = [


minor nit: KW_LIST is slightly clearer

AllanYangZhou · 2017-08-10T19:18:19Z

core/templates/dev/head/domain/classifier/PythonProgramTokenizer.js

+            }
+          }
+
+          // new statement


For consistency, capitalize and add a period (same applies for all other comments).

AllanYangZhou · 2017-08-12T21:53:09Z

@prasanna08 Looking good overall. I guess we should get all the checks fixed.

prasanna08 · 2017-08-14T02:12:42Z

@AllanYangZhou fixed tests. PTAL?

anmolshkl

@prasanna08 just a couple of comments here. I'm trusting you and the tests here :P

anmolshkl · 2017-08-14T13:21:47Z

core/templates/dev/head/domain/classifier/CountVectorizerService.js

+ * extractor of sklearn.
+ *
+ * IMPORTANT NOTE: The Vectorizer function uses the vocabulary that was
+ * extracted during training. During training scikit's


During the training,

anmolshkl · 2017-08-14T13:41:19Z

core/templates/dev/head/domain/classifier/PythonProgramTokenizer.js

+/**
+ * Tokenizer for Python code.
+ *
+ * IMPORTANT NOTE: The tokenizer is build using Python's own tokenizer module.


build -> built

anmolshkl · 2017-08-14T13:42:14Z

core/templates/dev/head/domain/classifier/PythonProgramTokenizer.js

+ * IMPORTANT NOTE: The tokenizer is build using Python's own tokenizer module.
+ * These functions are simply translated from Python code to JS code and they
+ * both do same task. The unnecessary code from Python's tokenizer module
+ * has been removed before translating it into JS andcCode relevant to


anmolshkl · 2017-08-14T13:48:15Z

core/templates/dev/head/domain/classifier/PythonProgramTokenizerSpec.js

+        'num in range(1000):\n  if num%7 == 0 or num%5 == 0:\n\ts +=x\n' +
+        'print s');
+
+      var expectedTokens = [


I believe that this matches the output produced by Python's tokenizer?

Not exactly. The python tokenizer produces \n tokens but this one fails to do so. Other than that all the important tokens are same. I think the issue with \n is that python's tokenizer uses StringIO for reading input line but here I am splitting the program with \n before passing it to tokenizer. But all other tokens are same as python's tokenizer.

so the count for '\n' in the count vector would always be 0 in the case of front-end count-vectorizer?

Yes, unfortunately, that's likely to happen.

Hmm, I'm not sure what impact this could have (the impact could be negligible or it might decrease the accuracy significantly). If we are splitting on '\n', then, can't we just add '\n' tokens later?

I just remembered this, we are already filtering out NL and COMMENT tokens (in backend as well as frontend) and all \n are NL tokens, so they are not present in backend either. That means we are not losing anything by not having \n tokens.

anmolshkl · 2017-08-14T13:58:47Z

core/templates/dev/head/domain/classifier/SVMPredictionService.js

+        startIndices[i] = startIndices[i - 1] + nSupport[i - 1];
+      }
+
+      // Find kernel values for supportVectors and given input. Assumes that


Shouldn't there be a check that flags an error if the dimensions are not same?

Added a check.

anmolshkl · 2017-08-14T14:06:16Z

core/templates/dev/head/domain/classifier/SVMPredictionServiceSpec.js

+      var svmData = {
+        classes: [0, 1], 
+        n_support: [2, 2], 
+        support_vectors: [[1.0, 1.0], [0.0, 0.0], [1.0, 0.0], [0.0, 1.0]], 


is this generated with scikit?

Yep, I trained XOR on scikit and extracted data using classifier_utils.extract_svm_parameter() function.

AllanYangZhou · 2017-08-14T21:28:49Z

@prasanna08 The code looks fine, the only concern I have is why the SVMPredictionServiceTests didn't catch the maxVoteIdx bug from earlier?

prasanna08 · 2017-08-15T01:22:00Z

@AllanYangZhou so the reason for that is that there were only 2 classes. So let me explain all the cases one by one.

Input [1, 1] produced votes [1, 0]. That is it gave 1 vote to output class 0 and 0 votes to output class 1.
Input [1, 0] produced votes [0, 1]
Input [0, 1] produced votes [0, 1]
Input [0, 0] produced votes [1, 0]

Now my old logic was like this:

maxVoteIdx = 0;
for (var i = 0; i < votes.length; i++) {
  if (votes[i] > maxVoteIdx) {
    maxVoteIdx = i;
  }
}

So let's see what happens in case of [1, 0] votes. There will be 2 passes of for loop.

votes[0] > 0 which is True (1 > 0). Therefore maxVoteIdx = 0;
votes[1] > 0 which is False (0 = 0). So no change.
Therefore returned value is class[maxVoteIdx] which is 0 and that is True in both cases when [1, 0] votes are generated.

Now let's see what happens for [0, 1]

votes[0] > 0. False.
votes[1] > 0. True (1 > 0). maxVoteIdx = 1;
Therefore returned class is 1 and that is True for both cases when votes are [0, 1].

So the result was correct in flawed logic because we only had 2 classes and votes were binary (either 0 vote or 1 vote). If we had few more classes then we would also have more votes per class and error may have been identified at that time. Is that understandable?

…nto code-classifier-prediction

…ier.

AllanYangZhou · 2017-08-16T16:45:27Z

@prasanna08 Alright, I understand now. Thanks!

AllanYangZhou · 2017-08-16T16:48:50Z

extensions/interactions/CodeRepl/CodeReplPredictionService.js

+
+          if (
+            tokenId == PythonProgramTokenType.NL ||
+            tokenId == PythonProgramTokenType.COMMENT ||


=== preferred (and elsewhere)

prasanna08 · 2017-08-18T01:55:57Z

@anmolshkl can you take a final look. So that this can be merged and I can enable code classifier in development instance for some end to end testing.

prasanna08 added 3 commits August 7, 2017 15:40

Implement frontend prediction API for code classifier.

b0393e2

Change module name to oppia

56411e0

Use $log.

7b6c609

prasanna08 changed the title ~~[NOT FOR MERGE]: Code classifier prediction API~~ [WIP]: Code classifier prediction API Aug 7, 2017

seanlip reviewed Aug 8, 2017

View reviewed changes

Add frontend tests for classifier service functions.

6db8aff

anmolshkl suggested changes Aug 8, 2017

View reviewed changes

AllanYangZhou reviewed Aug 9, 2017

View reviewed changes

Addressed review comments

d7316ff

Add more frontend test

3822ffb

Removed template literals from tokenizer

25bdaf6

AllanYangZhou suggested changes Aug 10, 2017

View reviewed changes

prasanna08 and others added 2 commits August 11, 2017 06:18

Addressed review comments.

40c2ab5

Fix non-embedded view

b48e5d6

prasanna08 added 2 commits August 13, 2017 21:13

Fix backend tests

d4ae15d

Fix e2e test.

c06a83c

Fix non-embedded view

846e135

anmolshkl suggested changes Aug 14, 2017

View reviewed changes

prasanna08 added 6 commits August 15, 2017 06:55

Addressed review comments.

d6b0f33

Merge branch 'oppia3612c' of https://github.com/utkarsh-dixit/oppia i…

23c5b7d

…nto code-classifier-prediction

Create tokens using tokenizer before filtering tokens in code classif…

df6b673

…ier.

Add separate tokenizer for count vector.

ae37dd7

REVERT 23c5b7d (accidental merger)

0fde9fd

Nit changes.

d7dbd38

AllanYangZhou suggested changes Aug 16, 2017

View reviewed changes

Addressed review comments

6c08be6

AllanYangZhou approved these changes Aug 17, 2017

View reviewed changes

prasanna08 changed the title ~~[WIP]: Code classifier prediction API~~ Code classifier frontend prediction API services. Aug 17, 2017

anmolshkl approved these changes Aug 18, 2017

View reviewed changes

anmolshkl merged commit 12df8f6 into oppia:develop Aug 18, 2017

		@@ -0,0 +1,45 @@
		// Copyright 2017 The Oppia Authors. All Rights Reserved.

Code classifier frontend prediction API services. #3719

Code classifier frontend prediction API services. #3719

Conversation

prasanna08 commented Aug 7, 2017 • edited

prasanna08 commented Aug 7, 2017

seanlip left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prasanna08 Aug 8, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prasanna08 Aug 9, 2017 • edited

Choose a reason for hiding this comment

AllanYangZhou commented Aug 8, 2017

prasanna08 commented Aug 8, 2017

codecov-io commented Aug 8, 2017 • edited

Codecov Report

prasanna08 commented Aug 8, 2017 • edited

anmolshkl left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seanlip commented Aug 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AllanYangZhou commented Aug 9, 2017

seanlip commented Aug 9, 2017

anmolshkl commented Aug 9, 2017 via email

seanlip commented Aug 9, 2017

prasanna08 commented Aug 9, 2017

anmolshkl commented Aug 9, 2017 via email

seanlip commented Aug 9, 2017

AllanYangZhou commented Aug 9, 2017

prasanna08 commented Aug 10, 2017

seanlip commented Aug 10, 2017

prasanna08 commented Aug 10, 2017 • edited

prasanna08 commented Aug 10, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AllanYangZhou commented Aug 12, 2017

prasanna08 commented Aug 14, 2017

anmolshkl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prasanna08 commented Aug 7, 2017 •

edited

prasanna08 Aug 8, 2017 •

edited

prasanna08 Aug 9, 2017 •

edited

codecov-io commented Aug 8, 2017 •

edited

prasanna08 commented Aug 8, 2017 •

edited

anmolshkl left a comment •

edited

prasanna08 commented Aug 10, 2017 •

edited

prasanna08 commented Aug 10, 2017 •

edited

prasanna08 Aug 16, 2017 •

edited