Improved the performance of the SGD classifier on sparse mutations by reducing the noise #71

htcai · 2016-12-02T01:25:23Z

In response to #52, I experimented a variety of methods to improve the performance of the SGD Classifier on sparse genes (e.g., RIT1, MAP2K1).

Mainly, I used PCA to reduce the noise and then employed LDA to compress the data to 2 dimensions. This strategy is justified by the apparent overfitting regarding many genes captured in #52.

Taking RIT1 as the example, the improvement is notable. The test AUROC is 0.837, comparing with the 0.65 in #52. For several other genes I explored (e.g., MAP2K1, SMAD2), the improvement is mostly between 0.07 and 0.10. Moreover, the total time needed for dimensionality reduction and model training is shorter than half a minute in Ubuntu on my ASUS laptop.

Still, there are a few issues. The optimal n_component of PCA for different genes is different. Therefore, it will be great if I can search over multiple values of n_components, which is probably viable using pipeline. However, Ubuntu (8GB RAM + 8GB swap) always runs out of memory when running the pipeline. My MacBook does not have this issue since it uses compressing memory, but it never ends running the pipeline. Hence, I didn't use pipeline in the file submitted.

Also, I am not sure how to interpret the probability plot at the end of the notebook.

Any suggestions and/or comments are welcome. Thanks in advance!

gwaybio

Nice work! Really cool idea combining PCA and LDA - it is surprising that the ROC curves look so good.

There are some plots (namely, the decision plot and probability plot) that are concerning. I made some comments inline.

I also have a general comment about RIT1:

It is an oncogene important in the RAS/MAPK pathway
Therefore, it would be interesting to test if a RIT1 classifier can also classify KRAS/NRAS/HRAS mutations.
Along the same lines, can a KRAS/NRAS/HRAS classifier predict RIT1?

I think these questions could be an alternative way of improving the performance of a RIT specific classifier - but may not work for other genes with low prevalence.

gwaybio · 2016-12-07T00:03:43Z

algorithms/scripts/RIT1-PCA+LDA-htcai.py

+# In[9]:
+
+# Typically, this can only be done where the number of mutations is large enough
+X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=0)


Looks like with this split there are only (6016 / 0.002874) * 0.1 = 1.7 gold standard positives in the test set. We may want to get a bit creative to overcome this sample size issue.

I decided to change the size of the testing data to 0.2 and set stratify=y, which probably is able to make the testing score more reliable.

gwaybio · 2016-12-07T00:08:44Z

algorithms/scripts/RIT1-PCA+LDA-htcai.py

+
+# In[10]:
+
+get_ipython().run_cell_magic('time', '', 'scale = StandardScaler()\nX_train_scale = scale.fit_transform(X_train)\nX_test_scale = scale.transform(X_test)')


any reason why you are fit_transforming X_train but not X_test? Same comment about PCA implementation below.

When we train the models for feature transformation and dimensionality reduction, I think I should only use the information from the train data. A sample is from Raschka 2015.

gwaybio · 2016-12-07T00:10:20Z

algorithms/scripts/RIT1-PCA+LDA-htcai.py

+
+# In[12]:
+
+get_ipython().run_cell_magic('time', '', '\nlda = LinearDiscriminantAnalysis(n_components=2)\nX_train_lda = lda.fit_transform(X_train_pca, y_train)\nX_test_lda = lda.transform(X_test_pca)')


wow! only 2 components! Can you add a scatter plot here where you color RIT1 mutation status?

Sorry that I misunderstood the component number for LDA. According to the documentation of sklearn.discriminant_analysis.LinearDiscriminantAnalysis, n_components cannot be larger than n_classes-1. Therefore, LDA cannot provide a one-dimensional data, which is probably not very informative. But now I am using PCA only, and keeping a 30-dimensional data.

gwaybio · 2016-12-07T00:14:34Z

algorithms/scripts/RIT1-PCA+LDA-htcai.py

+# Ignore numpy warning caused by seaborn
+warnings.filterwarnings('ignore', 'using a non-integer number instead of an integer')
+
+ax = sns.distplot(predict_df.query("status == 0").decision_function, hist=False, label='Negatives')


It looks like the LDA transform causes the decision function to shift far to the negative. Any reason for this?

gwaybio · 2016-12-07T00:16:12Z

algorithms/scripts/RIT1-PCA+LDA-htcai.py

+# In[24]:
+
+ax = sns.distplot(predict_df.query("status == 0").probability, hist=False, label='Negatives')
+ax = sns.distplot(predict_df.query("status == 1").probability, hist=False, label='Positives')


It looks like the positives all have extremely low probability of being positive but the negatives are flat. Looks fishy. Is there an intuition as to why this is happening?

htcai · 2016-12-07T00:50:35Z

@gwaygenomics thanks a lot for so many important comments and questions! I will try to answer the questions one by one.

KT12 · 2016-12-07T18:20:10Z

@htcai I'll just leave this link for reference since it contains discussion of PCA >> LDA for classification.

htcai · 2016-12-08T00:36:51Z

@KT12 thank you for the link! That is actually where I learned about this strategy. But now I believe that I need to re-read the post and the linked articles.

htcai · 2016-12-20T02:14:37Z

Following @dhimmel 's suggestion, I am using only PCA to remove the noise and reduce the dimensionality to 30. I renamed the file to RIT1-PCA-htcai.ipynb. As a side note, I corrected my misunderstanding of LDA. Specifically, n_components cannot be larger than n_class-1, according to the documentation. Therefore, LDA can only produce a 1-dimensional data in this case, which is not very (intuitively) informative.

Firstly, I set test_size=0.2 and stratify=y, since there are very few positive samples in the dataset, as @gwaygenomics pointed out.

Also, I scaled the data both before and after PCA. If without the second time of scaling, the range of decision_function is extremely large (e.g., > 50,000). Also, it actually improved the testing score. Now the testing score is 0.843, even better than the result from the earlier analysis involving LDA.

In addition, the plot of predict_proba looks bumpy, which may well be a consequence of PCA.

Please let me know what I should do to further improve the notebook. Thanks in advance!

Moreover, I am interested in @gwaygenomics 's comments on the the path. I would like to further explore the connection between RIT1 and KRAS/NRAS/HRAS.

dhimmel

Nice job with some really interesting and helpful results. Made some comments. We can discuss at meetup tomorrow if you can make it.

dhimmel · 2016-12-20T03:38:40Z

algorithms/scripts/RIT1-PCA-htcai.py

+# In[1]:
+
+import os
+import urllib


Can remove urllib and random imports

dhimmel · 2016-12-20T03:39:55Z

algorithms/scripts/RIT1-PCA-htcai.py

+# In[8]:
+
+# Here are the percentage of tumors with RIT1
+y.value_counts(True)


Can you also run y.value_counts()? There are so few positives I want to know the exact number... this is cool!

dhimmel · 2016-12-20T03:42:19Z

algorithms/scripts/RIT1-PCA-htcai.py

+get_ipython().run_cell_magic('time', '', 'scale_pre = StandardScaler()\nX_train_scale = scale_pre.fit_transform(X_train)\nX_test_scale = scale_pre.transform(X_test)')
+
+
+# ## Reducing noise via PCA


Let's get rid of %%time in this section for better diff viewing. And the "Feature Standardization" as well.

dhimmel · 2016-12-20T03:48:08Z

algorithms/scripts/RIT1-PCA-htcai.py

+
+# In[10]:
+
+get_ipython().run_cell_magic('time', '', 'scale_pre = StandardScaler()\nX_train_scale = scale_pre.fit_transform(X_train)\nX_test_scale = scale_pre.transform(X_test)')


I think you can write this all in a pipeline, which will simplify the code and make the process more clear. See example in 2.TCGA-MLexample.py.

We can discuss at the meetup, whether we should do cross-validation the right or wrong but faster way as discussed in #70 (comment).

dhimmel · 2016-12-20T03:51:58Z

algorithms/scripts/RIT1-PCA-htcai.py

+
+param_grid = {
+    'alpha': [10 ** x for x in range(-4, 1)],
+    'l1_ratio': [0, 0.2, 0.5, 0.8, 1],


I'd more densely sample alpha and reduce the l1_ratio to [0, 0.15]. See #56.

dhimmel · 2016-12-20T03:53:00Z

algorithms/scripts/RIT1-PCA-htcai.py

+
+# In[17]:
+
+# Cross-validated performance heatmap


This figure shows how hard the problem is. We're struggling to find a nice optimum of parameters. Performance is all over the place.

dhimmel · 2016-12-20T03:56:11Z

algorithms/scripts/RIT1-PCA-htcai.py

+
+# In[11]:
+
+get_ipython().run_cell_magic('time', '', 'n_components = 30\npca = PCA(n_components=n_components, random_state=0)\nX_train_pca = pca.fit_transform(X_train_scale)\nX_test_pca = pca.transform(X_test_scale)')


Would be nice to know what percentage of variance 30 components are capturing? It seems like 30 could be removing some important signals.

dhimmel · 2017-01-10T01:30:24Z

algorithms/RIT1-PCA-htcai.ipynb

+   ],
+   "source": [
+    "# Percentage of preserved variance\n",
+    "print('{:.4}'.format(sum(pca.explained_variance_ratio_)))"


So 35 components captures ~60% of variance. Can we increase the number of components until we capture at least 90% of variance?

As I mentioned below, increasing the dimensionality leads to over-fitting and extremely low testing score immediately. It appears that a large proportion of the variation in the dataset is probably noise.

htcai · 2017-01-10T01:33:22Z

I made the following main changes.

I increased the proportion of testing data to 30%, which leads to a more interpretable visualization of probability (i.e., the last plot). As @gwaygenomics pointed out, a small portion of negative samples were mistakenly predicted to be positive, which is displayed as a second (but small) peak in the high-probability range. In addition, although this strategy slightly reduces the training and testing score (to 0.763), the testing score is still better than the baseline SGD Classifier (0.65).
Following @dhimmel 's suggestion, I fixed the l1_ratio and searched within a more condensed range of alpha (np.linspace(10**-4, 10**-2, num=16)). The validation results display a complicated effect of alpha's change.
I calculated the ratio of explained variance of the 35 dimensions preserved in PCA (5 more than it was), which turns out to be 0.5996. This appears to be quite low. However, if more dimensions are kept, over-fitting will occur immediately. Moreover, I am applying PCA to reduce the noise in the data. A low value of explained_variance_ratio might not be unexpected.
Raw counts of positive samples are provided for both the testing set and the entire dataset.

dhimmel · 2017-01-10T16:48:29Z

Great work with this pull request -- this is interesting stuff!

Can you re-export RIT1-PCA-htcai.py?

The validation results display a complicated effect of alpha's change.

Here's the plot:

I don't actually think the effect of alpha is complicated. Instead I think we have a really noisy estimate of performance. It is concerning that we're not able to identify an optimal alpha range. Instead, the max performing values are scattered. Yuck! The bright side is if we can crack how to deal with these low number of positive situations, we're doing something really powerful.

I think we may be able to decrease the noise in our performance measurements by playing with the cv parameter of GridSearchCV. Let's start by trying out 10-fold crossvalidation (cv=10). If that still doesn't clarify things, we may need repeated cross validation, although we'll have to see how inconvenient that is in sklearn (very according to this answer although I don't think it should actually be that hard).

I calculated the ratio of explained variance of the 35 dimensions preserved in PCA (5 more than it was), which turns out to be 0.5996. This appears to be quite low. However, if more dimensions are kept, over-fitting will occur immediately.

Interesting. I'm worried that for some mutations the ~40% of uncaptured variance will be important. But I agree it's more important to actually be able to fit a model than include all the relevant features.

dhimmel · 2017-01-10T16:55:49Z

We could also use StratifiedShuffleSplit -- while not as elegant as repeated cross-validation, it should work.

On second thought, perhaps it's even more elegant!

htcai · 2017-01-10T17:21:44Z

@dhimmel Thanks a lot for your detailed and valuable comments! I will re-export the .py script after I further revise the notebook according to your suggestions. Unfortunately, pipeline still drain the memory even if I only cross-validate over two values of alpha.

In order to locate the optimal range of alpha, a cross-validation involving a larger number of values (e.g., 100, 200, etc) can be run. After the dimensionality reduction, it will probably take only less than one minute if alpha is the only hyper-parameter that is being tuned.

Training a classifier over an extremely imbalanced dataset has always been a thorny issue. I tried under-sampling and over-sampling several weeks ago, but it turned out to be a disaster. The performance is much worse than the baseline SGD classifier. I am happy to explore other methods later.

It is a good idea to try 10-fold cross-validation. My only concern is that there will be only 1 or 2 positive samples in each of the 10 folds. Still, only the experiment result can clarify whether that is a problem.

I agree that the removal of ~40% of the variance is a problem for other mutations. For the about 10 mutations I experimented, the optimal number of principal components is between 30 - 100. Therefore, if with sufficient computational resource, a GridSearchCV over n_components should work for our purpose of optimizing the classifier for a different mutation.

dhimmel · 2017-01-10T17:30:50Z

Therefore, if with sufficient computational resource, a GridSearchCV over n_components should work for our purpose of optimizing the classifier for a different mutation.

Great idea -- really insightful! Agree that computation time will be the drawback.

My only concern is that there will be only 1 or 2 positive samples in each of the 10 folds.

There will be more positives in the training folds, which is ultimately more important. The larger number of testing folds (10 versus 3) will counterbalance the fewer number of positives in each. This is why leave one out CV is best, even though there are often 0 positives in a given testing fold.

htcai · 2017-01-14T01:18:22Z

@dhimmel I used 10-fold GridSearchCV without StratifiedShuffleSplit and searched over a larger space of alpha: [x for x in np.linspace(10**-4, 10**-2, num=200)]. The performance measurement is still rather noisy. I probably need to try repeated cross-validation to reduce the noise.

On the other hand, StratifiedShuffleSplit tends to cause over-fitting and makes the selection prefer extremely large values.

dhimmel · 2017-01-14T01:34:34Z

@htcai interesting. I'd love to see 1 × 10-fold for larger alphas, like you did for StratifiedShuffleSplit. I'm wondering if these higher alphas are actually producing better models... I don't really see a way that StratifiedShuffleSplit could cause inflated CV performance.

Do you want to switch the notebook to StratifiedShuffleSplit, so I can look over that code? Then let's see the coefficients of the best model to see what's going on with these high-performing large-alphas.

An alternative would be 10 × 10-fold CV, but that could be a pain to implement. And I don't really think it's fundamentally different from StratifiedShuffleSplit -- just a little more efficient.

htcai · 2017-01-14T15:36:04Z

I updated the notebook with StratifiedShuffleSplit and searched for alpha over [2**x for x in range(-20, 60)]. Larger values of alpha are generally more preferable, but 0.5 is the optimal one.

It is good to see that the positive and the negative curves are more neatly segregated in the plots of decision_function and probability.

However, over-fitting seems to be salient.

dhimmel · 2017-01-14T17:31:42Z

algorithms/scripts/RIT1-PCA-htcai.py

+cv_folds = [[t, c] for t, c in cv_folds]
+
+
+# In[17]:

 get_ipython().run_cell_magic('time', '', "clf = SGDClassifier(random_state=0, class_weight='balanced', loss='log', penalty='elasticnet')\ncv = GridSearchCV(estimator=clf, param_grid=param_grid, n_jobs=-1, scoring='roc_auc')\ncv.fit(X = X_train_scale, y=y_train)")


Move everything besides cv.fit(X = X_train_scale, y=y_train) out of the %%time cell for better py export.

In GridSearchCV you need to specify cv=sss.

dhimmel · 2017-01-14T17:36:18Z

algorithms/scripts/RIT1-PCA-htcai.py

-# In[14]:
+# In[16]:
+
+sss = StratifiedShuffleSplit(n_splits=10, test_size=0.3, random_state=0)


If computation time doesn't become prohibitive, I'd love something more thorough such as:

StratifiedShuffleSplit(n_splits=100, test_size=0.1, random_state=0)

n_splits times test_size equals the average number of times each observation will be used for testing in CV. The higher this is the less variation we should have in our performance estimates. However, we want test_size to be low, so the percent of observations used for training is maximized. Therefore, we should increase n_splits up until the point where it starts taking a long time. Make sense?

Yes, this is very sensible. It does not take a long time to run a 100-fold Grid Search. Actually, I set n_split=150. Now the heat map looks much less noisy.

dhimmel · 2017-01-14T17:37:00Z

algorithms/scripts/RIT1-PCA-htcai.py

+
+sss = StratifiedShuffleSplit(n_splits=10, test_size=0.3, random_state=0)
+cv_folds = sss.split(X_train_scale, y_train)
+cv_folds = [[t, c] for t, c in cv_folds]


Computing cv_folds yourself may not be necessary.

dhimmel · 2017-01-16T15:52:53Z

Cool one last thing so we can understand what's going on the super high alphas. Can you export the coefficients from the best model with alpha = 2048?

htcai · 2017-01-16T17:37:06Z

That is a good idea! I visualized the coefficients of the best_estimator_ with a bar plot.

dhimmel

I'm going over these results with @cgreene right now. We think the predicted probabilities should be centered around 0.0029 (21 / (21 + 7285)). So we're trying to diagnose what's going on. Two changes to param_grid will help us figure this out... hopefully.

dhimmel · 2017-01-16T18:00:30Z

algorithms/scripts/RIT1-PCA-htcai.py

+
+param_grid = {
+    'alpha': [2**x for x in range(-20, 60)],
+    'l1_ratio': [0]


Can add 0.15 to this list and rerun?

dhimmel · 2017-01-16T18:05:54Z

algorithms/scripts/RIT1-PCA-htcai.py

+# In[15]:
+
+param_grid = {
+    'alpha': [2**x for x in range(-20, 60)],


Let's reduce the upper bound to 30 at most.

htcai · 2017-01-16T19:04:13Z

I searched over four values [0, 0.05, 0.1, 0.15] and the best parameters are {'alpha': 1, 'l1_ratio': 0.05}. Still, it seems that the predicted probabilities are higher than expected.

dhimmel

Yeah I'm a little confused by the probabilities. Using a pipeline would help us make sure the right transformations are being applied.

Otherwise these are fantastic results. Really like the new CV plot!

dhimmel · 2017-01-16T20:20:49Z

algorithms/scripts/RIT1-PCA-htcai.py

+
+# In[24]:
+
+X_transformed = scale_post.transform(pca.transform(scale_pre.transform(X)))


Did you want to put scale_pre, pca, and scale_post in a pipeline? This would really simplify things and help avoid errors.

htcai · 2017-01-19T01:07:46Z

I finally got it run with pipeline (and hopefully of correct order) including scaling, PCA and classifier. Within a small amount of time, I can only search over a smaller space of alpha and use a 3-fold cross validation. As a consequence, I decided to increase test_size in StratifiedShuffleSplit to 0.2 and only use 20% of the entire data set for testing. Also, the CV plot does not seem as good as it was.

dhimmel · 2017-01-29T21:20:40Z

Hey @htcai, I meant to suggest using a pipeline but preserving the exact functionality. Sorry I wasn't more clear. Feel free to revert to 544fd03.

I don't want to drag this pull request out, so either I can merge now, or we can talk Tuesday about the pipeline method I meant to suggest.

htcai · 2017-01-31T02:05:59Z

@dhimmel Thanks for your clarification. I guess you are suggesting pipelining the scaling and the PCA while leaving the classifier alone. If this is not the case, we should talk further about the implementation tomorrow.

I increased the number of principal components to 50, which is a plausible value if we want a uniform one for all of the sparse genes. The testing score was actually improved for RIT1 comparing with 30 and 35.

dhimmel · 2017-02-01T21:28:17Z

@htcai I remember that you made some progress last night at the meetup. Let me know when this is ready for me to take a look again.

htcai · 2017-02-02T02:01:44Z

@dhimmel I am not sure whether you are referring to pipelining the data standardization and PCA. If this is the case, then it is already contained in the latest commit.

I tried FastICA and NMF. Both of them, with different n_components, either skew the prediction of probability (as does PCA) or drop the testing score to some level below random guess (i.e., 50%).

In addition, I am currently running NMF with 500 components. It seems that it will not finish within a reasonable amount of time.

dhimmel · 2017-02-02T16:26:49Z

@dhimmel I am not sure whether you are referring to pipelining the data standardization and PCA. If this is the case, then it is already contained in the latest commit.

Great, that's what I was referring to. This PR is ready to merge. Let's do a separate for the other dimensionality reduction methods.

htcai added 2 commits December 1, 2016 19:52

improved the performance of the classifier by reducing the noise

a33e55f

generated the py file

a564a6b

gwaybio reviewed Dec 7, 2016

View reviewed changes

htcai added 2 commits December 19, 2016 20:45

use PCA only

9da4aa1

generated the py file

b827ad5

dhimmel reviewed Dec 20, 2016

View reviewed changes

enlarged the testing dataset

e44a6f2

dhimmel reviewed Jan 10, 2017

View reviewed changes

KT12 mentioned this pull request Jan 12, 2017

Reduce dimension with random projection #76

Merged

used StratifiedShuffleSplit

43a3967

dhimmel requested changes Jan 14, 2017

View reviewed changes

150-fold cross-validation

32c0839

joshlevy89 mentioned this pull request Jan 15, 2017

Marginal gain of gene expression data over covariates #67

Merged

htcai added 2 commits January 16, 2017 12:18

visualized the coefficients of the best estimator

c338112

re-exported the py file

d3e69d7

dhimmel reviewed Jan 16, 2017

View reviewed changes

consider more l1_ratio values

544fd03

dhimmel reviewed Jan 16, 2017

View reviewed changes

pipelined scaling and PCA

fb94e99

htcai force-pushed the sparsity branch from 065ee89 to fb94e99 Compare January 31, 2017 01:58

dhimmel approved these changes Feb 2, 2017

View reviewed changes

dhimmel merged commit 9b87c44 into cognoma:master Feb 2, 2017

dhimmel mentioned this pull request Jun 6, 2017

Evaluate dask-searchcv to speed up GridSearchCV #94

Closed

rdvelazquez mentioned this pull request Jul 11, 2017

Selecting the number of components returned by PCA #106

Closed

rdvelazquez mentioned this pull request Aug 17, 2017

Jobs fail due to too few positive samples cognoma/frontend#102

Closed


		# In[10]:

		get_ipython().run_cell_magic('time', '', 'scale = StandardScaler()\nX_train_scale = scale.fit_transform(X_train)\nX_test_scale = scale.transform(X_test)')


		# In[12]:

		get_ipython().run_cell_magic('time', '', '\nlda = LinearDiscriminantAnalysis(n_components=2)\nX_train_lda = lda.fit_transform(X_train_pca, y_train)\nX_test_lda = lda.transform(X_test_pca)')

		get_ipython().run_cell_magic('time', '', 'scale_pre = StandardScaler()\nX_train_scale = scale_pre.fit_transform(X_train)\nX_test_scale = scale_pre.transform(X_test)')


		# ## Reducing noise via PCA


		# In[11]:

		get_ipython().run_cell_magic('time', '', 'n_components = 30\npca = PCA(n_components=n_components, random_state=0)\nX_train_pca = pca.fit_transform(X_train_scale)\nX_test_pca = pca.transform(X_test_scale)')


		# In[24]:

		X_transformed = scale_post.transform(pca.transform(scale_pre.transform(X)))

Improved the performance of the SGD classifier on sparse mutations by reducing the noise #71

Improved the performance of the SGD classifier on sparse mutations by reducing the noise #71

Conversation

htcai commented Dec 2, 2016 • edited Loading

gwaybio left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

htcai commented Dec 7, 2016

KT12 commented Dec 7, 2016

htcai commented Dec 8, 2016

htcai commented Dec 20, 2016

dhimmel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

htcai Jan 10, 2017 • edited Loading

Choose a reason for hiding this comment

htcai commented Jan 10, 2017

dhimmel commented Jan 10, 2017

dhimmel commented Jan 10, 2017

htcai commented Jan 10, 2017

dhimmel commented Jan 10, 2017 • edited Loading

htcai commented Jan 14, 2017 • edited Loading

dhimmel commented Jan 14, 2017 • edited Loading

htcai commented Jan 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhimmel commented Jan 16, 2017 • edited Loading

htcai commented Jan 16, 2017

dhimmel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

htcai commented Jan 16, 2017

dhimmel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

htcai commented Jan 19, 2017

dhimmel commented Jan 29, 2017

htcai commented Jan 31, 2017

dhimmel commented Feb 1, 2017

htcai commented Feb 2, 2017 • edited Loading

dhimmel commented Feb 2, 2017

htcai commented Dec 2, 2016 •

edited

Loading

htcai Jan 10, 2017 •

edited

Loading

dhimmel commented Jan 10, 2017 •

edited

Loading

htcai commented Jan 14, 2017 •

edited

Loading

dhimmel commented Jan 14, 2017 •

edited

Loading

dhimmel commented Jan 16, 2017 •

edited

Loading

htcai commented Feb 2, 2017 •

edited

Loading