Add recommendation model #4175

yhliang2018 · 2018-05-04T20:08:15Z

Add recommendation model to the garden. Will update readme and add unit tests.

qlzh727 · 2018-05-04T20:14:39Z

official/recommendation/README.md

@@ -0,0 +1,35 @@
+# Recommendation Model
+This is an implementation of the Neural Collaborative Filtering (NCF) framework with Neural Matrix Factorization (NeuMF) model as described in the [Neural Collaborative Filtering](https://arxiv.org/abs/1708.05031) paper. Current implementation is based on the code from the authors' [NCF code](https://github.com/hexiangnan/neural_collaborative_filtering) and the Standford pytorch implementation at [mlperf repo](https://github.com/mlperf/reference/tree/master/recommendation/pytorch).


Probably should change "pytorch" to "PyTorch" which is their official name.

Or, better yet, just "the Stanford implementation." Also note the typo in Stanford.

Another nit: "implementation in the MLPerf Repo"

Will do. Thank you for the careful checking.

qlzh727 · 2018-05-04T20:31:25Z

official/recommendation/data_download.py

+import tensorflow as tf  # pylint: disable=g-bad-import-order
+
+import numpy as np
+import pandas as pd


Not sure if it worth the effort or not, maybe we can convert this to use the tf.data for importing CSV files.

See https://www.tensorflow.org/api_docs/python/tf/estimator/inputs/pandas_input_fn and the containing module for convenient converters

qlzh727 · 2018-05-04T20:34:21Z

official/recommendation/data_download.py

+    parse_file_to_csv(FLAGS.data_dir, FLAGS.dataset)
+
+
+if __name__ == '__main__':


This will probably need some update to use absl flags and app.

qlzh727 · 2018-05-04T20:38:54Z

official/recommendation/data_download.py

+if __name__ == '__main__':
+  parser = argparse.ArgumentParser()
+  parser.add_argument(
+      '--data_dir', type=str, default='/tmp/ml_data',


The default value "/tmp/ml_data" seems to be too generic. Can we give it a more specific name?

Agreed. ml_data could also stand for "machine learning data", which could confuse the user.

Aha, this ml is abbr. of "MoiveLens" which is the name of the dataset used for model training and evaluation. Maybe rename it as "movie_lens"?

qlzh727 · 2018-05-04T20:40:26Z

official/recommendation/data_download.py

+    file_path, _ = urllib.request.urlretrieve(
+        DATA_URL + dataset_zip, file_path, _progress)
+    statinfo = os.stat(file_path)
+    print('\nSuccessfully downloaded', file_path, statinfo.st_size, 'bytes.')


Leading \n in a string is bit weird, maybe just print('\n', 'Successfully downloaded', ...)

Or we can change to tf.logging to mitigate the line return issue.

qlzh727 · 2018-05-04T20:53:58Z

official/recommendation/data_download.py

+
+  # Get the info of users who have more than 20 ratings on items.
+  grouped = df.groupby(USER_COLUMN)
+  df = grouped.filter(lambda x: len(x) >= 20)


The magic number 20 need some clarification.

Hah, it's from the paper. Will add comments on it.

qlzh727 · 2018-05-04T20:56:03Z

official/recommendation/ncf.py

+
+  # Calculate HR score
+  def _get_hr(ranklist, gt_item):
+    for item in ranklist:


this should be simplified as:

return 1 if gt_item in ranklist else 0

qlzh727 · 2018-05-04T21:00:58Z

official/recommendation/ncf.py

+
+  # Calculate NDCG score
+  def _get_ndcg(ranklist, gt_item):
+    for i in range(len(ranklist)):


This can be updated to:

return 0 if gt_item not in ranklist else math.log(2) / math.log(ranklist.index(gt_item))

The expression should be math.log(2) / math.log(ranklist.index(gt_item) + 2)?

qlzh727 · 2018-05-04T21:03:00Z

official/recommendation/ncf.py

+
+
+def main(_):
+  tf.logging.set_verbosity(tf.logging.INFO)


The main function is way too long. Please extract them into sub functions, like load_data, create_model, and run_training.

qlzh727 · 2018-05-04T21:03:41Z

official/recommendation/ncf.py

+
+
+if __name__ == '__main__':
+  parser = argparse.ArgumentParser()


same, but probably in future PR, change to use absl flag and tf.logging.

k-w-w

Looks great and easy to understand overall. The model written with Keras is very clean.

k-w-w · 2018-05-04T21:50:17Z

official/recommendation/data_download.py

+
+  # serialize to csv file
+  df_train_ratings = pd.DataFrame(list(all_ratings))
+  df_train_ratings['fake_rating'] = 1


Why does a fake rating need to be added?

This fake rating is to indicate the interaction between the user and the item. Here the 'all_rating' only contains user_id and item_id fields, but the final csv file needs three fields of 'user_id, item_id, interaction'. I will add some comments here.

k-w-w · 2018-05-04T21:53:32Z

official/recommendation/ncf.py

+
+  Args:
+    est_model: The Estimator.
+    user_input: The user input for evaluation.


Expected types/shapes would be good for these arguments and return items

Will add it.

Thanks, and it would be helpful on most of the other functions as well.

k-w-w · 2018-05-04T22:03:00Z

official/recommendation/neumf_model.py

+
+
+def _validate_batch_size_for_multi_gpu(batch_size):
+  """For multi-gpu, batch-size must be a multiple of the number of GPUs.


Perhaps add an explanation about why the batch size needs to be a multiple (is the batch being split across each of the GPUs?)/

Yes. The batch will be evenly split across all GPUs. Will add comments there.

k-w-w · 2018-05-04T22:09:53Z

official/recommendation/data_download.py

+if __name__ == '__main__':
+  parser = argparse.ArgumentParser()
+  parser.add_argument(
+      '--data_dir', type=str, default='/tmp/ml_data',


Agreed. ml_data could also stand for "machine learning data", which could confuse the user.

karmel

Sorry for the delay

karmel · 2018-05-04T22:23:02Z

official/recommendation/README.md

@@ -0,0 +1,35 @@
+# Recommendation Model
+This is an implementation of the Neural Collaborative Filtering (NCF) framework with Neural Matrix Factorization (NeuMF) model as described in the [Neural Collaborative Filtering](https://arxiv.org/abs/1708.05031) paper. Current implementation is based on the code from the authors' [NCF code](https://github.com/hexiangnan/neural_collaborative_filtering) and the Standford pytorch implementation at [mlperf repo](https://github.com/mlperf/reference/tree/master/recommendation/pytorch).


Or, better yet, just "the Stanford implementation." Also note the typo in Stanford.

karmel · 2018-05-04T22:23:58Z

official/recommendation/README.md

@@ -0,0 +1,35 @@
+# Recommendation Model
+This is an implementation of the Neural Collaborative Filtering (NCF) framework with Neural Matrix Factorization (NeuMF) model as described in the [Neural Collaborative Filtering](https://arxiv.org/abs/1708.05031) paper. Current implementation is based on the code from the authors' [NCF code](https://github.com/hexiangnan/neural_collaborative_filtering) and the Standford pytorch implementation at [mlperf repo](https://github.com/mlperf/reference/tree/master/recommendation/pytorch).


Another nit: "implementation in the MLPerf Repo"

karmel · 2018-05-04T22:26:59Z

official/recommendation/README.md

+# Recommendation Model
+This is an implementation of the Neural Collaborative Filtering (NCF) framework with Neural Matrix Factorization (NeuMF) model as described in the [Neural Collaborative Filtering](https://arxiv.org/abs/1708.05031) paper. Current implementation is based on the code from the authors' [NCF code](https://github.com/hexiangnan/neural_collaborative_filtering) and the Standford pytorch implementation at [mlperf repo](https://github.com/mlperf/reference/tree/master/recommendation/pytorch).
+
+NCF is a general framework under which a neural network architecture is proposed to model latent features of users and items in collaborative filtering of recommendation. Unlike traditional models, NCF does not resort to Matrix Factorization (MF) with an inner product on latent features of users and items. It replaces the inner product with a multi-layer perceptron that can learn an arbitrary function from data.


I don't really understand precisely what the first sentence means here. Maybe, "NCF is a type of machine learning model used for collaborative filtering and recommendation in which a neural network is used to determine latent features representing users and items."?

Actually the first sentence is quoted from the paper. I will rephrase it better.

karmel · 2018-05-04T22:35:01Z

official/recommendation/data_download.py

+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Download and extract the movielens dataset from grouplens website.


MovieLens, GroupLens

karmel · 2018-05-04T22:38:07Z

official/recommendation/data_download.py

+import tensorflow as tf  # pylint: disable=g-bad-import-order
+
+import numpy as np
+import pandas as pd


See https://www.tensorflow.org/api_docs/python/tf/estimator/inputs/pandas_input_fn and the containing module for convenient converters

karmel · 2018-05-07T20:04:37Z

official/recommendation/ncf.py

+        .format(time.time()-t1, num_users, num_items, train.nnz,
+                len(test_ratings)))
+
+  # Create NeuMF model from tf.keras Model


This is really a Keras model that is returned, right? Create a keras model from the NeuMF wrapper? (But, see comment below-- why?)

karmel · 2018-05-07T20:05:37Z

official/recommendation/neumf_model.py

+    self.model_layers = model_layers
+    self.learning_rate = learning_rate
+
+  def __call__(self, multi_gpu, batch_size):


call is a reserved method that takes specific inputs and gives specific outputs. (See in Resnet, for example.) If this is just assembling a keras model, let's name it something more straightforward.

karmel · 2018-05-07T20:07:25Z

official/recommendation/neumf_model.py

+
+class NeuMF(object):
+  """Neural matrix factorization (NeuMF) model for recommendations."""
+


I don't really understand why this is a class. It seems that this is just a helper that gets the keras model. Why make it a class? And why with the given method names? Somewhat confused here, since this is not ever passed to the estimator.

Will update this class as a subclass of keras model.

karmel · 2018-05-07T21:42:35Z

official/recommendation/data_download.py

+import zipfile
+
+import numpy as np
+import pandas as pd


Not sure where @qlzh727 's comment went on pandas-- but, it occurs to me now that we are using pandas just to download the data and re-save as csv. I guess that's fine, especially since it's not on the training path, but, it does seem strange. Let's not worry now about fixing this, but maybe add a note that makes it explicit that the actual data loading for the model is with numpy; this is just a series of helpers for processing the data prior to training on it. Although if it's easy to do this without the additional pandas dependencies, that would be preferable.

karmel · 2018-05-07T22:37:03Z

official/recommendation/ncf_dataset.py

+Prepare input for model training and evaluation.
+"""
+import numpy as np
+import scipy.sparse as sp


This is another dependency-- note that it will have to be added, along with pandas, to the list of dependencies, and also to any BUILD files.

On second thought, starting to review this file, but I think we do this in TF instead of numpy/scipy. That would likely improve performance, and reduce dependencies. See https://www.tensorflow.org/get_started/datasets_quickstart#reading_a_csv_file , and then the rest should just be general matrix manipulation, which should be done in TF directly. Not going to review the rest of this file now, but will re-review when done as "TF-first".

Okay, will do.

robieta · 2018-05-07T21:03:15Z

official/recommendation/data_download.py

+  Returns:
+    A processed pandas DataFrame of the rating dataset.
+  """
+  names = ['user_id', 'item_id', 'rating', 'timestamp']


Make this a global constant?

robieta · 2018-05-07T22:17:23Z

official/recommendation/data_download.py

+  t2 = time.time()
+  test_ratings = []
+  test_negs = []
+  all_items = set(range(len(original_items)))


Could you add a comment about what exactly this is?

This basically tries to generate the 0-based index for all the items, and put it into a set for the later use. Comments have added in the code.

robieta · 2018-05-07T22:26:18Z

official/recommendation/data_download.py

+    test_item = user_to_items[user].pop()  # Get the latest one
+
+    all_ratings.remove((user, test_item))  # Remove the test item
+    all_negs = all_items - set(user_to_items[user])


Not sure how long this takes, but the .difference(x) set method is generally faster than - set(x).

Okay, will update it.

robieta · 2018-05-07T22:34:37Z

official/recommendation/data_download.py

+
+
+def main(_):
+  """Download and extract the data from grouplens website."""


Can you change this to use tempfile and tf.gfile so it works on distributed filesystems?

robieta · 2018-05-07T22:39:18Z

official/recommendation/ncf.py

+from six.moves import xrange  # pylint: disable=redefined-builtin
+import tensorflow as tf  # pylint: disable=g-bad-import-order
+
+import data_download


should these be official.recommendation.module_name?

Yes. Will update it in next PR with absl flags

robieta · 2018-05-07T22:41:13Z

official/recommendation/ncf.py

+    est_model: The Estimator.
+    user_input: The user input for evaluation.
+    item_input: The item input for evaluation.
+    gt_items: The test item for HR and NDCG calculation.


what does gt stand for?

I guess gt here is short for ground truth, as this one is the leave-one-out validation item. Will add comment on it.

robieta · 2018-05-07T22:49:48Z

official/recommendation/ncf.py

+  num_users = len(gt_items)
+  step = len(user_input) // num_users  # Step for each user
+  # Evaluation on each user
+  for idx in range(num_users):


undefined abbreviations make this hard to follow. Does idx have some significance/stand for something? If so could you spell it out, and if not could you make it clear that this is just a loop counter?

:) I think idx should be among the acceptable loop counters. idx is commonly used in numpy, elsewhere for index-- ie, a counter.

robieta · 2018-05-07T22:55:05Z

official/recommendation/ncf.py

+    start = idx * step
+    end = (idx + 1) * step
+
+    items = item_input[start:end]


Is is easy to refactor these into something like a list of lists? There is just extra mental overhead of reading "index into slice" code as opposed to something like predicted_scores=all_predicted_scores[user_id]. If it's too much trouble to change feel free to ignore this request.

Good idea. Done!

robieta · 2018-05-07T22:58:20Z

official/recommendation/ncf.py

+def main(_):
+  tf.logging.set_verbosity(tf.logging.INFO)
+
+  print('NeuMF arguments: {}'.format(FLAGS))


Please make the logic use a flags object rather than FLAGS following the pattern of the other main functions.

robieta · 2018-05-07T23:08:07Z

official/recommendation/ncf_dataset.py

+    return [int(tmp[0]), int(tmp[1]), float(tmp[2]) > 0]
+
+  lines = open(train_fname, 'r').readlines()[1:]
+  data = list(map(process_line, lines))


I generally prefer list comprehension to map() because the behavior changes from 2 to 3.

k-w-w · 2018-05-09T19:11:28Z

official/recommendation/README.md

+NCF is a general framework under which a neural network architecture is proposed to model latent features of users and items in collaborative filtering of recommendation. Unlike traditional models, NCF does not resort to Matrix Factorization (MF) with an inner product on latent features of users and items. It replaces the inner product with a multi-layer perceptron that can learn an arbitrary function from data.
+
+Two instantiations of NCF are Generalized Matrix Factorization (GMF) and Multi-Layer Perceptron (MLP). GMF applies a linear kernel to model the latent feature interactions, and and MLP uses a nonlinear kernel to learn the interaction function from data. NeuMF is a fused model of GMF and MLP to better model the complex user-item interactions, and unifies the strengths of linearity of MF and non-linearity of MLP for modeling the user-item latent structures. NeuMF allows GMF and MLP to learn separate embeddings, and combines the two models by concatenating their last hidden layer. [neumf_model.py](neumf_model.py) defines the architecture details.
+


It would be helpful to get a list of abbreviations used (like the docstring at the top of neumf_model.py), including HR, GT, NDCG, ml-1m, etc.

k-w-w · 2018-05-09T19:12:01Z

official/recommendation/README.md

+
+Note the ml-20m dataset is large (the rating file is ~500 MB), and it may take several minutes (<10 mins)for data preprocessing.
+
+### Train and Evaluate Model


Including how long the model takes the train and expected final metrics would be great

k-w-w · 2018-05-09T19:13:02Z

official/recommendation/data_download.py

+    dataset_name: The dataset name to be processed.
+  """
+
+  # Use random seed as parameter


I might be misunderstanding something, but the random seed is being set to 0, not a parameter.

k-w-w · 2018-05-09T19:15:00Z

official/recommendation/ncf.py

+
+  Args:
+    est_model: The Estimator.
+    user_input: The user input for evaluation.


Thanks, and it would be helpful on most of the other functions as well.

yhliang2018 · 2018-05-09T19:45:23Z

official/recommendation/neumf_model.py

+        inputs=[user_input, item_input], outputs=prediction)
+
+    # Use multiple gpus
+    if multi_gpu:


Will update this multi_gpu mode with distribution strategy, based on discussions with @robieta

yhliang2018 · 2018-05-18T17:56:00Z

Hi All,

Thanks a lot for the helpful comments. I have updated the data input pipeline with tf.dataset, and also add the distribution strategy for multiple-gpu mode. Now the Adam optimizer doesn't work well with distribution strategy, so we use GradientDescent optimizer for now. Will update it as the bug is fixed.

Will add unit test and official flags in next PR.

karmel

Almost there-- a few relatively minor comments. Thanks--

karmel · 2018-05-18T21:58:24Z

official/recommendation/data_download.py

+  return ratings
+
+
+def load_movielens_1million(file_name, sort=True):


nit: extra underscore (1_million, 20_million) would be clearer.

karmel · 2018-05-18T21:58:58Z

official/recommendation/data_download.py

+      - Ratings are made on a 5-star scale (whole-star ratings only)
+      - Timestamp is represented in seconds since midnight Coordinated Universal
+        Time (UTC) of January 1, 1970.
+      - Each user has at least 20 ratings


karmel · 2018-05-18T22:17:24Z

official/recommendation/dataset.py

+
+
+def load_data(file_name):
+  """Load data from a csv file."""


This says csv, but splits on \t.

karmel · 2018-05-18T22:19:38Z

official/recommendation/dataset.py

+
+def load_data(file_name):
+  """Load data from a csv file."""
+  lines = open(file_name, "r").readlines()


tf.gfile.GFile().

karmel · 2018-05-18T22:24:45Z

official/recommendation/dataset.py

+  """
+  # Load training positive instances into memory for later train data generation
+  train_data = load_data(train_fname)
+  num_users = max(train_data, key=lambda x: x[0])[0] + 1


Can we return a np.array from load_dataset instead? That allows us to grab max more efficiently than doing this. Also, this seems unreliable; if I have non-sequential uids, this is not true, right? Better to do unique()?

karmel · 2018-05-18T23:02:35Z

official/recommendation/ncf_main.py

+        cycle_index, total_training_cycle - 1))
+
+    # Train the model
+    train_cycle_begin = time.time()


Can be done in the followup, but probably not necessary to collect/print out this very rough timing info.

karmel · 2018-05-18T23:06:20Z

official/recommendation/ncf_main.py

+      help="Batch size.")
+  parser.add_argument(
+      "--num_factors", type=int, default=8,
+      help="Embedding size of MF model.")


embedding_size maybe? "Factors" is used a lot in MF proper, but our users are more likely deep-learning-first rather than MF-first.

karmel · 2018-05-18T23:06:43Z

official/recommendation/ncf_main.py

+      help="Size of hidden layers for MLP.")
+  parser.add_argument(
+      "--reg_mf", type=float, default=0,
+      help="Regularization for MF embeddings.")


mf_regularization?

karmel · 2018-05-18T23:07:46Z

official/recommendation/neumf_model.py

+      reg_mf: A floating number, the regularization for MF embeddings.
+
+    Raises:
+      ValueError: if the first model layer is not even.


"not a multiple of two" would be clearer

karmel · 2018-05-18T23:08:14Z

official/recommendation/neumf_model.py

+
+    # Input variables
+    user_input = tf.keras.layers.Input(
+        shape=(1,), dtype="int32", name=constants.USER)


It's weird that the dtype is a string here. tf.int32 doesn't work?

yhliang2018 · 2018-05-21T21:30:33Z

Hi Karmel,

Thank you for the helpful comments! I have addressed most of them. Others like remove print statements, unit test, and official flags will be done in next PR.

As our offline discussion, for input_fn we still use csv files for tf.dataset which is required by distribution strategy. Will create a CL for dataset_input_fn (similar to numpy_input_fn), or seeking other possible approaches later.

My issues have been addressed.

karmel · 2018-05-22T02:34:29Z

Thanks-- note that @qlzh727 is out, so you may have to assume he approves for now, and make any last requested changes in a follow-up.

Will make the requested changes in next PR as Karmel suggested

Add recommendation model

3b905cb

yhliang2018 requested review from a team and karmel as code owners May 4, 2018 20:08

googlebot added the cla: yes label May 4, 2018

qlzh727 previously requested changes May 4, 2018

View reviewed changes

k-w-w reviewed May 4, 2018

View reviewed changes

Fix pylints check error

89770cf

karmel suggested changes May 7, 2018

View reviewed changes

robieta previously requested changes May 7, 2018

View reviewed changes

k-w-w reviewed May 9, 2018

View reviewed changes

yhliang2018 commented May 9, 2018

View reviewed changes

yhliang2018 force-pushed the official/recommendation branch 2 times, most recently from a98dc2f to 89770cf Compare May 16, 2018 22:11

yhliang2018 added 3 commits May 16, 2018 15:13

Rename file

6466b7a

Address comments, update input pipeline, and add distribution strategy

4608811

Fix import error

c5747c8

karmel suggested changes May 18, 2018

View reviewed changes

yhliang2018 added 2 commits May 21, 2018 14:02

Address more comments

59d2fc4

Fix lints

fa486bf

karmel approved these changes May 22, 2018

View reviewed changes

yhliang2018 merged commit 81d7766 into master May 22, 2018

yhliang2018 deleted the official/recommendation branch May 22, 2018 05:31

		@@ -0,0 +1,35 @@
		# Recommendation Model
		This is an implementation of the Neural Collaborative Filtering (NCF) framework with Neural Matrix Factorization (NeuMF) model as described in the [Neural Collaborative Filtering](https://arxiv.org/abs/1708.05031) paper. Current implementation is based on the code from the authors' [NCF code](https://github.com/hexiangnan/neural_collaborative_filtering) and the Standford pytorch implementation at [mlperf repo](https://github.com/mlperf/reference/tree/master/recommendation/pytorch).

		parse_file_to_csv(FLAGS.data_dir, FLAGS.dataset)


		if __name__ == '__main__':



		if __name__ == '__main__':
		parser = argparse.ArgumentParser()



		def _validate_batch_size_for_multi_gpu(batch_size):
		"""For multi-gpu, batch-size must be a multiple of the number of GPUs.


		class NeuMF(object):
		"""Neural matrix factorization (NeuMF) model for recommendations."""



		def main(_):
		"""Download and extract the data from grouplens website."""

		NCF is a general framework under which a neural network architecture is proposed to model latent features of users and items in collaborative filtering of recommendation. Unlike traditional models, NCF does not resort to Matrix Factorization (MF) with an inner product on latent features of users and items. It replaces the inner product with a multi-layer perceptron that can learn an arbitrary function from data.

		Two instantiations of NCF are Generalized Matrix Factorization (GMF) and Multi-Layer Perceptron (MLP). GMF applies a linear kernel to model the latent feature interactions, and and MLP uses a nonlinear kernel to learn the interaction function from data. NeuMF is a fused model of GMF and MLP to better model the complex user-item interactions, and unifies the strengths of linearity of MF and non-linearity of MLP for modeling the user-item latent structures. NeuMF allows GMF and MLP to learn separate embeddings, and combines the two models by concatenating their last hidden layer. [neumf_model.py](neumf_model.py) defines the architecture details.


		Note the ml-20m dataset is large (the rating file is ~500 MB), and it may take several minutes (<10 mins)for data preprocessing.

		### Train and Evaluate Model

		return ratings


		def load_movielens_1million(file_name, sort=True):

Add recommendation model #4175

Add recommendation model #4175

Uh oh!

Conversation

yhliang2018 commented May 4, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yhliang2018 May 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k-w-w left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karmel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yhliang2018 May 8, 2018 •

edited

Loading

yhliang2018 May 9, 2018 •

edited

Loading