Add BLEU Score #222

abheesht17 · 2022-06-13T02:46:55Z

Resolves #65

Notebooks:

OLD: https://colab.research.google.com/drive/1TZ8XnrmMcU8ZE2J-3amb44p-U-hxER53?usp=sharing

Edit (commit e90bef3):

We have decided to go ahead with a tf.py_function implementation. Yet to add UTs, but otherwise, it's done.

Notebooks:

Graph ops vs Python ops: https://colab.research.google.com/drive/1TZ8XnrmMcU8ZE2J-3amb44p-U-hxER53?usp=sharing (preserving this here for posterity, in case someone attempts to implement BLEU score using Graph ops)
Trial run on WMT14 en-de dataset: https://colab.research.google.com/drive/1jBN56tc6EdCctr50Em7Xk7lx8_y45QV_?usp=sharing

mattdangerw

Thanks for all the work on this! Left a few questions and a few comments.

Feel free to start adding unit tests!

mattdangerw · 2022-06-30T19:01:15Z

keras_nlp/metrics/bleu.py

+    inputs of shapes `()`, `(batch_size,)` and `(batch_size, 1)`.
+
+    Args:
+        tokenizer: callable. A function that takes a string `tf.RaggedTensor`


What happens it you pass a tokenizer layer here, will that work? Say byte tokenizer for simplicity.

Hmmm, it won't work with byte tokeniser because we use tensor_to_string_list in the code. Do you want me to change that?

I think we should either support our tokenizers or not name this argument to something else.

Tokenizer means something specific in our library now, if we use that name but don't support our tokenizer class that is a bad look.

We do support our tokenisers. I've added a unit test here:

keras-nlp/keras_nlp/metrics/bleu_test.py

Line 105 in 0b6ebfa

def test_custom_tokenizer(self):

mattdangerw · 2022-06-30T19:01:54Z

keras_nlp/metrics/bleu.py

+class Bleu(keras.metrics.Metric):
+    """BLEU metric.
+
+    This class implements the BLEU metric. BLEU is generally used to evaluate


We should probably mention more prominently that this will replicate sacrebleu by default, but can be used with other tokenizers e.g. for other languages.

mattdangerw · 2022-06-30T19:02:58Z

keras_nlp/metrics/bleu.py

+                )
+
+        def _get_ngrams(segment, max_order):
+            """Extracts all n-grams upto a given maximum order from an input


summary sentence for a docstring should always fit on a single line.

mattdangerw · 2022-06-30T20:09:16Z

keras_nlp/metrics/bleu.py

+                    ngram_counts[ngram] += 1
+            return ngram_counts
+
+        def corpus_bleu(


Why is this public? do we expect this to be called explicitly? If so can you show the use case?

keras_nlp/metrics/bleu.py

mattdangerw · 2022-06-30T20:35:22Z

keras_nlp/metrics/bleu.py

+            considered. Defaults to 4.
+        smooth: bool. Whether to apply Lin et al. 2004 smoothing to the BLEU
+            score. Defaults to False.
+        variant: string. Either `"corpus_bleu"` or `"sentence_bleu"`. The former


It seems like corpus bleu is the better option here? I see that sacrebleu exposes methods for both of these, but does not seems to document the sentence one. Huggingface looks like it might not even have an option for this (is that true?).

I guess I'm wondering if it might make sense to not even expose this, and wait till someone asks for the sentence option.

After doing a survey, this is what I found:

NLTK exposes Sentence BLEU, but the function merely takes in one sample and calls the corpus BLEU function: https://github.com/nltk/nltk/blob/develop/nltk/translate/bleu_score.py#L20. This answer explains it well: https://stackoverflow.com/a/40650581. I think they expect the user to average the metric on their own.

PyTorch Ignite does provide an option for using Corpus BLEU/Sentence BLEU: https://pytorch.org/ignite/generated/ignite.metrics.Bleu.html (average = macro_average, and surprisingly, Sentence BLEU is the default).

Hugging Face's SacreBLEU implementation does not explicitly provide an option, but have a look at this parameter: https://github.com/huggingface/datasets/blob/master/metrics/sacrebleu/sacrebleu.py#L66.

Conclusion: I think the expectation is that, if users want to compute the Sentence BLEU score, they can do so by passing one sample at a time, and average over the returned scores.

Some additional notes
However, another point to note is that HF provides two options with all its metrics:

.compute() - user can pass one sample at a time, get the BLEU scores, and average over them for computing Sentence BLEU.

.add_batch() - will compute the Corpus BLEU score across all samples across batches.

We use Keras metrics similar to the add_batch() function. So, if the user wants to compute the Sentence BLEU score, he/she/they will have to re-initialise the metric for every sample. PyTorch Ignite metrics also work similar to the add_batch function, which is why they have provided an option for macro/micro-averaging. So, I am just wondering whether HF and NLTK do not provide explicit options to macro-average the BLEU scores because the user can average the BLEU scores. But with Ignite, the user can't do that without re-initialising before every sample, which is why an option has been provided,

mattdangerw · 2022-06-30T20:38:17Z

keras_nlp/metrics/bleu.py

+        max_order: int. The maximum n-gram order to use. For example, if
+            `max_order` is set to 3, unigrams, bigrams, and trigrams will be
+            considered. Defaults to 4.
+        smooth: bool. Whether to apply Lin et al. 2004 smoothing to the BLEU


Can we describe this better? Lin et al. 2004 with a period in the middle of the docstring does not read very well. Also please add to reference section.

mattdangerw · 2022-06-30T20:44:17Z

keras_nlp/metrics/bleu.py

+                reference_length += min(len(r) for r in references)
+                translation_length += len(translation)
+
+                merged_ref_ngram_counts = collections.Counter()


So we've confirmed at this point that the Counter approach is much more efficient correct? Can we open an issue to remove the py_function from bleu as a follow up?

Describe a bit of the slowdown we saw when trying to do that today. Probably not urgent, but maybe someday we could collaborate with tf.text to ship an efficient non-python version of this op.

Right! Opened an issue for this: #247 :)

mattdangerw · 2022-06-30T20:45:26Z

keras_nlp/metrics/bleu.py

+                f"Received: variant={variant}"
+            )
+
+        def default_tokenizer(inputs):


There are a lot of nested function that are quite long all over. Not very readable.

Can we either pull them out of the class entirely, or make them class methods?

Made a few of them class methods.

Maybe make this one a private method called _tokenizer that does.

if self.tokenizer: return self.tokenizer(x) for pattern, replacement...

That way saving self.tokenizer in the config would work as expected.

abheesht17

Replied to a few comments :)

abheesht17 · 2022-07-01T00:52:05Z

keras_nlp/metrics/bleu.py

+            considered. Defaults to 4.
+        smooth: bool. Whether to apply Lin et al. 2004 smoothing to the BLEU
+            score. Defaults to False.
+        variant: string. Either `"corpus_bleu"` or `"sentence_bleu"`. The former


After doing a survey, this is what I found:

NLTK exposes Sentence BLEU, but the function merely takes in one sample and calls the corpus BLEU function: https://github.com/nltk/nltk/blob/develop/nltk/translate/bleu_score.py#L20. This answer explains it well: https://stackoverflow.com/a/40650581. I think they expect the user to average the metric on their own.

PyTorch Ignite does provide an option for using Corpus BLEU/Sentence BLEU: https://pytorch.org/ignite/generated/ignite.metrics.Bleu.html (average = macro_average, and surprisingly, Sentence BLEU is the default).

Hugging Face's SacreBLEU implementation does not explicitly provide an option, but have a look at this parameter: https://github.com/huggingface/datasets/blob/master/metrics/sacrebleu/sacrebleu.py#L66.

Conclusion: I think the expectation is that, if users want to compute the Sentence BLEU score, they can do so by passing one sample at a time, and average over the returned scores.

Some additional notes
However, another point to note is that HF provides two options with all its metrics:

.compute() - user can pass one sample at a time, get the BLEU scores, and average over them for computing Sentence BLEU.

.add_batch() - will compute the Corpus BLEU score across all samples across batches.

We use Keras metrics similar to the add_batch() function. So, if the user wants to compute the Sentence BLEU score, he/she/they will have to re-initialise the metric for every sample. PyTorch Ignite metrics also work similar to the add_batch function, which is why they have provided an option for macro/micro-averaging. So, I am just wondering whether HF and NLTK do not provide explicit options to macro-average the BLEU scores because the user can average the BLEU scores. But with Ignite, the user can't do that without re-initialising before every sample, which is why an option has been provided,

abheesht17 · 2022-07-01T01:04:38Z

keras_nlp/metrics/bleu.py

+                reference_length += min(len(r) for r in references)
+                translation_length += len(translation)
+
+                merged_ref_ngram_counts = collections.Counter()


Right! Opened an issue for this: #247 :)

abheesht17

Addressed most comments. Thanks for the review, @mattdangerw!

abheesht17 · 2022-07-01T15:25:20Z

keras_nlp/metrics/bleu.py

+    inputs of shapes `()`, `(batch_size,)` and `(batch_size, 1)`.
+
+    Args:
+        tokenizer: callable. A function that takes a string `tf.RaggedTensor`


Hmmm, it won't work with byte tokeniser because we use tensor_to_string_list in the code. Do you want me to change that?

keras_nlp/metrics/bleu.py

abheesht17 · 2022-07-01T16:02:14Z

keras_nlp/metrics/bleu.py

+                f"Received: variant={variant}"
+            )
+
+        def default_tokenizer(inputs):


Made a few of them class methods.

chenmoneygithub

Thank you! Dropped some comments on style & interface design. Will take another pass on the implementation body.

chenmoneygithub · 2022-07-06T19:49:00Z

keras_nlp/metrics/bleu.py

+    """BLEU metric.
+
+    This class implements the BLEU metric. BLEU is generally used to evaluate
+    machine translation systems. by default, this implementation replicates


nit: capital case By default

keras_nlp/metrics/bleu.py

chenmoneygithub · 2022-07-06T19:55:05Z

keras_nlp/metrics/bleu.py

+
+    Note on input shapes:
+    For `y_true` and `y_pred`, this class supports the following shapes:
+    If `y_pred` is a scalar value, `y_true` has to be a 1D dense tensor.


This requirement on shape match is a little weird. Suppose both y_pred and y_true are scalar, can we just convert y_true for them?

Hmmm, the idea right now is that one sample can have multiple references. So, basically, a translation can have multiple reference sentences. That's why the rank of y_true = rank of y_pred + 1.

chenmoneygithub · 2022-07-06T19:56:36Z

keras_nlp/metrics/bleu.py

+    Args:
+        tokenizer: callable. A function that takes a string `tf.RaggedTensor`
+            (of any shape), and tokenizes the strings in the tensor. This
+            function should use TensorFlow graph ops. If the tokenizer is not


Is it necessary? If people are not interested in using model.evaluate(), can they just run it in pure eager mode?

True, but we call the tokeniser after converting the inputs to tensors. So, we have to use TF ops here such as tf.strings.regex_replace().

chenmoneygithub · 2022-07-06T20:49:25Z

keras_nlp/metrics/bleu.py

+        )
+
+    def _get_ngrams(self, segment, max_order):
+        """Extracts all n-grams upto a given maximum order from an input segment.


remember to fix this!

chenmoneygithub · 2022-07-06T20:50:46Z

keras_nlp/metrics/bleu.py

+        https://github.com/tensorflow/nmt/blob/master/nmt/scripts/bleu.py.
+
+        Args:
+            segment: string. Text segment from which n-grams will be


Is this a string or tensor of split tokens?

Yeah, this should be list.

keras_nlp/metrics/bleu.py

abheesht17

Thank you, @chenmoneygithub, for the review comments. I've addressed them!

abheesht17 · 2022-07-07T12:17:55Z

keras_nlp/metrics/bleu.py

+
+    Note on input shapes:
+    For `y_true` and `y_pred`, this class supports the following shapes:
+    If `y_pred` is a scalar value, `y_true` has to be a 1D dense tensor.


Hmmm, the idea right now is that one sample can have multiple references. So, basically, a translation can have multiple reference sentences. That's why the rank of y_true = rank of y_pred + 1.

abheesht17 · 2022-07-07T12:19:32Z

keras_nlp/metrics/bleu.py

+    Args:
+        tokenizer: callable. A function that takes a string `tf.RaggedTensor`
+            (of any shape), and tokenizes the strings in the tensor. This
+            function should use TensorFlow graph ops. If the tokenizer is not


True, but we call the tokeniser after converting the inputs to tensors. So, we have to use TF ops here such as tf.strings.regex_replace().

abheesht17 · 2022-07-07T12:20:19Z

keras_nlp/metrics/bleu.py

+        https://github.com/tensorflow/nmt/blob/master/nmt/scripts/bleu.py.
+
+        Args:
+            segment: string. Text segment from which n-grams will be


Yeah, this should be list.

mattdangerw

Thanks!

mattdangerw · 2022-07-08T04:55:20Z

keras_nlp/utils/tensor_utils.py

+
+    Args:
+        inputs: Input tensor, or dict/list/tuple of input tensors.
+        *args: Additional positional arguments.


remove *args and **kwargs

mattdangerw · 2022-07-08T04:58:57Z

keras_nlp/metrics/bleu_test.py

+from keras_nlp.tokenizers import ByteTokenizer
+
+
+class BleuTest(tf.test.TestCase):


can we add a test with model.compile to make sure we are ok in function compilation? I think we did that for rouge.

My bad - I forgot to add it.

mattdangerw · 2022-07-08T05:03:01Z

keras_nlp/metrics/bleu.py

+        )
+
+    def update_state(self, y_true, y_pred, sample_weight=None):
+        def validate_and_fix_rank(inputs, tensor_name, base_rank=0):


let's just split this into a separate private method on the layer, code can stay unchanged

I'll remove calculate_bleu_score from here. Let's keep validate_and_fix_rank inside this function for homogeneity (since we have done the same for ROUGE and Edit Distance)?

mattdangerw · 2022-07-08T05:06:41Z

keras_nlp/metrics/bleu.py

+            possible_matches,
+            translation_length,
+            reference_length,
+        ) = tf.py_function(


given that you are converting everything to numpy, would this do better as a numpy_function call?

Hmmm, I think it won't work because we have ragged tensors? Numpy has no support for ragged matrices, I think.

I think it fails to convert ragged input tensors to numpy arrays:

File "/home/abheesht/repos/keras-nlp/keras_nlp/metrics/bleu.py", line 224, in _corpus_bleu for (references, translation) in zip( TypeError: in user code: File "/home/abheesht/repos/keras-nlp/keras_nlp/metrics/bleu.py", line 349, in update_state * ( File "/home/abheesht/repos/keras-nlp/keras_nlp/metrics/bleu.py", line 302, in _calculate_bleu_score ) = self._corpus_bleu( File "/home/abheesht/repos/keras-nlp/keras_nlp/metrics/bleu.py", line 224, in _corpus_bleu for (references, translation) in zip( TypeError: iteration over a 0-d array

sg, let's stick with py_function for now

mattdangerw · 2022-07-08T05:06:58Z

keras_nlp/metrics/bleu.py

+                self.dtype,
+                self.dtype,
+                self.dtype,
+                self.dtype,


If you remove the trailing comma will this format to one line?

keras_nlp/metrics/bleu.py

mattdangerw · 2022-07-08T05:11:57Z

keras_nlp/metrics/bleu.py

+    inputs of shapes `()`, `(batch_size,)` and `(batch_size, 1)`.
+
+    Args:
+        tokenizer: callable. A function that takes a string `tf.RaggedTensor`


I think we should either support our tokenizers or not name this argument to something else.

Tokenizer means something specific in our library now, if we use that name but don't support our tokenizer class that is a bad look.

keras_nlp/metrics/bleu.py

mattdangerw · 2022-07-08T05:24:59Z

keras_nlp/metrics/bleu.py

+                f"Received: variant={variant}"
+            )
+
+        def default_tokenizer(inputs):


Maybe make this one a private method called _tokenizer that does.

if self.tokenizer: return self.tokenizer(x) for pattern, replacement...

That way saving self.tokenizer in the config would work as expected.

mattdangerw · 2022-07-08T05:28:03Z

keras_nlp/metrics/bleu.py

+    https://cloud.google.com/translate/automl/docs/evaluate#bleu.
+
+    Note on input shapes:
+    `y_pred` can be a scalar (of shape `()`), or a dense tensor of shape


How common is it to call bleu with only a single reference translation (is that unusual, or the normal case?). If it's usual, could we consider supporting the case that y_pred and y_true have the same shape? That might lead to a simpler overall usage in that case.

Open question, I'm not sure if that is true.

Well, it is called often with one reference...but all implementations take input the way I've mentioned in the doc-string, whether it be SacreBLEU, or HF, or even NLTK.

abheesht17

Thanks, @mattdangerw, for the comments!

abheesht17 · 2022-07-08T08:21:16Z

keras_nlp/metrics/bleu.py

+    https://cloud.google.com/translate/automl/docs/evaluate#bleu.
+
+    Note on input shapes:
+    `y_pred` can be a scalar (of shape `()`), or a dense tensor of shape


Well, it is called often with one reference...but all implementations take input the way I've mentioned in the doc-string, whether it be SacreBLEU, or HF, or even NLTK.

abheesht17 · 2022-07-08T08:23:30Z

keras_nlp/metrics/bleu.py

+    inputs of shapes `()`, `(batch_size,)` and `(batch_size, 1)`.
+
+    Args:
+        tokenizer: callable. A function that takes a string `tf.RaggedTensor`


We do support our tokenisers. I've added a unit test here:

keras-nlp/keras_nlp/metrics/bleu_test.py

Line 105 in 0b6ebfa

def test_custom_tokenizer(self):

abheesht17 · 2022-07-08T08:45:47Z

keras_nlp/metrics/bleu_test.py

+from keras_nlp.tokenizers import ByteTokenizer
+
+
+class BleuTest(tf.test.TestCase):


My bad - I forgot to add it.

abheesht17 · 2022-07-08T08:52:17Z

keras_nlp/metrics/bleu.py

+        )
+
+    def update_state(self, y_true, y_pred, sample_weight=None):
+        def validate_and_fix_rank(inputs, tensor_name, base_rank=0):


I'll remove calculate_bleu_score from here. Let's keep validate_and_fix_rank inside this function for homogeneity (since we have done the same for ROUGE and Edit Distance)?

abheesht17 · 2022-07-08T11:35:39Z

keras_nlp/metrics/bleu.py

+            possible_matches,
+            translation_length,
+            reference_length,
+        ) = tf.py_function(


Hmmm, I think it won't work because we have ragged tensors? Numpy has no support for ragged matrices, I think.

abheesht17 · 2022-07-08T12:36:28Z

keras_nlp/metrics/bleu.py

+            possible_matches,
+            translation_length,
+            reference_length,
+        ) = tf.py_function(


I think it fails to convert ragged input tensors to numpy arrays:

File "/home/abheesht/repos/keras-nlp/keras_nlp/metrics/bleu.py", line 224, in _corpus_bleu for (references, translation) in zip( TypeError: in user code: File "/home/abheesht/repos/keras-nlp/keras_nlp/metrics/bleu.py", line 349, in update_state * ( File "/home/abheesht/repos/keras-nlp/keras_nlp/metrics/bleu.py", line 302, in _calculate_bleu_score ) = self._corpus_bleu( File "/home/abheesht/repos/keras-nlp/keras_nlp/metrics/bleu.py", line 224, in _corpus_bleu for (references, translation) in zip( TypeError: iteration over a 0-d array

keras_nlp/metrics/bleu.py

mattdangerw

This looks great! Definitely a complex metric, so I expect we will have to keep iterating, but I think we are in a good spot to land. Last few comments.

mattdangerw · 2022-07-08T22:23:27Z

keras_nlp/metrics/bleu.py

+    `y_true` should be a tensor of shape `(num_references,)`. For batched
+    inputs, `y_pred` should be a tensor of shape `(batch_size,)`,
+    and `y_true` should be a tensor of shape `(batch_size, num_references)`. In
+    case of batched inputs, `y_true` can also be of shape `(batch_size, None)`


a ragged tensor with shape (batch_size, None)

mattdangerw · 2022-07-08T22:24:37Z

keras_nlp/metrics/bleu.py

+            possible_matches,
+            translation_length,
+            reference_length,
+        ) = tf.py_function(


sg, let's stick with py_function for now

mattdangerw · 2022-07-08T22:24:50Z

keras_nlp/metrics/bleu.py

+        )
+
+    def update_state(self, y_true, y_pred, sample_weight=None):
+        def validate_and_fix_rank(inputs, tensor_name, base_rank=0):


mattdangerw · 2022-07-08T22:25:25Z

keras_nlp/metrics/bleu.py

+        )
+
+    def _get_ngrams(self, segment, max_order):
+        """Extracts all n-grams upto a given maximum order from an input segment.


remember to fix this!

mattdangerw · 2022-07-11T23:23:31Z

LGTM from me! @chenmoneygithub feel free to merge whenever you are satisfied here!

abheesht17 added 9 commits June 13, 2022 07:22

Add rough BLEU Score implementation

f151982

Merge branch 'master' into bleu

7708cd9

Add BLEU score class

0f757d5

Merge branch 'keras-team:master' into bleu

e90bef3

Add arg for corpus BLEU and sentence BLEU

eface1e

Minor bug fixes

dc2110e

More bug fixes

e18cc50

Add doc-strings

d59058b

Add references

5ddcfa7

mattdangerw requested changes Jun 30, 2022

View reviewed changes

abheesht17 commented Jul 1, 2022

View reviewed changes

Merge branch 'keras-team:master' into bleu

b2d0822

abheesht17 commented Jul 1, 2022

View reviewed changes

abheesht17 added 3 commits July 1, 2022 21:49

Address reew comments - I

45136fb

Merge branch 'master' into bleu

23f9a2f

Add UTs, allow dtypes other than string tensors, remove Sentence BLEU

0217b71

chenmoneygithub suggested changes Jul 6, 2022

View reviewed changes

Address review comments-II, make shape changes

0b6ebfa

abheesht17 commented Jul 7, 2022

View reviewed changes

mattdangerw approved these changes Jul 8, 2022

View reviewed changes

abheesht17 commented Jul 8, 2022

View reviewed changes

abheesht17 added 3 commits July 8, 2022 19:15

Address review comments-III

be897dd

Serialise tokenizer

363da3a

Small fixes

fa2c658

mattdangerw approved these changes Jul 8, 2022

View reviewed changes

Doc-string changes

0acaae5

chenmoneygithub approved these changes Jul 11, 2022

View reviewed changes

chenmoneygithub merged commit 13a9f2d into keras-team:master Jul 11, 2022

		from keras_nlp.tokenizers import ByteTokenizer


		class BleuTest(tf.test.TestCase):

Add BLEU Score #222

Add BLEU Score #222

Conversation

abheesht17 commented Jun 13, 2022 • edited

mattdangerw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abheesht17 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abheesht17 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chenmoneygithub left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abheesht17 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattdangerw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abheesht17 Jul 8, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abheesht17 Jul 8, 2022 • edited

Choose a reason for hiding this comment

abheesht17 left a comment

Choose a reason for hiding this comment

abheesht17 Jul 8, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abheesht17 Jul 8, 2022 • edited

Choose a reason for hiding this comment

mattdangerw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abheesht17 commented Jun 13, 2022 •

edited

abheesht17 Jul 8, 2022 •

edited

abheesht17 Jul 8, 2022 •

edited

abheesht17 Jul 8, 2022 •

edited

abheesht17 Jul 8, 2022 •

edited