Random Deletion Layer #214

aflah02 · 2022-05-31T08:40:09Z

Fixes #152
Hey @mattdangerw @chenmoneygithub
I've added the layer and some tests, I'll add more tests in the meanwhile as the code is reviewed!
You can find a demo here
Once major obstacle is adding docstrings as they are also tested and that causes conflicts since the layer randomly deletes values the docstring output may or may not be what the layer outputs hence I could only give one example where probability is 1 and all the words are the same. Any workarounds for this?
Also I'm facing issues with datasets as .numpy() doesn't work in graph_mode. Simply trying to iterate and copy the tensor also fails in graph_mode, any workarounds for this?
I have a workaround which I can do while running it but I can't seem to figure out any changes to make to the code which would fix this -

tf.data.experimental.enable_debug_mode()
augmenter = RandomDeletion(
    probability = 0.5,
    max_deletions = 3,
)
inputs = ["Never gonna give you up", "Never gonna let you down", "Never gonna run around and desert you"]
ds = tf.data.Dataset.from_tensor_slices(inputs)
ds = ds.batch(3).map(lambda y: tf.py_function(
                          (lambda x: augmenter(x)),
                          inp=[y], Tout=tf.string
                      ))
ds.take(1).get_single_element()

Output -

<tf.Tensor: shape=(3,), dtype=string, numpy=array([b'Never you', b'let you', b'Never gonna run and'], dtype=object)>

mattdangerw

Thanks! Left some comments. At a high level, we should add some testing for compiling this layer in a tf function first (see comments on the test), then figure out how to simplify the layer call code so it's compilable.

mattdangerw · 2022-06-01T18:43:32Z

keras_nlp/layers/random_deletion.py

+
+
+class RandomDeletion(keras.layers.Layer):
+    """Augments input by randomly deleting words


period at end of sentence.

Also we should probably have a little more a description in a separate paragraph, that describes the flow of computation. E.g. split words, delete words, reforms words.

mattdangerw · 2022-06-01T18:44:56Z

keras_nlp/layers/random_deletion.py

+from tensorflow import keras
+
+
+class RandomDeletion(keras.layers.Layer):


I think before we ship this we should decide if we want to leave room for a separate character deletion layer, or if we would want to do that as attributes on this layer.

If a character deletion layer would be separate, we should probably call this "RandomCharacterDeletion" or something.

Agree here, we should consider the scalability.

My question is to make a character-level deletion layer, how much change would be required?

@chenmoneygithub I did discuss it with Matt here after his initial comment. We're now thinking of having them as 2 separate layers but would love to hear your thoughts on this!

I am fine with both! I raised the question because I want to check if possible to have a BaseClass and only do small customization on WordDelete and CharacaterDelete.

@chenmoneygithub That does seem like a more efficient design choice but not really sure about that. I'll get back if I find a good way for that

mattdangerw · 2022-06-01T18:51:38Z

keras_nlp/layers/random_deletion.py

+            inputs = inputs[0]
+        return inputs
+
+    def get_config(self) -> Dict[str, Any]:


I think we can leave the return annotation off. Let's stick to optional annotations for simple types only.

keras_nlp/layers/random_deletion.py

mattdangerw · 2022-06-01T19:01:27Z

keras_nlp/layers/random_deletion.py

+                > self.probability
+            )
+            # Iterate to check for any cases where deletions exceed the maximum
+            for i in range(len(row_splits) - 1):


I think we can clean this whole for loop up, though will need to think about how exactly a bit more.

One place for inspiration is probably the op code for tf text's RandomItemSelector. Which is also selecting a number of items based on a probability with a max cap.

I'm a little skeptical that this would function trace, you are doing a lot of looping and calling .numpy(), the later definitely won't work in a compiled context.

Reading about RandomItemSelector, I think briefly what it does is to:

Calculate how many to select, let's call it N.

shuffle the list/array/tensor's index array.

Pick the first N elements from index array, then use tf.gather to get the actual selected elements.

Thanks for sharing this @mattdangerw and for the concise summary @chenmoneygithub this seems like a much more smoother way to do it, will incorporate this idea in my code. Just to clarify do I need to cite this code file in the code as (inspired by) or something of that sort?

keras_nlp/layers/random_deletion_test.py

aflah02 · 2022-06-01T19:13:04Z

Thanks for the reviews, will get to work on this!

chenmoneygithub

Thanks for the PR! left some comments!

chenmoneygithub · 2022-06-02T00:29:03Z

keras_nlp/layers/random_deletion.py

+from tensorflow import keras
+
+
+class RandomDeletion(keras.layers.Layer):


Agree here, we should consider the scalability.

My question is to make a character-level deletion layer, how much change would be required?

chenmoneygithub · 2022-06-02T00:31:50Z

keras_nlp/layers/random_deletion.py

+    """Augments input by randomly deleting words
+
+    Args:
+        probability: probability of a word being chosen for deletion


Let's be consistent with the capital case after ":", also we need to note the type in the arg comment.

keras_nlp/layers/random_deletion.py

chenmoneygithub · 2022-06-02T00:38:02Z

keras_nlp/layers/random_deletion.py

+            dtype = tf.dtypes.as_dtype(kwargs["dtype"])
+            if not dtype.is_integer and dtype != tf.string:
+                raise ValueError(
+                    "Output dtype must be an integer type of a string. "


chenmoneygithub · 2022-06-02T00:38:17Z

keras_nlp/layers/random_deletion.py

+        self.max_deletions = max_deletions
+
+    def call(self, inputs):
+        """Augments input by randomly deleting words


period at end.

chenmoneygithub · 2022-06-02T00:42:08Z

keras_nlp/layers/random_deletion.py

+            row_splits = ragged_words.row_splits.numpy()
+            mask = (
+                tf.random.uniform(ragged_words.flat_values.shape)
+                > self.probability


the format looks a bit strange, is that formatted by black?

@chenmoneygithub Yup this was generated by black

chenmoneygithub · 2022-06-02T00:46:32Z

keras_nlp/layers/random_deletion.py

+                > self.probability
+            )
+            # Iterate to check for any cases where deletions exceed the maximum
+            for i in range(len(row_splits) - 1):


Reading about RandomItemSelector, I think briefly what it does is to:

Calculate how many to select, let's call it N.

shuffle the list/array/tensor's index array.

Pick the first N elements from index array, then use tf.gather to get the actual selected elements.

mattdangerw

A couple high level comments!

mattdangerw · 2022-06-09T23:43:01Z

keras_nlp/layers/random_word_deletion_test.py

+        output = augmenter(input)
+        self.assertAllEqual(output.shape, tf.convert_to_tensor(input).shape)
+
+    def test_shape_with_nested(self):


I'm not sure we actually need to support this. The problem is we are only supporting it for one level of nested rank, not multiple.

There are three crucial shapes to support as input: (), (batch_size,), and (batch_size, 1). We should be able to support all of these without the outer map_fn in your code above.

This could it would actually be reasonable to error out on, and ask for a support shape size.

That Makes Sense, I'll throw an error for the same

mattdangerw · 2022-06-10T00:08:27Z

keras_nlp/layers/random_word_deletion.py

+            if scalar_input:
+                inputs = tf.expand_dims(inputs, 0)
+
+            ragged_words = tf.strings.split(inputs)


I'm a little worried about how we are doing whitespace splitting here. There's two downsides with this approach.

It's destructive. Newlines, tabs, consecutive spaces will all get replaces with a single space.

It will group punctuation along with words and delete them both. So for an input "Are you OK?" It would delete "OK?" as a single token I think.

Do we want that behavior? Maybe in the context of EDA normal usage it is fine.

There might be an option to do something fancier here. Use tf_text.regex_split to split on whitespace and punctuation, but still keep the split characters. Use tf.strings.regex_full_match to exclude whitespace and punctation from deletion. Something like...

ragged_words = tf_text.regex_split(inputs, WHITESPACE_AND_PUNCTUATION_REGEX, WHITESPACE_AND_PUNCTUATION_REGEX) positions_flat = tf.range(tf.size(ragged_words.flat_values)) positions = ragged_words.with_flat_values(positions_flat) seperator_positions = tf.strings.regex_full_match(ragged_words, WHITESPACE_AND_PUNCTUATION_REGEX) positions = tf.ragged.boolean_mask(positions_flat, separator_positions == False)

Thanks @mattdangerw working on this rn, just to confirm in the last line it should be
tf.ragged.boolean_mask(positions, separator_positions == False) instead right?

@mattdangerw This method also does this weird thing where it preserves the whitespaces for the deleted texts too, as you can see in this demo the whitespaces in front of "like" and "to" are also preserved. This does satisfy the criteria of preserving spaces albeit in a very weird way is this the intended result?

chenmoneygithub

Thanks! My main concern is the current implementation of picking up the indices to delete is a little complex and use sequential computation.

chenmoneygithub · 2022-06-18T22:02:20Z

keras_nlp/layers/random_word_deletion.py

+class RandomWordDeletion(keras.layers.Layer):
+    """Augments input by randomly deleting words.
+
+    The layer works by splitting the words using `tf.strings.split` computes


This sentence is too long, let's refactor it a bit. One possible way is to break down by steps: 1) Split the words ... 2) randomly select ... 3) mask out... 4) join

chenmoneygithub · 2022-06-18T22:04:06Z

keras_nlp/layers/random_word_deletion.py

+    then removed before returning and the remaining tokens are joined back.
+
+    Args:
+        probability: probability of a word being chosen for deletion


Nit: argument type, and period at the end.

Also maybe call this selection_rate to be consistent with mlm_mask_generator

chenmoneygithub · 2022-06-18T22:05:25Z

keras_nlp/layers/random_word_deletion.py

+    ... )
+    >>> augmenter(["I like to fly kites, do you?",
+    ...     "Can we go fly some kites later?"])
+    <tf.Tensor: shape=(2,), dtype=string, numpy=array([b'I fly kites, do you?',


The output becomes byte string. I am wondering is it still a valid input to our tokenizers?

chenmoneygithub · 2022-06-18T22:06:11Z

keras_nlp/layers/random_word_deletion.py

+    <tf.Tensor: shape=(2,), dtype=string, numpy=array([b'fly kites, do you?',
+    b'we go some kites'], dtype=object)>
+
+    Batch the inputs and then Augment.


Nit: Augment => augment.

chenmoneygithub · 2022-06-18T22:08:39Z

keras_nlp/layers/random_word_deletion.py

+    def __init__(self, probability, max_deletions, name = None, **kwargs):
+        # Check dtype and provide a default.
+        if "dtype" not in kwargs or kwargs["dtype"] is None:
+            kwargs["dtype"] = tf.int32


All examples above have string inputs and string outputs, but here we allow integer inputs as well (we default to tf.int32). If integer inputs are expected, let's add one example to showcase, otherwise let's modify this type check part.

chenmoneygithub · 2022-06-18T22:14:14Z

keras_nlp/layers/random_word_deletion.py

+        def validate_and_fix_rank(inputs):
+            if not isinstance(inputs, (tf.Tensor, tf.RaggedTensor)):
+                inputs = tf.convert_to_tensor(inputs)
+                inputs = tf.cast(inputs, tf.string)


Do we allow integer tensors? If not, let's error out.

chenmoneygithub · 2022-06-18T22:16:09Z

keras_nlp/layers/random_word_deletion.py

+            elif inputs.shape.rank == 2:
+                if inputs.shape[1] != 1:
+                    raise ValueError(
+                        f"input must be of shape `[batch_size, 1]`. "


I am a little confused about this - if the input is tokenized input, such as [["Input", "must", "be", "string"], ["I", "don't", "know", "what", "I", "am", "writing"]], then we will reject the input?

@chenmoneygithub I think the confusion you are having is whether we allow input that is pre-split or if splitting happens on the layer.

If we split inside the layer, the input you are describing is invalid. And I think it is totally valid to support only shapes (bs, 1), (bs) and ().

If we split outside the layer, the input shapes we should support will change. Essentially we should support ragged with shape (bs, None) or dense with shape (None).

chenmoneygithub · 2022-06-18T22:32:31Z

keras_nlp/layers/random_word_deletion.py

+
+        # Shuffle and trim to items that are going to be selected.
+        def _shuffle_and_trim(x):
+            positions, top_n = x


For batched input, is this positions still having the batch dim? It looks not to me because you are taking the flat_values. But I might be wrong here.

chenmoneygithub · 2022-06-18T22:37:35Z

keras_nlp/layers/random_word_deletion.py

+            shuffled = tf.random.shuffle(positions)
+            return shuffled[:top_n]
+
+        selected_for_mask = tf.map_fn(


I am a little concerning about the current implementation - the current logic of picking up indices to delete looks complex to me, and you are using tf.map_fn to handle each element separately instead of doing a batch processing, which could get slow.

Rethinking about how we can do the index pick up, can we do the following? Let's assume we have already done the split, so the input is a 2D raggedTensor.

calculate how many words to pick up for each sequence, let's call it num_row_i for ith row.

For each row, pick up 1 - num_row_i indices.

for the selected indices in 2), we do a tf.gather to gather them into a new tensor, which are the words we want to keep.

reduce_join the output of 3) to reconstruct the string for each row.

In this way, we can keep the batch dimension along with the computation. But I am not 100% sure if this works, WDYT?

Let's chat about this tomorrow! Overall I think we need to make sure this can trace, but don't actually need to care about performance that much. If you are running an augmentation, it's because you have very little data. You don't actually care about speed.

I think the reason this is so complex (here and in tf.text), is that there is no good way to uniformly sample from a pool of candidate indices with a max cap without using a map call. It's actually the shuffle call that we can't run without map.

mattdangerw · 2022-06-21T23:44:43Z

One thing we need to add here is a seed argument. We support that in all our random layers, for example initializers.

I think the easiest way to do that will be to subclass keras.__internal__.layers.BaseRandomLayer code, and pass the seed to the super class. Then we can use a few functions to generate seeds for the ops themselves, self._random_generator.make_seed_for_stateless_op for the seed for the stateless_binomial call, and self._random_generator.make_legacy_seed for the seed for the shuffle call.

After we do all of this, I think we should see the following behavior...

tf.random.set_seed(1)
RandomDelection(0.2, seed=1)(inputs)

tf.random.set_seed(1)
RandomDelection(0.2, seed=1)(inputs) # output is the same as first call.

tf.random.set_seed(1)
RandomDelection(0.2, seed=2)(inputs) # output is different than the first call.

tf.random.set_seed(1)
RandomDelection(0.2, seed=2)(inputs) # output is the same as third call.

Overall, this is still a little more tricky than we would like, as the underlying random API for tensorflow changes.

…s-nlp into RandomDeletionLayer

mattdangerw

Few comments! Let's discuss re any sort of skip argument tomorrow

keras_nlp/layers/random_deletion.py

mattdangerw · 2022-07-06T01:34:13Z

keras_nlp/layers/random_deletion.py

+    >>> tf.random.get_global_generator().reset_from_seed(30)
+    >>> tf.random.set_seed(30)
+    >>> inputs = tf.strings.unicode_split(["Hey Dude", "Speed Up"], "UTF-8")
+    >>> augmenter = keras_nlp.layers.RandomDeletion(rate = 0.4,


I think you can show these examples without max_deletions, just to keep things simple

keras_nlp/layers/random_deletion.py

keras_nlp/layers/random_deletion_test.py

mattdangerw · 2022-07-06T01:47:48Z

keras_nlp/layers/random_deletion.py

+        self.rate = rate
+        self.max_deletions = max_deletions
+        self.seed = seed
+        self._random_generator = backend.RandomGenerator(seed)


just use this from the keras import, keras.backend.RandomGenerator

There is some weird behaviour here, if I do this I get AttributeError: module 'keras.api._v2.keras.backend' has no attribute 'RandomGenerator' not sure why

RandomGenerator is not exposed as a public API: https://github.com/keras-team/keras/blob/v2.9.0/keras/backend.py#L1823

There is actually an API in TF: tf.random.Generator (https://www.tensorflow.org/api_docs/python/tf/random/Generator), can we use that one?

mattdangerw · 2022-07-06T01:49:33Z

keras_nlp/layers/random_deletion.py

+        word_counts = tf.cast(inputs.row_lengths(), "float32")
+        num_to_select = tf.random.stateless_binomial(
+            shape=tf.shape(word_counts),
+            seed=tf.random.get_global_generator().make_seeds()[:, 0],


can't we do self._random_generator.make_seed_for_stateless_op()?

@mattdangerw I don't think we can. It returns a None right now unless I set the rng_type but as per the latest tf release it is not available currently and was added after the latest release

mattdangerw · 2022-07-06T01:50:26Z

keras_nlp/layers/random_deletion.py

+    Examples:
+
+    Word level usage
+    >>> tf.random.get_global_generator().reset_from_seed(30)


after we make the minor randomness changes suggested below, can we do tf.keras.utils.set_random_seed(30)? That would make this a little more readable with a single seed line.

mattdangerw

looking close! few comments

keras_nlp/layers/random_deletion.py

keras_nlp/layers/random_deletion_test.py

keras_nlp/layers/random_deletion.py

mattdangerw

Few more comments

keras_nlp/layers/random_deletion.py

mattdangerw

minor comments, lgtm pending these changes

keras_nlp/layers/random_deletion.py

mattdangerw · 2022-07-25T23:34:19Z

keras_nlp/layers/random_deletion.py

+            )
+        elif self.skip_py_fn:
+
+            def _preprocess_string_fn(word):


I think we can simplify this a bit. We should also rename word -> token. Something like...

string_fn = lambda x: self.skip_py_fn(x.numpy().decode("utf-8")) int_fn = lambda x: self.skip_py_fn(x.numpy()) py_fn = string_fn if ragged.dtype == tf.string else int_fn return tf.map_fn( lambda x: tf.py_function(py_fn, [x], tf.bool), inputs.flat_values, dtype=tf.bool, )

@mattdangerw This seems easier however it fails the style rules, the formatter gives this error - E731 do not assign a lambda expression, use a def. I guess this will persist till we keep assigning lambdas. Alternatively this works:

def string_fn(token): return self.skip_py_fn(token.numpy().decode("utf-8")) def int_fn(token): return self.skip_py_fn(token.numpy()) py_fn = string_fn if inputs.dtype == tf.string else int_fn

keras_nlp/layers/random_deletion_test.py

mattdangerw · 2022-07-25T23:49:22Z

keras_nlp/layers/random_deletion_test.py

+        for i in range(output.shape[0]):
+            self.assertAllEqual(output[i], exp_output[i])
+
+    def test_skip_options(self):


This looks like we are just testing roughly the same things as the dataset test below. Can we just move the skip_list test into the dataset test below and delete this test?

@mattdangerw The dataset tests already have skip options being tested so should I remove this altogether?

chenmoneygithub

LGTM! Let's wait for Matt's approval.

mattdangerw

LGTM!

aflah02 · 2022-07-27T15:37:28Z

Thanks!

aflah02 added 9 commits April 30, 2022 00:11

Random Deletion Working

bf3219f

Added to init

66c0de7

WOrking

d2508dc

Merge branch 'keras-team:master' into RandomDeletionLayer

9e2e243

Working

b48dc5a

Current Status

02ce27d

Working Layer More Tests to be Added

f137256

Fixed Scalar Case

c7dcb8a

Added Comments

515a1d3

mattdangerw requested changes Jun 1, 2022

View reviewed changes

aflah02 mentioned this pull request Jun 1, 2022

Random Deletion Layer - Data Augmentation #152

Closed

chenmoneygithub reviewed Jun 2, 2022

View reviewed changes

aflah02 added 5 commits June 3, 2022 20:26

Minor Fixes

6acfb62

Major Refactors and Fixes, ToDo - Docs, Tests

68fcae0

Fixed Shape Issues for Scalar Lists

058d572

Finalized Tests and DocString

7a57339

Ran Stylers Added More Descriptive DocString

9cf9a2a

aflah02 requested review from chenmoneygithub and mattdangerw June 9, 2022 07:12

Fixed Failing Docstring Tests

ce10e2d

mattdangerw requested changes Jun 10, 2022

View reviewed changes

aflah02 added 2 commits June 10, 2022 14:41

Removed Map Call and Unsupported Test

a1ab88e

Shape Fixes

17e2365

chenmoneygithub suggested changes Jun 18, 2022

View reviewed changes

aflah02 added 4 commits June 30, 2022 01:21

Working

b5cfe45

Working

839a770

Changing Parent Class

ec2d4ed

Changes

dbdd690

aflah02 added 4 commits July 2, 2022 00:59

Formatter Ran

fd856ad

Merge branch 'keras-team:master' into RandomDeletionLayer

292353b

Finalized

9c447dc

Merge branch 'RandomDeletionLayer' of https://github.com/aflah02/kera…

35cf31a

…s-nlp into RandomDeletionLayer

mattdangerw requested changes Jul 6, 2022

View reviewed changes

aflah02 added 7 commits July 6, 2022 12:31

Addresed Review Comments

b26dd24

Fornatter

983bfb3

Added new Tests

4813f1f

Fan Formatter

86bafbe

Skip Works

46eda29

New Randomness

906614a

All Testing Done

20df2b2

mattdangerw requested changes Jul 20, 2022

View reviewed changes

mattdangerw reviewed Jul 20, 2022

View reviewed changes

keras_nlp/layers/random_deletion.py Outdated Show resolved Hide resolved

aflah02 and others added 4 commits July 21, 2022 00:00

Review Changes

0c8fdc5

Addressed all Review Comments

3d87d14

Merge branch 'keras-team:master' into RandomDeletionLayer

5ee888a

Copy edits for docstrings

a77880e

mattdangerw requested changes Jul 22, 2022

View reviewed changes

keras_nlp/layers/random_deletion.py Outdated Show resolved Hide resolved

keras_nlp/layers/random_deletion.py Outdated Show resolved Hide resolved

keras_nlp/layers/random_deletion.py Outdated Show resolved Hide resolved

keras_nlp/layers/random_deletion.py Show resolved Hide resolved

aflah02 added 3 commits July 25, 2022 22:59

Finishes

9c1904c

Merge branch 'keras-team:master' into RandomDeletionLayer

b87394d

Changed Tokenizer Import

81ba7bf

mattdangerw requested changes Jul 25, 2022

View reviewed changes

Addressed Reviews

0da28dc

chenmoneygithub approved these changes Jul 26, 2022

View reviewed changes

Fix typo

d91a1bb

mattdangerw approved these changes Jul 27, 2022

View reviewed changes

mattdangerw merged commit 58f4bbb into keras-team:master Jul 27, 2022



		class RandomDeletion(keras.layers.Layer):
		"""Augments input by randomly deleting words

		from tensorflow import keras


		class RandomDeletion(keras.layers.Layer):

Random Deletion Layer #214

Random Deletion Layer #214

Conversation

aflah02 commented May 31, 2022 • edited

mattdangerw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aflah02 commented Jun 1, 2022

chenmoneygithub left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattdangerw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aflah02 Jun 10, 2022 • edited

Choose a reason for hiding this comment

chenmoneygithub left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattdangerw commented Jun 21, 2022

mattdangerw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattdangerw left a comment • edited

Choose a reason for hiding this comment

mattdangerw left a comment

Choose a reason for hiding this comment

mattdangerw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chenmoneygithub left a comment

Choose a reason for hiding this comment

mattdangerw left a comment

Choose a reason for hiding this comment

aflah02 commented Jul 27, 2022

aflah02 commented May 31, 2022 •

edited

aflah02 Jun 10, 2022 •

edited

mattdangerw left a comment •

edited