[PT_BREAK] Implement Tanh Gelu Approximation #3039

rdspring1 · 2021-07-14T20:35:07Z

Enables Tanh Gelu approximation via integer argument
This is to stay in consistency with PyTorch core changes that are introducing in pytorch/pytorch#61439

JackCaoG · 2021-07-14T20:39:12Z

Thanks Ryan! Feel free to @ me for review when upstream pr is ready and this pr is ready to be reviewed.

Summary: 1. Implements pytorch/pytorch#39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: pytorch/pytorch#61439 Reviewed By: mikaylagawarecki Differential Revision: D33744717 Pulled By: jbschlosser fbshipit-source-id: d64532a562ed53247bb4fa52bb16722634d5c187

Summary: 1. Implements #39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: #61439 Reviewed By: mikaylagawarecki Differential Revision: D33744717 Pulled By: jbschlosser fbshipit-source-id: d64532a562ed53247bb4fa52bb16722634d5c187

Summary: 1. Implements #39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: #61439 Reviewed By: mikaylagawarecki Differential Revision: D33744717 Pulled By: jbschlosser fbshipit-source-id: d64532a562ed53247bb4fa52bb16722634d5c187 (cherry picked from commit 4713dd9)

rdspring1 · 2022-01-28T17:38:12Z

@JackCaoG The upstream gelu PR was just merged, so this PR is ready for review.

JackCaoG · 2022-01-28T18:28:47Z

@rdspring1 Wait, does it mean if we don't merge this pr right now, CI will start fail for pytorch/xla?

rdspring1 · 2022-01-28T18:40:48Z

Well, the PR was just reverted. I need to rebase the latest changes. We've been working through some internal CI failures.

JackCaoG · 2022-01-28T18:53:39Z

@rdspring1 Please let me finish the review of this pr before merging the pytorch one. Ideally we should have both pr reviewed and merge at the same time to avoid the breakage.

Our head is broken due to the change

clang++-8 -MMD -MF /tmp/pytorch/xla/build/temp.linux-x86_64-3.7/torch_xla/csrc/aten_xla_type.o.d -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -I/tmp/pytorch/xla -I/tmp/pytorch/xla/third_party/tensorflow/bazel-tensorflow -I/tmp/pytorch/xla/third_party/tensorflow/bazel-bin -I/tmp/pytorch/xla/third_party/tensorflow/bazel-tensorflow/external/protobuf_archive/src -I/tmp/pytorch/xla/third_party/tensorflow/bazel-tensorflow/external/com_google_protobuf/src -I/tmp/pytorch/xla/third_party/tensorflow/bazel-tensorflow/external/eigen_archive -I/tmp/pytorch/xla/third_party/tensorflow/bazel-tensorflow/external/com_google_absl -I/tmp/pytorch -I/tmp/pytorch/torch/csrc -I/tmp/pytorch/torch/lib/tmp_install/include -I/opt/conda/envs/pytorch/lib/python3.7/site-packages/torch/include -I/opt/conda/envs/pytorch/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/pytorch/lib/python3.7/site-packages/torch/include/TH -I/opt/conda/envs/pytorch/lib/python3.7/site-packages/torch/include/THC -I/opt/conda/envs/pytorch/include/python3.7m -c -c /tmp/pytorch/xla/torch_xla/csrc/aten_xla_type.cpp -o /tmp/pytorch/xla/build/temp.linux-x86_64-3.7/torch_xla/csrc/aten_xla_type.o -std=c++14 -Wno-sign-compare -Wno-deprecated-declarations -Wno-return-type -Wno-macro-redefined -Wno-return-std-move -DNDEBUG -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_clang"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1002"' -DTORCH_EXTENSION_NAME=_XLAC -D_GLIBCXX_USE_CXX11_ABI=1
/tmp/pytorch/xla/torch_xla/csrc/aten_xla_type.cpp:1511:32: error: out-of-line definition of 'gelu' does not match any declaration in 'torch_xla::XLANativeFunctions'
at::Tensor XLANativeFunctions::gelu(const at::Tensor& self) {
                               ^~~~
/tmp/pytorch/xla/torch_xla/csrc/aten_xla_type.cpp:1516:32: error: out-of-line definition of 'gelu_backward' does not match any declaration in 'torch_xla::XLANativeFunctions'
at::Tensor XLANativeFunctions::gelu_backward(const at::Tensor& grad

and I am triggering a new one since it is reverted.

JackCaoG

Mostly LGTM, minor nits

JackCaoG · 2022-01-28T19:05:12Z

torch_xla/csrc/ops/ops.cpp

+  if (approximate == kTanh) {
+    // inner = math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(input, 3))
+    // input * 0.5 * (1.0 + torch.tanh(inner))
+    const float kBeta = M_SQRT2 * M_2_SQRTPI * 0.5;


maybe better to mark it as static?

JackCaoG · 2022-01-28T19:07:27Z

torch_xla/csrc/ops/ops.cpp

+  const int64_t kNone = 0;
+  const int64_t kTanh = 1;
+  if (approximate == kTanh) {
+    const float kBeta = M_SQRT2 * M_2_SQRTPI * 0.5;


JackCaoG · 2022-01-28T19:16:19Z

torch_xla/csrc/ops/ops.cpp

+
+    NodePtr tanh_derivative = one - tanh_inner * tanh_inner;
+    NodePtr inner_derivative = beta * (one + three * kappa * Pow(input, two));
+    NodePtr right_derivative = left * tanh_derivative * inner_derivative;


hmm, it really should be lower as a node class and lowering function if it gets this complicated. I will let this one go since upstream pr needs to merge and fix it latter.

Summary: 1. Implements pytorch/pytorch#39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: pytorch/pytorch#61439 Reviewed By: cpuhrsch Differential Revision: D33850228 Pulled By: jbschlosser fbshipit-source-id: 3cc33fb298e480d7ecc5c67716da019d60c6ab33

Summary: 1. Implements #39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: #61439 Reviewed By: cpuhrsch Differential Revision: D33850228 Pulled By: jbschlosser fbshipit-source-id: 3cc33fb298e480d7ecc5c67716da019d60c6ab33

Summary: 1. Implements #39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: #61439 Reviewed By: cpuhrsch Differential Revision: D33850228 Pulled By: jbschlosser fbshipit-source-id: 3cc33fb298e480d7ecc5c67716da019d60c6ab33 (cherry picked from commit 3a53b3e)

Summary: 1. Implements pytorch/pytorch#39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: pytorch/pytorch#61439 Reviewed By: mikaylagawarecki Differential Revision: D33744717 Pulled By: jbschlosser fbshipit-source-id: d64532a562ed53247bb4fa52bb16722634d5c187

Summary: 1. Implements pytorch/pytorch#39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: pytorch/pytorch#61439 Reviewed By: cpuhrsch Differential Revision: D33850228 Pulled By: jbschlosser fbshipit-source-id: 3cc33fb298e480d7ecc5c67716da019d60c6ab33

JackCaoG · 2022-02-02T23:00:42Z

@rdspring1 Will the upstream pytorch pr make it to the pytorch 1.11?

rdspring1 · 2022-02-03T03:35:12Z

No, I don't think so. There is some issue with the mobile builds.

Summary: 1. Implements pytorch/pytorch#39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: pytorch/pytorch#61439 Reviewed By: VitalyFedyunin Differential Revision: D33894937 Pulled By: jbschlosser fbshipit-source-id: b65e8fb6ea66168af8f34f45ed50e92737a33851

Summary: 1. Implements #39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: #61439 Reviewed By: VitalyFedyunin Differential Revision: D33894937 Pulled By: jbschlosser fbshipit-source-id: b65e8fb6ea66168af8f34f45ed50e92737a33851

Summary: 1. Implements #39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: #61439 Reviewed By: VitalyFedyunin Differential Revision: D33894937 Pulled By: jbschlosser fbshipit-source-id: b65e8fb6ea66168af8f34f45ed50e92737a33851 (cherry picked from commit 6e986f9)

Summary: 1. Implements pytorch#39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: pytorch#61439 Reviewed By: VitalyFedyunin Differential Revision: D33894937 Pulled By: jbschlosser fbshipit-source-id: b65e8fb6ea66168af8f34f45ed50e92737a33851 (cherry picked from commit 6e986f9)

Summary: 1. Implements pytorch/pytorch#39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: pytorch/pytorch#61439 Reviewed By: VitalyFedyunin Differential Revision: D33894937 Pulled By: jbschlosser fbshipit-source-id: b65e8fb6ea66168af8f34f45ed50e92737a33851 (cherry picked from commit 6e986f91a958dd73514b4e64984c0b149157dc6f)

Summary: 1. Implements pytorch#39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: pytorch#61439 Reviewed By: VitalyFedyunin Differential Revision: D33894937 Pulled By: jbschlosser fbshipit-source-id: b65e8fb6ea66168af8f34f45ed50e92737a33851

Summary: 1. Implements pytorch/pytorch#39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: pytorch/pytorch#61439 Reviewed By: mikaylagawarecki Differential Revision: D33744717 Pulled By: jbschlosser fbshipit-source-id: d64532a562ed53247bb4fa52bb16722634d5c187 (cherry picked from commit 4713dd9ccaa8983422bf3aa7b73df8d9ebd8cc02)

Summary: 1. Implements pytorch/pytorch#39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: pytorch/pytorch#61439 Reviewed By: cpuhrsch Differential Revision: D33850228 Pulled By: jbschlosser fbshipit-source-id: 3cc33fb298e480d7ecc5c67716da019d60c6ab33 (cherry picked from commit 3a53b3e94fd58190d1261efd3cf41b53506fb96e)

Summary: 1. Implements pytorch/pytorch#39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: pytorch/pytorch#61439 Reviewed By: VitalyFedyunin Differential Revision: D33894937 Pulled By: jbschlosser fbshipit-source-id: b65e8fb6ea66168af8f34f45ed50e92737a33851 (cherry picked from commit 6e986f91a958dd73514b4e64984c0b149157dc6f)

Summary: 1. Implements pytorch/pytorch#39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: pytorch/pytorch#61439 Reviewed By: mikaylagawarecki Differential Revision: D33744717 Pulled By: jbschlosser fbshipit-source-id: d64532a562ed53247bb4fa52bb16722634d5c187 (cherry picked from commit 4713dd9ccaa8983422bf3aa7b73df8d9ebd8cc02)

Summary: 1. Implements pytorch/pytorch#39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: pytorch/pytorch#61439 Reviewed By: cpuhrsch Differential Revision: D33850228 Pulled By: jbschlosser fbshipit-source-id: 3cc33fb298e480d7ecc5c67716da019d60c6ab33 (cherry picked from commit 3a53b3e94fd58190d1261efd3cf41b53506fb96e)

Summary: 1. Implements pytorch/pytorch#39853 2. Adds approximate boolean flag to Gelu 3. Enables Tanh Gelu approximation 4. Adds double backward support for Gelu 5. Enable Tanh Gelu in NvFuser ``` def gelu(x, approximate : str = 'none'): if approximate == 'tanh': # sqrt(2/pi) = 0.7978845608028654 return 0.5 * x * (1.0 + torch.tanh(0.7978845608028654 * (x + 0.044715 * torch.pow(x, 3.0)))) else: return x * normcdf(x) ``` Linking XLA PR - pytorch/xla#3039 Pull Request resolved: pytorch/pytorch#61439 Reviewed By: VitalyFedyunin Differential Revision: D33894937 Pulled By: jbschlosser fbshipit-source-id: b65e8fb6ea66168af8f34f45ed50e92737a33851 (cherry picked from commit 6e986f91a958dd73514b4e64984c0b149157dc6f)

This was referenced Jul 14, 2021

Implement Tanh Gelu Approximation pytorch/pytorch#61439

Closed

Implement Tanh Gelu csarofeen/pytorch#988

Merged

rdspring1 force-pushed the fast_gelu branch from dc661dc to 7536ca1 Compare October 6, 2021 05:54

rdspring1 changed the title ~~Implement Tanh Gelu Approximation~~ [PT_BREAK] Implement Tanh Gelu Approximation Oct 7, 2021

rdspring1 marked this pull request as ready for review October 8, 2021 00:40

rdspring1 force-pushed the fast_gelu branch from fc80324 to e716d65 Compare November 11, 2021 01:05

rdspring1 and others added 3 commits November 20, 2021 13:48

Implement Tanh Gelu Approximation

324708b

Rename x to input

b4ea547

Replace x*x with Pow(x,2)

b885b47

rdspring1 force-pushed the fast_gelu branch from e716d65 to 6aa2d40 Compare November 20, 2021 19:48

Implement "approximate" string argument for tanh gelu

405cb06

rdspring1 force-pushed the fast_gelu branch from 6aa2d40 to 405cb06 Compare November 20, 2021 21:51

JackCaoG reviewed Jan 28, 2022

View reviewed changes

JackCaoG merged commit 82a110f into pytorch:master Feb 14, 2022

jbschlosser mentioned this pull request Feb 14, 2022

Nvfuser code bump 2_1_2022 pytorch/pytorch#72127

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PT_BREAK] Implement Tanh Gelu Approximation #3039

[PT_BREAK] Implement Tanh Gelu Approximation #3039

rdspring1 commented Jul 14, 2021 •

edited

Loading

JackCaoG commented Jul 14, 2021

rdspring1 commented Jan 28, 2022

JackCaoG commented Jan 28, 2022

rdspring1 commented Jan 28, 2022 •

edited

Loading

JackCaoG commented Jan 28, 2022

JackCaoG left a comment

JackCaoG Jan 28, 2022

JackCaoG Jan 28, 2022

JackCaoG Jan 28, 2022

JackCaoG commented Feb 2, 2022

rdspring1 commented Feb 3, 2022 •

edited

Loading

[PT_BREAK] Implement Tanh Gelu Approximation #3039

[PT_BREAK] Implement Tanh Gelu Approximation #3039

Conversation

rdspring1 commented Jul 14, 2021 • edited Loading

JackCaoG commented Jul 14, 2021

rdspring1 commented Jan 28, 2022

JackCaoG commented Jan 28, 2022

rdspring1 commented Jan 28, 2022 • edited Loading

JackCaoG commented Jan 28, 2022

JackCaoG left a comment

Choose a reason for hiding this comment

JackCaoG Jan 28, 2022

Choose a reason for hiding this comment

JackCaoG Jan 28, 2022

Choose a reason for hiding this comment

JackCaoG Jan 28, 2022

Choose a reason for hiding this comment

JackCaoG commented Feb 2, 2022

rdspring1 commented Feb 3, 2022 • edited Loading

rdspring1 commented Jul 14, 2021 •

edited

Loading

rdspring1 commented Jan 28, 2022 •

edited

Loading

rdspring1 commented Feb 3, 2022 •

edited

Loading