http: add CONNECT-UDP support by jeongseokson · Pull Request #27714 · envoyproxy/envoy

jeongseokson · 2023-05-31T01:06:39Z

Commit Message:
This commit adds CONNECT-UDP (RFC 9298) support. UdpConnPool is added to create a UDP socket for a new CONNECT-UDP request, and UDPUpstream is added to maintain the socket and other relevant data associated with UDP upstreams.

We added an integration test for the terminating CONNECT-UDP proxy, but not the forwarding proxy in this commit. We are going to add test cases to cover the forwarding proxy scenario in a subsequent commit.

Additional Description:
Risk Level: Medium, the feature can only be enabled by the new configuration added in this commit.
Testing: Integration test
Runtime guard: envoy.reloadable_features.enable_connect_udp_support
Release Notes: added support for CONNECT-UDP (RFC 9298). Can be disabled by setting runtime feature envoy.reloadable_features.enable_connect_udp_support to false.

I also added a basic integration test of sending and receiving data between a client and UDP server through Envoy proxy using CONNECT-UDP. Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

repokitteh-read-only · 2023-05-31T01:06:51Z

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @lizan
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #27714 was opened by jeongseokson.

see: more, trace.

jeongseokson · 2023-05-31T01:10:25Z

/assign @DavidSchinazi

David will do a first pass review. Thank you!

jeongseokson · 2023-05-31T01:28:04Z

What's the right way to add api/envoy/extensions/upstreams/http/udp/v3/udp_connection_pool.proto? The check_and_fix_proto_format CI complains about it.

tyxia · 2023-06-01T04:07:13Z

What's the right way to add api/envoy/extensions/upstreams/http/udp/v3/udp_connection_pool.proto? The check_and_fix_proto_format CI complains about it.

Drive-by comment.... The proto BUILD file is generated by tool. Once you have the correct proto change (e.g., import, namespace, etc), you can run fix_proto_format. I used sudo ./ci/run_envoy_docker.sh '\''./ci/do_ci.sh fix_proto_format locally.

jeongseokson · 2023-06-01T17:09:56Z

What's the right way to add api/envoy/extensions/upstreams/http/udp/v3/udp_connection_pool.proto? The check_and_fix_proto_format CI complains about it.

Drive-by comment.... The proto BUILD file is generated by tool. Once you have the correct proto change (e.g., import, namespace, etc), you can run fix_proto_format. I used sudo ./ci/run_envoy_docker.sh '\''./ci/do_ci.sh fix_proto_format locally.

I tried it and it deleted both the proto and the BUILD file. Tried it with the proto file alone then it complained about missing the BUILD file. I guess I should have written the proto in a different way? I don't see any Envoy documentation about this.

DavidSchinazi

Thanks, this is overall in great shape!

DavidSchinazi · 2023-06-01T19:32:28Z

  virtual GenericConnPoolPtr
-  createGenericConnPool(Upstream::ThreadLocalCluster& thread_local_cluster, bool is_connect,
+  createGenericConnPool(Upstream::ThreadLocalCluster& thread_local_cluster,
+                        GenericConnPoolFactory::UpstreamProtocol is_connect,


should we rename is_connect to upstream_protocol?

RIght, done.

DavidSchinazi · 2023-06-01T19:36:49Z

+  // Rewrites the host of CONNECT-UDP requests.
+  if (HeaderUtility::isConnectUdp(*request_headers_) &&
+      !HeaderUtility::rewriteAuthorityForConnectUdp(*request_headers_)) {
+    sendLocalReply(Code::BadRequest, "Invalid URI Template in the path", nullptr, absl::nullopt,


I think this makes more sense to be Code::NotFound similar to CONNECT above – conceptually, the path the client used isn't invalid, we just don't expect CONNECT-UDP there.

Makes sense. Done.

DavidSchinazi · 2023-06-01T20:09:57Z


+namespace {
+
+const Regex::CompiledGoogleReMatcher& connectUdpPathMatcher() {


A regex seems pretty heavyweight for something this simple. I'd assume we'd get noticeably better performance by checking whether the path is prefixed by /.well-known/masque/udp/, removing the prefix and then splitting on /.

That said, this could be a premature optimization. Is there precedent for regex vs split in the codebase currently?

Regex is used in Envoy mainly for rewriting headers based on the user configuration. I agree that in this case, regex could be an overkill. I think this could make more sense in a later stage if we want to support more custom URI Templates. I will remove regex as you suggested.

DavidSchinazi · 2023-06-01T20:19:54Z

-    // TODO(https://github.com/envoyproxy/envoy/issues/23564): Http3 Datagram support should be
-    // turned on by returning quic::HttpDatagramSupport::kRfc once the CONNECT-UDP support work is
-    // completed.
+#ifdef ENVOY_ENABLE_HTTP_DATAGRAMS


Should this be a runtime guard instead of compile time? Ideally the presence of CONNECT-UDP in the config should enable this. Same comment for all other uses of this macro in this PR

So this macro is for excluding HTTP Capsule support from the Envoy Mobile builds. I think the right way would be adding the runtime guard as well as this macro guard. I will add the runtime guard accordingly.

Added the runtime guard.

What's the purpose of removing this from Envoy Mobile at compile time? They're tight on binary size but this is something that will be needed there eventually and a macro adds complexity

It was added in the previous PR as Alyssa and Ryan wanted to have an option to include or exclude Http Datagram support in the Envoy Mobile build. Here's the relevant discussion in the previous PR: #25925 (comment)

If we all agree that it's no longer necessary, I would like to remove it likewise. Perhaps it would be a separate PR to make this one concise.

DavidSchinazi · 2023-06-01T20:35:38Z

+bool UdpUpstream::OnCapsule(const quiche::Capsule& capsule) {
+  quiche::CapsuleType capsule_type = capsule.capsule_type();
+  if (capsule_type != quiche::CapsuleType::DATAGRAM) {
+    // Silently drops Datagram Capsules with an unknown type.


nit: these aren't datagram capsules

Suggested change

// Silently drops Datagram Capsules with an unknown type.

// Silently drops capsules with an unknown type.

Good point. Thanks.

DavidSchinazi · 2023-06-01T20:38:21Z

+
+  Network::SocketPtr createSocket(const Upstream::HostConstSharedPtr& host) {
+    return std::make_unique<Network::SocketImpl>(Network::Socket::Type::Datagram, host->address(),
+                                                 nullptr, Network::SocketCreationOptions{});


nit: /*variable_name=*/nullptr

DavidSchinazi · 2023-06-01T20:43:13Z

+  std::string received_capsule_fragment =
+      absl::HexStringToBytes("00"               // DATAGRAM capsule type
+                             "08"               // capsule length
+                             "a1a2a3a4a5a6a7a8" // HTTP Datagram payload


Why doesn't this have the 00 byte of the contextID=0?

I think I misunderstood the specification. Fixed the error in the test cases as well as the UdpUpstream.

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

jeongseokson

Thanks a lot for reviewing the PR, David. I will send another commit to complete the fix.

jeongseokson · 2023-06-02T17:09:35Z

+  // Rewrites the host of CONNECT-UDP requests.
+  if (HeaderUtility::isConnectUdp(*request_headers_) &&
+      !HeaderUtility::rewriteAuthorityForConnectUdp(*request_headers_)) {
+    sendLocalReply(Code::BadRequest, "Invalid URI Template in the path", nullptr, absl::nullopt,


Makes sense. Done.

jeongseokson · 2023-06-02T19:50:20Z

-    // TODO(https://github.com/envoyproxy/envoy/issues/23564): Http3 Datagram support should be
-    // turned on by returning quic::HttpDatagramSupport::kRfc once the CONNECT-UDP support work is
-    // completed.
+#ifdef ENVOY_ENABLE_HTTP_DATAGRAMS


So this macro is for excluding HTTP Capsule support from the Envoy Mobile builds. I think the right way would be adding the runtime guard as well as this macro guard. I will add the runtime guard accordingly.

jeongseokson · 2023-06-02T20:08:12Z

+      Event::FileReadyType::Read);
+}
+
+void UdpUpstream::encodeData(Buffer::Instance& data, bool /*end_stream*/) {


Your assumption is correct. Thanks for catching this. I thought Capsules could not span across multiple DATA frames. Done.

jeongseokson · 2023-06-02T20:29:50Z

+}
+
+Envoy::Http::Status UdpUpstream::encodeHeaders(const Envoy::Http::RequestHeaderMap&,
+                                               bool /*end_stream*/) {


If end_stream is set, it means the request is header only. The existing CONNECT (TCP) implementation in Envoy uses it only for supporting PROXY protocol, which I assume not relevant to CONNECT-UDP, so I didn't find the use of the end_stream in this method. Let me know if you think otherwise.

jeongseokson · 2023-06-02T20:48:02Z

+
+  Network::SocketPtr createSocket(const Upstream::HostConstSharedPtr& host) {
+    return std::make_unique<Network::SocketImpl>(Network::Socket::Type::Datagram, host->address(),
+                                                 nullptr, Network::SocketCreationOptions{});


jeongseokson · 2023-06-02T20:59:44Z

+  std::string received_capsule_fragment =
+      absl::HexStringToBytes("00"               // DATAGRAM capsule type
+                             "08"               // capsule length
+                             "a1a2a3a4a5a6a7a8" // HTTP Datagram payload


I think I misunderstood the specification. Fixed the error in the test cases as well as the UdpUpstream.

jeongseokson · 2023-06-02T21:11:22Z


+namespace {
+
+const Regex::CompiledGoogleReMatcher& connectUdpPathMatcher() {


Regex is used in Envoy mainly for rewriting headers based on the user configuration. I agree that in this case, regex could be an overkill. I think this could make more sense in a later stage if we want to support more custom URI Templates. I will remove regex as you suggested.

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

repokitteh-read-only · 2023-06-07T21:06:04Z

CC @envoyproxy/runtime-guard-changes: FYI only for changes made to (source/common/runtime/runtime_features.cc).

🐱

Caused by: #27714 was synchronize by jeongseokson.

see: more, trace.

jeongseokson

Resolved the rest of the issues such as IPv6 parsing. Can you please review my new changes, David? Thank you.

jeongseokson · 2023-06-07T21:07:06Z

-    // TODO(https://github.com/envoyproxy/envoy/issues/23564): Http3 Datagram support should be
-    // turned on by returning quic::HttpDatagramSupport::kRfc once the CONNECT-UDP support work is
-    // completed.
+#ifdef ENVOY_ENABLE_HTTP_DATAGRAMS


Added the runtime guard.

DavidSchinazi

Overall this looks great! Please send to wider review after addressing these last comments

DavidSchinazi · 2023-06-07T21:55:12Z

+  ASSERT(!ip_address.empty());
+  // Removes the Zone ID in the IPv6 address.
+  std::vector<absl::string_view> ip_address_split = absl::StrSplit(ip_address, '%');
+  if (ip_address_split.empty()) {


Should this be an assert? The array should have at least one element when the string isn't empty

Ah, yes this is not necessary because ip_address_split will be empty anyway if the ip_address is empty and the function will return false. Done.

DavidSchinazi · 2023-06-07T21:57:21Z

+  if (ip_address_split.empty()) {
+    return false;
+  }
+  sockaddr_in6 addr_in;


nit: seems like you don't need a full sockaddr_in6 here, just an in6_addr

DavidSchinazi · 2023-06-07T22:02:16Z

-    // TODO(https://github.com/envoyproxy/envoy/issues/23564): Http3 Datagram support should be
-    // turned on by returning quic::HttpDatagramSupport::kRfc once the CONNECT-UDP support work is
-    // completed.
+#ifdef ENVOY_ENABLE_HTTP_DATAGRAMS


What's the purpose of removing this from Envoy Mobile at compile time? They're tight on binary size but this is something that will be needed there eventually and a macro adds complexity

DavidSchinazi · 2023-06-07T22:13:55Z

+}
+
+Envoy::Http::Status UdpUpstream::encodeHeaders(const Envoy::Http::RequestHeaderMap&,
+                                               bool /*end_stream*/) {


Per the spec The lifetime of the socket is tied to the request stream so if we get a FIN on the stream then we should tear everything down. It would be silly for the client to FIN the stream right after the headers but if it happens we should tear everything down and not bother opening the socket at all. Maybe reply with status code 400?

DavidSchinazi · 2023-06-07T22:21:23Z

+TEST(HeaderIsValidTest, IsConnectUdp) {
+  EXPECT_TRUE(
+      HeaderUtility::isConnectUdp(Http::TestRequestHeaderMapImpl{{"upgrade", "connect-udp"}}));
+  // Extended CONNECT requests should be normalied to HTTP/1.1.


Suggested change

// Extended CONNECT requests should be normalied to HTTP/1.1.

// Extended CONNECT requests should be normalized to HTTP/1.1.

DavidSchinazi · 2023-06-07T22:23:29Z

+  EXPECT_TRUE(HeaderUtility::rewriteAuthorityForConnectUdp(connect_udp_request));
+  EXPECT_EQ(connect_udp_request.getHostValue(), "foo.lyft.com:80");
+  TestRequestHeaderMapImpl connect_udp_request_ipv6{
+      {":path", "/.well-known/masque/udp/2001:0db8:85a3::8a2e:0370:7334/80/"},


Per the spec if "target_host" contains an IPv6 literal, the colons (":") MUST be percent-encoded. For example, if the target host is "2001:db8::42", it will be encoded in the URI as "2001%3Adb8%3A%3A42".

DavidSchinazi · 2023-06-07T22:27:16Z

+    const std::string sent_capsule_fragment =
+        absl::HexStringToBytes("00"               // DATAGRAM capsule type
+                               "08"               // capsule length
+                               "00a1a2a3a4a5a6a7" // UDP Proxying HTTP Datagram payload


Suggested change

"00a1a2a3a4a5a6a7" // UDP Proxying HTTP Datagram payload

"00" // context ID

"a1a2a3a4a5a6a7" // UDP Proxying Payload

Same below

Neat! Done.

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

jeongseokson · 2023-06-17T05:10:11Z

/retest

RyanTheOptimist · 2023-06-17T18:09:44Z

Looks like some tidy failures here:

https://dev.azure.com/cncf/envoy/_build/results?buildId=140504&view=logs&jobId=b7634614-24f3-5416-e791-4f3affaaed6c&j=b7634614-24f3-5416-e791-4f3affaaed6c&t=d565ee52-b888-5d80-7fea-dfaed104c390

Also coverage:

Code coverage for source/extensions/upstreams is lower than limit of 96.6 (93.8)
Code coverage for source/extensions/upstreams/http is lower than limit of 96.6 (93.1)
Code coverage for source/extensions/upstreams/http/generic is lower than limit of 96.6 (84.6)
Code coverage for source/extensions/upstreams/http/udp is lower than limit of 96.6 (78.0)

Detailed coverage results here:
https://storage.googleapis.com/envoy-pr/e9f2ada/coverage/index.html

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

RyanTheOptimist · 2023-06-20T17:52:41Z

/wait
on CI

I also added the extensions/upstreams/http/generic to the exception list for the coverage test due to the coverage bug of excluding brackets of case statements. Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

repokitteh-read-only · 2023-06-22T00:25:17Z

CC @envoyproxy/coverage-shephards: FYI only for changes made to (test/per_file_coverage.sh).
envoyproxy/coverage-shephards assignee is @RyanTheOptimist

🐱

Caused by: #27714 was synchronize by jeongseokson.

see: more, trace.

jeongseokson · 2023-06-22T00:28:43Z

Hi @RyanTheOptimist

I added more test cases to increase the test coverage.

Because of the coverage bug of excluding nested brackets of case statements, it's not possible to reach the coverage threshold in extensions/upstreams/http/generic/config.cc (https://storage.googleapis.com/envoy-pr/e9f2ada/coverage/source/extensions/upstreams/http/generic/config.cc.gcov.html). I added an additional test case to cover "return nullptr;" but still the maximum coverage I can achieve is 88.5 because of the bug of the coverage CI.

I added it to the exception list to make it pass the coverage test. I set the threshold as 85.0 to leave some headroom. Let me know if you have any comments or questions. Thank you!

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

jeongseokson · 2023-06-22T03:54:17Z

/retest

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

Commit Message: This commit adds CONNECT-UDP (RFC 9298) support. UdpConnPool is added to create a UDP socket for a new CONNECT-UDP request, and UDPUpstream is added to maintain the socket and other relevant data associated with UDP upstreams. We added an integration test for the terminating CONNECT-UDP proxy, but not the forwarding proxy in this commit. We are going to add test cases to cover the forwarding proxy scenario in a subsequent commit. Additional Description: Risk Level: Medium, the feature can only be enabled by the new configuration added in this commit. Testing: Integration test Runtime guard: envoy.reloadable_features.enable_connect_udp_support Release Notes: added support for CONNECT-UDP (RFC 9298). Can be disabled by setting runtime feature envoy.reloadable_features.enable_connect_udp_support to false. Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com> Co-authored-by: asingh-g <abhisinghx@google.com> Signed-off-by: asheryer <asheryer@amazon.com>

Commit Message: This commit adds CONNECT-UDP (RFC 9298) support. UdpConnPool is added to create a UDP socket for a new CONNECT-UDP request, and UDPUpstream is added to maintain the socket and other relevant data associated with UDP upstreams. We added an integration test for the terminating CONNECT-UDP proxy, but not the forwarding proxy in this commit. We are going to add test cases to cover the forwarding proxy scenario in a subsequent commit. Additional Description: Risk Level: Medium, the feature can only be enabled by the new configuration added in this commit. Testing: Integration test Runtime guard: envoy.reloadable_features.enable_connect_udp_support Release Notes: added support for CONNECT-UDP (RFC 9298). Can be disabled by setting runtime feature envoy.reloadable_features.enable_connect_udp_support to false. Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com> Co-authored-by: asingh-g <abhisinghx@google.com> Signed-off-by: Ryan Eskin <ryan.eskin89@protonmail.com>

jeongseokson and others added 10 commits May 31, 2023 00:36

Add CONNECT-UDP support.

1befcd4

I also added a basic integration test of sending and receiving data between a client and UDP server through Envoy proxy using CONNECT-UDP. Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

"Fix typo that breaks build"

2085b5f

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

Add Capsule support.

7b90eee

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

Revmoe state printing.

ccf6cd5

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

Add potential integration test cases.

0bf459b

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

Add URI Template path and QUIC Datagram support.

f65c8ef

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

Add timeout unit tests for udp tunneling test

484395c

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

Fix stream idle timeout test

d413a4b

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

Cleanup code and add some comments.

61f513d

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

Fix format.

c33f563

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

repokitteh-read-only Bot added the api label May 31, 2023

repokitteh-read-only Bot assigned lizan May 31, 2023

repokitteh-read-only Bot assigned DavidSchinazi May 31, 2023

silverstar194 reviewed May 31, 2023

View reviewed changes

Comment thread source/common/http/conn_manager_impl.cc

DavidSchinazi reviewed Jun 1, 2023

View reviewed changes

Fix some of the errors pointed by David.

55b91ab

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

jeongseokson commented Jun 2, 2023

View reviewed changes

ravenblackx added the waiting label Jun 6, 2023

jeongseokson added 2 commits June 7, 2023 20:59

Resolve David's comments and add the runtime flag.

1dc111b

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

Format fix.

84ba656

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

repokitteh-read-only Bot removed the waiting label Jun 7, 2023

jeongseokson commented Jun 7, 2023

View reviewed changes

DavidSchinazi reviewed Jun 7, 2023

View reviewed changes

Resolved further comments from David.

4c4f440

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

jeongseokson added 2 commits June 16, 2023 22:37

Merge branch 'main' into connect_udp

aff5dda

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

Fix a format error.

d233e0a

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

RyanTheOptimist previously approved these changes Jun 16, 2023

View reviewed changes

Fix TSAN error.

3daf78d

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

jeongseokson dismissed RyanTheOptimist’s stale review via 3daf78d June 17, 2023 03:57

Fix a typo.

3b67d84

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

jeongseokson added 2 commits June 20, 2023 05:16

Add unit tests for UdpUpstream and fix clang_tidy.

b24d5e9

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

Fix ASAN error.

fa691d3

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

repokitteh-read-only Bot added the waiting label Jun 20, 2023

Add more tests.

f87d2fc

I also added the extensions/upstreams/http/generic to the exception list for the coverage test due to the coverage bug of excluding brackets of case statements. Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

repokitteh-read-only Bot removed the waiting label Jun 22, 2023

jeongseokson added 2 commits June 22, 2023 01:18

Merge branch 'main' into connect_udp

82cd2bf

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

Fix the interface mismatch.

29edf3f

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

Fix clang tidy.

86756a1

Signed-off-by: Jeongseok Son <jeongseok.son@gmail.com>

RyanTheOptimist approved these changes Jun 22, 2023

View reviewed changes

RyanTheOptimist merged commit b4f3755 into envoyproxy:main Jun 22, 2023

jeongseokson mentioned this pull request Jun 26, 2023

connect-udp support (RFC 9298) #23564

Closed

emaildanwilson mentioned this pull request Aug 8, 2023

UDP support istio/istio#1430

Open

DavidSchinazi mentioned this pull request Sep 12, 2023

quic: fix build with option --//bazel:http3=False #29561

Merged

alyssawilk mentioned this pull request Jan 23, 2024

envoy_reloadable_features_enable_connect_udp_support deprecation #31975

Closed

asedeno mentioned this pull request Feb 12, 2025

CONNECT-UDP: set remote_address in udp upstream socket #38427

Merged


		namespace {

		const Regex::CompiledGoogleReMatcher& connectUdpPathMatcher() {

	// Silently drops Datagram Capsules with an unknown type.
	// Silently drops capsules with an unknown type.

	// Extended CONNECT requests should be normalied to HTTP/1.1.
	// Extended CONNECT requests should be normalized to HTTP/1.1.

	"00a1a2a3a4a5a6a7" // UDP Proxying HTTP Datagram payload
	"00" // context ID
	"a1a2a3a4a5a6a7" // UDP Proxying Payload

Conversation

jeongseokson commented May 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

repokitteh-read-only Bot commented May 31, 2023

Uh oh!

jeongseokson commented May 31, 2023

Uh oh!

jeongseokson commented May 31, 2023

Uh oh!

Uh oh!

tyxia commented Jun 1, 2023

Uh oh!

jeongseokson commented Jun 1, 2023

Uh oh!

DavidSchinazi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeongseokson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

repokitteh-read-only Bot commented Jun 7, 2023

Uh oh!

jeongseokson left a comment

Choose a reason for hiding this comment

Uh oh!

jeongseokson commented May 31, 2023 •

edited

Loading