Storage: Add Oracle Cloud Infrastructure Object Storage Bucket support #4661

aarontams · 2021-09-16T06:04:09Z

I added CHANGELOG entry for this change.
Change is not relevant to the end user.

Changes

Add Oracle Cloud Infrastructure (OCI) Object Storage Bucket support

OCI Object Storage supports three different providers, default, instance principle, and raw. A lot of OCI users like to use Instance principle to security. It is very important to support instance principle for the OCI users. S3 doesn't support OCI instance principle.

Support multiple prometheus instances with a single bucket

By default, like other clients, this new Thanos OCI Object Storage client will storage the data from a single prometheus instance to the top directory of the bucket. This means each bucket can only store one prometheus instance data.

In OCI, a lot of users like to put different type of data in a single bucket.
To support multiple prometheus instances with a single bucket, a new object_base_path option is introduced in this OCI Object Storage client. If defined, the value will be used as the prefix of the prometheus data in OCI bucket.

For example, if you want to use the same bucket for 2 prometheus instances, you can define object_base_path as /path/for/prometheus1 for the first prometheus instance, and /path/for/prometheus2 for the second prometheus instance.

Three tests are included here to demo how the object_base_path work.

Verification

Running similar OCI client against 3 prometheus instances for over 50 days without any issue.
For each prometheus instance

average 4.5GB (2GB in chunks and 2.5G index) per 2-hour prometheus block
over 50GB per day
over 2TB prometheus data uploaded to OCI Object Storage

Testing

export THANOS_TEST_OBJSTORE_SKIP=GCS,S3,AZURE,SWIFT,COS,ALIYUNOSS,BOS
go test -v -run TestObjStore_AcceptanceTest_e2e ./pkg/...

See the test steps in storage.md under OCI section.

Ran three tests with different OCI_OBJECT_BASE_PATH settings. Below are the screenshots from Oracle Cloud Infrastructure console to show the object paths when the tests were running.

Test 1 - OCI_OBJECT_BASE_PATH is not set

Test 2 - OCI_OBJECT_BASE_PATH=/my/base/path

Test 3 - OCI_OBJECT_BASE_PATH=test3/base-path

pkg/objstore/oci/helper.go

pkg/objstore/oci/oci.go

aarontams · 2021-09-16T12:45:35Z

Here is docs / Documentation check error:

error: 		docs/storage.md:423: "https://cloud.baidu.com/doc/BOS/index.html" not accessible; status code 0: Get "https://cloud.baidu.com/doc/BOS/index.html": context deadline exceeded

The https://cloud.baidu.com/doc/BOS/index.html link is not belong to my MR. The error is from the original codes.

aarontams · 2021-09-16T12:53:13Z

For ci/circleci testing
To test OCI, you will need to provide real OCI Objectstorage bucket, otherwise the test will fail.

aarontams · 2021-09-16T12:56:07Z

For go / Linters (Static Analysis) for Go

cannot 'detected not clean work tree before running lint: you have unstaged changes.
M	pkg/objstore/oci/helper.go
M	pkg/objstore/oci/oci.go
Please commit or stash them.

I assume this is normal for a PR. Please let me know if this is not the case here. Thx!

GiedriusS · 2021-09-16T13:05:52Z

Please add the DCO (:

aarontams · 2021-09-23T05:10:33Z

Please let me know if there is any additional information that I need to provide to help the review process. Thx!

wiardvanrij · 2021-10-04T11:43:39Z

Hi, looks great and I think it makes sense to provide wider support for different clouds.

By default, like other clients, this new Thanos OCI Object Storage client will storage the data from a single prometheus instance to the top directory of the bucket. This means each bucket can only store one prometheus instance data.

Is it though? I'm quite confident that a specific ULID is created and it should not be a problem to use the same object store for n-amount of instances. If one would like to separate the data, one could use a different object store instance.

I'm not really a fan of the object_base_path. It's prone for (user) error; what if someone does NOT change the value of it, what happens then?

Perhaps you could elaborate a bit more on it for me, it might be that I'm missing the actual point of course :)

aarontams · 2021-10-05T18:58:58Z

Hi, looks great and I think it makes sense to provide wider support for different clouds.

By default, like other clients, this new Thanos OCI Object Storage client will storage the data from a single prometheus instance to the top directory of the bucket. This means each bucket can only store one prometheus instance data.

Is it though? I'm quite confident that a specific ULID is created and it should not be a problem to use the same object store for n-amount of instances. If one would like to separate the data, one could use a different object store instance.

I'm not really a fan of the object_base_path. It's prone for (user) error; what if someone does NOT change the value of it, what happens then?

Perhaps you could elaborate a bit more on it for me, it might be that I'm missing the actual point of course :)

To answer your initial question.
object_base_path is an optional field. If it is not provided, this new client will act the same as all other existing storage clients. So, if you don't want it, you can completely ignore it in your store setup.

Here is use case for our team:
In our production env, we have over 40 K8s clusters. Some of the K8s cluster have over 100 prometheus instances. For each K8s cluster, we create one bucket in OCI. With the object_base_path, we can potentially capture data from all 100 prometheus instances in a single bucket.

We use OCI instance principle to grant bucket access to the K8s cluster. We also create/delete prometheus instances dynamically (on demand) all the time. With the object_base_path, we can save some time/work to set up or delete OCI buckets when the number of prometheus instances changed.

In additional, with the object_base_path, we can easily find and delete all the data for a specific prometheus. For example:

prometheus_001
prometheus_002
....
prometheus_123

I am open for suggestion for my MR. If you really think object_base_path shouldn't be part of the offer even it is just an optional field, I will consider to remove it (or hide it from the configuration) so I can have the basic implementation live in the Thanos github repo. In this case, I just need to add the little enhancement each time I upgrade Thanos in our production env. Please let me know what you think.

I have the similar code running in few of my prometheus instances for over couple months with hiccup 🤞

Again, thanks for reviewing my MR.

wiardvanrij · 2021-10-06T11:07:35Z

If you really think object_base_path shouldn't be part of the offer even it is just an optional field, I will consider to remove it

To be clear, you have equal say in how we should do things! :) I think your use-case makes sense and perhaps it should even be a thing to implement that path for each backend we have. Let's wait for some more reviews. I'll reminder myself of this PR and will ping others within 2 weeks if there is no other review to get this moving :)

aarontams · 2021-10-06T17:43:36Z

Thanks. Looking forward to seeing more reviewers.

If it helps, I am happy to do a quick demo during the Thursday US community meeting in couple weeks so reviewers/people can better understand the motivation of this MR and how things work in our team and OCI. Just let me know here and feel free to slack me in #thanos-dev.

aarontams · 2021-10-28T23:26:52Z

Hi @wiardvanrij, @GiedriusS, @bwplotka
I just rebase my PR to master and all tests passed without changing the code.
Please let me know if a short demo during Community meeting will help.

aarontams · 2021-11-19T03:40:34Z

Hi @wiardvanrij @GiedriusS @bwplotka
I tried to attend the Thanos community meeting yesterday to get some help for this PR, but the meeting got cancelled. Please let me know if there is anything that I can do to move this PR forward.
Thanks in advance.

docs/storage.md

GiedriusS · 2021-11-19T06:52:21Z

docs/storage.md

+The default HTTP Client comes with this Thanos OCI Object Storage should be able to handle most of the use cases.
+
+By default, like other clients, Thanos OCI Object Storage client will storage the data from a single prometheus instance to the top directory of the bucket. 
+This means each bucket can only store one prometheus instance data.


What does this mean? Why only one? 🤔 Block UUIDs are quite random and it is very unlikely that there would be a clash.

Sorry for the poor description.
When thanos uploads all the blocks from a prometheus instance, it will create a directory structure similar to the following in OCI objectstore

|-- 01FF8QX35297NG9HCDHGH1BN1 | |-- chunks | |-- 000001 | |-- 000002 | |-- .... |-- 01FFPFKMAQQNF4N945X7JWPQ8 | |-- chunks |. |-- .... |-- .... other prometheus blocks | |-- debug |-- meta |-- .....

I consider above as one prometheus instance data.

In this PR, a new option called object_base_path is added.
If defined, the whole directory structure above will be uploaded under the object_base_path prefix.

In second screenshot in this PR, it holds 4 prometheus instances.
Prometheus A - object_base_path is set to thanos.dev.thanos
Prometheus B - object_base_path is set to thanos.meta.sa/0
Prometheus C - object_base_path is set to thanos.meta.sa/1
Prometheus D - object_base_path is set to thanos.meta.sa/2

Here is the illustration in OCI objectstorage bucket.

|-- thanos.dev.thanos |-- <<Promethus-A with all blocks and debug meta>> |-- thanos.meta.sa |-- 0 <<Promethus-B with all blocks and debug meta>> |-- 1 <<Promethus-C with all blocks and debug meta>> |-- 2 <<Promethus-D with all blocks and debug meta>> |-- 01FF8QX35297NG9HCDHGH1BN1 | |-- chunks | |-- 000001 | |-- 000002 | |-- .... |-- 01FFPFKMAQQNF4N945X7JWPQ8 | |-- chunks |. |-- .... |-- .... other prometheus blocks | |-- debug |-- meta |-- .....

If you want to provide a better description for the Thanos community, I will be happy to change it in my PR.

Yes, that's what I mean - this #1318 has been a long-standing issue. I think this kind of functionality needs to be added "one level above" i.e. not at the raw object storage calls level. Otherwise, it would be weird to have duplicate functionality. I think others would agree as well. My suggestion would be to remove this prefix functionality and move it one level above so that it could be used with any kind of object storage. It could be done in a separate pull request. We could double check in #thanos-dev, though

Got it. Yes, adding the feature to one level up is a great idea for all Objectstorage clients.

Since allowing the same bucket to hold multiple prometheus instances' data is the main use case of our team and we would like to use Thanos master branch instead of my forked branch.

I can think of three different approaches (order by my preferences) to avoid duplicate functionality in my PR:
Approach # 1

hide the object_base_path configuration from storage.md documentation

without the object_base_path setting, my PR will act exactly the same as other ObjecctStorage Clients

the hidden feature can still be used by the insider who know the existence of this feature

after the code check it, whoever want to work on *: Support multi-tenancy on object storage using custom prefix #1318 will have an example how OCI client is implemented.

Approach # 2

hide the object_base_path configuration from storage.md documentation

hard code the object_base_path to empty string and won't accept any hidden customization

Approach # 3
Remove all the object_base_path throughout the PR.

Please let me know which approach is acceptable for you.
Thanks!

Sorry for the delay again, bit busy times.
As I like the different approaches I'm not that fan of hidden features in a way that one needs to know the existence in an open source project. Also I took some time to read the long standing issue and the consensus after the discussion was this:

Looks like the decision is that we are ok for simple custom prefix on config level (for all objstorages) in YAML (:

So, I would say that if we implement it, it should be "that". Still open for discussion though, but then it should require a lot more discussion from everyone else again IMO ^^

I would say, let's just implement this on level higher as discussed in that main issue. For all the storage backends, implement at least the config in a way that it can say "This is not supported yet for this object store" if one would configure it. With then the exception for OCI, in which you did implement it.

This gives a bit more motivation to get that issue fixed, and we can have iterations per objectstorage to actually implement it. I think by saying "It needs to be done for them all in one go", is really limiting us to get it moving forward.

What do you think? Also, I'm actually happy to help on that. I got some spare time soon.

I really think we should do (3) approach here. Then, you could implement it in a follow up PR. We have interfaces for object storage so IMHO it should be fairly easy to implement prefix functionality.

I will remove the prefix option when I have a chance.

Don't forget to remove this sentence as it is not true in practice, it is very hard to get a UUID collision.

docs/storage.md

pkg/objstore/oci/helper.go

pkg/objstore/oci/oci.go

aarontams · 2021-11-20T02:48:56Z

Thanks @GiedriusS for the review.
I will update my MR and provide more info for your comments when I have a chance.

aarontams · 2022-06-29T15:01:52Z

Hi @wiardvanrij
I rebased to master and fixed the link. Thanks for looking into it.

Djelibeybi

Because the acronym OCI is overloaded, it would be better to expand out all the visible uses of OCI to Oracle Cloud Infrastructure to avoid any potential confusion. I've also had a go at tidying up the docs and error messages, but please test this before updating the PR! :)

docs/storage.md

pkg/objstore/oci/oci.go

aarontams · 2022-06-30T22:10:00Z

@wiardvanrij
FYI
The doc check has 3 errors which are unrelated to my PR.

error: 3 errors:
[125](https://github.com/thanos-io/thanos/runs/7124986832?check_suite_focus=true#step:5:126)
		docs/quick-tutorial.md:3: "https://katacoda.com/thanos/courses/thanos" not accessible; status code 404: Not Found
[126](https://github.com/thanos-io/thanos/runs/7124986832?check_suite_focus=true#step:5:127)
		docs/getting-started.md:82: "https://katacoda.com/thanos/courses/thanos/1-globalview" not accessible; status code 404: Not Found
[127](https://github.com/thanos-io/thanos/runs/7124986832?check_suite_focus=true#step:5:128)
		CONTRIBUTING.md:77: "https://katacoda.com/thanos" not accessible; status code 404: Not Found
[128](https://github.com/thanos-io/thanos/runs/7124986832?check_suite_focus=true#step:5:129)

wiardvanrij · 2022-07-01T13:21:33Z

@wiardvanrij FYI The doc check has 3 errors which are unrelated to my PR.

error: 3 errors:
[125](https://github.com/thanos-io/thanos/runs/7124986832?check_suite_focus=true#step:5:126)
		docs/quick-tutorial.md:3: "https://katacoda.com/thanos/courses/thanos" not accessible; status code 404: Not Found
[126](https://github.com/thanos-io/thanos/runs/7124986832?check_suite_focus=true#step:5:127)
		docs/getting-started.md:82: "https://katacoda.com/thanos/courses/thanos/1-globalview" not accessible; status code 404: Not Found
[127](https://github.com/thanos-io/thanos/runs/7124986832?check_suite_focus=true#step:5:128)
		CONTRIBUTING.md:77: "https://katacoda.com/thanos" not accessible; status code 404: Not Found
[128](https://github.com/thanos-io/thanos/runs/7124986832?check_suite_focus=true#step:5:129)

Yea, katacoda closed it's doors (: - We have fixed this on main now (#5454) , please have an other rebase (sorry about that :D )

wiardvanrij · 2022-07-01T13:24:51Z

Also feel free to create a changelog entry already ^^

aarontams · 2022-07-01T23:11:14Z

Hi @wiardvanrij
Thanks again for fixing the doc in master.

Rebased. Hopefully, that will resolve the last failed doc check.

I also checked the I added CHANGELOG entry for this change. box.

aarontams · 2022-07-02T13:06:21Z

FYI
The latest doc check error is not related to my PR.

error: 		docs/getting-started.md:114: "https://build.thebeat.co/monitoring-the-beat-microservices-a-tale-of-evolution-4e246882606e" not accessible even after retry; status code 0: Get "https://build.thebeat.co/monitoring-the-beat-microservices-a-tale-of-evolution-4e246882606e?gi=242b3805957d": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
[125](https://github.com/thanos-io/thanos/runs/7160297573?check_suite_focus=true#step:5:126)
make: *** [Makefile:243: check-docs] Error 1

GiedriusS · 2022-07-11T10:40:49Z

pkg/objstore/oci/helper.go

+	"github.com/thanos-io/thanos/pkg/objstore"
+)
+
+func DefaultTransport(config Config) *http.Transport {


There's a common function for this now - could you please remove this and use that function?

Hi @GiedriusS,
Sorry for the late response, I am on vacation this week.

This DefaultTransport func allow user to define some customize setting here

type: OCI config: provider: "default" bucket: "" compartment_ocid: "" part_size: "" // Optional part size to override the OCI default of 128 MiB, value is in bytes. max_request_retries: "" // Optional maximum number of retries for a request. request_retry_interval: "" // Optional sleep duration in seconds between retry requests. http_config: idle_conn_timeout: 1m30s // Optional maximum amount of time an idle (keep-alive) connection will remain idle before closing itself. Zero means no limit. response_header_timeout: 2m // Optional amount of time to wait for a server's response headers after fully writing the request. tls_handshake_timeout: 10s // Optional maximum amount of time waiting to wait for a TLS handshake. Zero means no timeout. expect_continue_timeout: 1s // Optional amount of time to wait for a server's first response headers. Zero means no timeout and causes the body to be sent immediately. insecure_skip_verify: false // Optional. If true, crypto/tls accepts any certificate presented by the server and any host name in that certificate. max_idle_conns: 100 // Optional maximum number of idle (keep-alive) connections across all hosts. Zero means no limit. max_idle_conns_per_host: 100 // Optional maximum idle (keep-alive) connections to keep per-host. If zero, DefaultMaxIdleConnsPerHost=2 is used. max_conns_per_host: 0 // Optional maximum total number of connections per host. disable_compression: false // Optional. If true, prevents the Transport from requesting compression. client_timeout: 90s // Optional time limit for requests made by the HTTP Client.

Do you want me to ONLY use the DefaultTransport func in net/http/transport.go to construct the Transport for the httpclient?
If this is the case, I will need to remove the customize config fields (which are hard coded) from the PR.

For now, I renamed the DefaultTransport to CustomTransport.

Please advice.

Sorry for the late response, I am on vacation this week.

Sorry @aarontams for the earlier message, just saw this one. Maybe you can open another PR against the new repo when you get back.

GiedriusS

There are some suggestions by @Djelibeybi to the documentation + one nit. Could you please look through those, rebase, and then we can merge this?

Signed-off-by: aaron.tam <aaron.tam@oracle.com>

aarontams · 2022-07-15T07:15:29Z

There are some suggestions by @Djelibeybi to the documentation + one nit. Could you please look through those, rebase, and then we can merge this?

It was already done in previous update from 2 weeks ago Thx!

kakkoyun · 2022-07-15T15:28:05Z

@aarontams Could you please rebase the PR? I would like to merge these changes and sync them to thanos-io/objstore before the #5254 is merged.

kakkoyun · 2022-07-15T16:43:36Z

Closing this to move it to thanos-io/objstore#14

aarontams · 2022-07-15T21:21:26Z

@kakkoyun @GiedriusS @wiardvanrij
I am confused, I rebased my PR in the last 24 hours. Is there any reason @kakkoyun needed to close my PR? Was that by mistake?

Djelibeybi · 2022-07-16T00:37:59Z

@aarontams it was moved to thanos-io/objstore#14

aarontams · 2022-07-16T03:55:25Z

@Djelibeybi
thanos-io/objstore#14 is the issue in https://github.com/thanos-io/objstore. Not PR. My PR got closed here.

@kakkoyun
@aarontams Could you please rebase the PR? I would like to merge these changes and sync them to thanos-io/objstore before the #5254 is merged.

I thought @kakkoyun wanted to merge this PR to https://github.com/thanos-io/thanos repo first. Instead this PR is closed. I am confused. Am I missing something obvious?

Djelibeybi · 2022-07-16T03:56:41Z

Ah, right. Sorry, I just saw the link and didn't look too closely. Guess we'll have to wait and see.

yeya24 · 2022-07-16T04:48:59Z

I thought @kakkoyun wanted to merge this PR to https://github.com/thanos-io/thanos repo first. Instead this PR is closed. I am confused. Am I missing something obvious?

Sorry about that. I totally missed this pr and enabled auto-merge for #5510. This causes all the object-store related code to be removed from this codebase.

I think @kakkoyun intended to merge this pr first so that's why he asked you for a rebase. But I just merged that #5510 so that Kemal had to only close this pr. I am so sorry about it. @aarontams Would you mind opening a pr on the repo for this change? Thanks!

kakkoyun · 2022-07-16T07:07:03Z

Sorry for not clearly communicating my actions here. It is exactly as @yeya24 explained.

I saw that you're on vacation and I didn't want to push you to rebase. So I moved on. We still want to have these changes. That's why I have opened an issue on the new objstore repo for this.

I know it's a bit of more work. Could you please open a new PR on the new repo? I hadn't done it because the package structure has changed and it's more involved than syncing over some files.

Sorry for the inconvenience again @aarontams

aarontams · 2022-07-17T06:59:50Z

Hi @kakkoyun and @yeya24
No problem. With all the bumps that I have faced since I opened this PR last September, this little bump is not a big deal.

I have a new PR ready in thanos-io/objstore#15 with new test result.

Hopefully, it will be a smooth ride from here. We will move the future conversation to the new PR.

sonatype-lift bot reviewed Sep 16, 2021

View reviewed changes

pkg/objstore/oci/helper.go Show resolved Hide resolved

sonatype-lift bot reviewed Sep 16, 2021

View reviewed changes

pkg/objstore/oci/oci.go Show resolved Hide resolved

sonatype-lift bot reviewed Sep 16, 2021

View reviewed changes

pkg/objstore/oci/oci.go Show resolved Hide resolved

sonatype-lift bot reviewed Sep 16, 2021

View reviewed changes

pkg/objstore/oci/oci.go Show resolved Hide resolved

aarontams force-pushed the add-oci-object-storage-bucket-support_20210915 branch from 16a3cfd to 21e07f2 Compare September 16, 2021 06:32

aarontams force-pushed the add-oci-object-storage-bucket-support_20210915 branch from 0970286 to 91db551 Compare September 16, 2021 14:37

aarontams force-pushed the add-oci-object-storage-bucket-support_20210915 branch 2 times, most recently from 3ef73f7 to e67c44d Compare September 28, 2021 12:56

aarontams force-pushed the add-oci-object-storage-bucket-support_20210915 branch from e67c44d to 729f80b Compare October 28, 2021 23:14

aarontams force-pushed the add-oci-object-storage-bucket-support_20210915 branch from 729f80b to 83084b1 Compare November 18, 2021 15:55

GiedriusS reviewed Nov 19, 2021

View reviewed changes

aarontams force-pushed the add-oci-object-storage-bucket-support_20210915 branch from 83084b1 to c6c1e11 Compare November 21, 2021 22:56

aarontams force-pushed the add-oci-object-storage-bucket-support_20210915 branch 6 times, most recently from f0edce8 to 091134f Compare December 5, 2021 06:42

aarontams force-pushed the add-oci-object-storage-bucket-support_20210915 branch from 399c53c to 98e4346 Compare June 29, 2022 15:00

Djelibeybi suggested changes Jun 29, 2022

View reviewed changes

aarontams force-pushed the add-oci-object-storage-bucket-support_20210915 branch from 98e4346 to f972f82 Compare June 30, 2022 04:48

aarontams force-pushed the add-oci-object-storage-bucket-support_20210915 branch from f972f82 to e5c3c12 Compare July 1, 2022 23:08

aarontams force-pushed the add-oci-object-storage-bucket-support_20210915 branch from e5c3c12 to 8ad71ac Compare July 2, 2022 06:41

kakkoyun added the package: objstore Mark PRs and issues needs to be migrated to thanos-io/objstore label Jul 8, 2022

GiedriusS reviewed Jul 11, 2022

View reviewed changes

GiedriusS previously approved these changes Jul 11, 2022

View reviewed changes

Add OCI Object Storage Bucket support

575d30e

Signed-off-by: aaron.tam <aaron.tam@oracle.com>

aarontams dismissed GiedriusS’s stale review via 575d30e July 15, 2022 07:12

aarontams force-pushed the add-oci-object-storage-bucket-support_20210915 branch from 8ad71ac to 575d30e Compare July 15, 2022 07:12

kakkoyun mentioned this pull request Jul 15, 2022

Add Oracle Cloud Infrastructure Object Storage Bucket support thanos-io/objstore#14

Closed

kakkoyun closed this Jul 15, 2022

aarontams mentioned this pull request Jul 17, 2022

Add OCI Object Storage Bucket support thanos-io/objstore#15

Merged

2 tasks

Storage: Add Oracle Cloud Infrastructure Object Storage Bucket support #4661

Storage: Add Oracle Cloud Infrastructure Object Storage Bucket support #4661

Conversation

aarontams commented Sep 16, 2021 • edited

Changes

Add Oracle Cloud Infrastructure (OCI) Object Storage Bucket support

Support multiple prometheus instances with a single bucket

Verification

Testing

Test 1 - OCI_OBJECT_BASE_PATH is not set

Test 2 - OCI_OBJECT_BASE_PATH=/my/base/path

Test 3 - OCI_OBJECT_BASE_PATH=test3/base-path

aarontams commented Sep 16, 2021

aarontams commented Sep 16, 2021

aarontams commented Sep 16, 2021

GiedriusS commented Sep 16, 2021

aarontams commented Sep 23, 2021

wiardvanrij commented Oct 4, 2021

aarontams commented Oct 5, 2021

wiardvanrij commented Oct 6, 2021

aarontams commented Oct 6, 2021

aarontams commented Oct 28, 2021 • edited

aarontams commented Nov 19, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aarontams commented Nov 20, 2021

aarontams commented Jun 29, 2022

Djelibeybi left a comment

Choose a reason for hiding this comment

aarontams commented Jun 30, 2022

wiardvanrij commented Jul 1, 2022

wiardvanrij commented Jul 1, 2022

aarontams commented Jul 1, 2022

aarontams commented Jul 2, 2022

Choose a reason for hiding this comment

aarontams Jul 15, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GiedriusS left a comment

Choose a reason for hiding this comment

aarontams commented Jul 15, 2022

kakkoyun commented Jul 15, 2022

kakkoyun commented Jul 15, 2022

aarontams commented Jul 15, 2022 • edited

Djelibeybi commented Jul 16, 2022

aarontams commented Jul 16, 2022 • edited

Djelibeybi commented Jul 16, 2022

yeya24 commented Jul 16, 2022

kakkoyun commented Jul 16, 2022

aarontams commented Jul 17, 2022 • edited

aarontams commented Sep 16, 2021 •

edited

aarontams commented Oct 28, 2021 •

edited

aarontams Jul 15, 2022 •

edited

aarontams commented Jul 15, 2022 •

edited

aarontams commented Jul 16, 2022 •

edited

aarontams commented Jul 17, 2022 •

edited