LEP: Object Support for Longhorn #5832

jecluis · 2023-04-30T22:57:27Z

We are proposing Object support for Longhorn, by integrating aquarist-labs/s3gw into Longhorn, and seamlessly provide S3-compatible object endpoints from Longhorn volumes.

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

Please Note: While we believe this to be a decent state, there are many things that are still unknowns and we would love the feedback, and any constructive criticism or suggestions on how to improve will be immensely appreciated!

enhancements/20230430-object-endpoint.md

innobead · 2023-05-08T09:05:12Z

@shuo-wu @PhanLe1010 review this after 1.5.0.

l-mb · 2023-05-08T09:13:12Z

@shuo-wu @PhanLe1010 review this after 1.5.0.

Hi @innobead, would it be possible to give the team a casual review of the approach already? If so, we could get started on a tentative/draft PR of the actual controller already, which might drive further discussion without waiting a few weeks. We just don't want to head down the entirely wrong way :)

innobead · 2023-05-08T09:17:04Z

@l-mb , NP. I think members will try to review it when available. I have reviewed it in general, and it should be fine generally but some details need to craft over time. Having a controller is correct as described in the PRD.

enhancements/20230430-object-endpoint.md

longhorn/longhorn#5832

- Add config options and flags for object gateway - Add CRDs for the object endpoint - Regenerate depoy/longhorn.yaml Related-to: longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

enhancements/20230430-object-endpoint.md

PhanLe1010

I have left some questions.

I am still learning S3GW and COSI, so please forgive me if my questions don't make sense or if they are naive

enhancements/20230430-object-endpoint.md

m-ildefons · 2023-07-21T14:17:16Z

For the current POC implementation I've been playing around with different settings that may or may not be useful in the ObjectEndpoint CRD. This led me to the conclusion that for the features described in this LEP, the following form of a CRD would be best suited:

---
apiVersion: longhorn.io/v1beta2
kind: ObjectEndpoint
metadata:
  name: test-endpoint
  namespace: longhorn-system
spec:
  storageClass: longhorn
  credentials:
    accessKey: foo
    secretKey: bar
  size: 5Gi

The differences to the proposed CRD are:

The Image field is missing. This field is unnecessary as the Longhorn manager already knows which image it should use to deploy the s3gw, as that is given as command line option.
PublicDomain and SSLCertificate are missing because I haven't implemented the features making use of these settings yet.
Volume field has been replaced by the optional storageClass and size fields. These two fields describe the desired properties of the backing volume sufficiently. The controller can then automatically deploy the resources necessary to realize this. This makes it easier on a user to deploy an object endpoint, since they don't have to deploy a volume first and take care of the association with the object endpoint manually.

giubacc · 2023-07-21T14:42:25Z

For the current POC implementation I've been playing around with different settings that may or may not be useful in the ObjectEndpoint CRD. This led me to the conclusion that for the features described in this LEP, the following form of a CRD would be best suited:
---
apiVersion: longhorn.io/v1beta2
kind: ObjectEndpoint
metadata:
  name: test-endpoint
  namespace: longhorn-system
spec:
  storageClass: longhorn
  credentials:
    accessKey: foo
    secretKey: bar
  size: 5Gi
The differences to the proposed CRD are:

The Image field is missing. This field is unnecessary as the Longhorn manager already knows which image it should use to deploy the s3gw, as that is given as command line option.

PublicDomain and SSLCertificate are missing because I haven't implemented the features making use of these settings yet.

Volume field has been replaced by the optional storageClass and size fields. These two fields describe the desired properties of the backing volume sufficiently. The controller can then automatically deploy the resources necessary to realize this. This makes it easier on a user to deploy an object endpoint, since they don't have to deploy a volume first and take care of the association with the object endpoint manually.

This sounds nicely. 👍

Some random questions:

Could it make sense to add Longhorn specific settings for controlling the volume properties?
Or here are we saying that we rely completely on the properties defined in the storage-class?
The use case here seems to suggest that is responsibility of the user to define his storage classes. Could this be automated someway when s3gw is used with LH (that is the privileged scenario)? Or is this the intended design?

m-ildefons · 2023-07-24T07:14:08Z

Could it make sense to add Longhorn specific settings for controlling the volume properties? Or here are we saying that we rely completely on the properties defined in the storage-class? The use case here seems to suggest that is responsibility of the user to define his storage classes. Could this be automated someway when s3gw is used with LH (that is the privileged scenario)? Or is this the intended design?

Both variants, expecting a user to pre-create storage classes or offering all settings, have their pros and cons. Expecting the user to create storage classes certainly is much simpler.
One noteworthy thing here is, that when creating a volume through the Longhorn UI, the users is asked for various settings directly, which completely circumvents the storage classes.
I'm not sure if we want to copy this behavior, but it would be a reasonable choice to be consistent with the volume creation dialogue.

We're also not necessarily confined to either/or exclusively. We could create an abstraction that can handle both, it would just be more work, e.g.:

longhorn.ObjectEndpoint{
  ObjectMeta metav1.ObjectMeta {
    Name: <string>
  },
  Spec: longhorn.ObjectEndpointSpec{
    Storage: <longhorn.StorageSettings>
  },
}

longhorn.StorageSettings{
  Size: <resource.Quantity>
  
  //+optional
  StorageClass: <string>
  
  //+optional
  VolumeSettings: <longhorn.VolumeSettings>
},

longhorn.VolumeSettings{
  numberOfReplicas: <string>
  staleReplicaTimeout: <string>
  fsType: <string>
  allowVolumeExpansion: <bool>
  [...]
},

This way both a storage class or volume settings could be specified directly. It's then the task of the controller to ensure that a) no conflicting settings are given (i.e. either the storage class or the volume settings) and b) to provision the correct resources and connect them up.
This mechanism is certainly the most involved on the controller side, but it's within the realm of technical possibilities.

A strong argument for supporting volume settings directly instead of through a storage class is, that as far as I can tell the Longhonr UI does not have a dialogue for creating storage classes. It's up to the user to create them through the use of YAML files and kubectl, which is a bit of an ugly spot on the user experience IMO, especially when we expect the user to do that to make use of all the settings for the object endpoint. But a dialogue for creating storage classes from the Longhorn UI isn't in the scope of this LEP.

jecluis · 2023-07-24T10:41:01Z

Why would we need to have the user defining the storage class? Can't we create one if there isn't one ourselves? Wouldn't that also give us more control over how the data is kept?

m-ildefons · 2023-07-25T14:13:28Z

Why would we need to have the user defining the storage class? Can't we create one if there isn't one ourselves? Wouldn't that also give us more control over how the data is kept?

There's a default storage class that we can fall back onto, in case the user hasn't deleted it.

Unrelated to that, another question: Should the ObjectEndpoint resource be namespace or cluster scoped?
The argument for being cluster scoped would be that this would allow us to keep the deployments, services, PVCs confined to the longhorn-system namespace, hiding them from the user.
Making it namespace scoped has advantages for users who utilize (cluster-)roles to limit permissions of actors on the cluster. I'm not sure how important that is.

giubacc · 2023-07-25T14:56:46Z

The argument for being cluster scoped would be that this would allow us to keep the deployments, services, PVCs confined to the longhorn-system namespace, hiding them from the user.

I haven't a strong opinion pro or cons one approach.
I would prefer technical s3gw resources to not pollute the user's namespace ... but being more pro or cons one way could become more obvious in practice.

enhancements/20230430-object-endpoint.md

longhorn/longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

PhanLe1010 · 2023-08-16T00:24:46Z

enhancements/20230430-object-endpoint.md

+* Integration of s3gw UI for administration and management purposes. Such an
+  Enhancement Proposal should be a standalone LEP by its own right.
+* Providing Object endpoints for multiple volumes. In this proposal we limit one
+  object endpoint per Longhorn volume (see longhorn/longhorn#5444).
+* Multiple Object endpoints for a single volume, either in Read-Write Many, or
+  as active/passive for HA failover. While we believe it to be of the utmost
+  importance, we are considering it future work that should be addressed in its
+  own LEP.
+* Specify how COSI can be implemented for Longhorn and s3gw. This too should be
+  addressed in a specific LEP.


While these are non-goals for this LEP. I feel that we should keep them in mind while designing this PR to make it easier for future work

m-ildefons · 2023-08-29T08:37:13Z

enhancements/20230430-object-endpoint.md

+During resource creation we will also need to explicitly create a
+`Persitent Volume` that will be bound to the `Persistent Volume Claim` mentioned


We will also need to create the Longhorn volume explicitly.

m-ildefons · 2023-08-29T08:37:59Z

enhancements/20230430-object-endpoint.md

+```golang
+type ObjectEndpointSpec struct {
+    Credentials     ObjectEndpointCredentials
+    StorageClass    string


The StorageClass property is ignored[1], since we manage the volume explicitly ourselves in the controller. Perhaps it should be removed or replaced by other properties that are more meaningful in this context.

[1]:
When explicitly creating a PV and PVC together, the storage class property must match or be an empty string, but all other aspects of the storage class are ignored.
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reserving-a-persistentvolume

Agree to remove the StorageClass property. We still need to hardcode a fixed storageclass like longhorn-objectstore (without any parameter) so that features like PVC expansion can work. This is important because user will need to be able to resize the PVC. Ref the POC https://github.com/longhorn/longhorn-manager/blob/c30fb8b1250ee6d2cecd3e818d3ce7b60719fa23/controller/object_endpoint_controller.go#L433-L434

m-ildefons · 2023-08-29T08:48:14Z

enhancements/20230430-object-endpoint.md

+While handling `starting`, the resources required to have already been created
+through the Kubernetes API, but the controller still has to wait for them to be
+ready and healthy before the transition to `running` can be performed.
+Coincidentally, this is also the behavior expected when the endpoint is in state
+`error`, given the controller will need to wait for resources to be healthy
+before being able to move the endpoint to state `running`.


Is this a requirement or is this documenting what the current iteration of the controller is doing?

It is incorporating what the current implementation is doing.

m-ildefons · 2023-08-29T08:52:15Z

enhancements/20230430-object-endpoint.md

+Integrating Longhorn with s3gw will require the creation of mechanisms that
+allow us to 1) describe an object endpoint; 2) deploy an `s3gw` pod consuming
+a volume; and 3) deploy an `s3gw-ui` pod for management and administration. For
+the purposes of this LEP, there will always be one single `s3gw-ui` pod
+associated with an `s3gw` service, hence one `s3gw-ui` pod per Object Endpoint.
+We do not exclude eventually allowing one single `s3gw-ui` pod being associated
+with multiple Object Endpoints, but that should be considered as future work.


This LEP now includes the UI integration as well?

Integration, no. That presumes the administration and management UI becomes part of the Longhorn UI. That has been stated from the beginning that is not part of this LEP.

s3gw-ui support on the other hand has always been part of this LEP.

m-ildefons · 2023-08-29T08:55:58Z

enhancements/20230430-object-endpoint.md

+### Test plan
+
+It is not clear at this moment how this can be tested, much due to lack of
+knowledge on how Longhorn testing works. Help on this topic would be much
+appreciated.


There is an integration test suite in longhorn/longhorn-tests which we can utilize to test the controller's behavior from a K8s client point of view.
There are also unit tests in the controller it self, which can be used to test individual functions against a mocked/faked K8s API.

m-ildefons · 2023-08-29T08:57:35Z

enhancements/20230430-object-endpoint.md

+* For publicly accessible endpoints, the user must specify a domain name to be
+  used


Is being cluster-externally accessible a requirement for this LEP?

Potentially. I don't see why not. It shouldn't be adding a lot more complexity, I think, and requiring another LEP to make stuff externally accessible seems a bit of overkill to me.

It needs to be - otherwise we can't support the backup-from-another-LH-cluster-to-s3 use case.

PhanLe1010 · 2023-08-29T18:40:41Z

enhancements/20230430-object-endpoint.md

+    * This can, potentially, be randomly generated as well.
+* For publicly accessible endpoints, the user must specify a domain name to be
+  used
+* The user must provide SSL certificates to be used by the endpoint


Why does the user must provide an SSL certificate?
I thought that the SSL cert, and ingress stuff are handled by users (not Longhorn) when they want to expose the s3 endpoint to outside of the cluster?

In-cluster deployments may also require SSL -- we know of at least such a user. We can drop this requirement and have the user deploy SSL themselves, but natively supporting this seems the path of least resistance. Is there anything you are worried about?

Can't we document / automate a cert-manager integration? We need SSL for the UIs anyway too.

PhanLe1010 · 2023-08-29T19:12:08Z

enhancements/20230430-object-endpoint.md

+We define a new Object Endpoint Custom Resource Definition, as follows:
+
+```golang
+type ObjectEndpoint struct {


When I did some demos about this s3 feature to other team members. There are comments about the name of the CRD. ObjectEndpoint sounds like an URL or link. How about the name ObjectStore which indicate a storage system for s3 objects? Looking at rook, looks like they are also using ObjectStore name https://rook.io/docs/rook/v1.9/CRDs/Object-Storage/ceph-object-store-crd/

Amazon calls theirs "S3 endpoints". And, well, they are the API endpoints associated with the S3 API too, and we can have multiple ones of them. I think either phrasing works and we shouldn't get too invested in this argument :-)

I agree with Phan here. Using ObjectStore would leave endpoint free to describe TLS and ingress/domain configuration, allowing perhaps the use of multiple endpoints for one store, e.g. like so:

kind: ObjectStore apiVersion: longhorn.io/v1beta2 metadata: [...] spec: credentials: accessKey: ... secretKey: ... storage: size: 100Gi numberOfReplicas: 3 diskSelector: - ssd endpoints: - name: public domainName: foobar.example.com tls: ... - name: private domainName: foobar.longhorn-system.svc.cluster.local

PhanLe1010 · 2023-08-29T20:59:56Z

enhancements/20230430-object-endpoint.md

+
+Integrating Longhorn with s3gw will require the creation of mechanisms that
+allow us to 1) describe an object endpoint; 2) deploy an `s3gw` pod consuming
+a volume; and 3) deploy an `s3gw-ui` pod for management and administration. For


How can we support deploying S3GW ui here given the fact that we don't help user to deploy ingress. tls cert by default?

If we can deploy s3gw-ui without having deploying ingress, middleware for CORS, and tls cert, that would be great. But are we able to do this currently in s3gw?

I believe this has been discussed previously. There are on-going efforts to make the s3gw-ui not depend on a specific ingress to handle CORS. This LEP accounts for those efforts.

PhanLe1010 · 2023-08-30T01:57:46Z

enhancements/20230430-object-endpoint.md

+### Backup and Restore
+
+Given the data being kept by an Object Endpoint is stored in a Longhorn volume,
+we rely on Longhorn's backup and restore capabilities.
+
+However, an Object Endpoint in this context is more than just the data held by a
+given volume: there's meta state that needs to be backed up and restored, in the
+form of Secrets, endpoint names and their associated volume, etc.
+
+At this point in time we don't yet have a solution for this, but we believe this
+should rely on whatever mechanisms Longhorn has to backup and restore its own
+state. Insights are greatly appreciated.


I am working on designing this part

It'd be very interesting if we could restore an endpoint to a different/new one - so someone can pull up the "old" version while the new one remains around.

l-mb · 2023-08-30T15:08:21Z

enhancements/20230430-object-endpoint.md

+During resource creation we will also need to explicitly create a
+`Persitent Volume` that will be bound to the `Persistent Volume Claim` mentioned
+above. We need to do this so we can opinionate on the file system being used; in
+this case, XFS, which `s3gw` requires for performance reasons.


(As a note, we need the reflink support so we can effectively merge multipart uploads etc, which ext4 can't do.)

longhorn/longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Add config options and flags for object gateway - Add CRDs for the object endpoint - Regenerate depoy/longhorn.yaml Related-to: longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

longhorn/longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

- Add config options and flags for object gateway - Add CRDs for the object endpoint - Regenerate depoy/longhorn.yaml Related-to: longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

votdev · 2023-10-04T11:28:11Z

enhancements/20230430-object-endpoint.md

+the existing `/v1/volumes` API endpoints, given they are semantically distinct
+from what we are trying to achieve.
+
+We thus propose the creation of a `/v1/endpoint/object` API endpoint. This route


This section does not match the reality of the PR https://github.com/longhorn/longhorn-manager/pull/2136/files#diff-cb740c237beef9090d8b3a15ca8b76aac83a3af2d52deadc382b4309916a715c and needs to be adapted.

votdev · 2023-10-06T07:50:04Z

enhancements/20230430-object-endpoint.md

+#### Creating a new "Object Store"
+
+* The user clicks on *Create*
+* A modal dialog is shown, with the various Object Endpoint related fields:


These fields are taken from https://github.com/longhorn/longhorn-manager/pull/2136/files#diff-c75a796f2c69c35ee5ab6293f23db5c8d5af9a241f4f2345e96b1fc41d72ddceR55

Is it necessary to request all that fields from the user or can we hardcode some of them?

Referecens: - longhorn/longhorn-ui#649 - https://github.com/longhorn/longhorn-manager/pull/2136/files#diff-cb740c237beef9090d8b3a15ca8b76aac83a3af2d52deadc382b4309916a715c - https://github.com/longhorn/longhorn-manager/pull/2136/files#diff-c75a796f2c69c35ee5ab6293f23db5c8d5af9a241f4f2345e96b1fc41d72ddceR41 Signed-off-by: Volker Theile <vtheile@suse.com>

- Add config options and flags for object gateway - Add CRDs for the object endpoint - Regenerate depoy/longhorn.yaml Related-to: longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

longhorn/longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

innobead · 2023-12-12T04:54:54Z

Closing first, and this will be revisited in 1.7.

jecluis requested a review from a team as a code owner April 30, 2023 22:57

jecluis marked this pull request as draft April 30, 2023 22:57

l-mb reviewed May 2, 2023

View reviewed changes

enhancements/20230430-object-endpoint.md Outdated Show resolved Hide resolved

l-mb reviewed May 2, 2023

View reviewed changes

enhancements/20230430-object-endpoint.md Outdated Show resolved Hide resolved

l-mb reviewed May 2, 2023

View reviewed changes

enhancements/20230430-object-endpoint.md Outdated Show resolved Hide resolved

innobead requested review from PhanLe1010 and shuo-wu May 8, 2023 09:04

innobead assigned jecluis Jun 19, 2023

m-ildefons reviewed Jul 14, 2023

View reviewed changes

enhancements/20230430-object-endpoint.md Outdated Show resolved Hide resolved

innobead requested a review from derekbit July 14, 2023 09:16

m-ildefons added a commit to m-ildefons/longhorn-manager that referenced this pull request Jul 14, 2023

Object Endpoint

b65606e

longhorn/longhorn#5832

m-ildefons reviewed Jul 18, 2023

View reviewed changes

enhancements/20230430-object-endpoint.md Outdated Show resolved Hide resolved

m-ildefons reviewed Jul 18, 2023

View reviewed changes

enhancements/20230430-object-endpoint.md Outdated Show resolved Hide resolved

PhanLe1010 reviewed Jul 20, 2023

View reviewed changes

votdev reviewed Jul 25, 2023

View reviewed changes

enhancements/20230430-object-endpoint.md Show resolved Hide resolved

m-ildefons added a commit to m-ildefons/longhorn-manager that referenced this pull request Aug 11, 2023

Object Endpoint

572c5d6

longhorn/longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

PhanLe1010 reviewed Aug 16, 2023

View reviewed changes

jecluis force-pushed the lep-20230430-object-endpoint branch from 774c067 to f992d3b Compare August 28, 2023 21:28

jecluis marked this pull request as ready for review August 28, 2023 21:30

jecluis force-pushed the lep-20230430-object-endpoint branch from f992d3b to 7e55591 Compare August 28, 2023 21:31

m-ildefons reviewed Aug 29, 2023

View reviewed changes

PhanLe1010 reviewed Aug 30, 2023

View reviewed changes

l-mb reviewed Aug 30, 2023

View reviewed changes

PhanLe1010 pushed a commit to PhanLe1010/longhorn-manager that referenced this pull request Sep 6, 2023

Object Endpoint

305b89d

longhorn/longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

PhanLe1010 pushed a commit to PhanLe1010/longhorn-manager that referenced this pull request Sep 6, 2023

Object Endpoint

dfcd0e6

longhorn/longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

This was referenced Sep 6, 2023

Object Storage (LEP 20230430) #6640

Closed

Object Storage (LEP 20230430) longhorn/longhorn-manager#2136

Closed

Object Storage (LEP 20230430) longhorn/longhorn-ui#649

Closed

m-ildefons mentioned this pull request Sep 6, 2023

[Draft] [DNM] Object Storage longhorn/longhorn-tests#1523

Closed

m-ildefons added a commit to m-ildefons/longhorn-manager that referenced this pull request Sep 6, 2023

Object Endpoint

60a1642

longhorn/longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

m-ildefons added a commit to m-ildefons/longhorn-manager that referenced this pull request Sep 6, 2023

Object Endpoint

18fa003

longhorn/longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

This was referenced Sep 14, 2023

Object Storage (LEP 20230430) longhorn/longhorn-engine#934

Closed

Object Storage (LEP 20230430) longhorn/backupstore#148

Closed

Object Storage (LEP 20230430) longhorn/longhorn-instance-manager#285

Closed

m-ildefons added a commit to m-ildefons/longhorn-manager that referenced this pull request Sep 19, 2023

Object Endpoint

2f70fba

longhorn/longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

votdev suggested changes Oct 4, 2023

View reviewed changes

votdev reviewed Oct 6, 2023

View reviewed changes

votdev force-pushed the lep-20230430-object-endpoint branch from ac2494e to 75de654 Compare October 12, 2023 08:43

m-ildefons added a commit to m-ildefons/longhorn-manager that referenced this pull request Nov 2, 2023

Object Endpoint

bd989de

longhorn/longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

m-ildefons added a commit to m-ildefons/longhorn-manager that referenced this pull request Nov 9, 2023

Object Endpoint

5187bb3

longhorn/longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

m-ildefons added a commit to m-ildefons/longhorn-manager that referenced this pull request Nov 17, 2023

Object Endpoint

420881d

longhorn/longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

m-ildefons added a commit to m-ildefons/longhorn-manager that referenced this pull request Nov 20, 2023

Object Endpoint

b6168df

longhorn/longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

m-ildefons mentioned this pull request Nov 22, 2023

[FEATURE] S3/Object Gateway Support - Experimental #4154

Open

m-ildefons added a commit to m-ildefons/longhorn-manager that referenced this pull request Nov 29, 2023

Object Endpoint

9b58eef

longhorn/longhorn#5832 Signed-off-by: Moritz Röhrich <moritz.rohrich@suse.com>

innobead closed this Dec 12, 2023

		During resource creation we will also need to explicitly create a
		`Persitent Volume` that will be bound to the `Persistent Volume Claim` mentioned

		* For publicly accessible endpoints, the user must specify a domain name to be
		used

LEP: Object Support for Longhorn #5832

LEP: Object Support for Longhorn #5832

Conversation

jecluis commented Apr 30, 2023 • edited

innobead commented May 8, 2023

l-mb commented May 8, 2023

innobead commented May 8, 2023 • edited

PhanLe1010 left a comment • edited

Choose a reason for hiding this comment

m-ildefons commented Jul 21, 2023

giubacc commented Jul 21, 2023

m-ildefons commented Jul 24, 2023

jecluis commented Jul 24, 2023

m-ildefons commented Jul 25, 2023

giubacc commented Jul 25, 2023

PhanLe1010 Aug 16, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PhanLe1010 Sep 7, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

innobead commented Dec 12, 2023

jecluis commented Apr 30, 2023 •

edited

innobead commented May 8, 2023 •

edited

PhanLe1010 left a comment •

edited

PhanLe1010 Aug 16, 2023 •

edited

PhanLe1010 Sep 7, 2023 •

edited