[PIP 79][client] Add lazy-loading feature to PartitionedProducer #10279

equanz · 2021-04-20T08:30:37Z

Master Issue: https://github.com/apache/pulsar/wiki/PIP-79%3A-Reduce-redundant-producers-from-partitioned-producer

Motivation

Please see the PIP document.
This is a part of implementations. ~~I will submit the next PR about PartitionedTopicStats later.~~ #10534 is the next PR.

Modifications

Partitioned producer connects to partitions lazily
- only support it at Shared mode to support this specification
Add round-robin with limit number of partitions routing mode

Verifying this change

Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

Added test for lazy-loading
Added test for routing mode class

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (no)
The public API: (no)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation

Does this pull request introduce a new feature? (yes)
If yes, how is the feature documented? (not documented)

equanz · 2021-04-20T09:40:22Z

/pulsarbot run-failure-checks

eolivelli

Thanks this awesome proposal.
I left some early feedback after doing a first pass.
I am not sure about concurrent aspects during producer creation. I will take another look soon

eolivelli · 2021-04-26T05:16:18Z

pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTest2.java

@@ -1521,6 +1522,17 @@ public void testMaxTopicsPerNamespace() throws Exception {
        }

        // check producer/consumer auto create partitioned topic
+        final Function<Producer<byte[]>, Producer<byte[]>> send = (p) -> {


Why are you modifying an existing test?
It is okay to refactor and reuse an existing test but in this case probably you are altering the behaviour of the existing guest and also you are not sure that you are testing your feature properly

Why are you modifying an existing test?

Because this feature is modifying existing behavior. Currently, a partitioned producer connects to all partitions. Some cases of tests suppose "partitioned producer connects to all partitions(and also create all partition automatically)".
In this change, I modified this procedure to lazy-loading and modified test cases to follow it(here is a part of test codes for this feature).

If we should not change existing behavior, I'll change this feature to be able to stop.

eolivelli · 2021-04-26T05:20:06Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/PartitionedProducerImpl.java

+                    log.error("[{}] Could not create internal producer. partitionIndex: {}", topic, partition,
+                            createException);
+                    try {
+                        producers.remove(partition).close();


We are not sure that we are removing the newly create producer here.

The reason for it is to recreate internal producer when producer creation is failed. This behavior is tested here.

I mean that we should test the result for "producers.remove(partition)" for equality ('==') with prod

nkurihar · 2021-07-15T06:29:14Z

@sijie @eolivelli @merlimat
Could you please review this PR?

Though there are conflicts, we want to confirm the motivation and solution make sense before resolving conflicts.

Vanlightly · 2021-08-05T13:35:42Z

I have submitted a PR for lazy producers in the C++ client (#11570), I did the work before seeing this PIP-79. The approach I took was to create the producers without starting the lookup and connection procedure. So the collection of producers is as big as the number of partitions, and changes when the number partitions change, but the lookup and connect procedure is only started on a producer's first message. It does not block in sendAsync but kicks off the procedure asynchronously allowing messages to be buffered until the producer is connected. In the C++ client, deadlocks were an issue so I avoided any blocking code.

I don't know if that approach makes sense for the Java client as I have not studied how the Java client works in detail yet. But that seems to be the main difference between my C++ changes and this Java client change.

Fixes #11496 also matches part of PIP 79. C++ implementation that closely matches the proposed Java client changes from reducing partitioned producer connections and lookups: PR 10279 ### Motivation Producers that send messages to partitioned topics start a producer per partition, even when using single partition routing. For topics that have the combination of a large number of producers and a large number of partitions, this can put strain on the brokers. With say 1000 partitions and single partition routing with non-keyed messages, 999 topic owner lookups and producer registrations are performed that could be avoided. PIP 79 also describes this. I wrote this before realising that PIP 79 also covers this. This implementation can be reviewed and contrasted to the Java client implementation in #10279. ### Modifications Allows partitioned producers to start producers for individual partitions lazily. Starting a producer involves a topic owner lookup to find out which broker is the owner of the partition, then registering the producer for that partition with the owner broker. For topics with many partitions and when using SinglePartition routing without keyed messages, all of these lookups and producer registrations are a waste except for the single chosen partition. This change allows the user to control whether a producer on a partitioned topic uses this lazy start or not, via a new config in ProducerConfiguration. When ProducerConfiguration.setLazyStartPartitionedProducers(true) is set, the PartitionedProducerImpl.start() becomes a synchronous operation that only does housekeeping (no network operations). The producer of any given partition is started (which includes a topic owner lookup and registration) upon sending the first message to that partition. While the producer starts, messages are buffered. The sendTimeout timer is only activated once a producer has been fully started, which should give enough time for any buffered messages to be sent. For very short send timeouts, this setting could cause send timeouts during the start phase. The default of 30s should however not cause this issue.

equanz · 2021-08-30T07:58:18Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/PartitionedProducerImpl.java

-        return producers.get(0).getProducerName();
+        return producers.get(firstPartitionIndex).getProducerName();


This change breaks existing behavior. If it is not approved, I'll be changed to create the partition-0 producer for backward compatibility.

shouldn't be the same producerName for all the producers ?
why not computing the name in the constructor and cache it ?

shouldn't be the same producerName for all the producers ?

In my understanding, currently producerName is calculated for each partition if doesn't set it at ProducerBuilder.

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/ServerCnx.java

Lines 1110 to 1111 in fe4cd09

final String producerName = cmdProducer.hasProducerName() ? cmdProducer.getProducerName()

: service.generateUniqueProducerName();

pulsar/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java

Lines 1364 to 1366 in fe4cd09

if (this.producerName == null) {

this.producerName = producerName;

}

why not computing the name in the constructor and cache it ?

If needed, I'll add the behavior that compute and cache producerName to below.

pulsar/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PartitionedProducerImpl.java

Lines 134 to 160 in fe4cd09

producer.producerCreatedFuture().handle((prod, createException) -> {

if (createException != null) {

setState(State.Failed);

createFail.compareAndSet(null, createException);

}

// we mark success if all the partitions are created

// successfully, else we throw an exception

// due to any

// failure in one of the partitions and close the successfully

// created partitions

if (completed.incrementAndGet() == topicMetadata.numPartitions()) {

if (createFail.get() == null) {

setState(State.Ready);

log.info("[{}] Created partitioned producer", topic);

producerCreatedFuture().complete(PartitionedProducerImpl.this);

} else {

log.error("[{}] Could not create partitioned producer.", topic, createFail.get().getCause());

closeAsync().handle((ok, closeException) -> {

producerCreatedFuture().completeExceptionally(createFail.get());

client.cleanupProducer(this);

return null;

});

}

}

return null;

});

equanz · 2021-08-30T08:59:56Z

/pulsarbot run-failure-checks

equanz · 2021-08-30T09:55:26Z

Rebased to current master commit
PTAL

equanz · 2021-09-01T11:48:46Z

@eolivelli @Vanlightly
Could you please review it?

eolivelli

I left another round of comments

@merlimat @codelipenghui @rdhabalia can you please take a look ?

eolivelli · 2021-09-01T13:01:06Z

pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicAutoCreationTest.java

+        for (int i = 0; i < 3; i++) {
+            try {
+                producer.newMessage().value("msg".getBytes()).send();
+            } catch (Throwable e) {


the test will fail anyway, no need to catch the exception

eolivelli · 2021-09-01T13:02:18Z

pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicPoliciesTest.java

+                    p.newMessage().value("msg".getBytes()).send();
+                } catch (Throwable e) {
+                    log.info("Exception: ", e);
+                    fail();


what about

catch (Exception e) { log.info("Exception: ", e); throw new RuntimeException(e); }

eolivelli · 2021-09-01T13:02:26Z

pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicPoliciesTest.java

+            for (int i = 0; i < 2; i++) {
+                try {
+                    p.newMessage().value("msg".getBytes()).send();
+                } catch (Throwable e) {


eolivelli · 2021-09-01T13:02:49Z

pulsar-broker/src/test/java/org/apache/pulsar/broker/service/PersistentTopicE2ETest.java

                .newProducer(Schema.JSON(Schemas.BytesRecord.class))
                .topic(topic)
+                .enableBatching(false)


why do you need to change this test ?

In this PR, I'll introduce internal producer lazy-loading feature. When internal producer is elected by MessageRouter and isn't created yet, create it lazily.
https://github.com/equanz/pulsar/blob/844cca7cf84b76b7d27f12de4dbf36e23cbf29e9/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PartitionedProducerImpl.java#L193-L211
Therefore, if it isn't elected by MessageRouter, doesn't create producer and also topic in this test case.

For above reason, we should add Producer#send.

When enableBatching, not easier to elect all of internal producers at client side.

pulsar/pulsar-client/src/main/java/org/apache/pulsar/client/impl/RoundRobinPartitionMessageRouterImpl.java

Lines 82 to 84 in fe4cd09

if (isBatchingEnabled) { // if batching is enabled, choose partition on `partitionSwitchMs` boundary.

long currentMs = clock.millis();

return signSafeMod(currentMs / partitionSwitchMs + startPtnIdx, topicMetadata.numPartitions());

eolivelli · 2021-09-01T13:03:27Z

pulsar-broker/src/test/java/org/apache/pulsar/broker/service/SubscriptionSeekTest.java

@@ -418,7 +418,11 @@ public void testSeekTimeOnPartitionedTopic() throws Exception {
        long resetTimeInMillis = TimeUnit.SECONDS
                .toMillis(RelativeTimeUtil.parseRelativeTimeInSeconds(resetTimeStr));
        admin.topics().createPartitionedTopic(topicName, partitions);
-        Producer<byte[]> producer = pulsarClient.newProducer().topic(topicName).create();
+        Producer<byte[]> producer = pulsarClient.newProducer().topic(topicName).enableBatching(false).create();


why do you need to change this test ? (enableBatching(false))

eolivelli · 2021-09-01T13:04:15Z

pulsar-broker/src/test/java/org/apache/pulsar/client/impl/TopicsConsumerImplTest.java

@@ -863,7 +863,7 @@ public void testConsumerDistributionInFailoverSubscriptionWhenUpdatePartitions()
                .messageRouter(new MessageRouter() {
                    @Override
                    public int choosePartition(Message<?> msg, TopicMetadata metadata) {
-                        return Integer.parseInt(msg.getKey()) % metadata.numPartitions();
+                        return msg.hasKey() ? Integer.parseInt(msg.getKey()) % metadata.numPartitions() : 0;


this change looks unrelated to this patch, please revert

I'll add this behavior for authn/authz (please see also #11570 (comment) ). Elect first partition index by MessageRouter with blank message.
https://github.com/equanz/pulsar/blob/844cca7cf84b76b7d27f12de4dbf36e23cbf29e9/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PartitionedProducerImpl.java#L87-L88

If we shouldn't change this test and behavior,

add pseudo message key like xxx to blank message

handle exceptions in PartitionedProducerImpl

select first partition index without MessageRouter (e.g. randomly select)

if message router elects partition from part of partitions like singlepartition routing mode, create redundant producer for this change

etc.

eolivelli · 2021-09-01T13:04:49Z

...integration/src/test/java/org/apache/pulsar/tests/integration/compaction/TestCompaction.java

@@ -182,7 +182,7 @@ public void testPublishCompactAndConsumePartitionedTopics(Supplier<String> servi
                .messageRouter(new MessageRouter() {
                    @Override
                    public int choosePartition(Message<?> msg, TopicMetadata metadata) {
-                        return Integer.parseInt(msg.getKey()) % metadata.numPartitions();
+                        return msg.hasKey() ? Integer.parseInt(msg.getKey()) % metadata.numPartitions() : 0;


why do you need to change this test ?

eolivelli · 2021-09-01T13:06:34Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/PartitionedProducerImpl.java

-        return producers.get(0).getProducerName();
+        return producers.get(firstPartitionIndex).getProducerName();


shouldn't be the same producerName for all the producers ?
why not computing the name in the constructor and cache it ?

eolivelli · 2021-09-01T13:07:45Z

pulsar-client/src/main/java/org/apache/pulsar/client/impl/PartitionedProducerImpl.java

+                    log.error("[{}] Could not create internal producer. partitionIndex: {}", topic, partition,
+                            createException);
+                    try {
+                        producers.remove(partition).close();


I mean that we should test the result for "producers.remove(partition)" for equality ('==') with prod

equanz · 2021-09-02T10:28:07Z

@eolivelli
Thank you for your comments! I'll address your comments.
And could you please check my replies?

equanz · 2021-09-03T10:32:21Z

/pulsarbot run-failure-checks

eolivelli

I believe that the changes in the test cases are showing too many behaviour changes.

We should add a configuration option to enable this behaviour explicitly

@joefk @merlimat @rdhabalia @codelipenghui @sijie WDYT ?

eolivelli

Lgtm

equanz · 2021-10-12T13:31:21Z

/pulsarbot run-failure-checks

equanz · 2021-10-12T13:36:22Z

/pulsarbot run-failure-checks

equanz · 2021-10-13T08:21:26Z

/pulsarbot run-failure-checks

equanz · 2021-10-13T10:12:15Z

@eolivelli Thank you for your review!

@massakam @k2la Rebased to resolve conflicts. Also, add fc83a5d commit to follow #12287 feature even if lazy-loading feature is enabled. PTAL.

pulsar-broker/src/test/java/org/apache/pulsar/client/impl/PartialPartitionedProducerTest.java

.../main/java/org/apache/pulsar/client/impl/customroute/PartialRoundRobinMessageRouterImpl.java

pulsar-client/src/main/java/org/apache/pulsar/client/impl/PartitionedProducerImpl.java

equanz · 2021-10-15T00:56:45Z

/pulsarbot run-failure-checks

equanz · 2021-10-15T02:00:12Z

/pulsarbot run-failure-checks

equanz · 2021-10-15T05:04:52Z

@massakam PTAL

…e#11570) Fixes apache#11496 also matches part of PIP 79. C++ implementation that closely matches the proposed Java client changes from reducing partitioned producer connections and lookups: PR 10279 ### Motivation Producers that send messages to partitioned topics start a producer per partition, even when using single partition routing. For topics that have the combination of a large number of producers and a large number of partitions, this can put strain on the brokers. With say 1000 partitions and single partition routing with non-keyed messages, 999 topic owner lookups and producer registrations are performed that could be avoided. PIP 79 also describes this. I wrote this before realising that PIP 79 also covers this. This implementation can be reviewed and contrasted to the Java client implementation in apache#10279. ### Modifications Allows partitioned producers to start producers for individual partitions lazily. Starting a producer involves a topic owner lookup to find out which broker is the owner of the partition, then registering the producer for that partition with the owner broker. For topics with many partitions and when using SinglePartition routing without keyed messages, all of these lookups and producer registrations are a waste except for the single chosen partition. This change allows the user to control whether a producer on a partitioned topic uses this lazy start or not, via a new config in ProducerConfiguration. When ProducerConfiguration.setLazyStartPartitionedProducers(true) is set, the PartitionedProducerImpl.start() becomes a synchronous operation that only does housekeeping (no network operations). The producer of any given partition is started (which includes a topic owner lookup and registration) upon sending the first message to that partition. While the producer starts, messages are buffered. The sendTimeout timer is only activated once a producer has been fully started, which should give enough time for any buffered messages to be sent. For very short send timeouts, this setting could cause send timeouts during the start phase. The default of 30s should however not cause this issue.

…che#10279) * feat: add lazy loading feature to PartitionedProducerImpl * feat: add partial round robin routing mode * test: add tests for lazy-loading * fix: fix producer closing code at lazy-loading * test: remove unnecessary handling, fail from test codes * feat: add enableLazyStartPartitionedProducers config * test: fix test for lazy-loading config * fix: address comments * fix: add partition-change interceptor * fix: address comments

Fixes #11496 also matches part of PIP 79. C++ implementation that closely matches the proposed Java client changes from reducing partitioned producer connections and lookups: PR 10279 ### Motivation Producers that send messages to partitioned topics start a producer per partition, even when using single partition routing. For topics that have the combination of a large number of producers and a large number of partitions, this can put strain on the brokers. With say 1000 partitions and single partition routing with non-keyed messages, 999 topic owner lookups and producer registrations are performed that could be avoided. PIP 79 also describes this. I wrote this before realising that PIP 79 also covers this. This implementation can be reviewed and contrasted to the Java client implementation in apache/pulsar#10279. ### Modifications Allows partitioned producers to start producers for individual partitions lazily. Starting a producer involves a topic owner lookup to find out which broker is the owner of the partition, then registering the producer for that partition with the owner broker. For topics with many partitions and when using SinglePartition routing without keyed messages, all of these lookups and producer registrations are a waste except for the single chosen partition. This change allows the user to control whether a producer on a partitioned topic uses this lazy start or not, via a new config in ProducerConfiguration. When ProducerConfiguration.setLazyStartPartitionedProducers(true) is set, the PartitionedProducerImpl.start() becomes a synchronous operation that only does housekeeping (no network operations). The producer of any given partition is started (which includes a topic owner lookup and registration) upon sending the first message to that partition. While the producer starts, messages are buffered. The sendTimeout timer is only activated once a producer has been fully started, which should give enough time for any buffered messages to be sent. For very short send timeouts, this setting could cause send timeouts during the start phase. The default of 30s should however not cause this issue.

nkurihar added area/client type/PIP type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages type/feature The PR added a new feature or issue requested a new feature labels Apr 20, 2021

equanz force-pushed the pip_79_lazy_loading branch from 164d1b6 to d49aab4 Compare April 21, 2021 09:03

eolivelli requested changes Apr 26, 2021

View reviewed changes

sijie assigned equanz May 4, 2021

sijie added this to the 2.8.0 milestone May 4, 2021

sijie requested review from codelipenghui, jiazhai and merlimat May 4, 2021 03:25

equanz mentioned this pull request May 11, 2021

[PIP 79][common,broker,client] Change PartitionedTopicStats to support partial partitioned producer #10534

Closed

1 task

codelipenghui modified the milestones: 2.8.0, 2.9.0 May 21, 2021

equanz mentioned this pull request Aug 4, 2021

Reduce producer lookups and connections in partitioned producers #11496

Closed

Vanlightly mentioned this pull request Aug 5, 2021

[Issue 11496][C++] Allow partitioned producers to start lazily #11570

Merged

equanz force-pushed the pip_79_lazy_loading branch from d49aab4 to 844cca7 Compare August 30, 2021 07:54

equanz commented Aug 30, 2021

View reviewed changes

equanz requested a review from eolivelli September 1, 2021 11:46

eolivelli reviewed Sep 1, 2021

View reviewed changes

eolivelli requested changes Sep 3, 2021

View reviewed changes

equanz added 8 commits October 12, 2021 17:21

feat: add partial round robin routing mode

eac2982

test: add tests for lazy-loading

41cef47

fix: fix producer closing code at lazy-loading

6dd79e9

test: remove unnecessary handling, fail from test codes

3bd0672

feat: add enableLazyStartPartitionedProducers config

07676ff

test: fix test for lazy-loading config

384a54d

fix: address comments

bb63f0f

fix: add partition-change interceptor

fc83a5d

equanz force-pushed the pip_79_lazy_loading branch from 8c20c02 to fc83a5d Compare October 12, 2021 08:23

eolivelli approved these changes Oct 12, 2021

View reviewed changes

massakam reviewed Oct 14, 2021

View reviewed changes

pulsar-broker/src/test/java/org/apache/pulsar/client/impl/PartialPartitionedProducerTest.java Outdated Show resolved Hide resolved

massakam reviewed Oct 14, 2021

View reviewed changes

.../main/java/org/apache/pulsar/client/impl/customroute/PartialRoundRobinMessageRouterImpl.java Show resolved Hide resolved

massakam reviewed Oct 14, 2021

View reviewed changes

pulsar-client/src/main/java/org/apache/pulsar/client/impl/PartitionedProducerImpl.java Outdated Show resolved Hide resolved

fix: address comments

2f101dd

massakam approved these changes Oct 15, 2021

View reviewed changes

massakam merged commit 1f8945a into apache:master Oct 15, 2021

equanz deleted the pip_79_lazy_loading branch October 15, 2021 06:38

equanz mentioned this pull request Nov 11, 2021

[PIP 79][common,broker,client] Change PartitionedTopicStats to aggregate by producerName #12401

Merged

2 tasks

coderzc mentioned this pull request Sep 5, 2022

[feat][client-c++] Support Exclusive Producer access mode for c++ #17439

Merged

4 tasks

equanz mentioned this pull request Nov 7, 2022

[fix][broker] Deprecated aggregatePublisherStatsByProducerName config #18254

Closed

13 tasks

		return producers.get(0).getProducerName();
		return producers.get(firstPartitionIndex).getProducerName();

	final String producerName = cmdProducer.hasProducerName() ? cmdProducer.getProducerName()
	: service.generateUniqueProducerName();

	if (this.producerName == null) {
	this.producerName = producerName;
	}

	producer.producerCreatedFuture().handle((prod, createException) -> {
	if (createException != null) {
	setState(State.Failed);
	createFail.compareAndSet(null, createException);
	}
	// we mark success if all the partitions are created
	// successfully, else we throw an exception
	// due to any
	// failure in one of the partitions and close the successfully
	// created partitions
	if (completed.incrementAndGet() == topicMetadata.numPartitions()) {
	if (createFail.get() == null) {
	setState(State.Ready);
	log.info("[{}] Created partitioned producer", topic);
	producerCreatedFuture().complete(PartitionedProducerImpl.this);
	} else {
	log.error("[{}] Could not create partitioned producer.", topic, createFail.get().getCause());
	closeAsync().handle((ok, closeException) -> {
	producerCreatedFuture().completeExceptionally(createFail.get());
	client.cleanupProducer(this);
	return null;
	});
	}
	}

	return null;
	});

	if (isBatchingEnabled) { // if batching is enabled, choose partition on `partitionSwitchMs` boundary.
	long currentMs = clock.millis();
	return signSafeMod(currentMs / partitionSwitchMs + startPtnIdx, topicMetadata.numPartitions());

[PIP 79][client] Add lazy-loading feature to PartitionedProducer #10279

[PIP 79][client] Add lazy-loading feature to PartitionedProducer #10279

Conversation

equanz commented Apr 20, 2021 • edited Loading

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

equanz commented Apr 20, 2021

eolivelli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

equanz Apr 27, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

equanz Apr 27, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nkurihar commented Jul 15, 2021

Vanlightly commented Aug 5, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

equanz Sep 2, 2021 • edited Loading

Choose a reason for hiding this comment

equanz commented Aug 30, 2021

equanz commented Aug 30, 2021

equanz commented Sep 1, 2021

eolivelli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

equanz Sep 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

equanz commented Sep 2, 2021

equanz commented Sep 3, 2021

eolivelli left a comment

Choose a reason for hiding this comment

eolivelli left a comment

Choose a reason for hiding this comment

equanz commented Oct 12, 2021

equanz commented Oct 12, 2021

equanz commented Oct 13, 2021

equanz commented Oct 13, 2021

equanz commented Oct 15, 2021

equanz commented Oct 15, 2021

equanz commented Oct 15, 2021

equanz commented Apr 20, 2021 •

edited

Loading

equanz Apr 27, 2021 •

edited

Loading

equanz Apr 27, 2021 •

edited

Loading

equanz Sep 2, 2021 •

edited

Loading

equanz Sep 2, 2021 •

edited

Loading