Add support for Max Series Per Query for block storage and streaming ingesters #4179

treid314 · 2021-05-12T22:38:27Z

What this PR does:
This PR adds support for max_series_per_query option for block storage and streaming ingesters
Which issue(s) this PR fixes:
Fixes #3669

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

pracucci

Good job! I did a very quick 1st pass review and I haven't reviewed tests yet. I also suggest you to double check if passing the limiter via the context is really required (I'm not sure).

Please add a CHANGELOG entry.

pracucci · 2021-05-13T15:33:26Z

pkg/util/limiter/per_query_limiter.go

+
+// NewPerQueryLimiter makes a new per-query rate limiter. Each per-query limiter
+// is configured using the `maxSeriesPerQuery` and `maxChunkBytesPerQuery` limits.
+func NewPerQueryLimiter(maxSeriesPerQuery int, maxChunkBytesPerQuery int) *PerQueryLimiter {


I would completely remove maxChunkBytesPerQuery from this PR.

pracucci · 2021-05-13T15:34:32Z

pkg/util/limiter/per_query_limiter.go

+	"sync"
+)
+
+type PerQueryLimiter struct {


[nit] We can simplify the name and remove the Per prefix everywhere. QueryLimiter looks self-explanatory. WDYT?

pracucci · 2021-05-13T15:35:20Z

pkg/util/limiter/per_query_limiter.go

+	maxSeriesPerQuery     int
+	maxChunkBytesPerQuery int
+}
+type perQueryLimiterCtxMarker struct{}


[nit] I would call it ...CtxKey instead of CtxMarker given it's a context key.

pracucci · 2021-05-13T15:36:05Z

pkg/util/limiter/per_query_limiter.go

+	}
+}
+
+func NewPerQueryLimiterOnContext(ctx context.Context, maxSeriesPerQuery int, maxChunkBytesPerQuery int) context.Context {


[nit] Do we need this function at all? The logic can be achieve concatenating the AddPerQueryLimiterToContext() and NewPerQueryLimiter() together.

pracucci · 2021-05-13T15:36:48Z

pkg/util/limiter/per_query_limiter.go

+// AddFingerPrint Add a label adapter fast fingerprint to the map of unique fingerprints. If the
+// added series label causes us to go over the limit of maxSeriesPerQuery we will
+// return a validation error
+func (pql *PerQueryLimiter) AddFingerPrint(labelAdapter []cortexpb.LabelAdapter, matchers []*labels.Matcher) error {


[nit] I would name AddSeries(). You're not adding a fingerprint, but a series. The fingerprint thing is an internal implementation that shouldn't be exposed to the caller.

pracucci · 2021-05-13T15:54:03Z

pkg/querier/blocks_store_queryable.go

@@ -3,6 +3,8 @@ package querier
 import (
 	"context"
 	"fmt"
+	"github.com/cortexproject/cortex/pkg/cortexpb"


We group imports. Please configure GoLand File Watchers as follow:

pkg/distributor/query.go

pracucci · 2021-05-13T15:56:22Z

pkg/util/limiter/per_query_limiter.go

+
+// PerQueryLimiterFromContext returns a Per Query Limiter from the current context.
+// IF there is Per Query Limiter on the context we will return ???
+func PerQueryLimiterFromContext(ctx context.Context) *PerQueryLimiter {


Remove this function please. Directly call FromContextWithFallback().

pracucci · 2021-05-13T15:56:31Z

pkg/util/limiter/per_query_limiter.go

+	}
+
+	pql.uniqueSeriesMx.Lock()
+	//Unlock after return


No need for such comments :)

pracucci · 2021-05-13T15:57:12Z

pkg/util/limiter/per_query_limiter.go

+	mapL := len(pql.uniqueSeries)
+	return mapL


Suggested change

mapL := len(pql.uniqueSeries)

return mapL

return len(pql.uniqueSeries)

…ingesters and update limits.go to reflect changes Signed-off-by: Tyler Reid <treid@Tylers-MacBook-Air.local>

Signed-off-by: Tyler Reid <treid@Tylers-MacBook-Air.local>

Signed-off-by: Tyler Reid <treid@Tylers-MacBook-Air.local> Signed-off-by: Tyler Reid <tyler.reid@grafana.com>

Signed-off-by: Tyler Reid <tyler.reid@grafana.com>

pracucci

Very good job! I left many nits but the logic looks good. However, while suggesting updated CLI flag I've just realised we're basically introducing a breaking change. Let's discuss about this offline.

pracucci · 2021-05-14T16:26:09Z

pkg/util/validation/limits.go

@@ -130,7 +130,7 @@ func (l *Limits) RegisterFlags(f *flag.FlagSet) {
 	f.BoolVar(&l.EnforceMetricName, "validation.enforce-metric-name", true, "Enforce every sample has a metric name.")
 	f.BoolVar(&l.EnforceMetadataMetricName, "validation.enforce-metadata-metric-name", true, "Enforce every metadata has a metric name.")

-	f.IntVar(&l.MaxSeriesPerQuery, "ingester.max-series-per-query", 100000, "The maximum number of series for which a query can fetch samples from each ingester. This limit is enforced only in the ingesters (when querying samples not flushed to the storage yet) and it's a per-instance limit. This limit is ignored when running the Cortex blocks storage.")
+	f.IntVar(&l.MaxSeriesPerQuery, "ingester.max-series-per-query", 100000, "The maximum number of series for which a query can fetch samples from each ingester and block storage. This limit is enforced in the ingesters and block storage and it's a per-instance limit.")


While writing this I just realized we're introducing a breaking change. I need to discuss it next week with Peter/Goutham, but we may need to introduce a new limit for the blocks storage to avoid introducing a breaking change in the logic.

Suggested change

f.IntVar(&l.MaxSeriesPerQuery, "ingester.max-series-per-query", 100000, "The maximum number of series for which a query can fetch samples from each ingester and block storage. This limit is enforced in the ingesters and block storage and it's a per-instance limit.")

f.IntVar(&l.MaxSeriesPerQuery, "ingester.max-series-per-query", 100000, "The maximum number of series for which a query can fetch samples from ingesters and storage. When running the chunks storage, this limit is enforced only in the ingesters and it's a per-instance limit. When running the blocks storage, this limit is enforced in the querier on samples read both from ingesters and storage and it's a per-query limit.")

docs/configuration/arguments.md

CHANGELOG.md

pracucci · 2021-05-14T16:30:56Z

pkg/util/limiter/query_limiter.go

+// AddSeries Add labels for series to the count of unique series. If the
+// added series label causes us to go over the limit of maxSeriesPerQuery we will
+// return a validation error
+func (ql *QueryLimiter) AddSeries(labelAdapter []cortexpb.LabelAdapter, matchers []*labels.Matcher) error {


I would simplify this and not pass matchers. It's OK if you do not specify the matchers in the error.

pkg/util/limiter/query_limiter_test.go

pracucci · 2021-05-14T16:44:26Z

pkg/util/limiter/query_limiter_test.go

+	for n := 0; n < b.N; n++ {
+		limiter := NewQueryLimiter(10000)
+		for _, s := range series {
+			limiter.AddSeries(cortexpb.FromLabelsToLabelAdapters(s), matchers)
+		}
+	}


What do you want to benchmark here? If it's the AddSeries() performances, then b.N should be the number of series you add in the test.

pkg/distributor/distributor_test.go

pracucci · 2021-05-14T16:52:07Z

pkg/distributor/distributor_test.go

+
+	limits := &validation.Limits{}
+	flagext.DefaultValues(limits)
+	limits.MaxSeriesPerQuery = maxSeriesLimit


You can set this, but it's ignored, because what's used is the limiter in the context.

Co-authored-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Tyler Reid <tyler.reid@grafana.com>

Signed-off-by: Tyler Reid <tyler.reid@grafana.com>

…ad of ingester.max-series-per-query for series per query limit Signed-off-by: Tyler Reid <tyler.reid@grafana.com>

pracucci

Good job! I can approve once last comments are addressed, thanks!

The PR is still in Draft. Can you switch it to "Ready to review", please?

Note to other reviewers: as discussed offline with Tyler, I think passing the limiter via context is not required (we should be able to workaround it with a custom interface extending storage.Querier) but I suggested to attempt the refactoring in a follow-up PR.

pracucci · 2021-05-18T09:16:22Z

pkg/util/validation/limits.go

 // chunks from ingesters.
 func (o *Overrides) MaxChunksPerQueryFromIngesters(userID string) int {
 	return o.getOverridesForUser(userID).MaxChunksPerQuery
 }

+// MaxFetchedSeriesPerQuery returns the maximum number of series allowed per query when fetching
+// chunks from ingesters and block storage


Suggested change

// chunks from ingesters and block storage

// chunks from ingesters and blocks storage.

pracucci · 2021-05-18T09:16:40Z

pkg/util/validation/limits.go

@@ -144,6 +145,7 @@ func (l *Limits) RegisterFlags(f *flag.FlagSet) {
 	f.IntVar(&l.MaxGlobalMetadataPerMetric, "ingester.max-global-metadata-per-metric", 0, "The maximum number of metadata per metric, across the cluster. 0 to disable.")
 	f.IntVar(&l.MaxChunksPerQueryFromStore, "store.query-chunk-limit", 2e6, "Deprecated. Use -querier.max-fetched-chunks-per-query CLI flag and its respective YAML config option instead. Maximum number of chunks that can be fetched in a single query. This limit is enforced when fetching chunks from the long-term storage only. When running the Cortex chunks storage, this limit is enforced in the querier and ruler, while when running the Cortex blocks storage this limit is enforced in the querier, ruler and store-gateway. 0 to disable.")
 	f.IntVar(&l.MaxChunksPerQuery, "querier.max-fetched-chunks-per-query", 0, "Maximum number of chunks that can be fetched in a single query from ingesters and long-term storage: the total number of actual fetched chunks could be 2x the limit, being independently applied when querying ingesters and long-term storage. This limit is enforced in the ingester (if chunks streaming is enabled), querier, ruler and store-gateway. Takes precedence over the deprecated -store.query-chunk-limit. 0 to disable.")
+	f.IntVar(&l.MaxFetchedSeriesPerQuery, "querier.max-series-per-query", 0, "The maximum number of series for which a query can fetch samples from each ingesters and block storage. When running in block storage mode this limit is enforced on the querier and counts series returned from ingesters and block storage as a per-query limit.")


By convention, new CLI flags have the same exact naming of the YAML config with _ replaced with -. So the CLI flag name should be querier.max-fetched-series-per-query.

Remember to update CHANGELOG and doc accordingly.

The maximum number of series for which a query can fetch samples from each ingesters and block storage. When running in block storage mode this limit is enforced on the querier and counts series returned from ingesters and block storage as a per-query limit.

Some little adjustments:

The maximum number of unique series for which a query can fetch samples from each ingesters and long-term storage. This limit is enforced in the querier only when running Cortex with blocks storage. 0 to disable.

Please also update the CLI flag description of ingester.max-series-per-query to mention to use -querier.max-fetched-series-per-query when running the blocks storage.

pracucci · 2021-05-18T09:20:23Z

docs/configuration/arguments.md

+    When running Cortex chunks storage: limit enforced in the ingesters only and it's a per-instance limit.
+    When running Cortex blocks storage: limit enforced in the queriers both on samples fetched from ingesters and store-gateways (long-term storage).


This is no more true. Please update the doc accordingly.

pracucci · 2021-05-18T09:21:38Z

pkg/distributor/distributor_test.go

+	})
+	defer stopAll(ds, r)
+
+	// Push a number of series below the max series limit. Each series has 1 sample.


[nit] The "Each series has 1 sample." comment is irrelevant here.

Suggested change

// Push a number of series below the max series limit. Each series has 1 sample.

// Push a number of series below the max series limit.

pracucci · 2021-05-18T09:24:06Z

pkg/distributor/query.go

+		chunksLimit  = d.limits.MaxChunksPerQueryFromIngesters(userID)
+		chunksCount  = atomic.Int32{}
+		queryLimiter = limiter.QueryLimiterFromContextWithFallback(ctx)
+		matchers, _  = ingester_client.FromLabelMatchers(req.Matchers)


Please move this back to the original place. No more required to have it here.

pracucci · 2021-05-18T09:29:09Z

pkg/util/limiter/query_limiter.go

+
+var (
+	qlCtxKey        = &queryLimiterCtxKey{}
+	errMaxSeriesHit = "The query hit the max number of series limit while fetching chunks (limit: %d)"


[nit] I would further simplify the error message. The final user has no idea what a chunk is. We should just say that the query hit the max number of series limit.

Suggested change

errMaxSeriesHit = "The query hit the max number of series limit while fetching chunks (limit: %d)"

errMaxSeriesHit = "The query hit the max number of series limit (limit: %d)"

pracucci · 2021-05-18T09:29:49Z

pkg/util/limiter/query_limiter.go

+
+	ql.uniqueSeries[client.FastFingerprint(seriesLabels)] = struct{}{}
+	if len(ql.uniqueSeries) > ql.maxSeriesPerQuery {
+		// Format error with query and max limit


[nit] Not much useful comment and also "with query" is not more true.

Suggested change

// Format error with query and max limit

pracucci · 2021-05-18T09:30:01Z

pkg/util/limiter/query_limiter.go

+	return nil
+}
+
+// UniqueSeries returns the count of unique series seen by this query limiter.


Suggested change

// UniqueSeries returns the count of unique series seen by this query limiter.

// uniqueSeriesCount returns the count of unique series seen by this query limiter.

pstibrany

Thank you, this looks good! I've left some nit comments, but nothing blocking.

pstibrany · 2021-05-18T13:57:59Z

pkg/util/limiter/query_limiter.go

+// is configured using the `maxSeriesPerQuery` limit.
+func NewQueryLimiter(maxSeriesPerQuery int) *QueryLimiter {
+	return &QueryLimiter{
+		uniqueSeriesMx: sync.RWMutex{},


Nit: This line is not needed. uniqueSeriesMx will be initialized to default value, which happens to be sync.RWMutex{}.

pstibrany · 2021-05-18T13:58:36Z

pkg/util/limiter/query_limiter.go

+type queryLimiterCtxKey struct{}
+
+var (
+	qlCtxKey        = &queryLimiterCtxKey{}


Nit: contextKey would be just fine.

pstibrany · 2021-05-18T14:02:36Z

pkg/util/limiter/query_limiter.go

+)
+
+type QueryLimiter struct {
+	uniqueSeriesMx sync.RWMutex


Given that Read-lock is only used from tests and not production code, I'd suggest to use sync.Mutex, which is faster than sync.RWMutex. (In your benchmark by about 7.5%)

pstibrany · 2021-05-18T14:07:00Z

pkg/util/limiter/query_limiter.go

+	ql.uniqueSeriesMx.Lock()
+	defer ql.uniqueSeriesMx.Unlock()
+
+	ql.uniqueSeries[client.FastFingerprint(seriesLabels)] = struct{}{}


Nit: To minimize time during which the lock is held, it is preferable to do computations that don't need the lock before locking. In this case it's call to client.FastFingerprint. It's unlikely to be a big deal here, but still a good practice.

I also find it little strange that we always add to the map before checking the limit. If entries in the map were larger, it could lead to unexpected memory problems. Here it's not a problem.

I also find it little strange that we always add to the map before checking the limit.

I figured since we don't know if the series has a unique fingerprint and will put us over the limit, we can't check until after we've added/not added the passed in series fingerprint

pkg/util/limiter/query_limiter_test.go

pstibrany · 2021-05-18T14:13:20Z

pkg/querier/querier.go

@@ -223,6 +223,8 @@ func NewQueryable(distributor QueryableWithFilter, stores []QueryableWithFilter,
 		if err != nil {
 			return nil, err
 		}
+		// Take the set tenant limits


I don't understand this comment. :(

pstibrany · 2021-05-18T14:16:05Z

pkg/distributor/query.go

+		chunksLimit  = d.limits.MaxChunksPerQueryFromIngesters(userID)
+		chunksCount  = atomic.Int32{}
+		queryLimiter = limiter.QueryLimiterFromContextWithFallback(ctx)
+		matchers, _  = ingester_client.FromLabelMatchers(req.Matchers)


This line was moved from other place, but comment associated with the line was left at original place. Plus, as Marco points out, this is only used when building error message.

Signed-off-by: Tyler Reid <tyler.reid@grafana.com>

pstibrany

Thanks for addressing my feedback.

pstibrany · 2021-05-19T10:46:21Z

pkg/util/limiter/query_limiter.go

+	ql.uniqueSeries[fingerprint] = struct{}{}
+	if len(ql.uniqueSeries) > ql.maxSeriesPerQuery {
+		// Format error with max limit
+		return validation.LimitError(fmt.Sprintf(errMaxSeriesHit, ql.maxSeriesPerQuery))


Nit: translation to validation.LimitError should not be a responsibility of query limiter, but code in the querier package. We only do that so that Prometheus API layer returns correct status code.

I suggest to merge as is and then having Tyler addressing this in a follow-up PR.

Signed-off-by: Tyler Reid <tyler.reid@grafana.com>

pracucci

LGTM, thanks!

pracucci · 2021-05-20T10:39:32Z

pkg/util/limiter/query_limiter.go

+	ql.uniqueSeries[fingerprint] = struct{}{}
+	if len(ql.uniqueSeries) > ql.maxSeriesPerQuery {
+		// Format error with max limit
+		return validation.LimitError(fmt.Sprintf(errMaxSeriesHit, ql.maxSeriesPerQuery))


I suggest to merge as is and then having Tyler addressing this in a follow-up PR.

Signed-off-by: Marco Pracucci <marco@pracucci.com>

* Add per-user query metrics for series and bytes returned Add stats included in query responses from the querier and distributor for measuring the number of series and bytes included in successful queries. These stats are emitted per-user as summaries from the query frontends. These stats are picked to add visibility into the same resources limited as part of #4179 and #4216. Fixes #4259 Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Formatting fix Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Fix changelog to match actual changes Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Typo Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Code review changes, rename things for clarity Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Apply suggestions from code review Co-authored-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Code review changes, remove superfluous summaries Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> Co-authored-by: Marco Pracucci <marco@pracucci.com>

…ct#4343) * Add per-user query metrics for series and bytes returned Add stats included in query responses from the querier and distributor for measuring the number of series and bytes included in successful queries. These stats are emitted per-user as summaries from the query frontends. These stats are picked to add visibility into the same resources limited as part of cortexproject#4179 and cortexproject#4216. Fixes cortexproject#4259 Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Formatting fix Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Fix changelog to match actual changes Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Typo Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Code review changes, rename things for clarity Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Apply suggestions from code review Co-authored-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> * Code review changes, remove superfluous summaries Signed-off-by: Nick Pillitteri <nick.pillitteri@grafana.com> Co-authored-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Alvin Lin <alvinlin@amazon.com>

pull-request-size bot added the size/L label May 12, 2021

treid314 force-pushed the series-limits branch from 4e5f321 to 77fac51 Compare May 13, 2021 15:18

pracucci reviewed May 13, 2021

View reviewed changes

treid314 force-pushed the series-limits branch from 844307a to 7d85509 Compare May 13, 2021 21:30

Tyler Reid and others added 5 commits May 14, 2021 11:18

Add support for Max Series Per Query for block storage and streaming …

12a12b7

…ingesters and update limits.go to reflect changes Signed-off-by: Tyler Reid <treid@Tylers-MacBook-Air.local>

Fix spacing issues

9148855

Signed-off-by: Tyler Reid <treid@Tylers-MacBook-Air.local>

Review and linter changes

60571d2

Signed-off-by: Tyler Reid <treid@Tylers-MacBook-Air.local>

Update docs

ad00485

Signed-off-by: Tyler Reid <treid@Tylers-MacBook-Air.local> Signed-off-by: Tyler Reid <tyler.reid@grafana.com>

Add changelog for series per query limit

01e59ea

Signed-off-by: Tyler Reid <tyler.reid@grafana.com>

treid314 force-pushed the series-limits branch from 1a2384d to 01e59ea Compare May 14, 2021 16:24

pracucci reviewed May 14, 2021

View reviewed changes

Apply suggestions from code review

4f31df8

Co-authored-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Tyler Reid <tyler.reid@grafana.com>

treid314 force-pushed the series-limits branch from 9ec3ad6 to 4f31df8 Compare May 14, 2021 20:35

Tyler Reid added 5 commits May 14, 2021 15:36

address review comments

f62d32d

Signed-off-by: Tyler Reid <tyler.reid@grafana.com>

Added integration test for series per query limit

e2e1c6c

Signed-off-by: Tyler Reid <tyler.reid@grafana.com>

Fix up integration test

04d51cd

Signed-off-by: Tyler Reid <tyler.reid@grafana.com>

Merge remote-tracking branch 'upstream/master' into series-limits

0d9af4f

Add new config option querier.max-series-per-query and use that inste…

38005dd

…ad of ingester.max-series-per-query for series per query limit Signed-off-by: Tyler Reid <tyler.reid@grafana.com>

pracucci reviewed May 18, 2021

View reviewed changes

pstibrany approved these changes May 18, 2021

View reviewed changes

Tyler Reid added 2 commits May 18, 2021 12:31

Update docs to show new cli flag and other review changes

6ec0b09

Signed-off-by: Tyler Reid <tyler.reid@grafana.com>

Merge remote-tracking branch 'upstream/master' into series-limits

f4d0d4c

treid314 marked this pull request as ready for review May 18, 2021 17:39

pstibrany approved these changes May 19, 2021

View reviewed changes

Tyler Reid added 2 commits May 19, 2021 08:59

Check error to resolve linter issue

2b8cb1d

Signed-off-by: Tyler Reid <tyler.reid@grafana.com>

Merge remote-tracking branch 'upstream/master' into series-limits

076571c

pracucci approved these changes May 20, 2021

View reviewed changes

Clean white noise

29a0fcf

Signed-off-by: Marco Pracucci <marco@pracucci.com>

pracucci enabled auto-merge (squash) May 20, 2021 10:40

pracucci merged commit da11bff into cortexproject:master May 20, 2021

56quarters mentioned this pull request Jul 6, 2021

Add per-user query metrics for series and bytes returned #4343

Merged

3 tasks

	mapL := len(pql.uniqueSeries)
	return mapL
	return len(pql.uniqueSeries)

	// chunks from ingesters and block storage
	// chunks from ingesters and blocks storage.

		When running Cortex chunks storage: limit enforced in the ingesters only and it's a per-instance limit.
		When running Cortex blocks storage: limit enforced in the queriers both on samples fetched from ingesters and store-gateways (long-term storage).

	// Push a number of series below the max series limit. Each series has 1 sample.
	// Push a number of series below the max series limit.

	errMaxSeriesHit = "The query hit the max number of series limit while fetching chunks (limit: %d)"
	errMaxSeriesHit = "The query hit the max number of series limit (limit: %d)"

	// UniqueSeries returns the count of unique series seen by this query limiter.
	// uniqueSeriesCount returns the count of unique series seen by this query limiter.

Add support for Max Series Per Query for block storage and streaming ingesters #4179

Add support for Max Series Per Query for block storage and streaming ingesters #4179

Conversation

treid314 commented May 12, 2021 • edited Loading

pracucci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pracucci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pracucci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pstibrany left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pstibrany left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pracucci left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

treid314 commented May 12, 2021 •

edited

Loading