Spark 3.3: Support storage-partitioned joins #6371

aokolnychyi · 2022-12-06T23:42:32Z

This PR adds support for storage-partitioned joins in Spark 3.3.

aokolnychyi · 2022-12-07T00:14:00Z

core/src/main/java/org/apache/iceberg/Partitioning.java

   * @param specs one or many specs
   * @return the constructed grouping key type
   */
-  public static StructType groupingKeyType(Collection<PartitionSpec> specs) {
-    return buildPartitionProjectionType("grouping key", specs, commonActiveFieldIds(specs));
+  public static StructType groupingKeyType(Schema schema, Collection<PartitionSpec> specs) {


We need to take into account the schema as we may project only particular columns.

aokolnychyi · 2022-12-07T00:14:35Z

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/Spark3Util.java

@@ -255,74 +256,90 @@ public static org.apache.iceberg.Table toIcebergTable(Table table) {
    return sparkTable.table();
  }

+  public static Transform[] toTransforms(Schema schema, List<PartitionField> fields) {
+    SpecTransformToSparkTransform visitor = new SpecTransformToSparkTransform(schema);


I pulled the anonymous class below into SpecTransformToSparkTransform to reuse here.

aokolnychyi · 2022-12-07T00:18:05Z

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java

@@ -42,4 +42,9 @@ private SparkSQLProperties() {}
  // Controls whether to check the order of fields during writes
  public static final String CHECK_ORDERING = "spark.sql.iceberg.check-ordering";
  public static final boolean CHECK_ORDERING_DEFAULT = true;
+
+  // Controls whether to preserve the existing grouping of data while planning splits
+  public static final String PRESERVE_DATA_GROUPING =


One alternative name was spark.sql.iceberg.split.preserve-partition-boundaries. I discarded it because we are not really preserving partition boundaries if there are multiple specs. Also, Spark will push down join keys in the future so we won't really respect partition boundaries as such (arguable).

That being said, I'll think more tomorrow. Ideas are always welcome.

Name is fine to me, Ideally this isn't something that get's actually configured by the end user

I am afraid this one will be pretty public. Users will have to explicitly enable this as we don't know if Spark can benefit from the reported distribution and skip shuffles. Hence, we disable it by default to avoid any performance regressions caused by less dense packing of splits.

I think this name makes sense.

aokolnychyi · 2022-12-07T00:20:24Z

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java

+  }
+
+  @SuppressWarnings("unchecked")
+  protected synchronized List<T> tasks() {


Made tasks and taskGroups synchronized to follow SparkCopyOnWriteScan that consumes this logic now.

aokolnychyi · 2022-12-07T00:21:32Z

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScanBuilder.java

@@ -366,9 +365,9 @@ public Scan buildCopyOnWriteScan() {

    Schema expectedSchema = schemaWithMetadataColumns();

-    TableScan scan =
+    BatchScan scan =


Had to change this so that SparkCopyOnWriteScan can use the same task planning.

aokolnychyi · 2022-12-07T00:22:09Z

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/StructInternalRow.java

@@ -356,4 +357,23 @@ private <T> GenericArrayData fillArray(

    return new GenericArrayData(array);
  }
+
+  @Override
+  public boolean equals(Object other) {


Had to implement as Spark collects grouping keys and checks their equality.

What is the contract for equals for InternalRow? This probably works, but does it indicate a bug in Spark where Spark assumes that InternalRow implementations consider one another equal?

It is a bug in Spark that will be fixed. Spark uses groupBy in one place, which relies on equals.
All other places use InternalRowSet.

aokolnychyi · 2022-12-07T00:22:49Z

spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/SparkTestBase.java

+    return executeAndKeepPlan(() -> sql(query, args));
+  }
+
+  protected SparkPlan executeAndKeepPlan(Action action) {


A utility method to fetch what SparkPlan was actually executed.

spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestStoragePartitionedJoins.java

aokolnychyi · 2022-12-07T00:38:50Z

...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestAlterTablePartitionFields.java

@@ -404,15 +404,15 @@ public void testSparkTableAddDropPartitions() throws Exception {
    assertPartitioningEquals(sparkTable(), 1, "bucket(16, id)");

    sql("ALTER TABLE %s ADD PARTITION FIELD truncate(data, 4)", tableName);
-    assertPartitioningEquals(sparkTable(), 2, "truncate(data, 4)");
+    assertPartitioningEquals(sparkTable(), 2, "truncate(4, data)");


I had to change Spark3Util.toTransforms to match TruncateFunction, which expects width first.

aokolnychyi · 2022-12-07T00:41:18Z

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/Spark3Util.java

+    @Override
+    public Transform truncate(String sourceName, int sourceId, int width) {
+      NamedReference column = Expressions.column(quotedName(sourceId));
+      return Expressions.apply("truncate", Expressions.literal(width), column);


Passing width first to match TruncateFunction. Otherwise, the function catalog won't resolve.

So this is ok as a change because the previous version wouldn't have actually worked correct? Just making sure we aren't breaking any api here

I think the only place that would change is string output of partitioning in SparkTable. Otherwise, we handle both combinations in the TruncateTransform extractor.

aokolnychyi · 2022-12-07T00:42:31Z

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java

+  // Controls whether to preserve the existing grouping of data while planning splits
+  public static final String PRESERVE_DATA_GROUPING =
+      "spark.sql.iceberg.split.preserve-data-grouping";
+  public static final boolean PRESERVE_DATA_GROUPING_DEFAULT = false;


The default is false to avoid introducing a performance regression as split planning may be less dense.

My question here would be should we really have the default be false? I have no problem with the name but It feels like this is probably always the right decision for a scan with possible joins. We should probably look into this more in the future but my guess is that if a query requires partition columns, we should group by those columns.

That's my long way to say i'm fine with the default being false for now, but I think it should probably be true.

My worry is performance regressions that this may cause. There may be substantially more splits if this config is on. In order to benefit from SPJ, joins must have equality conditions on partition columns. Spark will propagate join conditions in the future. Our long-term plan may be to check if v2 bucketing enabled and whether we have join conditions on partition columns to return true by default. Even that will be suboptimal cause we don't know if SPJ would actually apply or we simply produced less dense splits.

I think long term we will want to get rid of this, but that will require some help from Spark. Ideally, Spark should tell the source whether or not it cares about preserving grouping, and on which columns it matters. If we had that information we wouldn't need this at all.

aokolnychyi · 2022-12-07T00:45:12Z

cc @sunchao @flyrain @szehon-ho @RussellSpitzer @rdblue

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java

spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestStoragePartitionedJoins.java

RussellSpitzer · 2022-12-07T16:17:54Z

spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestStoragePartitionedJoins.java

+
+  // TODO: add tests for truncate transforms once SPARK-40295 is released
+  // TODO: add tests for cases when one side contains a subset of keys once Spark supports this
+


Should we have some tests with multiple buckets? or multiple partition transforms?

I see the example below

Does this work with a non Iceberg Source as part of the join?

In theory it's possible. Spark will check if both sides of the SPJ have compatible partition transforms, via checking whether the V2 function identifiers are the same. So if the non-Iceberg source reports the same functions to Spark and use them in partition transforms, it could work (although I'm not sure whether this is a common use case).

We will definitely need to support distributing the other side using the Iceberg function catalog. A common use case for this is MERGE. The incoming relation may be a view, which will never be distributed in a compatible way with Iceberg tables. I hope Spark would be smart enough to shuffle just the smaller relation using the reported partitioning from the target table.

Unfortunately, it does not work even if I distribute the incoming data manually using the Iceberg function catalog. We must have KeyGroupPartitioning on both sides, which can only be reported by data sources.

It uses Inner Join for merges without a "when not matching"

I would think for outer join we would similarly just keep all of the non-matching partitions separate but I haven't really looked at the code

Maybe we could just distribute all the non-matching rows "randomly" into the N partitions, but I'll need to check more closely on Spark side. It's a bit complicated.

@sunchao, we can't use the newly added logic for pushing down common keys cause we don't know the keys in the relation we will shuffle (until we actually shuffle it)?

I'd be great to at least support cases when users manually repartition the incoming source relation using the Iceberg function catalog. I believe that would produce HashPartitioning or RangePartitioning on the other side.

@sunchao, we can't use the newly added logic for pushing down common keys cause we don't know the keys in the relation we will shuffle (until we actually shuffle it)?

Correct. We don't know the keys until runtime, which is too late.

I'd be great to at least support cases when users manually repartition the incoming source relation using the Iceberg function catalog. I believe that would produce HashPartitioning or RangePartitioning on the other side.

Let me keep a note, and do some experiment on these ideas.

core/src/main/java/org/apache/iceberg/Partitioning.java

RussellSpitzer

This looks good to me and it's exciting to see the culmination of all the work @aokolnychyi and @sunchao have been doing!

I have a few small notes,
1.) On the grouping key type method in Partitioning.java, we probably want to write a little more documentation about the "schema" argument and what that means when it is null.
2.) There are a lot of Spark Sql configurations required in the test suite, how many of these are actually required for SPJ, are any of those potentially being removed in the future?
3.) Does SPJ work when one side is not an Iceberg Table? All the tests look like Iceberg tables vs Iceberg tables.

zinking · 2022-12-12T11:37:44Z

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java

+        List<T> plannedTasks = Lists.newArrayList();
+
+        for (ScanTask task : taskIterable) {
+          ValidationException.check(


is this desirable to do the validation in the for LOOP? the exception will be thrown anyway.

This check is to provide a reasonable exception if there is a task of an unsupported type. Since we may have a combination of different tasks, each task is being validated.

I don't think it would be super expensive to perform this validation but it is kind of optional too.

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java

aokolnychyi · 2022-12-19T22:00:58Z

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java

+  }
+
+  @Override
+  protected StructType groupingKeyType() {


One debatable point if groupingKeyType should be part of equals and hashCode.

core/src/main/java/org/apache/iceberg/Partitioning.java

api/src/main/java/org/apache/iceberg/types/Types.java

rdblue · 2022-12-20T16:34:21Z

core/src/test/java/org/apache/iceberg/TestPartitioning.java

+    TestTables.TestTable table =
+        TestTables.create(tableDir, "test", SCHEMA, BY_CATEGORY_DATA_SPEC, V1_FORMAT_VERSION);
+
+    Schema projectedSchema = table.schema().select("id", "data");


Ah, so is the idea to use the pushed projection to limit the fields used? I think that's a great idea!

Correct. Otherwise, we will break Spark because we can report non-projected columns.

rdblue · 2022-12-20T16:37:33Z

...park-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteDelete.java

+    Assert.assertEquals("Should have 3 snapshots", 3, Iterables.size(table.snapshots()));
+
+    Snapshot currentSnapshot = table.currentSnapshot();
+    validateCopyOnWrite(currentSnapshot, "1", "1", "1");


Nit: it would be nice to not pass the counts as strings...

Not that we should fix it in this PR. But maybe we can introduce a version of validateCopyOnWrite that accepts ints and calls the string version.

This is what we did historically to simplify the validation as we compare summary map (i.e. strings).
I'll check.

Doing this properly would touch lots of existing places. I'll follow up to fix this separately.

rdblue · 2022-12-20T16:39:56Z

core/src/main/java/org/apache/iceberg/Partitioning.java

-  public static StructType groupingKeyType(Collection<PartitionSpec> specs) {
-    return buildPartitionProjectionType("grouping key", specs, commonActiveFieldIds(specs));
+  public static StructType groupingKeyType(Schema schema, Collection<PartitionSpec> specs) {
+    return buildPartitionProjectionType("grouping key", specs, commonActiveFieldIds(schema, specs));


How do we limit the specs passed to this method to just the ones that are used by manifests that are scanned during planning? (I may answer this myself later, but I want to write the question down)

It is determined based on scan tasks that match our filter in SparkPartitioningAwareScan.

If a table has multiple specs but we scan only tasks that belong to one spec, we should take into account only the one that is being queried.

...park-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteUpdate.java

rdblue · 2022-12-20T17:13:20Z

...ava/org/apache/iceberg/spark/extensions/TestStoragePartitionedJoinsInRowLevelOperations.java

+              executeAndKeepPlan(
+                  "DELETE FROM %s t WHERE "
+                      + "EXISTS (SELECT 1 FROM %s s WHERE t.id = s.id AND t.dep = s.dep) AND "
+                      + "dep = 'hr'",


Why restrict this to just hr? It seems like that could cause everything to fit in a single task, which Spark handles specially in some cases. If we were to test a delete in both departments, then I think this would be more likely to test what we want in Spark.

SPJ requires the same set of keys on both sides. The delta relation contains only records in hr. If I skip this, SPJ would not apply as the target has both hr and hardware. I make sure we don't have a single partition by configuring the open file cost and writing more than 1 file.

aokolnychyi · 2022-12-20T17:29:09Z

spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestStoragePartitionedJoins.java

+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class TestStoragePartitionedJoins extends SparkTestBaseWithCatalog {


@rdblue, this is the primary test suite that covers different transforms. I had just a few tests for row-level operations to make sure it works there too.

aokolnychyi · 2022-12-20T18:30:28Z

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java

+    return groupingKeyType;
+  }
+
+  private StructType computeGroupingKeyType() {


This method only exists to avoid Spotless formatting that splits this into multiple lines.

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java

rdblue · 2022-12-23T00:12:14Z

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/Spark3Util.java

@@ -473,6 +490,10 @@ private static String parentName(String[] fieldNames) {
    return null;
  }

+  public static String describe(List<org.apache.iceberg.expressions.Expression> exprs) {
+    return exprs.stream().map(Spark3Util::describe).collect(Collectors.joining(", "));


Not a blocker, but should we sanitize the expressions using ExpressionUtil.sanitize here? There is a risk of logging IDs if we log this.

Actually, this is used by toString, so it looks like the right behavior is to not sanitize because this is put into Spark plans where the full filter should be shown. We may want to introduce a sanitized describe later though, for logging purposes.

That's exactly my thought process too. I wanted to sanitize first but then realized it is used in the plan and kept the existing behavior.

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java

rdblue · 2022-12-23T00:40:50Z

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java

+  protected synchronized List<T> tasks() {
+    if (tasks == null) {
+      try (CloseableIterable<? extends ScanTask> taskIterable = scan.planFiles()) {
+        List<T> plannedTasks = Lists.newArrayList();


Minor: It looks like this was already done in SparkBatchQueryScan, but I generally prefer to keep things as a CloseableIterable as long as possible. This could probably return a CloseableIterable instead. That might help if we want to actually forget about tasks and gc them after runtime filtering.

I agree with you but I am not sure it applies here. We eagerly loaded tasks from day 1. They are used in lots of places and will need to be materialized immediately to compute stats, determine grouping key, etc.

I think we will be able to gc filtered out tasks in the current implementation as well.

rdblue · 2022-12-23T00:42:54Z

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java

+                groupingKeyType());
+        StructLikeSet plannedGroupingKeys = groupingKeys(plannedTaskGroups);
+
+        LOG.debug(


Is this useful? Maybe the number of unique grouping keys and the type.

It may be helpful to know how many splits we had before and after runtime filtering.

rdblue

This all looks good to me. I had a couple of additional questions in the new SparkPartitionginAwareScan about the transform construction, but I think that should be fairly simple to solve. Nice work, @aokolnychyi and @sunchao!

aokolnychyi · 2022-12-24T18:40:50Z

Thanks for reviewing, @RussellSpitzer @sunchao @zinking @rdblue!

github-actions bot added core spark labels Dec 6, 2022

aokolnychyi commented Dec 7, 2022

View reviewed changes

spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestStoragePartitionedJoins.java Show resolved Hide resolved

aokolnychyi force-pushed the spj branch from ddb12ea to d62a5a1 Compare December 7, 2022 00:36

aokolnychyi commented Dec 7, 2022

View reviewed changes

RussellSpitzer reviewed Dec 7, 2022

View reviewed changes

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java Show resolved Hide resolved

RussellSpitzer reviewed Dec 7, 2022

View reviewed changes

spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestStoragePartitionedJoins.java Show resolved Hide resolved

RussellSpitzer reviewed Dec 7, 2022

View reviewed changes

core/src/main/java/org/apache/iceberg/Partitioning.java Outdated Show resolved Hide resolved

RussellSpitzer approved these changes Dec 7, 2022

View reviewed changes

zinking reviewed Dec 12, 2022

View reviewed changes

aokolnychyi force-pushed the spj branch from d62a5a1 to 700ff2e Compare December 19, 2022 16:08

Spark 3.3: Support storage-partitioned joins

e365355

aokolnychyi force-pushed the spj branch from 700ff2e to e365355 Compare December 19, 2022 16:10

Minor updates

fc181f9

github-actions bot added the API label Dec 19, 2022

aokolnychyi commented Dec 19, 2022

View reviewed changes

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java Outdated Show resolved Hide resolved

aokolnychyi commented Dec 19, 2022

View reviewed changes

rdblue reviewed Dec 20, 2022

View reviewed changes

core/src/main/java/org/apache/iceberg/Partitioning.java Show resolved Hide resolved

rdblue reviewed Dec 20, 2022

View reviewed changes

api/src/main/java/org/apache/iceberg/types/Types.java Outdated Show resolved Hide resolved

rdblue reviewed Dec 20, 2022

View reviewed changes

...park-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestCopyOnWriteUpdate.java Outdated Show resolved Hide resolved

rdblue reviewed Dec 20, 2022

View reviewed changes

aokolnychyi commented Dec 20, 2022

View reviewed changes

rdblue reviewed Dec 22, 2022

View reviewed changes

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkSQLProperties.java Outdated Show resolved Hide resolved

rdblue reviewed Dec 23, 2022

View reviewed changes

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java Outdated Show resolved Hide resolved

rdblue reviewed Dec 23, 2022

View reviewed changes

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java Outdated Show resolved Hide resolved

rdblue reviewed Dec 23, 2022

View reviewed changes

spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkPartitioningAwareScan.java Outdated Show resolved Hide resolved

rdblue reviewed Dec 23, 2022

View reviewed changes

rdblue approved these changes Dec 23, 2022

View reviewed changes

Review

4f18833

github-actions bot removed the API label Dec 24, 2022

aokolnychyi merged commit 31c8011 into apache:master Dec 24, 2022

aokolnychyi mentioned this pull request Dec 24, 2022

Support bucket table for Iceberg #430

Closed

shrukul mentioned this pull request Apr 18, 2023

[Feature Request] Add support for Storage Partitioned Joins (SPJ) introduced in Spark v3.3 delta-io/delta#1698

Open

3 tasks

sunchao pushed a commit to sunchao/iceberg that referenced this pull request May 10, 2023

Spark 3.3: Support storage-partitioned joins (apache#6371)

7649595


		// TODO: add tests for truncate transforms once SPARK-40295 is released
		// TODO: add tests for cases when one side contains a subset of keys once Spark supports this

Spark 3.3: Support storage-partitioned joins #6371

Spark 3.3: Support storage-partitioned joins #6371

Conversation

aokolnychyi commented Dec 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aokolnychyi Dec 7, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aokolnychyi Dec 7, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aokolnychyi Dec 24, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aokolnychyi Dec 7, 2022 • edited

Choose a reason for hiding this comment

aokolnychyi Dec 7, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aokolnychyi Dec 7, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aokolnychyi commented Dec 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aokolnychyi Dec 7, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussellSpitzer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdblue Dec 20, 2022 • edited

Choose a reason for hiding this comment

aokolnychyi Dec 20, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aokolnychyi Dec 20, 2022 • edited

Choose a reason for hiding this comment

aokolnychyi Dec 20, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdblue Dec 23, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdblue left a comment

Choose a reason for hiding this comment

aokolnychyi commented Dec 24, 2022

aokolnychyi Dec 7, 2022 •

edited

aokolnychyi Dec 7, 2022 •

edited

aokolnychyi Dec 24, 2022 •

edited

aokolnychyi Dec 7, 2022 •

edited

aokolnychyi Dec 7, 2022 •

edited

aokolnychyi Dec 7, 2022 •

edited

aokolnychyi Dec 7, 2022 •

edited

rdblue Dec 20, 2022 •

edited

aokolnychyi Dec 20, 2022 •

edited

aokolnychyi Dec 20, 2022 •

edited

aokolnychyi Dec 20, 2022 •

edited

rdblue Dec 23, 2022 •

edited