Add sequence processing to warp operator #3879

stiepan · 2022-05-09T21:23:19Z

Category:

New feature (non-breaking change which adds functionality)

Description:

Adds FHWC FDHWC processing to the warp affine operator. The transformation matrix can be specified per-frame and provided either as a positional or named argument.

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

If enabling sequence processing and per-frame named (CPU) matrix argument only, the PR would have 2 lines of code.
However, matrix can be specified as a positional argument that can reside in GPU memory. This required extending the SequenceOperator functionality with broadcasting of per-sample positional arguments. However, for now, we still require GPU batches to be contiguous, which further required the SequenceOperator to actually allocate some tensorlists and perform copies. Further, the workspace is not cleared between iterations now.

The utility for testing sequence processing ops with per-frame args required some changes to specify positional arguments and the intended device:
#3901

To mark positional GPU argument as per-frame, per-frame operator was extended to work with GPU backend here:
#3900

The order property of TensorVector is not reflected properly by move assignment/constructor for TensorVector:
#3899

Checklist

Tests

Documentation

DALI team only

Requirements

Implements new requirements
Affects existing requirements
N/A

REQ IDs: WAFFINE.09

JIRA TASK: DALI-2506

mzient · 2022-05-13T12:12:16Z

dali/pipeline/operator/sequence_operator.h

@@ -55,12 +110,13 @@ inline bool is_per_frame(const TensorVector<CPUBackend> &arg_input) {
 * resize the outputs.
 */
 template <typename Backend>
-class SequenceOperator : public Operator<Backend> {
+class SequenceOperator : public Operator<Backend>, public SampleBroadcasting<Backend> {


Wouldn't this suffice?

Suggested change

class SequenceOperator : public Operator<Backend>, public SampleBroadcasting<Backend> {

class SequenceOperator : public Operator<Backend>, protected SampleBroadcasting<Backend> {

You're re-exposing the function BroadcastSamples anyway.

klecki

Looks like there is copy-paste error in the ExpandInput when unfolding, otherwise mostly stylistic comments and nitpicks.

klecki · 2022-05-16T11:05:32Z

dali/pipeline/workspace/workspace.h

@@ -72,6 +72,10 @@ class ArgumentWorkspace {
    return *it->second.tvec;
  }

+  TensorVector<CPUBackend>& UnsafeMutableArgumentInput(const std::string &arg_name) {
+    return const_cast<TensorVector<CPUBackend>&>(ArgumentInput(arg_name));


Not sure if I'm fan of this const cast (+ calling a const method from non-const one).

Yeah, I think we should rethink what constness means in workspaces. But it's beyond the scope of this PR and we already have a similar functions for regular inputs.

klecki · 2022-05-16T11:16:55Z

dali/pipeline/operator/sequence_operator.h

+template <typename BatchType>
+struct BatchBackend;
+
+template <template <typename> class BatchContainer, typename Backend>
+struct BatchBackend<BatchContainer<Backend>> {
+  using Type = Backend;
+};
+
+template <typename BatchType>
+using batch_backend_t =
+    typename BatchBackend<std::remove_cv_t<std::remove_reference_t<BatchType>>>::Type;


This looks more generic than just sequence_operator.

Does a particular suitable file come to your head?

Actually, I have mixed feelings, because it can backfire terribly. Effectively, it's just a utility that extracts the type template argument from a single-type template. Just see:

std::mutex m; std::lock_guard<std::mutex> g(m); static_assert(std::is_same<batch_backend_t<decltype(g)>, std::mutex>::value);

If we were to make it a general utility, we could make Backend a typedef inside classes templated against Backend (Buffer, TensorXxx, Operator) and then this trait could be defined as

template <typename X> using backend_t = typename X::Backend;

or possibly

template <typename X> using backend_t = typename std::remove_cv_t<std::remove_reference_t<X>>::Backend;

That would be safe against accidental use on something that's not related to backend.

Having said all that, I'm fine with keeping it as is in this limited scope - perhaps this utility should even be moved inside the class, so it's not (ab)used elsewhere.

Okay, so I moved it to the type_traits.h and added enable_if_t<is_batch_container<...>> guard, this way it can be reused and not abused.

klecki · 2022-05-16T12:17:24Z

dali/pipeline/operator/sequence_shape.h

+}
+
+template <typename Type>
+void setup_expanded_like(const Type &batch, Type &expanded_batch) {


I have a general remark, which might be considered harmful for this PR, but I think we tend to suggest the FunctionName(outputs, inputs) order. (The camel case also, but we are totally inconsistent with it, and I also write some utils with snake case.)

Okay, I even had it scribbled in my notes to change the order of arguments to match kernels/operators output/inputs, missed it.

Yup, memcpy order is preferred.

klecki · 2022-05-16T12:37:08Z

dali/pipeline/operator/sequence_shape.h

-  const auto &shape = data.shape();
-  TensorList<Backend> tl;
-  tl.ShareData(data);
+void broadcast_samples(const TensorVector<Backend> &batch, TensorVector<Backend> &expanded_batch,


Can you add a group comment for those overloads of broadcast_samples? I had to look up the num_expanded_samples in the tv_builder_like to see what it is used for. Also, if we expand according to the expand_extents, why do we need the num_expanded_samples?

Also, what is the format of the expand_extents - it's the shapes of the F[C] prefix, right?

As to the format: effectively it is as you say, more broadly the layout does not matter at this point: from batch (batch.num_samples() == expand_extents.num_samples()) we get expanded_batch.num_samples() == expanded_extents.num_elements() by repeating corresponding samples volume(expanded_extents[i]) times.

You are right that we could just compute the num_expanded_samples, it is just that it is already computed in the ExpandDesc (not to recompute it for validation etc.), so I pass it here as an argument. Later on there is even relevant assert in broadcast shapes (assert(num_expanded_samples == expand_extents.num_elements());)

klecki · 2022-05-16T13:26:13Z

dali/pipeline/operator/sequence_shape.h

@@ -266,19 +277,102 @@ inline TensorListShape<> unfold_outer_dims(const TensorListShape<> &shape, int n
  }
 }

+inline TensorListShape<> broadcast_samples(const TensorListShape<> &shape, int num_expanded_samples,


Maybe broadcast_sample_shapes? And please docstring :)

klecki · 2022-05-16T14:09:44Z

dali/test/python/sequences_test_utils.py

+
+
+class ArgCb:
+    def __init__(self, name: Union[str, int], cb: Callable[[SampleDesc], np.ndarray], is_per_frame: bool, dest_device: str = "cpu"):


We lost the documentation of what the callback is supposed to be doing. We know it's just a single arg callback.

klecki · 2022-05-16T14:50:53Z

dali/pipeline/operator/sequence_operator.h

@@ -55,12 +110,13 @@ inline bool is_per_frame(const TensorVector<CPUBackend> &arg_input) {
 * resize the outputs.
 */
 template <typename Backend>
-class SequenceOperator : public Operator<Backend> {
+class SequenceOperator : public Operator<Backend>, public SampleBroadcasting<Backend> {


One can also say, that adding functionality via inheritance instead of composition is not desired, but I'm no OOP evangelist :P

I had in mind the empty base optimization "pattern" (as in CPU case we do not add any members), but it probably makes precisely no noticeable difference here and you have a good point. On the other hand, can this hurt us in this case? Maybe I'll just follow Michał's suggestion to make it protected inheritance.

I got composition suggested once or twice, but I'm not asking for a rewrite, just saying it's a bit artificial

I'm fine either way, with inheritance having the benefit of less typing (and being already implemented).

changed it to protected inheritance

klecki · 2022-05-16T15:34:27Z

dali/pipeline/operator/sequence_operator.h

-    auto expanded_arg = ExpandArgumentLikeInput(arg_input, arg_name, input_idx);
-    auto expanded_handle = std::make_shared<TensorVector<CPUBackend>>(std::move(expanded_arg));
-    ExpandedAddArgument(arg_name, std::move(expanded_handle));
+    ExpandArgumentLikeInput(arg_input, ExpandedArg(arg_name), arg_name, input_idx);


This is mostly the change from recreating the workspace every time to setting it up once and than just putting new expanded data in the tensor vector in every iteratorion?

Yes, it is exactly that.

klecki · 2022-05-16T15:56:28Z

dali/pipeline/operator/sequence_operator.h

-                                [&](const auto &input) { return UnfoldBatch(input, input_desc); });
+      ProcessInput(ws, input_idx, [&](const auto &input) {
+        auto &expanded_input = ExpandedInput<detail::batch_backend_t<decltype(input)>>(input_idx);
+        UnfoldBatch(input, expanded_input, ref_expand_desc);


The input_desc changed to ref_expand_desc. Is this correct?

klecki · 2022-05-16T16:24:29Z

dali/test/python/sequences_test_utils.py

@@ -15,6 +15,7 @@
 import os
 import random
 import numpy as np
+from typing import List, Union, Callable


Btw, I am not reviewing this file too much, as it's in the other PR.

mzient · 2022-05-19T11:03:32Z

dali/pipeline/operator/sequence_operator.h

+    }
+  }
+
+  void SetupExpandedWorkspace(const workspace_t<Backend> &ws) {


Since it's only run once, how about

Suggested change

void SetupExpandedWorkspace(const workspace_t<Backend> &ws) {

void InitializeExpandedWorkspace(const workspace_t<Backend> &ws) {

?
(and in all related places)

mzient · 2022-05-19T11:05:07Z

dali/pipeline/operator/sequence_operator.h

@@ -607,6 +691,7 @@ class SequenceOperator : public Operator<Backend> {
 private:
  int expand_like_idx_ = -1;
  bool is_expanding_ = false;
+  bool is_expanded_set_up_ = false;


expanded what?

Suggested change

bool is_expanded_set_up_ = false;

bool is_expanded_ws_initialized_ = false;

mzient · 2022-05-19T11:09:00Z

dali/pipeline/operator/sequence_shape.h

+template <typename Type>
+void setup_expanded_like(const Type &batch, Type &expanded_batch) {


This isn't a generic function that can work with anything - it has very specific requirements for the target type and that should be somehow reflected in the template parameter name.

Suggested change

template <typename Type>

void setup_expanded_like(const Type &batch, Type &expanded_batch) {

template <typename BatchObject>

void setup_expanded_like(const BatchObject &batch, BatchObject &expanded_batch) {

mzient · 2022-05-19T11:09:56Z

dali/pipeline/operator/sequence_shape.h

+template <typename Type>
+std::shared_ptr<Type> expanded_like(const Type &batch) {


Suggested change

template <typename Type>

std::shared_ptr<Type> expanded_like(const Type &batch) {

template <typename BatchObject>

std::shared_ptr<BatchObject> expanded_like(const BatchObject &batch) {

mzient · 2022-05-19T11:15:27Z

dali/pipeline/operator/sequence_shape.h


-  void push(const SliceView &view) {
+  void SetNext(const SliceView &view) {


Suggested change

void SetNext(const SliceView &view) {

void SetNext(const SliceView &view) {

assert(size < tv_.num_samples());

mzient · 2022-05-19T11:15:44Z

dali/pipeline/operator/sequence_shape.h

-  TensorVector<Backend> take() {
-    assert(size == tv_.num_samples());
-    return std::move(tv_);
+  int Size() {


It's not really a size - that is constant and defined by tv_.num_samples(), isn't it?

Suggested change

int Size() {

int NextFrameIdx() const {

mzient · 2022-05-19T11:17:02Z

dali/pipeline/operator/sequence_shape.h

  }

-  TensorVector<Backend> tv_;
+ private:
+  TensorVector<Backend> &tv_;
  int size = 0;


Suggested change

int size = 0;

int next_ = 0;

?

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com> Add warp video test for named matrix param Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com> Enable positional arguments testing in the sequence test suite Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com> Accomodate existing per frame tests to the updated sequence util Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

This reverts commit f9be826. Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

stiepan · 2022-05-23T16:15:48Z

!build

dali-automaton · 2022-05-23T16:20:08Z

CI MESSAGE: [4911387]: BUILD STARTED

dali-automaton · 2022-05-23T17:38:57Z

CI MESSAGE: [4911387]: BUILD PASSED

mzient · 2022-05-26T14:48:43Z

dali/pipeline/data/type_traits.h

+template <template <typename> class BatchContainer, typename Backend>
+struct BatchBackend<BatchContainer<Backend>,


Meh. I won't fiight for it, but I think that adding a Backend typedef to relevant types would be a little better.

mzient · 2022-05-26T15:32:17Z

dali/pipeline/operator/sequence_shape.h

+  const auto &shape = batch.shape();
+  auto broadcast_shape = broadcast_sample_shapes(shape, num_expanded_samples, expand_extents);
+  expanded_batch.SetLayout(batch.GetLayout());
+  expanded_batch.Resize(broadcast_shape, batch.type());


Suggested change

expanded_batch.Resize(broadcast_shape, batch.type());

expanded_batch.set_order(stream);

expanded_batch.Resize(broadcast_shape, batch.type());

Now that we deal with GPU stuff...

mzient · 2022-05-27T12:42:12Z

dali/pipeline/operator/sequence_shape_test.cc

+template <typename BatchType>
+struct BatchBackend;
+
+template <template <typename> class BatchContainer, typename Backend>
+struct BatchBackend<BatchContainer<Backend>> {
+  using Type = Backend;
+};
+
+template <typename BatchType>
+using batch_backend_t =
+    typename BatchBackend<std::remove_cv_t<std::remove_reference_t<BatchType>>>::Type;
+


This is a duplicate - please remove.

Good catch!
done

mzient · 2022-05-27T12:46:21Z

dali/pipeline/operator/sequence_shape_test.cc

+  this->TestUnfolding(*expanded_batch, batch, 2);
+  batch.Resize({{2, 2, 6}, {13, 4, 11}}, DALI_FLOAT);
+  batch.SetLayout("XYZ");
+  this->TestUnfolding(*expanded_batch, batch, 2);


Suggested change

this->TestUnfolding(*expanded_batch, batch, 2);

this->TestUnfolding(*expanded_batch, batch, 1);

this->TestUnfolding(*expanded_batch, batch, 2);

this->TestUnfolding(*expanded_batch, batch, 3);

Can you also check unfolding of just one dim? I think these tests are cheap and unfolding 1 dim is where we want the best possible coverage.
I also suggest testing the corner case when we unfold down to scalars.

There is a separate test case that just unfolds 1 dim (and the same for all three). I added two cases with multiple iters that test 1, 2 and 3 unfolds in a single test case between resizes.

mzient · 2022-05-27T12:47:27Z

dali/pipeline/operator/sequence_shape_test.cc

+  this->TestUnfolding(*expanded_batch, batch, 2);
+  batch.Resize({{2, 2, 6}, {13, 4, 11}, {13, 4, 11}, {13, 4, 11}}, DALI_FLOAT);
+  batch.SetLayout("XYZ");
+  this->TestUnfolding(*expanded_batch, batch, 2);


mzient · 2022-05-27T12:53:00Z

dali/pipeline/operator/sequence_shape_test.cc

+  batch.Resize({{2, 2, 6}, {13, 4, 11}, {13, 4, 11}, {13, 4, 11}}, DALI_FLOAT);
+  batch.SetLayout("XYZ");
+  this->template FillBatch<float>(batch, 7);
+  this->template TestBroadcasting<float>(*expanded_batch, batch, {{}, {}, {}, {}});


Please test broadcasting of scalars.

batch.Resize({{}, {}, {}, {}}});

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

stiepan · 2022-05-29T19:53:34Z

!build

dali-automaton · 2022-05-29T19:55:28Z

CI MESSAGE: [4962767]: BUILD STARTED

dali-automaton · 2022-05-29T20:08:05Z

CI MESSAGE: [4962767]: BUILD FAILED

stiepan · 2022-05-30T07:42:31Z

!build

dali-automaton · 2022-05-30T07:45:29Z

CI MESSAGE: [4965559]: BUILD STARTED

dali-automaton · 2022-05-30T09:13:34Z

CI MESSAGE: [4965559]: BUILD PASSED

* Enable sequence support and per-frame arguments for warp * Add support for broadcasting of accompanying non-sequence inputs * Do not clear the expanded workspace between iterations Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

stiepan force-pushed the warp_sequence_processing branch from 562afde to f3a78bb Compare May 9, 2022 21:31

stiepan marked this pull request as ready for review May 10, 2022 09:21

stiepan changed the title ~~Warp sequence processing~~ Add sequence processing to warp operator May 10, 2022

stiepan marked this pull request as draft May 10, 2022 09:49

stiepan marked this pull request as ready for review May 12, 2022 09:14

mzient self-assigned this May 12, 2022

jantonguirao assigned klecki May 12, 2022

mzient reviewed May 13, 2022

View reviewed changes

This was referenced May 13, 2022

Support specifying per-frame positional arguments in sequence processing test utility #3901

Merged

Enable per-frame operator on GPU #3900

Merged

stiepan force-pushed the warp_sequence_processing branch from f4eb224 to 2af4b08 Compare May 16, 2022 12:24

klecki reviewed May 16, 2022

View reviewed changes

mzient reviewed May 19, 2022

View reviewed changes

stiepan added 11 commits May 20, 2022 16:53

Enable sequence support and per-frame arguments for warp

4fef91f

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Cpu positional input broadcasting

beded02

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Broadcast GPU arg

db3441b

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Disable convolutions

feb6ae8

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Add a method to access mutable argument input in the workspace

eb29b70

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Reusing containers from expanded workspace

c45a421

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Clean-up, assure setting required properties

7781f98

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Separate GPU broadcasting from CPU case

2007416

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Revert "Disable convolutions"

8175f8e

This reverts commit f9be826. Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Cleanup

0606cee

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

stiepan added 4 commits May 20, 2022 16:57

Add broadcasting tests

235b61e

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

TensorList broadcasting tests

dddbe17

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Move safeguarded batch_backend_t to type traits

6780b04

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Use protected inheritance for broadcasting mixin

088db0c

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

stiepan force-pushed the warp_sequence_processing branch from 2af4b08 to 088db0c Compare May 23, 2022 08:55

stiepan added 4 commits May 23, 2022 11:08

Use input_desc not ref_input_desc to unfold batch

2a337c3

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Renaming

a2d4e18

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Use memcpy order when passing expanded batch and batch

3f4ffb8

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Add docs

6ed1c2e

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

klecki approved these changes May 24, 2022

View reviewed changes

mzient reviewed May 26, 2022

View reviewed changes

mzient reviewed May 27, 2022

View reviewed changes

stiepan added 2 commits May 29, 2022 20:50

Set ws order to boradcast tensorlist, remove duplicate backend_t

e9c3021

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

Add a few more unfold tests and broadcasting scalar tests

a89bf3e

Signed-off-by: Kamil Tokarski <ktokarski@nvidia.com>

mzient approved these changes May 30, 2022

View reviewed changes

stiepan merged commit c30da52 into NVIDIA:main May 30, 2022

	class SequenceOperator : public Operator<Backend>, public SampleBroadcasting<Backend> {
	class SequenceOperator : public Operator<Backend>, protected SampleBroadcasting<Backend> {



		class ArgCb:
		def __init__(self, name: Union[str, int], cb: Callable[[SampleDesc], np.ndarray], is_per_frame: bool, dest_device: str = "cpu"):

	void SetupExpandedWorkspace(const workspace_t<Backend> &ws) {
	void InitializeExpandedWorkspace(const workspace_t<Backend> &ws) {

	bool is_expanded_set_up_ = false;
	bool is_expanded_ws_initialized_ = false;

		template <typename Type>
		void setup_expanded_like(const Type &batch, Type &expanded_batch) {

		template <typename Type>
		std::shared_ptr<Type> expanded_like(const Type &batch) {


		void push(const SliceView &view) {
		void SetNext(const SliceView &view) {

	void SetNext(const SliceView &view) {
	void SetNext(const SliceView &view) {
	assert(size < tv_.num_samples());

		template <template <typename> class BatchContainer, typename Backend>
		struct BatchBackend<BatchContainer<Backend>,

	expanded_batch.Resize(broadcast_shape, batch.type());
	expanded_batch.set_order(stream);
	expanded_batch.Resize(broadcast_shape, batch.type());

Add sequence processing to warp operator #3879

Add sequence processing to warp operator #3879

Conversation

stiepan commented May 9, 2022 • edited Loading

Category:

Description:

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

Checklist

Tests

Documentation

DALI team only

Requirements

Choose a reason for hiding this comment

klecki left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mzient May 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stiepan May 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stiepan May 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stiepan May 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stiepan May 23, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mzient May 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mzient May 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mzient May 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mzient May 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stiepan commented May 23, 2022

dali-automaton commented May 23, 2022

dali-automaton commented May 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mzient May 27, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stiepan commented May 9, 2022 •

edited

Loading

mzient May 20, 2022 •

edited

Loading

stiepan May 23, 2022 •

edited

Loading

stiepan May 23, 2022 •

edited

Loading

stiepan May 17, 2022 •

edited

Loading

stiepan May 23, 2022 •

edited

Loading

mzient May 19, 2022 •

edited

Loading

mzient May 19, 2022 •

edited

Loading

mzient May 19, 2022 •

edited

Loading

mzient May 19, 2022 •

edited

Loading

mzient May 27, 2022 •

edited

Loading

mzient May 27, 2022 •

edited

Loading