End-to-end test coverage for CLI commands output #304

hidmic · 2019-07-19T15:27:40Z

This pull request aims to add full end-to-end test coverage to ~~all~~ several ros2 CLI commands:

	topic	service	action	~~param~~	msg	srv	interface	node	~~run~~	~~component~~	~~lifecycle~~	pkg	~~daemon~~	~~launch~~
Correct output	✓	✓	✓	✗	✓	✓	✓	✓	✗	✗	✗	✓	✗	✗
Graceful termination	✓	✓	✓	✗	✓	✓	✓	✓	✗	✗	✗	✓	✗	✗
Error handling	✓	✓	✓	✗	✓	✓	✓	✓	✗	✗	✗	✓	✗	✗

~~Note: this table will be updated as progress is made.~~ Test coverage for commands that have been struck through i.e. param, component, lifecycle, etc., will not be provided here but in follow-up PRs. These commands have side effects beyond their execution lifetime that have to be checked in ways that are specific to each command. Thus, these don't lend themselves well to be integrated with the "standardized" testing device used for other commands.

Connected to ros2/ros2#739.

hidmic · 2019-07-19T15:29:21Z

There's still the discussion as to whether this is enough for end-to-end testing (let alone unit tests which are not covered by this PR) or even the behavior currently shown is appropriate. We'll have to iterate on it as we move forward.

hidmic · 2019-07-19T15:40:55Z

Running CI to uncover non-Linux, platform specific issues (build up to test_ros2cli, test test_ros2cli, including opensplice):

Linux (Linux-only first to save CI time)
Linux-aarch64
macOS
Windows

hidmic · 2019-07-26T18:28:40Z

Ok, I'll stop here for the reasons exposed in this PR's description and because this is getting larger than I'm usually comfortable with.

dirk-thomas

I haven't read the full patch yet - just a few high level comments:

The size of the tests makes me wonder if we really can't write them more compact? Is there anything we could do API / feature wise which would help reduce this?
What is the reason that all the tests are bundled in the separate package test_ros2cli? What is preventing them to be implemented in the package which command/verb they are testing?

ros2node/ros2node/verb/list.py

dirk-thomas · 2019-07-26T22:03:10Z

test_ros2cli/package.xml

@@ -12,11 +12,21 @@
  <test_depend>action_tutorials</test_depend>
  <test_depend>ament_lint_auto</test_depend>
  <test_depend>ament_lint_common</test_depend>
+  <test_depend>demo_nodes_py</test_depend>


This will create a repository level cycle since packages in the repository containing demo_nodes_py depend on packages in this repo. Imo we shouldn't do that.

Ok, that is true for ros2run. It's another instance of an issue we have elsewhere e.g. launch_ros, and that I was consciously eluding to avoid scope creep. IMO we shouldn't be using demo nodes for system or integration testing, nor duplicate ros2/system_tests talkers, listeners, servers, etc., and others. We need a package with test nodes that ros2/ros2cli, ros2/launch*, ros2/system_tests, can all depend on. Probably in its own repo, like we did for test_interface_files.

I can put this on hold, do that and come back, or fix it afterwards. What would you rather do?

Imo a package/repo like test_nodes sounds like a good idea.

dirk-thomas · 2019-07-26T22:05:48Z

test_ros2cli/CMakeLists.txt

+      if("${TEST_RMW_IMPLEMENTATION}" STREQUAL "rmw_connext_cpp")
+        # Connext startup is too slow. It needs a few extra seconds till
+        # discovery completes.
+        set(TEST_SETUP_DELAY 5.0)


That looks problematic because it:

is fragile in case Connext might just take a second longer

is slowing down the tests in case Connext happens to be faster than the threshold

needs to be done for potentially other RMW impl

Instead we should find a way to write the test to work without this delay.

It is problematic, I know. Truth to be told, all these tests have a delay to deal with CPU load fluctuations, uneven discovery times, etc., which makes them all flakey. I had to remove publishing/echoing rate tests because it's completely inconsistent, time wise.

Now, to fix this. Many tests don't need this delay at all e.g. ros2 msg list, so we can cut down time there. For those that do need it, I see two kinds: those that are bound to communication latency and those that are bound to discovery time.

For the first kind, we can wait for specific output or events e.g. for ros2 topic bw wait till we have any (non filtered) output to start testing it.

For the second kind e.g. ros2 topic list, well, it gets harder. Assuming the daemon is there, we may repeatedly execute the command, though that isn't really less fragile than a delay (and I can't figure how to pull that off using ros_testing as it is today). We may also wait for the ROS graph to show some nodes, but since the CLI command, the CLI daemon and launch test are three separate processes, it may not work as intended.

Alternatively, I think that we could make our CLI more predictable. Just like ros2 service call awaits for the server to be available, ros2 service list could have an optional ROS graph condition to wait for e.g. ros2 service list --when "'my_server' in nodes". The same goes for all discovery time bound commands. What do you think?

Assuming the daemon is there, we may repeatedly execute the command, though that isn't really less fragile than a delay.

Why would that not be less fragile? It will pass the test as soon as the command returns the desired output and only runs to a maximum timeout if the command doesn't do so.

Just like ros2 service call awaits for the server to be available, ros2 service list could have an optional ROS graph condition to wait for e.g. ros2 service list --when "'my_server' in nodes".

This is a good example how not to add conditions imo. If you want to wait for X (services of a node being available) you can't use a condition Y (check if a node has been discovered). Simply because there is a still a race since the node might become available first shortly before the services are actually advertised.

Why would that not be less fragile? It will pass the test as soon as the command returns the desired output and only runs to a maximum timeout if the command doesn't do so.

I meant neither option makes it not fragile. You may not delay enough or you may not wait enough. Sure, we can increase the timeout a lot to increase the likelihood of discovery to find all peers for free, but that doesn't make it not fragile. If CI nodes are running with enough load, tests may still flake.

Anyways, even if we go down that route, we lack support in launch testing to do it so we have to go back to it first.

Simply because there is a still a race since the node might become available first shortly before the services are actually advertised.

Yes, it doesn't make it not fragile either. I should've said that. I'm not well versed enough on DDS discovery mechanisms, is it atomic in any aspect e.g. some information chunks that are guaranteed to arrive in single block?

Sure, we can increase the timeout a lot to increase the likelihood of discovery to find all peers for free, but that doesn't make it not fragile. If CI nodes are running with enough load, tests may still flake.

I don't think that is the definition of flaky. The fact that we have to define an upper bound how long we are willing to wait is fine.

test_ros2cli/CMakeLists.txt

hidmic · 2019-07-29T15:54:50Z

The size of the tests makes me wonder if we really can't write them more compact?

What do you mean by size? These tests leverage the commonalities in the CLI commands to have single test_cli.py.in instance per command, parameterized N times. Thus the logic in test_cli.py.in. We could find other ways to reuse and instead have N tests. That'll yield shorter tests but the sheer size of this PR will increase due to boilerplate code duplication. Is that what you had in mind @dirk-thomas ?

What is the reason that all the tests are bundled in the separate package test_ros2cli?

There's currently no way to setup launch tests in pure Python packages (see ros2/launch#237).

dirk-thomas · 2019-07-29T16:11:13Z

test_ros2cli/test/test_cli.py.in

+        OpaqueFunction(function=lambda context: ready_fn())
+    )
+    launch_description = LaunchDescription([
+        # Always restart daemon to isolate tests.


Maybe instead of doing this it would be simpler to add an option --no-daemon to all the commands to not start the daemon on demand?

Hmm, that is a good idea, but won't it result in test instances affecting each other through daemon state?

dirk-thomas · 2019-07-29T16:15:04Z

test_ros2cli/CMakeLists.txt

+    add_custom_cli_test(
+      test_ros2interface
+      CONFIG_MODULE ros2interface_test_configuration
+      TIMEOUT 240)


The sum of all timeouts is 31 min. Is that really necessary?

It's insane. I was going to bring it up in tomorrow's meeting. And yes, it used to be half of it and tests would timeout in CI. Cutting down delays we may be able to reduce it, but I wouldn't expect a large reduction. Connext is very slow.

dirk-thomas · 2019-07-29T17:36:49Z

What do you mean by size?

I was wondering why are TestCommandFinishesInAFiniteAmountOfTime as well as TestCommandOutput necessary at all? I would expect that to be provided by launch_testing. Otherwise each package implementing test would need to implement that logic.

The length of each actual test file is also super long and verbose. I don't see an obvious way to reduce that though.

There's currently no way to setup launch tests in pure Python packages

I think it would be very important to make it possible to write a test function which can run a launch test. The fact that this PR moves all the tests into a separate package is a good indicator that not having that functionality is resulting in new code to be written which immediately introduces tech debt.

hidmic · 2019-07-29T18:17:07Z

I was wondering why are TestCommandFinishesInAFiniteAmountOfTime as well as TestCommandOutput necessary at all? I would expect that to be provided by launch_testing. Otherwise each package implementing test would need to implement that logic.

If we like this way of specifying tests we may standardize it, but no, it's not provided by launch_testing out of the box.

I think it would be very important to make it possible to write a test function which can run a launch test.

I agree with that, though it's not a blocker for these tests to land.

ivanpauno

I left some minimal comments.

ivanpauno · 2019-07-26T20:09:34Z

ros2node/ros2node/verb/list.py

@@ -37,4 +37,4 @@ def main(self, *, args):
        if args.count_nodes:
            print(len(node_names))
        elif node_names:
-            print(*[n.full_name for n in node_names], sep='\n')
+            print(*sorted(n.full_name for n in node_names), sep='\n')


Why sorting them? Was the order random?

I believe they show in discovery order, which is non-deterministic, yes.

ivanpauno · 2019-07-26T20:55:57Z

test_ros2cli/test/ros2action_test_configuration.py

+            output.extend(('    ' + yaml.dump({
+                'partial_sequence': sequence
+            })).splitlines())
+            output.append('')


ivanpauno · 2019-07-29T19:40:02Z

test_ros2cli/test/ros2action_test_configuration.py

+
+sys.path.append(os.path.dirname(__file__))
+
+from cli_test_configuration import CLITestConfiguration  # noqa


Prefer noqa: I100 than wildcard noqa (I know that tests were doing that, and that I wrote it 😄 ).

Same in many places.

ivanpauno · 2019-07-29T19:41:15Z

test_ros2cli/test/ros2action_test_configuration.py

+        'Waiting for an action server to become available...',
+        'Sending goal:',
+        '     order: {}'.format(order),
+        '',


ivanpauno · 2019-07-29T19:41:21Z

test_ros2cli/test/ros2action_test_configuration.py

+        '     order: {}'.format(order),
+        '',
+        re.compile('Goal accepted with ID: [a-f0-9]+'),
+        '',


ivanpauno · 2019-07-29T19:53:34Z

test_ros2cli/test/ros2interface_test_configuration.py

+        ),
+        CLITestConfiguration(
+            command='interface',
+            arguments=['show', 'std_msgs/msg/String'],


show should be tested with an action and a srv too.

ivanpauno · 2019-07-29T19:58:42Z

test_ros2cli/test/ros2msg_test_configuration.py

+        ),
+        CLITestConfiguration(
+            command='msg',
+            arguments=['show', 'std_msgs/msg/NotAMessageType'],


I would add a test case doing:

arguments=['show', 'std_msgs/srv/NotAMessageType']

and another doing:

arguments=['show', 'std_msgs/action/NotAMessageType']

Actually, I'm curious if they work correctly.

Well, this is my result:

ros2 msg show std_msgs/action/String string data

... it isn't working fine.

and this one is fun:

ros2 msg show std_msgs/asd/String string data

That's interesting.

@ivanpauno I can confirm interface introspection CLIs ignore the interface type namespace. I've commented out those tests, we should address that in a follow up PR.

ivanpauno · 2019-07-29T20:21:42Z

test_ros2cli/test/test_cli.py.in

+    def @TEST_NAME@(self, proc_info, command_under_test, test_configuration):
+        """Test that the command under test finished in a finite amount of time."""
+        success = proc_info.waitForShutdown(
+            process=command_under_test, timeout=test_configuration.timeout


waitForShutdown could take an optional keyword argument: assert: bool = False. In that way, we wouldn't need the extra if ...: assert ... here.
I think it may be useful.
Note: We have also assertWaitForShutdown, but it's less versatile than an extra argument.
Note: I won't delete the original waitForShutdown

ivanpauno · 2019-07-29T20:29:19Z

test_ros2cli/test/ros2service_test_configuration.py

+                             get_add_two_ints_server_action(service_name='_add_two_ints')],
+            expected_output=itertools.chain(
+                # Cope with launch internal ROS 2 node.
+                itertools.repeat(re.compile(r'/launch_ros/.*parameter.*'), 6),


We are checking for an exact number of lines here (6), of something that may easily change.
I don't know if it's better checking just if some lines are in the output, or checking for exact output.
This apply in many places, but I don't have an strong opinion about what's better.

Check lines in output pro: Updating the test is needed less often
Check lines in output con: Maybe, the test should have been updated (adding extra checks) after a change in other place, and it wasn't.

Check lines in output con: Maybe, the test should have been updated (adding extra checks) after a change in other place, and it wasn't.

That's the main reason why I went for a restrictive rather than a permissive test. If we trust CLI release testing on these tests, I'm more comfortable being as precise as possible regarding how they should behave.

ivanpauno · 2019-07-29T20:37:19Z

test_ros2cli/test/ros2topic_test_configuration.py

+        ),
+        CLITestConfiguration(
+            command='topic',
+            arguments=['pub', '/my_ns/chatter', 'std_msgs/msg/String', '{data: test}'],


Should the test also subscribe to the topic and check that the message is received?
I think that checking output is not enough in this case

That's why there's a listener, to check its output. Yes, we're trusting the listener in the process...

ivanpauno · 2019-07-29T20:47:01Z

The size of the tests makes me wonder if we really can't write them more compact? Is there anything we could do API / feature wise which would help reduce this?

I think that most of the code is just test cases for parameterizing the test.
It will be difficult to do it more compact.

Instead we should find a way to write the test to work without this delay.

I agree with this point. It's probably a bit difficult to implement.

I was wondering why are TestCommandFinishesInAFiniteAmountOfTime as well as TestCommandOutput necessary at all?

I did a comment about improving waitForShutdown, so TestCommandFinishesInAFiniteAmountOfTime will be shorter.
launch_testing methods for asserting process output could be better. Probably, something similar to TestCommandOutput could be provided by launch_testing.

hidmic · 2019-07-31T21:19:08Z

It will be difficult to do it more compact.

That is true @dirk-thomas. Test parameterizations are as few and compact as they can be. We simply have way to many combinations that need to be tested. But the test itself could use an API simplification.

hidmic · 2019-10-15T16:42:07Z

@ivanpauno @dirk-thomas rebased and ready for re-review. bf24f0d is a major refactor of this PR, so anything before that commit is likely outdated. PTAL when you have time. Splitting is also an option if you find this PR hard to grok.

hidmic · 2019-10-18T19:55:09Z

Considering how CI looks like atm, I don't think we'll get to merge this patch today. There are test fixture executables failing on specific platforms in a variety of ways, some even segfault in MacOS, while nightlies are all green. That means this is going to require further investigation on each platform.

CC @mjcarroll @dirk-thomas

ivanpauno · 2019-10-18T20:04:16Z

IMO, this can be merged after the feature freeze. It's just adding tests, no features.
It would be good to merge ros2/launch#279 if it's ready, though.

hidmic · 2019-10-18T20:18:04Z

It would be good to merge ros2/launch#279 if it's ready, though.

The same applies to that one. Test failures are mostly unrelated to this PR, but to other packages that make use of launch_testing and now fail in ways that are apparently unrelated to ros2/launch#279. But since it's the testing system itself that is under scrutiny, the only thing we can do is to go over each test in the right environment.

hidmic · 2019-10-21T14:06:23Z

Alright, recent builds from master show that segmentation faults on MacOS w/ Opensplice (http://ci.ros2.org/job/ci_osx/6892/) and flaky pendulum control tests on Linux-aarch64 w/ Opensplice (http://ci.ros2.org/job/ci_linux-aarch64/4368/) are already there and unrelated to this patch.

CI also experienced some issues and thus the current failed builds. I'll re-trigger after solving some other failures related to ros2cli tests on MacOS.

hidmic · 2019-10-21T15:47:10Z

Rebased to resolve conflicts.

hidmic · 2019-10-21T15:49:38Z

CI (full, against Fast-RTPS, Connext and Opensplice)

Linux
Linux-aarch64
macOS
Windows
- (only up to ros2topic after 2feb1dd)

- action command - service command - topic command - interface command - node command - msg command - srv command Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Improved test coverage for: - ros2action - ros2service - ros2topic - ros2msg - ros2srv - ros2interface - ros2node - ros2pkg Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

hidmic · 2019-10-22T13:15:21Z

Alright, @ivanpauno @mjcarroll may I ask for a final review? Test failures on Linux and MacOS are all unrelated to this patch and I had to re-trigger Windows CI because it failed for unrelated reasons (yet again). I'd like to merge this ASAP to avoid further conflicts with any new changes to ros2/ros2cli .

ivanpauno

LGTM!

It would be good to immediately follow-up in the TODOs (which are bugs unrelated to this PR that should be fixed).

ivanpauno · 2019-10-22T14:45:36Z

Please, double check that all CI failures are unrelated.

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

hidmic · 2019-10-22T19:44:27Z

Alright, finally green! Though that means there's some flakiness that needs to be addressed. @mjcarroll may I proceed with this PR and any follow-up ones after it?

hidmic added this to In progress in Eloquent via automation Jul 19, 2019

This was referenced Jul 19, 2019

Make some demos handle SIGINT gracefully. ros2/demos#377

Merged

Support CLI commands testing ros2/launch#279

Merged

hidmic mentioned this pull request Jul 19, 2019

Release Testing Automation - Iteration #1 ros2/ros2#739

Closed

hidmic force-pushed the hidmic/cli-test-coverage branch from cbbaa69 to 07a0883 Compare July 26, 2019 18:20

hidmic changed the title ~~[WIP] End-to-end test coverage for CLI commands~~ End-to-end test coverage for CLI commands output Jul 26, 2019

hidmic requested review from ivanpauno and dirk-thomas July 26, 2019 18:29

hidmic mentioned this pull request Jul 26, 2019

Fix type name queries for string types. ros2/rosidl_python#77

Merged

dirk-thomas reviewed Jul 26, 2019

View reviewed changes

This was referenced Jul 29, 2019

Alphasort ros2 node list output. #305

Merged

Fix ros2 topic bw output units. #306

Merged

dirk-thomas reviewed Jul 29, 2019

View reviewed changes

ivanpauno reviewed Jul 29, 2019

View reviewed changes

ivanpauno mentioned this pull request Aug 2, 2019

Add action send_goal prototype completer #301

Merged

hidmic mentioned this pull request Aug 5, 2019

[launch testing] Support repeated execution of executables under test ros2/launch#295

Closed

hidmic force-pushed the hidmic/cli-test-coverage branch from 08d5088 to e75d98e Compare October 4, 2019 18:42

hidmic force-pushed the hidmic/cli-test-coverage branch from e75d98e to 8a16abd Compare October 15, 2019 16:33

hidmic mentioned this pull request Oct 15, 2019

Add Python API for RMW implementation lookups ros2/rmw_implementation#73

Merged

hidmic force-pushed the hidmic/cli-test-coverage branch from a15df96 to 7c6ae38 Compare October 15, 2019 16:40

This was referenced Oct 17, 2019

Handle bad or missing package on ros2 interface show #366

Merged

Warn that old-style ready_fn and test attributes will be deprecated ros2/launch#346

Merged

hidmic force-pushed the hidmic/cli-test-coverage branch from 8f2f2d1 to 6dc5d0c Compare October 21, 2019 15:46

hidmic added 9 commits October 21, 2019 21:39

Add end-to-end CLI output tests for ros2:

60b2932

- action command - service command - topic command - interface command - node command - msg command - srv command Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Set nonzero default timeouts for ros2cli tests.

6ea88b3

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Refactor end-to-end CLI tests.

1441f4c

Improved test coverage for: - ros2action - ros2service - ros2topic - ros2msg - ros2srv - ros2interface - ros2node - ros2pkg Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Remove test_ros2cli package.

00b7858

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Drop Opensplice service types TODO

f7e4777

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Fix output mismatches and extend timeouts.

97bec5c

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Normalize ros2pkg destination directory in tests.

69f5f16

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Fix ros2msg, ros2srv and ros2interface CLI tests after (#371).

18d3d5f

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

Fix ros2pkg and ros2topic CLI.

e6f5893

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

hidmic force-pushed the hidmic/cli-test-coverage branch from 6dc5d0c to e6f5893 Compare October 22, 2019 00:47

Rename ros2interface test case.

01a17a9

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

ivanpauno approved these changes Oct 22, 2019

View reviewed changes

Make timeout longer for ros2 topic tests.

2feb1dd

Signed-off-by: Michel Hidalgo <michel@ekumenlabs.com>

mjcarroll approved these changes Oct 22, 2019

View reviewed changes

hidmic merged commit 309f9c5 into master Oct 22, 2019

Eloquent automation moved this from Reviewer approved to Done Oct 22, 2019

delete-merged-branch bot deleted the hidmic/cli-test-coverage branch October 22, 2019 20:06

dirk-thomas mentioned this pull request Jun 30, 2020

use transient_local and longer keep-alive for pub tests #546

Merged


		sys.path.append(os.path.dirname(__file__))

		from cli_test_configuration import CLITestConfiguration # noqa

End-to-end test coverage for CLI commands output #304

End-to-end test coverage for CLI commands output #304

Conversation

hidmic commented Jul 19, 2019 • edited

hidmic commented Jul 19, 2019 • edited

hidmic commented Jul 19, 2019 • edited

hidmic commented Jul 26, 2019

dirk-thomas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hidmic Jul 29, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hidmic Jul 29, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hidmic commented Jul 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hidmic Jul 29, 2019 • edited

Choose a reason for hiding this comment

dirk-thomas commented Jul 29, 2019

hidmic commented Jul 29, 2019

ivanpauno left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hidmic Jul 31, 2019 • edited

Choose a reason for hiding this comment

ivanpauno commented Jul 29, 2019

hidmic commented Jul 31, 2019

hidmic commented Oct 15, 2019 • edited

hidmic commented Oct 18, 2019

ivanpauno commented Oct 18, 2019

hidmic commented Oct 18, 2019 • edited

hidmic commented Oct 21, 2019 • edited

hidmic commented Oct 21, 2019

hidmic commented Oct 21, 2019 • edited

hidmic commented Oct 22, 2019

ivanpauno left a comment

Choose a reason for hiding this comment

ivanpauno commented Oct 22, 2019

hidmic commented Oct 22, 2019

hidmic commented Jul 19, 2019 •

edited

hidmic commented Jul 19, 2019 •

edited

hidmic commented Jul 19, 2019 •

edited

hidmic Jul 29, 2019 •

edited

hidmic Jul 29, 2019 •

edited

hidmic Jul 29, 2019 •

edited

hidmic Jul 31, 2019 •

edited

hidmic commented Oct 15, 2019 •

edited

hidmic commented Oct 18, 2019 •

edited

hidmic commented Oct 21, 2019 •

edited

hidmic commented Oct 21, 2019 •

edited