Posix shell: rebased & ready #10173

bkueng · 2018-08-06T07:38:21Z

This brings back the great work from @julianoes which was never finished in #5162.
I rebased (sorry Julian I had to squash it, there were many conflicts), and finished it to a point where everything is in a good state (meaning mergeable but not everything is converted or finished yet).

Features

all posix scripts run through bash now - there is already a unified script for iris and the typhoon_h480. The rest are still using the existing scripts, but already run through bash.
Running through bash (or a shell) allows to use full scripting power like on NuttX and thus unification of first all posix scripts and then also with the NuttX scripts (at least partly). This reduces duplication and allows the same functionality on posix targets as on NuttX.
support for multi-instances: it will simplify multi-vehicle simulations
supports an arbitrary amount of px4 shells: this makes PX4 modules 'feel' like individual executables and can be run as such when added to the PATH (e.g. px4-listener sensor_accel).

Try it out

To see the magic in action, start the simulation, then in a new shell:

cd <firmware>/build/posix_sitl_default
./bin/px4-commander takeoff
./bin/px4-uorb top

Testing Done

I did extensive testing:

SITL with jMAVSim and gazebo for different vehicles
replay works
RPi works
stress-test: I ran variations of while true; do ./bin/px4-ekf2 status; done in about 5 shells in parallel, and it works as expected. Memory usage stays stable.
not tested: snapdragon, bepob & ocpoc

TODO

Left todo before merging:

make sure MAVROS still works (tested via CI).
rebase and make required changes for the BeagleBone board if the PR is merged before

Future work

continue unification of SITL scripts
make sure MAVROS uses the new unified rcS script
merge scripts with NuttX
avoid printf for status output and use PX4_INFO_RAW instead

julianoes · 2018-08-06T07:41:12Z

@bkueng wooooah! That's awesome, I wasn't expecting this to ever happen!

julianoes · 2018-08-06T07:42:48Z

posix-configs/SITL/init/shell/iris

@@ -1,3 +1,8 @@
+#!/usr/bin/bash


We should probably use #/usr/bin/env bash everywhere though.

julianoes · 2018-08-06T07:43:21Z

ROMFS/px4fmu_common/init.d-posix/10016_iris

+#
+# @type Quadrotor Wide
+#
+# @maintainer Julian Oes <julian@oes.ch>


julianoes · 2018-08-06T07:51:37Z

posix-configs/eagle/init/rcS

+	#qshell df_isl29501_wrapper start
+
+# Qualcomm 200qx microheli platform
+elif param compare SYS_AUTOSTART 4200


We need to check if these two are still the to be supported snappy cases.

LorenzMeier · 2018-08-06T07:59:02Z

Awesome work! Can you rebase?

julianoes · 2018-08-06T07:56:18Z

platforms/posix/src/main.cpp


 	// SIGFPE
 	struct sigaction sig_fpe {};
-	sig_int.sa_handler = sig_fpe_handler;
-	sig_int.sa_flags = 0;// not SA_RESTART!;
+	sig_fpe.sa_handler = sig_fpe_handler;


julianoes · 2018-08-06T08:04:19Z

platforms/posix/src/main.cpp

@@ -506,24 +531,26 @@ void print_usage()
 {
 	printf("Usage for Server/daemon process: \n");
 	printf("\n");
-	printf("    px4 [-h|-d] [-s <startup_file>] [-d <test_data_directory>] [<rootfs_directory>]\n");
+	printf("    px4 [-h|-d] [-s <startup_file>] [-d <test_data_directory>] [<rootfs_directory>] [-i <instance>]\n");


Hang on? So we have -d for daemon mode, and we have -d <dir> for the test data directory?

And shouldn't there be a -i <instance> in the printf?

Right, it's just a documentation fail, it should be -t. I'll fix that.

julianoes · 2018-08-06T08:06:05Z

src/platforms/common/px4_log.c

@@ -106,22 +106,39 @@ __EXPORT void px4_log_modulename(int level, const char *moduleName, const char *

 			if (is_atty) { pos += snprintf(buffer + pos, max_length - pos, "%s", __px4_log_level_color[level]); }

+			if (pos >= max_length) { return; }


But if this check happens, we have already overflowed the buffer, right?

And nothing bad happens then?

No, because we pass the right buffer size to snprintf. I ran into the case where it was overflowing, which is how I found the problem and this fixed it then.
The only thing that is not nice here is the return. We could split into several messages, but the code would get really messy.

julianoes · 2018-08-06T08:06:52Z

platforms/posix/src/px4_daemon/pipe_protocol.h

@@ -70,7 +73,7 @@ struct client_send_packet_s {
 // We have per client receiver a pipe with the uuid in its file path.
 struct client_recv_packet_s {
 	struct message_header_s {
-		enum class e_msg_id : uint8_t {
+		enum class e_msg_id : int {


Nice catch.

julianoes · 2018-08-06T08:59:03Z

platforms/posix/src/px4_daemon/server.cpp

+			int payload_to_read = sizeof(packet)
+					      - sizeof(packet.header)
+					      - sizeof(packet.payload)
+					      + packet.header.payload_length;


Isn't sizeof(packet) - sizeof(packet.header) - sizeof(packet.payload) equal to 0?

Not necessarily if there are padding bytes between the header and the payload. Or at the struct end.

Hm, ok are padding bytes counted in sizeof?

Yes for the padding bytes that are inside the struct.

bkueng · 2018-08-06T12:28:45Z

platforms/posix/CMakeLists.txt

@@ -73,20 +85,12 @@ elseif ("${BOARD}" STREQUAL "bbblue")
 	target_link_libraries(px4 PRIVATE robotcontrol)

 	add_custom_target(upload
-		COMMAND scp -r ${PX4_SOURCE_DIR}/posix-configs/bbblue/*.config  debian@BBBluePX4:/home/debian/px4/posix-configs
-		COMMAND scp -r ${PX4_SOURCE_DIR}/ROMFS   $<TARGET_FILE:px4>     debian@BBBluePX4:/home/debian/px4
+		COMMAND rsync -arh --progress ${CMAKE_RUNTIME_OUTPUT_DIRECTORY} ${PX4_SOURCE_DIR}/posix-configs/bblue/*.config ${PX4_SOURCE_DIR}/ROMFS debian@BBBluePX4:/home/debian/px4


@UAV-Pilot This changes the uploading for bbblue: rsync is faster and the directory structure is a bit different. Can you test it please?
You need to run px4 like this:

cd px4 ./bin/px4 -s px4.config

After changing "posix-configs/bblue/" to "posix-configs/bbblue/", uploading with rsync worked as expected. Also note that rsync did not ask for password of the user on BeagleBone Blue board probably due to prior SSH authorized_keys setup.

Verified that without explicitly putting px4-alias.sh in the PATH, px4-alias.sh was executed in all of the following 3 scenarios (although it might not be necessary in daemon mode):

manual start: sudo ./bin/px4 -s px4.config (librobotcontrol currently requires root)

auto start via /etc/rc.local

auto start via systemd service

Nice, thanks for the quick and extensive testing! I'll fix up the path and have a look at the daemon mode.

dagar · 2018-08-06T13:07:50Z

platforms/posix/cmake/px4_impl_os.cmake

+			set(ln_name "${PREFIX}${MAIN}")
+			add_custom_command(TARGET ${TARGET}
+				POST_BUILD
+				COMMAND ln -s -f ${TARGET} ${ln_name}


${CMAKE_COMMAND} -E create_symlink

dagar · 2018-08-06T13:35:15Z

platforms/posix/cmake/px4_impl_os.cmake

+
+
+# TODO: document API
+function(px4_posix_generate_symlinks)


I think treating this like an install would be the "right" cmake way to do this. I can revisit that later.

dagar · 2018-08-06T13:39:26Z

Great work @bkueng!

I'm really looking forward to reducing as much platform fragmentation as possible.

dagar · 2018-08-06T13:43:27Z

Try it out
To see the magic in action, start the simulation, then in a new shell:

cd /build/posix_sitl_default
./bin/px4-commander takeoff
./bin/px4-uorb top

Explicit start?

bkueng · 2018-08-06T13:51:58Z

Explicit start?

What do you mean?

dagar · 2018-08-06T14:27:38Z

Tools/sitl_multiple_run.sh

@@ -43,7 +43,7 @@ while [ $n -le $sitl_num ]; do

 	pushd "$working_dir" &>/dev/null
 	echo "starting instance $n in $(pwd)"
-	sudo -b -u $user ../src/firmware/posix/px4 -d "$src_path" rcS >out.log 2>err.log
+	sudo -b -u $user ../bin/px4 -i $n -d "$src_path" -s rcS >out.log 2>err.log


I was wondering about that too. I just kept it the same now.

dagar · 2018-08-06T14:35:23Z

platforms/posix/src/main.cpp

-	sig_int.sa_handler = _SigIntHandler;
-	sig_int.sa_flags = 0;// not SA_RESTART!;
-	sigaction(SIGINT, &sig_int, nullptr);
+int create_symlinks_if_needed(std::string &data_path)


We shouldn't be creating symlinks from main. All of this should be put in place by the install.

We can do that as a later step, there are too many moving parts otherwise.

Yes, that's fine I just wanted to call it out.

dagar · 2018-08-06T14:39:41Z

Explicit start?

What do you mean?

I was wondering about when and where we explicitly start the daemon, but I see that now. The entrypoint for starting px4 and scripting is still weird, but definitely an improvement.

dagar · 2018-08-06T14:44:11Z

Why bother with the px4- prefix on all the commands?

bkueng · 2018-08-06T14:48:17Z

Why bother with the px4- prefix on all the commands?

Because otherwise we interfere with system commands (like shutdown)

dagar · 2018-08-06T15:10:55Z

Because otherwise we interfere with system commands (like shutdown)

Yes that's the downside of bringing this out to a real system shell and things like shutdown are a special case. NuttX shutdown shouldn't be treated interchangeably with PX4 sitl shutdown.

bkueng · 2018-08-07T18:00:51Z

@dagar Can you look into CI? I need to update the log path but I don't see from where process_logdata_ekf.py is called.

[tests_feature_shell_rebased-EFW66T23W3TPIPSQOC7HSJ3UKSSR5BD4ABWBOYSZLARO5N4K6BFQ] Running shell script

+ px4-posix_sitl_default-7e38a5d/px4/Tools/ecl_ekf/process_logdata_ekf.py .ros/rootfs/fs/microsd/log/*/*.ulg

Traceback (most recent call last):

  File "px4-posix_sitl_default-7e38a5d/px4/Tools/ecl_ekf/process_logdata_ekf.py", line 38, in <module>

    ulog = ULog(args.filename, None)

  File "/usr/local/lib/python2.7/dist-packages/pyulog/core.py", line 107, in __init__

    self._load_file(file_name, message_name_filter_list)

  File "/usr/local/lib/python2.7/dist-packages/pyulog/core.py", line 422, in _load_file

    self._file_handle = open(file_name, "rb")

IOError: [Errno 2] No such file or directory: '.ros/rootfs/fs/microsd/log/*/*.ulg'

Should be '.ros/log/*/*.ulg

lamping7 · 2018-08-07T22:35:51Z

Can you look into CI? I need to update the log path but I don't see from where process_logdata_ekf.py is called.

There you go @bkueng. Jenkins got restructured recently.

bkueng · 2018-08-08T05:54:09Z

Thanks @lamping7! My grepper ignores hidden files which is why I did not see it.

squashed & rebased version, not including: - listener changes - src/firmware renaming Commits: tag_to_version.py: fix Python3 error subprocess.communicate returns bytes instead of a str which is not the same for Python3. Therefore, we need to decode the bytes. cmake: remove folder src/firmware The folder src/firmware was not intuitive. Why would the binaries for SITL be inside a src and why even inside a src/firmware folder. Also, the rootfs was put there which made it even more confusing. The CMakeLists.txt files are moved into cmake/ and get now called from the main CMakeLists.txt. qshell: support for return value Instead of just sending commands, qshell will now also wait until the command has finished on QURT and sent back a return value. This will allow all modules on the DSP side to be spawned from the Linux side meaning that we only need one config/startup file instead of two. adb_upload: create folders before pushing Previously the script failed if the folder on the destination was not already existing. This therefore makes pushing easier. posix: spawn PX4 modules in bash This adds the possibility to spawn PX4 modules out of bash. Basically, the main executable can now be started as a server/daemon or as a client. The server replaces the existing functionality of the main exe with the pxh shell, however, it also opens a pipe that clients can talk to. Clients can run or spawn PX4 modules or commands by connecting to the server over the pipe. They clients will get the stdout and return value of their commands via a client specific pipe back. This work will allow to start all modules using a bash script similar to the way it is done in NuttX where the NuttShell scripts the startup scripts and starts the modules. SITL: use new client shell in SITL This is a first step to use the new shell capabilities for SITL. The new startup bash script rcS merges (and therefore replaces) the two existing scripts rcS_gazebo_iris and rcS_jmavsim_iris. More cleanup will be necessary for the rest of the SITL startup scripts. Snapdragon: use new shell to start all modules Instead of different mainapp.config and px4.config files, we can now use a unified rcS bash script which starts all the modules based on parameters, mainly the SYS_AUTOSTART param. Snapdragon: fix the airframe description pxh: argv needs to end with a nullptr The comment was wrong that argv needs an additional 0 termination. Instead it needs a nullptr at the end. px4_posix_tasks: variable cleanup The px4_task_spawn_cmd function got a cleanup while debugging, however, no functional changes. Snapdragon: move some drivers to 4100 config These drivers are supported by the community, so they go into the 4100 config. Snapdragon: update 210qc platform px4_daemon: use doxygen comments apps.h_in: fix string printf: use .c_str() px4_daemon: \b -> \n in printf px4_daemon: handle error in generate_uuid (close the file on error) posix main: some clarifications in comment (it's the symlinks not the script aliases) cmake: remove new install command again This one was probably wrong and untested. Installing needs revisiting. POSIX: remove argument USES_TERMINAL POSIX: copy init and mixer files for SITL Instead of using non-working install commands, the mixer and startup files are now copied as part of the build in cmake. adb_upload.sh: remove leftover commented printf POSIX main: just the pointer instead of memmove POSIX main: remove chroot chroot is removed because it hasn't been used anywhere and seems untested. px4_daemon: remove client pipe when cleaning up px4_daemon: fail if the client pipe already exists The client pipe is supposed to be specific (by UUID), so the path shouldn't exist already. history: limit the number of history entries This is a protection to avoid filling the memory if we are entering a lot of commands (e.g. auto-generated). px4_daemon: add a threadsafe map and use it px4_daemon: whitespace px4_daemon: fix client parsing Sometimes the client ends up reading more than one packet in one read. The parsing is not made for this and would require a (ring)buffer for it. The solution of this commit just reads as much as needed from the pipe which avoids having to do buffering and parsing. posix: changes sitl_run.sh and main.cpp cleanup This changes the paths in sitl_run.sh quite a bit to allow the px4 binary to run in the rootfs directory which should make it convenient and very close to the NuttX variant. Also main.cpp got a big cleanup after the big rebase with some conflicts. Quite some functionality was removed but it has yet to be seen if it needs to be re-added. px4_log: cleanup log levels, now they make sense Before DEBUG and INFO log levels where inverted which didn't make much sense in my eyes. dataman: fix path for bash shell logger: fix paths for bash shell mavlink: fix paths for bash shell param: fix path for bash shell inav: fix paths for bash shell sdlog2: fix paths for bash shell ROMFS: add forgotten mixer to list SITL init: more models, more options - Support for different models using the unified startup script rcS. - Support to choose the estimator by setting the environment variable PX4_ESTIMATOR. - Support to choose the logger by setting the environment variable PX4_LOGGER. rcS: fix string comparison listener: use template file Instead of having all of the C++ code inside the Python file it is nicer to have a separate template file with the C++ headers, etc. px4_log: add PX4_INFO_RAW for raw printfs This allows to do custom formatting but is still transported over sockets to clients. topic_listener: use PX4_INFO_RAW instead of printf commander: use PX4_INFO_RAW for status listener: rewrite to classes and factory posix: fix some argument warnings generate_listener.py: by accident changed shebang listener: big refactor of the generator Hopefully this makes it easier to read and change in the future. rcS: manually take over rebase changes listener: remove leftover try listener: properly clean up topic instance rcS: take over some vehicle specific changes posix-configs: vehicle specifics to separate files posix-configs: remove leftover lines uORBDevices: new PX4_INFO_RAW instead of printf px4_log: just use printf on NuttX listener: use less binary space, strip on NuttX generate_listener.py: remove commented code cmake: fix syntax error from merge px4_daemon: fixes after rebase of apps.h/cpp fix px4_daemon: namespace missing posix: only create stub for fsync on QURT unitests: reduce dependencies of param test This makes the unit test compile and link again after the bash changes. QURT: some compile fixes after a rebase SITL: arg change for sitl_run.sh to use rcS_test This allows to use a custom startup file for testing. SITL: add the folder test_data SITL: implement shutdown command as systemcmd The shutdown command needs to be a proper systemcmd, otherwise the alias and symlink generation doesn't work and we end up calling shutdown of the host computer which is to be avoided. px4fmu_test: same IO_pass mixer as px4fmu_default px4fmu_test: use normal quad x mixer There is no good reason to use a specific test mixer, except more cmake code around it. Therefore just use the same mixer as default, and at some point px4fmu_test and px4fmu_default can get merged POSIX: cleanup, dir and symlink fixes This cleans up the logic behind the symlinking and creating directories. POSIX: correct arg order in usage info tests: fix paths for SITL tests POSIX: printf fix sitl_run.sh: try to make this run on Mac as well cmake: try to make jenkins happier Path cleanup, the bin is no longer in src/firmware POSIX: fix symlink logic SITL: prefix all exported env variables cmake: fix path for ROS tests integrationtests: fix log path launch: try to make tets with ROS working again px4_defines: fix after wrong merge deconflicting px4_defines: get paths for POSIX correct cmake: fix cmake arguments This was fine with cmake 3.6 but did not work with cmake 3.2.2 cmake: use cp instead of cmake -E copy cmake -E copy does not support copying multiple files with versions < 3.5. Therefore, just use cp for now. ROMFS: fix build error after rebase cmake: fix paths in configs launch: use `spawn_model` again cmake: various fixes after big rebase param: path fixes after rebase posix platform: fixes after rebase test_mixer: fix screwed up rebase

dagar · 2018-08-08T14:55:53Z

It was done intentionally because the Jenkinsfiles end up being specific to our particular infrastructure. I don't mind moving them if someone feels strongly about it.

bkueng · 2018-08-09T06:09:40Z

I don't mind moving them if someone feels strongly about it.

Yes please do so. Hidden folders make it unnecessary harder to understand the CI system and find things.

julianoes · 2018-08-09T07:14:55Z

Awesome, 2 years and 10 days later and it's merged, thanks @bkueng! 🎉

Needed after PX4/PX4-Autopilot#10173 was merged.

hamishwillee · 2018-08-10T01:21:25Z

@bkueng If needed, and it sounds like it would be great, can you add the necessary docs? I am happy to help subedit.

dagar · 2018-08-10T15:00:34Z

I don't mind moving them if someone feels strongly about it.

Yes please do so. Hidden folders make it unnecessary harder to understand the CI system and find things.

Feel free to open a PR if you have an opinion where these things should live. We already have plenty of other hidden files (travis, clang-tidy, ack, you complete me, etc) that occasionally need updates. Remember the difference in this case is these are (unavoidably) going to continue to become more and more specific to particular CI infrastructure.

asimonov · 2018-08-27T17:20:05Z

I think this has broken Parrot Bebop 2. Just built from master and see this trying to run this on bebop:

/ # /data/ftp/internal_000/px4 /home/root/px4.config

______  __   __    ___ 
| ___ \ \ \ / /   /   |
| |_/ /  \ V /   / /| |
|  __/   /   \  / /_| |
| |     / /^\ \ \___  |
\_|     \/   \/     |_/

px4 starting.

INFO  [Unknown] Calling startup script: bash etc/init.d/rcS 0
sh: bash: not found
ERROR [Unknown] Startup script returned with return value: 32512

dagar · 2018-08-27T17:33:24Z

@asimonov could you please open a new issue and we'll follow up.
@mrpollo FYI

asimonov · 2018-08-27T18:29:12Z

#10342

mrpollo · 2018-08-27T19:08:08Z

@dagar Dronecode can sponsor a Bebop 2 to guarantee some level of support, would be good to have Parrot engineering support.

PX4/PX4-Autopilot#10173

Some threads do not exit and are still running when trying to exit SITL running under Windows in Cygwin. This problem was introduced with: - posix shell #10173 because of strating a child process for the startup script and mixing up the signal handling (only Ctrl+C broken) - lockstep #10648 because of simulator threads not stopping gracefully anymore with timing race conditions (no graceful exit possible anymore) I leave the SIGINT handler on its default implementation for Cygwin which kills the process and all its threads when pressing Ctrl+C. This hotfix at least allows the usage of Ctrl+C again instead of forcing the user to use the task manager to shut down px4.exe going crazy on CPU load instead of exiting everytime.

julianoes reviewed Aug 6, 2018

View reviewed changes

bkueng force-pushed the feature_shell_rebased branch from b1c96e8 to 59badae Compare August 6, 2018 12:24

bkueng commented Aug 6, 2018

View reviewed changes

dagar self-requested a review August 6, 2018 13:06

dagar reviewed Aug 6, 2018

View reviewed changes

dagar added the Admin: Enhancement (improvement) 💡 label Aug 6, 2018

dagar reviewed Aug 6, 2018

View reviewed changes

bkueng force-pushed the feature_shell_rebased branch from 89bc336 to 66a014b Compare August 6, 2018 19:18

UAV-Pilot mentioned this pull request Aug 7, 2018

Port PX4 to BeagleBone Blue #9635

Closed

bkueng force-pushed the feature_shell_rebased branch 5 times, most recently from 4447868 to 7e38a5d Compare August 7, 2018 17:22

dagar approved these changes Aug 8, 2018

View reviewed changes

LorenzMeier merged commit ffffcae into master Aug 8, 2018

LorenzMeier deleted the feature_shell_rebased branch August 8, 2018 19:09

bkueng added a commit to PX4/PX4-Devguide that referenced this pull request Aug 9, 2018

RPi: update PX4 start instructions

d115a05

Needed after PX4/PX4-Autopilot#10173 was merged.

bkueng mentioned this pull request Aug 9, 2018

RPi: update PX4 start instructions PX4/PX4-Devguide#576

Merged

bkueng added a commit to PX4/PX4-Devguide that referenced this pull request Aug 9, 2018

RPi: update PX4 start instructions

d55a1c8

Needed after PX4/PX4-Autopilot#10173 was merged.

dagar mentioned this pull request Aug 10, 2018

Flight task custom #10199

Closed

RomanBapst mentioned this pull request Aug 16, 2018

snapdragon801 ERROR [Unknown] Error opening startup file, does not exist: etc/init.d/rcS #10244

Closed

bkueng mentioned this pull request Aug 20, 2018

Multi-Vehicle Simulation with jmavsim PX4/PX4-Devguide#585

Closed

MaEtUgR mentioned this pull request Aug 22, 2018

jMAVSim SITL with Windows Cygwin gets stuck #10098

Closed

asimonov mentioned this pull request Aug 27, 2018

bebop px4 seem to require bash which is not available on bebop/BusyBox #10342

Closed

bkueng mentioned this pull request Aug 31, 2018

SITL: minor cleanup #10380

Merged

bkueng added a commit to PX4/PX4-Devguide that referenced this pull request Sep 6, 2018

add documentation for the Posix shell

099371a

PX4/PX4-Autopilot#10173

bkueng mentioned this pull request Sep 6, 2018

add documentation for the Posix shell PX4/PX4-Devguide#600

Merged

lamping7 mentioned this pull request Oct 2, 2018

Missing Topic: How to integrate customised Gazebo copter models into the code base PX4/PX4-Devguide#398

Open

MaEtUgR mentioned this pull request Jan 26, 2019

Hotfix for PX4 application not exiting anymore on Cygwin #11305

Merged

		@@ -106,22 +106,39 @@ __EXPORT void px4_log_modulename(int level, const char moduleName, const char

		if (is_atty) { pos += snprintf(buffer + pos, max_length - pos, "%s", __px4_log_level_color[level]); }

		if (pos >= max_length) { return; }

Posix shell: rebased & ready #10173

Posix shell: rebased & ready #10173

Conversation

bkueng commented Aug 6, 2018 • edited by AuterionWrikeBot Loading

Features

Try it out

Testing Done

TODO

Future work

julianoes commented Aug 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LorenzMeier commented Aug 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dagar commented Aug 6, 2018

dagar commented Aug 6, 2018

bkueng commented Aug 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dagar commented Aug 6, 2018

dagar commented Aug 6, 2018

bkueng commented Aug 6, 2018

dagar commented Aug 6, 2018

bkueng commented Aug 7, 2018

lamping7 commented Aug 7, 2018

bkueng commented Aug 8, 2018

dagar commented Aug 8, 2018

bkueng commented Aug 9, 2018

julianoes commented Aug 9, 2018

hamishwillee commented Aug 10, 2018

dagar commented Aug 10, 2018

asimonov commented Aug 27, 2018

dagar commented Aug 27, 2018

asimonov commented Aug 27, 2018

mrpollo commented Aug 27, 2018

bkueng commented Aug 6, 2018 •

edited by AuterionWrikeBot

Loading