High ping on WRT1900ACS #74

lefoyer · 2016-04-17T14:56:52Z

Two days ago, I bought this device, and flashed firmware (first test OpenWRT 15.05.1, next probe http://personalpages.tds.net/~davidc502/mvebu/Kernel4.4.6/4.4WNandPatchShelby/openwrt-mvebu-armada-385-linksys-shelby-squashfs-factory.img). I Tried 2Ghz & 5Ghz everywhere high ping (12-30ms+), to router and all connected to router devices and internet. Distance from the laptop to the router 3 meters, no walls.
From time to time the device reboots itself. Speed on 5Gnz falls from 320Mbit/s to 64Mbit/s on copy file from NAS (but Wifi speed 866.5Mbit/s).
-55 dbm -89 dbm 234.0 Mbit/s, MCS 0, 20Mhz 866.7 Mbit/s, MCS 0, 20MHz
I am find identical problem https://dev.openwrt.org/ticket/22130
Previous router flashed by OpenWRT RB2011UAS-2HnD-IN there were no problems? ping smolest 1 ms.

Razerwire · 2016-04-18T15:53:08Z

Thank god, i thought i was the only one experiencing this.....

I'm experiencing the same issue (high latency when connected via WiFi) with clients running Linux Kernel 4.5 and newer and Windows-10, in combination with Intel Wireless-AC 7265 and and Intel AC 8260.

I'm not seeing high latency when using my Netgear R7000 so it isn't my internet connection that is at fault.

lefoyer · 2016-04-18T16:11:24Z

My client: Win 7 Pro, Broadcom BCM94352HMB (AW-CE123H).

itrankolov · 2016-04-27T10:00:45Z

Hi,
I have the same problem. It was introduced with driver version > 10.3.0.12. The issue is related to WMM. If I disable WMM the latency goes down to < 1ms, but the link speed goes to 54M only as well, so this is not a solution.
This is happening with 3 different laptops (with different wifi adapters).

MrDoh · 2016-04-27T20:30:33Z

Been seeing long ping times with the WRT1900ACv1 on one of my Apple clients on every version of the wireless driver that I've used, maybe even before 10.3.0.12, including on 10.3.0.17. The internet ping time on my Apple iPad Air 2 is more than double the time that I see on other routers (37ms. versus <=15ms. on other routers). And now I'm also getting the same ballpark long ping time on my Nexus 6P phone. This is very frustrating, especially since I don't see this on the stock Linksys firmware. I'm glad to see it finally being reported and discussed. Love to see this finally get fixed.

itrankolov · 2016-04-27T21:28:43Z

The issue in my opinion is related to AMPDU priorities. For some reason the driver is treating everything as low priority traffic (best effort).

suihkulokki · 2016-04-28T08:57:23Z

If you have linux, you can try different ToS values with ping using the -Q parameter.
ping -Q 0x2E
if using different values decrease latency for you, then it's indeed a priority related issue. For values to try, see:
https://www.bintec-elmeg.com/portal/downloadcenter/dateien/workshops/current_en/ws_wlan_html_en_HTML/vowlan_infra_qos_wmm.html

itrankolov · 2016-04-28T09:17:15Z

Hi,
I've already tried that and that pointed to issues with AMPDU:

root@debian:~# ping 192.168.1.100 -c 10
PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data.
64 bytes from 192.168.1.100: icmp_seq=1 ttl=128 time=17.6 ms
64 bytes from 192.168.1.100: icmp_seq=2 ttl=128 time=15.0 ms
64 bytes from 192.168.1.100: icmp_seq=3 ttl=128 time=32.6 ms
64 bytes from 192.168.1.100: icmp_seq=4 ttl=128 time=30.2 ms
64 bytes from 192.168.1.100: icmp_seq=5 ttl=128 time=28.6 ms
64 bytes from 192.168.1.100: icmp_seq=6 ttl=128 time=27.7 ms
64 bytes from 192.168.1.100: icmp_seq=7 ttl=128 time=25.2 ms
64 bytes from 192.168.1.100: icmp_seq=8 ttl=128 time=22.6 ms
64 bytes from 192.168.1.100: icmp_seq=9 ttl=128 time=21.4 ms
64 bytes from 192.168.1.100: icmp_seq=10 ttl=128 time=20.1 ms

--- 192.168.1.100 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9017ms
rtt min/avg/max/mdev = 15.005/24.139/32.629/5.435 ms
root@debian:~# ping 192.168.1.100 -c 10 -Q 48
PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data.
64 bytes from 192.168.1.100: icmp_seq=1 ttl=128 time=2.49 ms
64 bytes from 192.168.1.100: icmp_seq=2 ttl=128 time=3.05 ms
64 bytes from 192.168.1.100: icmp_seq=3 ttl=128 time=3.60 ms
64 bytes from 192.168.1.100: icmp_seq=4 ttl=128 time=3.08 ms
64 bytes from 192.168.1.100: icmp_seq=5 ttl=128 time=3.21 ms
64 bytes from 192.168.1.100: icmp_seq=6 ttl=128 time=2.99 ms
64 bytes from 192.168.1.100: icmp_seq=7 ttl=128 time=3.07 ms
64 bytes from 192.168.1.100: icmp_seq=8 ttl=128 time=3.03 ms
64 bytes from 192.168.1.100: icmp_seq=9 ttl=128 time=2.90 ms
64 bytes from 192.168.1.100: icmp_seq=10 ttl=128 time=3.17 ms

--- 192.168.1.100 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9017ms
rtt min/avg/max/mdev = 2.492/3.061/3.600/0.271 ms

yuhhaurlin · 2016-04-28T09:18:31Z

This issue will be enhanced for next release.

yuhhaurlin · 2016-04-28T09:18:50Z

It is related to AMSDU.

itrankolov · 2016-04-28T09:22:05Z

Another sample to router from physical machine:
root@openwrt-builder:~# ping 192.168.1.1 -Q 48 -c 10
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=1.92 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=2.90 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=2.12 ms
64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=2.63 ms
64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=2.09 ms
64 bytes from 192.168.1.1: icmp_seq=6 ttl=64 time=2.46 ms
64 bytes from 192.168.1.1: icmp_seq=7 ttl=64 time=2.52 ms
64 bytes from 192.168.1.1: icmp_seq=8 ttl=64 time=1.81 ms
64 bytes from 192.168.1.1: icmp_seq=9 ttl=64 time=2.45 ms
64 bytes from 192.168.1.1: icmp_seq=10 ttl=64 time=2.78 ms

--- 192.168.1.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9126ms
rtt min/avg/max/mdev = 1.814/2.372/2.909/0.353 ms
root@openwrt-builder:~# ping 192.168.1.1 -c 10
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=25.9 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=17.7 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=22.6 ms
64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=26.8 ms
64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=30.7 ms
64 bytes from 192.168.1.1: icmp_seq=6 ttl=64 time=15.6 ms
64 bytes from 192.168.1.1: icmp_seq=7 ttl=64 time=19.8 ms
64 bytes from 192.168.1.1: icmp_seq=8 ttl=64 time=23.7 ms
64 bytes from 192.168.1.1: icmp_seq=9 ttl=64 time=28.3 ms
64 bytes from 192.168.1.1: icmp_seq=10 ttl=64 time=26.2 ms

--- 192.168.1.1 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9119ms
rtt min/avg/max/mdev = 15.651/23.795/30.774/4.573 ms

itrankolov · 2016-04-28T09:24:37Z

@yuhhaurlin
Thanks for the update.
Hopefully it will resolve the issues, because that forced me to buy another device, that I am not happy with ....

yuhhaurlin · 2016-04-28T09:24:48Z

However, this issue will not affect throughput.

itrankolov · 2016-04-28T09:26:15Z

Yes it does not affect throughput, but it affects latency and that is a problem when you're making voice calls, playing games and doing other stuff that is heavy reliant on latency.

yuhhaurlin · 2016-04-28T09:34:37Z

The problem is that if traffic is not heavy and AMSDU is on. AMSDU packets must wait for flush function to send them out.

yuhhaurlin · 2016-04-28T09:42:20Z

Basically, this issue will only affect single ping. However, I will enhance it on next release.

itrankolov · 2016-04-28T09:45:21Z

The interesting thing is that it is not obeying the setting on the parameter in the wireless adapter advanced settings - U-APSD Support. It does not matter if I set it to disabled.
I don't know if there is an option to disable AMPDU support on the adapter in OpenWRT? This may be useful as a workaround.

itrankolov · 2016-04-28T09:47:27Z

@yuhhaurlin it is not affecting only ping .... RDP, Voice, PCoIP and others are affected as well. For example if I disable WMM I get PCoIP latency of <2ms. With it enabled it is above 20ms.
The only thing it is not affecting is huge file transfers/downloads.

yuhhaurlin · 2016-04-28T10:05:05Z

Flush function is called around 10ms, the minimum sample interval of CODEC is 10ms. I think the driver should be all right for VoIP application.

yuhhaurlin · 2016-04-28T10:06:38Z

BTW, only AMSDU will affect ping latency. AMPDU will not.

itrankolov · 2016-04-28T10:45:09Z

I am not sure ... I mean, I am just guessing about the protocol responsible for the latency. My observations on the other side are correct. When is the new driver expected to arrive?

yuhhaurlin · 2016-05-20T08:07:20Z

I think running with 10.3.0.17-20160520, this problem should be gone.

johnnysl · 2016-05-20T11:28:21Z

I never have really experienced the issue, But gave the newest commit some quick testing:

Very first ping to router via 5ghz:
Pinging 192.168.1.1 with 32 bytes of data:
Reply from 192.168.1.1: bytes=32 time=21ms TTL=64
Reply from 192.168.1.1: bytes=32 time=27ms TTL=64
Reply from 192.168.1.1: bytes=32 time=32ms TTL=64
Reply from 192.168.1.1: bytes=32 time=38ms TTL=64

Ping statistics for 192.168.1.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 21ms, Maximum = 38ms, Average = 29ms

so that is not good.
tried google after that:

Pinging www.google.com [74.125.136.105] with 32 bytes of data:
Reply from 74.125.136.105: bytes=32 time=10ms TTL=49
Reply from 74.125.136.105: bytes=32 time=10ms TTL=49
Reply from 74.125.136.105: bytes=32 time=10ms TTL=49
Reply from 74.125.136.105: bytes=32 time=10ms TTL=49

thats really nice.

retried the router:
Pinging 192.168.1.1 with 32 bytes of data:
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64

that looks a lot better. Pinging the router stayed fast after that.

only thing i observe is that the initial ping is slow after a longer pause (5min) between the ping tries;
e.g.
Pinging www.google.com [74.125.136.105] with 32 bytes of data:
Reply from 74.125.136.105: bytes=32 time=151ms TTL=49
Reply from 74.125.136.105: bytes=32 time=10ms TTL=49
Reply from 74.125.136.105: bytes=32 time=10ms TTL=49
Reply from 74.125.136.105: bytes=32 time=10ms TTL=49

Roughly the same with pinging the router, there i see initial pings going up to 10/11 msec. (if pause using the wifi for a minute or 5)

[edit] copied the wrong info here and there, corrected it now

yuhhaurlin · 2016-05-21T00:16:37Z

Thanks. I will close this one.

MrDoh · 2016-05-21T01:11:49Z

How can you close this issue when the person that emaled said that "I never
have really experienced the issue," I think that you should get some
testing from someone who has experienced the issue before closing it, don't
you? Closing this issue without real testing and verification doesn't make
any sense to me.

Thanks.

-Roger

On Fri, May 20, 2016 at 5:16 PM, yuhhaurlin notifications@github.com
wrote:

Thanks. I will close this one.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#74 (comment)

yuhhaurlin · 2016-05-21T01:22:43Z

Thanks. Reopen it.

MrDoh · 2016-05-21T03:20:53Z

Thank you very much!

-Roger

On Fri, May 20, 2016 at 6:22 PM, yuhhaurlin notifications@github.com
wrote:

Thanks. Reopen it.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#74 (comment)

bill1228 · 2016-05-21T13:59:39Z

I agree with MrDoh.
I do have high ping times from one device to the router via 5GHz wireless others work fine. The one that doesn't is a TP-Link T4U USB wireless adapter with latest driver from TP-Link. This adapter is a Ralink chip for what it is worth.. This should not be closed until it is proven by more then one person/test that there is not a problem. If I tested with any of the other devices I have you would say it was fixed, but really fixed in my opinion. There are too many folks complaining about high ping times with the latest driver to not think there is still an issue.

Thanks for reopening it. Am looking forward to the new driver.
--bill

yuhhaurlin · 2016-05-21T14:04:21Z

Does anyone still get high ping Times for the latest driver?

ONjAXX · 2017-05-13T15:22:40Z

I guess, disabling AMSDU won't help then, since ping is much higher than mentioned added latency of 10ms, up to 30-50ms.
Also note than if there's no other network activity except ping, the latency is ~1ms, so AMSDU is again, not delaying it to 10ms.

Is there any debug info I can provide for you, you're welcome!

yuhhaurlin · 2017-05-13T15:43:20Z

No need to debug. If there is no traffic, BA stream will be tear down. If application is running. flushing of AMSDU won't take action. I suggest you disable power save of client, the ping won't be so high. If application is running, client won't enter power save mode.

ONjAXX · 2017-05-13T16:12:16Z

Powersaving is disabled, of course. I tried several clients, intel 7265, 8265, and atheros on 2.4. All the same.
Switching to Linksys fw brings things to normal.

yuhhaurlin · 2017-05-15T08:01:34Z

Can you cat /sys/kernel/debug/ieee80211/phy0 or phy1/mwlwifi/ampdu to see if AMPDU is established?

ONjAXX · 2017-05-15T10:08:16Z

Both read 0.
Additional info: if I run ping to router + some youtube stream - ping rises up to 60ms, but
If I also add Speedtest + youtube - ping to router backs to 1-3ms. Seems like buffer is getting flushed very often in this case and latency is ok!
ps: My wan is 50 mbit up/down, no bufferbloat ever detected by dslreports, regardless of sqm setup, wlan is at 650mbit, up to 50mb/s from router hdd

yuhhaurlin · 2017-05-15T10:11:14Z

If this is the case, it means there is no BA stream established. In fact, if traffic is pretty low with client, BA stream will be removed. I think your problem does not due to AMSDU (AMSDU must run on AMDPU, that means BA stream must be established). If you ping to router, the response time should be quick.

BrainSlayer · 2017-05-15T10:19:47Z

@ONjAXX if WMM is still disabled there will be no ba stream. it must be enabled for testing

ONjAXX · 2017-05-15T10:25:30Z

WMM is enabled. I did clean Lede install with latest mwlwifi

ONjAXX · 2017-05-15T10:30:28Z

A am sorry, I posted wrong data:
cat /sys/kernel/debug/ieee80211/phy0/mwlwifi/ampdu
stream: 0
idx: 0
state: 3
mac address: 00:28:xxxxxxxxx
tid: 0

yuhhaurlin · 2017-05-15T10:32:43Z

Yes, WMM must enable. BA streams of 88W8864 are limited. If traffic with a client is low, BA stream will be removed. 88W8964 has no this mechanism, it can support full BA streams for all TIDs of all associated stations.

yuhhaurlin · 2017-05-15T10:36:18Z

Did you have traffic with this client? If you stop traffic and only ping out, the BA stream will be removed. That is, if BA stream is established, it means traffic running with client, there should be no need to flush AMSDU if traffic is running.

ONjAXX · 2017-05-15T10:39:39Z

Yes, once I stop the youtube stream, BA stream is getting removed (cat reads nothing) and ping backs to 1ms. That means, that my problem is indeed AMSDU? Can you point me out please, how can I safely disable it (maybe in sources) to test the latency and possible harm to bandwidth? In my case latency is much more critical. Thanks a lot!

ONjAXX · 2017-05-15T10:41:45Z

Maybe there's some threshold when this technology kicks in to play with, without fully disabling?

yuhhaurlin · 2017-05-15T10:41:48Z

I am just curious: what kind of application that you encounter problems?

yuhhaurlin · 2017-05-15T10:44:53Z

All right, I think I will add debug file to control AMPDU and AMSDU. But it will be done later. 88W8964 has higher priority. Thanks.

ONjAXX · 2017-05-15T10:46:29Z

Basically everything reads higher pings, like online gaming, where I cleary see 90-100ms with added jerkines versus sub 50 ms on stock fw. This is not a placebo. When the most critical path (lan) is delayed, everything will be a bit slower. Thank you for considering adding a tune!
Meanwhile, I'll try to dig into sources and try myself, will report if I'll find something

ONjAXX · 2017-05-15T11:53:13Z

One addition: I also notice packet drops when AMSDU kicks in:
Reply from 192.168.1.1: bytes=32 time=1ms TTL=64
Reply from 192.168.1.1: bytes=32 time<1ms TTL=64
Reply from 192.168.1.1: bytes=32 time<1ms TTL=64
Reply from 192.168.1.1: bytes=32 time<1ms TTL=64
Reply from 192.168.1.1: bytes=32 time=2ms TTL=64
Reply from 192.168.1.1: bytes=32 time<1ms TTL=64
Reply from 192.168.1.1: bytes=32 time<1ms TTL=64
Request timed out.
Reply from 192.168.1.1: bytes=32 time=23ms TTL=64
Reply from 192.168.1.1: bytes=32 time=28ms TTL=64
Reply from 192.168.1.1: bytes=32 time=55ms TTL=64
Reply from 192.168.1.1: bytes=32 time=42ms TTL=64

ONjAXX · 2017-05-15T15:00:08Z

Ok, I recompiled with
mac80211.c:
sta_info->is_amsdu_allowed = false

Ping is OK now (~1-3ms), same as stock. WAN throughput is unaffected on 50mbit channel, will check LAN throughput and report back.

edit: WLAN speed is also seems the same. I will keep this config.

yuhhaurlin · 2017-05-15T17:12:17Z

Thanks. I will check it later.

starcms · 2017-05-25T13:14:57Z

@yuhhaurlin, I notice the same thing here on my WRT1200AC as @ONjAXX reported. Using 5GHz (didn't try 2.4), when pinging router with nothing else running, ping is fine (1-3ms). But simply reloading a webpage, or watching a youtube video causes pings to go to the 20-40ms range, with some packet loss as well.

Basically, whenever the output of cat /sys/kernel/debug/ieee80211/phy0/mwlwifi/ampdu is blank, pings are normal, but whenever there is activity and cat /sys/kernel/debug/ieee80211/phy0/mwlwifi/ampdu has output, pings go to the 20-40ms range (with occasional packet loss).

I am using the very latest version of mwlwifi with all the latest commits baked in on @davidc502's latest build of LEDE.

Thanks!

ONjAXX · 2017-06-13T17:08:29Z

I hope this issue wasn't forgotten, maybe it is worth getting it reopen?

Toetje585 · 2017-07-22T23:05:53Z

Wow, I'm not sure if it's related but since I switched to a wrt1900acs v2 gaming on my wifi is not possible anymore, goes up to 70. Also while playing rocket league I notice lots of lag spikes. And i'm also using @ @davidc502's latest build of lede. I also notice while reloading a webpage or anything else that generates a bit of load the latency goes up to around 300 now and then. For the info only one device is connected to the 5ghz band.

ONjAXX · 2017-07-23T11:19:39Z

@Toetje585 You may recompile the driver with AMSDU disabled:

mac80211.c:
sta_info->is_amsdu_allowed = false

I run such config while waiting for the proper solution.

Toetje585 · 2017-07-23T11:53:18Z

@ONjAXX Ohh boy, Never did compile a thing, might need to read a bit about it and try it. Thanks for all the info!

starcms · 2017-07-30T02:41:23Z

@ONjAXX, how exactly do you compile the driver with:

mac80211.c:
sta_info->is_amsdu_allowed = false

When I look at the mac802.11c file, I see:

if (sta->ht_cap.ht_supported) {
        sta_info->is_ampdu_allowed = true;
        sta_info->is_amsdu_allowed = false;
        if (sta->ht_cap.cap & IEEE80211_HT_CAP_MAX_AMSDU)
            sta_info->amsdu_ctrl.cap = MWL_AMSDU_SIZE_8K;
        else
            sta_info->amsdu_ctrl.cap = MWL_AMSDU_SIZE_4K;
        if ((sta->tdls) && (!sta->wme))
            sta->wme = true;

This shows it already set to false and all other references to amsdu are also already set to false in the mac80211.c file. What am I missing here?

starcms · 2017-07-30T03:09:45Z

@yuhhaurlin, could you please revisit this issue?

ONjAXX · 2017-07-30T11:03:16Z

@starcms Line 733
sta_info->is_amsdu_allowed = params->amsdu;
->
sta_info->is_amsdu_allowed = false

starcms · 2017-08-01T03:18:47Z

@ONjAXX thanks so much! The fix works perfectly on my 1200AC! Any news if @yuhhaurlin will look into amsdu and find the actual problem?

yuhhaurlin closed this as completed May 21, 2016

yuhhaurlin reopened this May 21, 2016

This was referenced Aug 16, 2017

5GHz band dropping clients sporadically under light sustained load #206

Closed

Latency of ping is long if AMSDU is on for 88W8864 #207

Closed

High ping on WRT1900ACS #74

High ping on WRT1900ACS #74

Comments

lefoyer commented Apr 17, 2016

Razerwire commented Apr 18, 2016 • edited

lefoyer commented Apr 18, 2016

itrankolov commented Apr 27, 2016 • edited

MrDoh commented Apr 27, 2016 • edited

itrankolov commented Apr 27, 2016

suihkulokki commented Apr 28, 2016

itrankolov commented Apr 28, 2016

yuhhaurlin commented Apr 28, 2016

yuhhaurlin commented Apr 28, 2016

itrankolov commented Apr 28, 2016

itrankolov commented Apr 28, 2016

yuhhaurlin commented Apr 28, 2016

itrankolov commented Apr 28, 2016

yuhhaurlin commented Apr 28, 2016

yuhhaurlin commented Apr 28, 2016

itrankolov commented Apr 28, 2016

itrankolov commented Apr 28, 2016 • edited

yuhhaurlin commented Apr 28, 2016

yuhhaurlin commented Apr 28, 2016

itrankolov commented Apr 28, 2016

yuhhaurlin commented May 20, 2016

johnnysl commented May 20, 2016 • edited

yuhhaurlin commented May 21, 2016

MrDoh commented May 21, 2016

yuhhaurlin commented May 21, 2016

MrDoh commented May 21, 2016

bill1228 commented May 21, 2016

yuhhaurlin commented May 21, 2016

ONjAXX commented May 13, 2017 • edited

yuhhaurlin commented May 13, 2017

ONjAXX commented May 13, 2017

yuhhaurlin commented May 15, 2017

ONjAXX commented May 15, 2017 • edited

yuhhaurlin commented May 15, 2017

BrainSlayer commented May 15, 2017

ONjAXX commented May 15, 2017

ONjAXX commented May 15, 2017

yuhhaurlin commented May 15, 2017

yuhhaurlin commented May 15, 2017

ONjAXX commented May 15, 2017

ONjAXX commented May 15, 2017

yuhhaurlin commented May 15, 2017

yuhhaurlin commented May 15, 2017

ONjAXX commented May 15, 2017 • edited

ONjAXX commented May 15, 2017

ONjAXX commented May 15, 2017 • edited

yuhhaurlin commented May 15, 2017

starcms commented May 25, 2017

ONjAXX commented Jun 13, 2017

Toetje585 commented Jul 22, 2017

ONjAXX commented Jul 23, 2017

Toetje585 commented Jul 23, 2017

starcms commented Jul 30, 2017 • edited

starcms commented Jul 30, 2017

ONjAXX commented Jul 30, 2017

starcms commented Aug 1, 2017 • edited

Razerwire commented Apr 18, 2016 •

edited

itrankolov commented Apr 27, 2016 •

edited

MrDoh commented Apr 27, 2016 •

edited

itrankolov commented Apr 28, 2016 •

edited

johnnysl commented May 20, 2016 •

edited

ONjAXX commented May 13, 2017 •

edited

ONjAXX commented May 15, 2017 •

edited

ONjAXX commented May 15, 2017 •

edited

ONjAXX commented May 15, 2017 •

edited

starcms commented Jul 30, 2017 •

edited

starcms commented Aug 1, 2017 •

edited