Can't stop powerd++ on recent CURRENT #3

z2v · 2016-12-19T08:13:59Z

service powerd++ stop hangs forever.

truss -p while stopping shows only repeating blocks like this:

nanosleep({ 0.500000000 })                       = 0 (0x0)
__sysctl(0x60b7a0,0x2,0x801a220a0,0x7fffffffea08,0x0,0x0) = 0 (0x0)
__sysctl(0x60b784,0x3,0x60b790,0x7fffffffeb78,0x0,0x0) = 0 (0x0)
__sysctl(0x801a16000,0x4,0x7fffffffebbc,0x7fffffffea18,0x0,0x0) = 0 (0x0)

$ uname -a
FreeBSD vzakharov 12.0-CURRENT FreeBSD 12.0-CURRENT #12 r310173: Sat Dec 17 00:18:14 MSK 2016     root@vzakharov:/home/obj/usr/src/sys/GENERIC-NODEBUG  amd64

Reinstalling from port does not help.

$ pkg info | grep powerd
powerdxx-0.1.5                 CPU clock speed/frequency daemon

The text was updated successfully, but these errors were encountered:

lonkamikaze · 2016-12-19T23:18:00Z

Does powerd++ perform its job while running? Does it react to kill -9?

z2v · 2016-12-20T08:10:35Z

Does powerd++ perform its job while running?

It seems, it does not.

$ ps aux | grep powerd
root    1550   0,0  0,0   19920   3280  -  SXs  11:01    0:00,28 /usr/local/sbin/powerd++ -a maximum -b adaptive
$ sysctl -a | grep "dev.cpu...freq"
dev.cpu.0.freq_levels: 2201/15000 2200/15000 2100/14088 2000/13193 1800/11602 1700/10901 1600/10075 1500/9404 1300/7966 1200/7204 1100/6591 1000/5860 900/5145 700/3890 600/3346 500/2693
dev.cpu.0.freq: 2201
[disconnect AC power line]
$ sysctl -a | grep "dev.cpu...freq"
dev.cpu.0.freq_levels: 2201/15000 2200/15000 2100/14088 2000/13193 1800/11602 1700/10901 1600/10075 1500/9404 1300/7966 1200/7204 1100/6591 1000/5860 900/5145 700/3890 600/3346 500/2693
dev.cpu.0.freq: 2201

Does it react to kill -9?

Yes, it dies.

z2v · 2016-12-20T08:15:28Z

BTW, powerd++ started to hang on stopping about a week ago. I update system every 2-3 days.

z2v · 2016-12-20T08:24:58Z

powerd++ built from HEAD does not work either. The only difference in truss -p output: argument of nanosleep call changes on every call:

nanosleep({ 0.489696449 })                       = 0 (0x0)
__sysctl(0x60bd08,0x2,0x801a1c000,0x7fffffffa798,0x0,0x0) = 0 (0x0)
__sysctl(0x801a16008,0x4,0x7fffffffa7cc,0x7fffffffa798,0x0,0x0) = 0 (0x0)
__sysctl(0x60bcf0,0x3,0x7fffffffa7e0,0x7fffffffa7b8,0x0,0x0) = 0 (0x0)
nanosleep({ 0.460852926 })                       = 0 (0x0)
__sysctl(0x60bd08,0x2,0x801a1c000,0x7fffffffa798,0x0,0x0) = 0 (0x0)
__sysctl(0x801a16008,0x4,0x7fffffffa7cc,0x7fffffffa798,0x0,0x0) = 0 (0x0)
__sysctl(0x60bcf0,0x3,0x7fffffffa7e0,0x7fffffffa7b8,0x0,0x0) = 0 (0x0)
nanosleep({ 0.492645268 })                       = 0 (0x0)
__sysctl(0x60bd08,0x2,0x801a1c000,0x7fffffffa798,0x0,0x0) = 0 (0x0)
__sysctl(0x801a16008,0x4,0x7fffffffa7cc,0x7fffffffa798,0x0,0x0) = 0 (0x0)
__sysctl(0x60bcf0,0x3,0x7fffffffa7e0,0x7fffffffa7b8,0x0,0x0) = 0 (0x0)

lonkamikaze · 2016-12-20T08:46:16Z

What happens if you run it in foreground, in verbose mode?

powerd++ -vf

z2v · 2016-12-20T08:54:57Z

FYI, I've updated local port to 0.2.4.

# /usr/local/sbin/powerd++ -vf -a maximum -b adaptive
powerd++: cannot access sysctl: dev.cpu.1.freq
powerd++: cannot access sysctl: dev.cpu.2.freq
powerd++: cannot access sysctl: dev.cpu.3.freq
Terminal Output
        verbose:               yes
        foreground:            yes
Load Sampling
        load samples:          4
        polling interval:      500 ms
        load average over:     2000 ms
Frequency Limits
        battery:               [0 MHz, 1000000 MHz]
        online:                [0 MHz, 1000000 MHz]
        unknown:               [0 MHz, 1000000 MHz]
CPU Cores
        CPU cores:             4
Core Groups
        0: [0, 3]
Core Frequency Limits
        0: [500 MHz, 2201 MHz]
Load Targets
        battery power target:  50% load
        online power target:   1000000 MHz
        unknown power target:  38% load
power:  online, load:   41 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  114 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  114 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
...

After disconnecting power line still shows

power:  online, load:  296 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  218 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  203 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  337 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  390 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz

lonkamikaze · 2016-12-20T09:01:11Z

Wow, this is so totally wrong!

Maybe a bug in an libc++ update (std::max, or std::min must be broken). Can you try to build upstream using gcc5?

If you have lang/gcc5 installed, just run make clean g++5. You can execute directly from the build directory: obj/powerd++ -vf

lonkamikaze · 2016-12-20T09:10:06Z

Oh, I just saw you set -a maximum. So it does what it's supposed to. How does it react to CTRL+C (i.e. SIGINT)?

z2v · 2016-12-20T09:11:45Z

vovka@vzakharov:~/projects/contrib/powerdxx > make clean g++5
rm -f *.o powerd++ loadrec libloadplay.so
cd "/home/vovka/projects/contrib/powerdxx" && make CXX="g++5" CXXFLAGS="-O2 -pipe -std=c++11 -Wall -Werror -pedantic -fmax-errors=2 -Wold-style-cast -Wl,-rpath=/usr/local/lib/gcc5"
g++5  -O2 -pipe -std=c++11 -Wall -Werror -pedantic -fmax-errors=2 -Wold-style-cast -Wl,-rpath=/usr/local/lib/gcc5 -c /home/vovka/projects/contrib/powerdxx/src/powerd++.cpp -o powerd++.o
g++5 -lutil -O2 -pipe -std=c++11 -Wall -Werror -pedantic -fmax-errors=2 -Wold-style-cast -Wl,-rpath=/usr/local/lib/gcc5 -o powerd++ powerd++.o
g++5  -O2 -pipe -std=c++11 -Wall -Werror -pedantic -fmax-errors=2 -Wold-style-cast -Wl,-rpath=/usr/local/lib/gcc5 -c /home/vovka/projects/contrib/powerdxx/src/loadrec.cpp -o loadrec.o
g++5 -O2 -pipe -std=c++11 -Wall -Werror -pedantic -fmax-errors=2 -Wold-style-cast -Wl,-rpath=/usr/local/lib/gcc5-o loadrec loadrec.o
g++5 -c -O2 -pipe -std=c++11 -Wall -Werror -pedantic -fmax-errors=2 -Wold-style-cast -Wl,-rpath=/usr/local/lib/gcc5 -fPIC -o loadplay.o /home/vovka/projects/contrib/powerdxx/src/loadplay.cpp
g++5 -lpthread -shared -O2 -pipe -std=c++11 -Wall -Werror -pedantic -fmax-errors=2 -Wold-style-cast -Wl,-rpath=/usr/local/lib/gcc5 -o libloadplay.so loadplay.o
vovka@vzakharov:~/projects/contrib/powerdxx > sudo obj/powerd++ -vf -a maximum -b adaptive
powerd++: cannot access sysctl: dev.cpu.1.freq
powerd++: cannot access sysctl: dev.cpu.2.freq
powerd++: cannot access sysctl: dev.cpu.3.freq
Terminal Output
        verbose:               yes
        foreground:            yes
Load Sampling
        load samples:          4
        polling interval:      500 ms
        load average over:     2000 ms
Frequency Limits
        battery:               [0 MHz, 1000000 MHz]
        online:                [0 MHz, 1000000 MHz]
        unknown:               [0 MHz, 1000000 MHz]
CPU Cores
        CPU cores:             4
Core Groups
        0: [0, 3]
Core Frequency Limits
        0: [500 MHz, 2201 MHz]
Load Targets
        battery power target:  50% load
        online power target:   1000000% load
        unknown power target:  38% load
power:  online, load:   58 MHz, cpu0.freq:  500 MHz, wanted: 1000000 MHz
power:  online, load:   92 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:   92 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:   98 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  140 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  115 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  132 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  202 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  245 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
...

Still no reaction on power line disconnecting.

CTRL+C is ignored. Have to run kill -9 to stop.

lonkamikaze · 2016-12-20T09:12:27Z

Check sysctl hw.acpi.acline. You probably have an acpi problem that causes your system not to know the power was disconnected. I have the same issue if I use EFI boot. This is not powerd++ related.

lonkamikaze · 2016-12-20T09:13:07Z

In verbose mode it's supposed to tell you about the signals it receives. So this is definitely a bug.

z2v · 2016-12-20T09:20:41Z

Yep, hw.acpi.acline always equal to 1.

I've found this messages in /var/log/messages:

ACPI Error: Needed type [Reference], found [Processor] 0xfffff800043b8980 (20161117/exresop-111)
ACPI Exception: AE_AML_OPERAND_TYPE, While resolving operands for [OpcodeName unavailable] (20161117/dswexec-498)
ACPI Error: Method parse/execution failed [\134_SB.PCI0.LPCB.EC0.PPNT] (Node 0xfffff80004396640), AE_AML_OPERAND_TYPE (20161117/psparse-560)
ACPI Error: Method parse/execution failed [\134_SB.PCI0.LPCB.EC0._Q04] (Node 0xfffff80004396c40), AE_AML_OPERAND_TYPE (20161117/psparse-560)
acpi_ec0: evaluation of query method _Q04 failed: AE_AML_OPERAND_TYPE

I'll try to revert r308953 (Merge ACPICA 20161117) and recheck.

lonkamikaze · 2016-12-20T09:23:47Z

I cannot help you with the acpi problem (I work around it by using BIOS mode, I've got all kinds of acpi problems when booting through EFI).

I will concentrate on the signal() issue.

z2v · 2016-12-20T09:25:19Z

Ok

lonkamikaze · 2016-12-20T10:05:17Z

A friend managed to reproduce the problem.

lonkamikaze · 2016-12-28T10:19:29Z

I just committed a workaround. I am 99% sure this is a FreeBSD userland bug, so I will keep this open until the workaround can be removed.

See commit a97ab9e.

z2v · 2016-12-28T10:30:33Z

Ok, seems to be working now:

$ git log -1 --oneline --decorate
 2d80d99 (HEAD -> master, origin/master, origin/HEAD) Document linkflags
$ make clean && make
...
$  sudo obj/powerd++ -vf -a maximum -b adaptive
[...]
power:  online, load:  139 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
^Cpower:  online, load:  190 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
powerd++: signal 2 received, exiting ...
$

lonkamikaze · 2017-01-09T16:54:27Z

I succeeded in providing a minimal test case (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215826#c1) and a test case that shows what goes wrong (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215826#c2).

Turns out that after an exception is thrown and caught, the sigprocmask is changed.

z2v · 2017-01-10T11:37:14Z

Well done, thanks. My ACPI problems has been fixed recently (by ACPICA 20161222 merge), so powerdxx seems to be working fully again:

$ sudo obj/powerd++ -vf -a maximum -b adaptive
powerd++: cannot access sysctl: dev.cpu.1.freq
powerd++: cannot access sysctl: dev.cpu.2.freq
powerd++: cannot access sysctl: dev.cpu.3.freq
Terminal Output
        verbose:               yes
        foreground:            yes
Load Sampling
        load samples:          4
        polling interval:      500 ms
        load average over:     2000 ms
Frequency Limits
        battery:               [0 MHz, 1000000 MHz]
        online:                [0 MHz, 1000000 MHz]
        unknown:               [0 MHz, 1000000 MHz]
CPU Cores
        CPU cores:             4
Core Groups
        0: [0, 3]
Core Frequency Limits
        0: [500 MHz, 2201 MHz]
Load Targets
        battery power target:  50% load
        online power target:   1000000% load
        unknown power target:  38% load
power:  online, load:  246 MHz, cpu0.freq: 2100 MHz, wanted: 1000000 MHz
power:  online, load:  481 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  682 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  848 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  901 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  883 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  929 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  823 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  788 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  783 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  827 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  880 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  835 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  866 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  877 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load: 1171 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power: battery, load: 1132 MHz, cpu0.freq: 2201 MHz, wanted: 2264 MHz
power: battery, load: 1311 MHz, cpu0.freq: 2201 MHz, wanted: 2622 MHz
power: battery, load: 1448 MHz, cpu0.freq: 2201 MHz, wanted: 2896 MHz
power: battery, load: 1328 MHz, cpu0.freq: 2201 MHz, wanted: 2656 MHz
power: battery, load: 1202 MHz, cpu0.freq: 2201 MHz, wanted: 2404 MHz
power: battery, load: 1071 MHz, cpu0.freq: 2201 MHz, wanted: 2142 MHz
power: battery, load:  779 MHz, cpu0.freq: 2100 MHz, wanted: 1558 MHz
power: battery, load:  709 MHz, cpu0.freq: 1600 MHz, wanted: 1418 MHz
power: battery, load:  757 MHz, cpu0.freq: 1500 MHz, wanted: 1514 MHz
power: battery, load:  696 MHz, cpu0.freq: 1500 MHz, wanted: 1392 MHz
power: battery, load:  617 MHz, cpu0.freq: 1300 MHz, wanted: 1234 MHz
power: battery, load:  566 MHz, cpu0.freq: 1200 MHz, wanted: 1132 MHz
power: battery, load:  594 MHz, cpu0.freq: 1100 MHz, wanted: 1188 MHz
power: battery, load:  559 MHz, cpu0.freq: 1200 MHz, wanted: 1118 MHz
power: battery, load:  504 MHz, cpu0.freq: 1100 MHz, wanted: 1008 MHz
power: battery, load:  495 MHz, cpu0.freq: 1000 MHz, wanted:  990 MHz
power: battery, load:  463 MHz, cpu0.freq: 1000 MHz, wanted:  926 MHz
power: battery, load:  465 MHz, cpu0.freq:  900 MHz, wanted:  930 MHz
power: battery, load:  455 MHz, cpu0.freq:  900 MHz, wanted:  910 MHz
power: battery, load:  486 MHz, cpu0.freq:  900 MHz, wanted:  972 MHz
power: battery, load:  491 MHz, cpu0.freq: 1000 MHz, wanted:  982 MHz
power: battery, load:  498 MHz, cpu0.freq: 1000 MHz, wanted:  996 MHz
power: battery, load:  504 MHz, cpu0.freq: 1000 MHz, wanted: 1008 MHz
power: battery, load:  544 MHz, cpu0.freq: 1000 MHz, wanted: 1088 MHz
power: battery, load:  502 MHz, cpu0.freq: 1100 MHz, wanted: 1004 MHz
power: battery, load:  517 MHz, cpu0.freq: 1000 MHz, wanted: 1034 MHz
power:  online, load:  497 MHz, cpu0.freq: 1000 MHz, wanted: 1000000 MHz
power:  online, load:  621 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  706 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  801 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  891 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:  982 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
^Cpower:  online, load:  981 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
powerd++: signal 2 received, exiting ...

lonkamikaze · 2017-01-10T12:02:06Z

I really should make the output for fixed freq mode prettier.

lonkamikaze · 2017-01-10T23:15:11Z

A fix has been committed to FreeBSD head: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215826#c6

z2v · 2017-01-11T10:56:10Z

I've updated FreeBSD to r311921 and rebuilt powerd++ without linking it to pthread:

vovka@vzakharov:~/projects/contrib/powerdxx > uname -a
FreeBSD vzakharov 12.0-CURRENT FreeBSD 12.0-CURRENT #20 r311921: Wed Jan 11 13:28:42 MSK 2017     root@vzakharov:/home/obj/usr/src/sys/GENERIC-NODEBUG  amd64
vovka@vzakharov:~/projects/contrib/powerdxx > git diff
      1 diff --git a/Makefile b/Makefile
      2 index 282d1a3..80a9605 100644
      3 --- a/Makefile
      4 +++ b/Makefile
      5 @@ -34,10 +34,10 @@ loadplay.o:
      6  #       linked in, e.g. head/r310361 is affected by this.
      7
      8  powerd++: ${.TARGET}.o
      9 -       ${CXX} ${CXXFLAGS} -o ${.TARGET} ${.ALLSRC} -lpthread -lutil
     10 +       ${CXX} ${CXXFLAGS} -o ${.TARGET} ${.ALLSRC} -lutil
     11
     12  loadrec: ${.TARGET}.o
     13 -       ${CXX} ${CXXFLAGS} -o ${.TARGET} ${.ALLSRC} -lpthread
     14 +       ${CXX} ${CXXFLAGS} -o ${.TARGET} ${.ALLSRC}
     15
     16  libloadplay.so: ${.TARGET:C/^lib//:C/\.so$//}.o
     17         ${CXX} ${CXXFLAGS} -o ${.TARGET} ${.ALLSRC} -lpthread -shared
vovka@vzakharov:~/projects/contrib/powerdxx > make clean
vovka@vzakharov:~/projects/contrib/powerdxx > make
c++  -O2 -pipe -std=c++11 -Wall -Werror -pedantic -c /home/vovka/projects/contrib/powerdxx/src/powerd++.cpp -o powerd++.o
c++ -O2 -pipe -std=c++11 -Wall -Werror -pedantic -o powerd++ powerd++.o -lutil
c++  -O2 -pipe -std=c++11 -Wall -Werror -pedantic -c /home/vovka/projects/contrib/powerdxx/src/loadrec.cpp -o loadrec.o
c++ -O2 -pipe -std=c++11 -Wall -Werror -pedantic -o loadrec loadrec.o
c++ -c -O2 -pipe -std=c++11 -Wall -Werror -pedantic -fPIC -o loadplay.o /home/vovka/projects/contrib/powerdxx/src/loadplay.cpp
c++ -O2 -pipe -std=c++11 -Wall -Werror -pedantic -o libloadplay.so loadplay.o -lpthread -shared
vovka@vzakharov:~/projects/contrib/powerdxx > sudo obj/powerd++ -f -a maximum -b adaptive
power:  online, load:   46 MHz, cpu0.freq: 2100 MHz, wanted: 1000000 MHz
power:  online, load:   89 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
power:  online, load:   89 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
^Cpower:  online, load:  120 MHz, cpu0.freq: 2201 MHz, wanted: 1000000 MHz
powerd++: signal 2 received, exiting ...
vovka@vzakharov:~/projects/contrib/powerdxx >

It works!

lonkamikaze · 2017-01-25T14:15:30Z

Fix has been MFC'ed yesterday.

I'll give it two weeks after which I will remove the workaround and close this issue.

lonkamikaze self-assigned this Dec 28, 2016

lonkamikaze added the bug label Jan 6, 2017

lonkamikaze closed this as completed in 655382c Feb 8, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't stop powerd++ on recent CURRENT #3

Can't stop powerd++ on recent CURRENT #3

z2v commented Dec 19, 2016 •

edited

Loading

lonkamikaze commented Dec 19, 2016

z2v commented Dec 20, 2016

z2v commented Dec 20, 2016 •

edited

Loading

z2v commented Dec 20, 2016

lonkamikaze commented Dec 20, 2016

z2v commented Dec 20, 2016

lonkamikaze commented Dec 20, 2016

lonkamikaze commented Dec 20, 2016 •

edited

Loading

z2v commented Dec 20, 2016

lonkamikaze commented Dec 20, 2016

lonkamikaze commented Dec 20, 2016

z2v commented Dec 20, 2016

lonkamikaze commented Dec 20, 2016

z2v commented Dec 20, 2016

lonkamikaze commented Dec 20, 2016

lonkamikaze commented Dec 28, 2016 •

edited

Loading

z2v commented Dec 28, 2016

lonkamikaze commented Jan 9, 2017 •

edited

Loading

z2v commented Jan 10, 2017

lonkamikaze commented Jan 10, 2017

lonkamikaze commented Jan 10, 2017

z2v commented Jan 11, 2017

lonkamikaze commented Jan 25, 2017

Can't stop powerd++ on recent CURRENT #3

Can't stop powerd++ on recent CURRENT #3

Comments

z2v commented Dec 19, 2016 • edited Loading

lonkamikaze commented Dec 19, 2016

z2v commented Dec 20, 2016

z2v commented Dec 20, 2016 • edited Loading

z2v commented Dec 20, 2016

lonkamikaze commented Dec 20, 2016

z2v commented Dec 20, 2016

lonkamikaze commented Dec 20, 2016

lonkamikaze commented Dec 20, 2016 • edited Loading

z2v commented Dec 20, 2016

lonkamikaze commented Dec 20, 2016

lonkamikaze commented Dec 20, 2016

z2v commented Dec 20, 2016

lonkamikaze commented Dec 20, 2016

z2v commented Dec 20, 2016

lonkamikaze commented Dec 20, 2016

lonkamikaze commented Dec 28, 2016 • edited Loading

z2v commented Dec 28, 2016

lonkamikaze commented Jan 9, 2017 • edited Loading

z2v commented Jan 10, 2017

lonkamikaze commented Jan 10, 2017

lonkamikaze commented Jan 10, 2017

z2v commented Jan 11, 2017

lonkamikaze commented Jan 25, 2017

z2v commented Dec 19, 2016 •

edited

Loading

z2v commented Dec 20, 2016 •

edited

Loading

lonkamikaze commented Dec 20, 2016 •

edited

Loading

lonkamikaze commented Dec 28, 2016 •

edited

Loading

lonkamikaze commented Jan 9, 2017 •

edited

Loading