2014-11 Mac strange pausing issue #1956

davelab6 · 2014-11-27T12:39:13Z

@monkeyiq @frank-trampe The latest mac build has a crazy pause going on... I made a video to show the behaviour:

DomT4 · 2014-11-27T12:52:34Z

I can't seem to make this happen in the Homebrew version at all, which is weird. Is this from the app bundle or built from source?

frank-trampe · 2014-11-27T12:53:50Z

Dave told me that it happened in the .app version. He is currently testing the Homebrew version.

DomT4 · 2014-11-27T12:57:41Z

Cheers. It'd be odd if something about the .app bundle itself was causing the pause 😕

monkeyiq · 2014-11-27T13:17:20Z

@davelab6 I got the slowdown on 27_1239 but it seems that 27_1820 is ok for me. The former was git master the later is from the tag for today.

davelab6 · 2014-11-27T14:19:07Z

FontForge.app 201411 26_0425.zip seems ok

davelab6 · 2014-11-27T14:49:01Z

I can't install via brew...

$ ./bootstrap 
bootstrap: running: glibtoolize --quiet
bootstrap: running: git clone --depth 365 'git://git.sv.gnu.org/gnulib' 'gnulib'
Cloning into 'gnulib'...
fatal: unable to connect to git.sv.gnu.org:
git.sv.gnu.org[0: 140.186.70.72]: errno=Connection refused

DomT4 · 2014-11-27T14:53:55Z

Weird. Worked fine earlier. Might be an upstream connection issue, unless you're sat behind a firewall that blocks the git ports. Try again in a little bit I guess :/. Is this stable or head?

Sent from my iPhone

On 27 Nov 2014, at 14:49, Dave Crossland notifications@github.com wrote:

I can't install via brew...

$ ./bootstrap
bootstrap: running: glibtoolize --quiet
bootstrap: running: git clone --depth 365 'git://git.sv.gnu.org/gnulib' 'gnulib'
Cloning into 'gnulib'...
fatal: unable to connect to git.sv.gnu.org:
git.sv.gnu.org[0: 140.186.70.72]: errno=Connection refused
—
Reply to this email directly or view it on GitHub.

frank-trampe · 2014-11-27T15:11:10Z

I'm confused. At present, master is at 20141126. How would a build from master differ from a build from 20141126?

adrientetar · 2014-11-27T15:41:38Z

gnulib has been flaky recently, and it seems like the sv.gnu.org link is just a redirect to git://git.savannah.gnu.org/gnulib.git. Maybe we should use this canonical URL, otherwise it’s a problem with the availability of their servers.

davelab6 · 2014-11-27T20:34:40Z

Master worked perfectly for the workshop, I'll post a photo on my g+
On 27 Nov 2014 15:41, "Adrien Tétar" notifications@github.com wrote:

gnulib has been flaky recently, and it seems like the sv.gnu.org link is
just a redirect to git://git.savannah.gnu.org/gnulib.git. Maybe we should
use this canonical URL, otherwise it’s a problem with the availability of
their servers.

—
Reply to this email directly or view it on GitHub
#1956 (comment)
.

davelab6 · 2014-11-27T20:40:49Z

I put the problem free master Mac build on the release page
On 27 Nov 2014 20:34, "Dave Crossland" dave@lab6.com wrote:

Master worked perfectly for the workshop, I'll post a photo on my g+
On 27 Nov 2014 15:41, "Adrien Tétar" notifications@github.com wrote:

gnulib has been flaky recently, and it seems like the sv.gnu.org link is
just a redirect to git://git.savannah.gnu.org/gnulib.git. Maybe we
should use this canonical URL, otherwise it’s a problem with the
availability of their servers.

—
Reply to this email directly or view it on GitHub
#1956 (comment)
.

monkeyiq · 2014-11-27T22:39:09Z

@frank-trampe I also wondered about that. With master being the same as the date tag when I did the master build. I noticed that the app.zip for both had a different md5, and with testing here one of them had the slowdown and the other didn't. Though didn't investigate more (as yet) as I just wanted one with a recent build that didn't slow down for the workshop.

larsenwork · 2014-11-30T17:45:11Z

The latest release on http://fontforge.github.io/en-US/downloads/mac/ still has slowdown/lag issues after 30 seconds or so

Version

frank-trampe · 2014-11-30T20:27:58Z

Hi, @AndreasLarsen.

For some reason, I am unable to reproduce the problem on my Macintosh. Could you perhaps run this in debug mode in order to see where it is spending its time?

Launch FontForge in debug mode (/Applications/FontForge.app/Contents/MacOS/FontForge --debug).
Start FontForge from lldb (run).
Wait for the slowdown.
Break by using Control + C in the terminal with lldb.
See what is happening (where).
Continue (continue).
Break, check, and continue a few more times.
Report results to Frank.

larsenwork · 2014-11-30T20:44:39Z

This is weird - doesn't happen when I'm in debug mode 😕

frank-trampe · 2014-11-30T20:48:39Z

And the problem was never solved. 😩

Do you know how to attach lldb to a running process? That might help here.

larsenwork · 2014-11-30T21:15:27Z

Never really tried it - not much of a programmer I'm afraid.
(lldb) file /Applications/FontForge.app error: '/Applications/FontForge.app' doesn't contain any 'host' platform architectures: x86_64, i386

frank-trampe · 2014-11-30T21:24:30Z

You need to give lldb the path to the FontForge binary. It is /Applications/FontForge.app/Resources/opt/local/bin/fontforge. Once you have specified the file, grep for the pid (ps -Af | grep fontforge), get the pid from the second column, and use the attach command in lldb.

larsenwork · 2014-11-30T22:10:47Z

Ok, will look into this tomorrow - just downloaded the latest HEAD or what it's called. Works without issues so far...

monkeyiq · 2014-12-03T08:13:37Z

good to see that things seem to be working at the moment. It might be useful to have a script that lets poeple attach to the running fontforge. It is unlikely that folks are so interested in doing it manually, but a "run this" then control-c the terminal and type bt might provide a quick route to helping resolve issues in the future.

I'll have a quick dig at that now.

monkeyiq · 2014-12-03T10:12:03Z

I've been playing around in 201412/03_1840 for a while now (at least in the minutes 5+) drawing stars and opening and closing the charview window. Can't seem to trigger the slowdown here on that version. FWIW using XQuartz 2.7.7 and osx 10.9.5.

larsenwork · 2014-12-06T15:52:37Z

Latest HEAD version, quartz 2.7.7 and OS X 10.10 issue still persist - haven't been able to figure out what causes it since it doesn't happen every time.
A debug script of some sort would be great :)

pathumego · 2014-12-07T17:06:52Z

Yep, it works in debug mode so far. Otherwise its 5 to 30 sec pause.

BenTalagan · 2014-12-07T22:52:13Z

I have observed the following behaviour (don't know if it will help), under MacOsX 10.9.5: reinstalling FontForge fixes the issue, but on first launch only. After then, every time I try to launch FF, I face the same freeze problem.

I can also see a difference between the very first launch and the other ones:

On the very first launch, only one app icon is displayed in the dock: xquartz.
On the next launches, two icons are shown in the dock: xquartz and fontforge.

When the lag starts to occur, clicking on the FF icon unfreezes immediately the app. It's quite strange, looks like there are interferences between xquartz, mac os x, and fontforge

Add: Launching FF in debug mode as explained in the thread works for me; I have only one icon in the dock then: xquartz.

pathumego · 2014-12-08T12:33:56Z

@venildo verify this behaviour on OSx10.10.1. I experience the same "When the lag starts to occur, clicking on the FF icon unfreezes immediately the app. "

davelab6 · 2014-12-10T01:21:18Z

Me too on 10.9

davelab6 · 2014-12-10T01:33:32Z

I can reliably reproduce when I open FontForge on a file, and do things for about 30s:

$ open -a FontForge font.ttf;
$ lldb;
(lldb) attach /Applications/FontForge.app/Resources/opt/local/bin/fontforge
error: attach failed: lost connection
(lldb)
$

davelab6 · 2014-12-15T22:45:44Z

Spoke to @frank-trampe to find out exactly how to get a backtrace on this without starting in --debug mode. If we can post several back traces using this method that are 'caught' with Ctrl-C at the moment when it is hanging, its possible Frank can figure it out :)

First open the Fontforge app via terminal:

$ open -a FontForge font.ttf;

Then find its process id (pid):

$ ps aux | grep fontforge | head -n1 | awk '{print $2}'
1234
$

Then run lldb;

(lldb) file /Applications/FontForge.app/Resources/opt/local/bin/fontforge
Current executable set to '/Applications/FontForge.app/Contents/Resources/opt/local/bin/fontforge' (x86_64).
(lldb) attach 1234
Process 1234 stopped
(lldb) c
Process 1234 resuming
(llldb)

And then run FontForge until the pausing occurs, and then press Ctrl-C in the lldb terminal window, and then run

^C
Process 1234 stopped
* thread #1: tid = 0x43b13a, 0x00007fff94109a1a libsystem_kernel.dylib`mach_msg_trap + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x00007fff94109a1a libsystem_kernel.dylib`mach_msg_trap + 10
libsystem_kernel.dylib`mach_msg_trap + 10:
-> 0x7fff94109a1a:  retq   
   0x7fff94109a1b:  nop    

libsystem_kernel.dylib`mach_msg_overwrite_trap:
   0x7fff94109a1c:  movq   %rcx, %r10
   0x7fff94109a1f:  movl   $0x1000020, %eax
(lldb) bt
* thread #1: tid = 0x43b13a, 0x00007fff94109a1a libsystem_kernel.dylib`mach_msg_trap + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007fff94109a1a libsystem_kernel.dylib`mach_msg_trap + 10
    frame #1: 0x00007fff94108d18 libsystem_kernel.dylib`mach_msg + 64
    frame #2: 0x00007fff8c548f15 CoreFoundation`__CFRunLoopServiceMachPort + 181
    frame #3: 0x00007fff8c548539 CoreFoundation`__CFRunLoopRun + 1161
    frame #4: 0x00007fff8c547e75 CoreFoundation`CFRunLoopRunSpecific + 309
    frame #5: 0x00007fff98110a0d HIToolbox`RunCurrentEventLoopInMode + 226
    frame #6: 0x00007fff981107b7 HIToolbox`ReceiveNextEventCommon + 479
    frame #7: 0x00007fff9815e5cd HIToolbox`AcquireNextEventInMode + 51
    frame #8: 0x00007fff98252c2d HIToolbox`RunApplicationEventLoop + 170
    frame #9: 0x000000010015e93c libfontforgeexe.2.dylib`fontforge_main(argc=<unavailable>, argv=<unavailable>) + 6642 at startui.c:1362
    frame #10: 0x000000010000ef2c fontforge`start + 52
(lldb)

(The above is just an example, I couldn't reproduce the issue just yet.)

pathumego · 2014-12-16T15:09:47Z

Ok I did this. I don't have to wait FF to pause. It just pauses at launch. And I noticed when I click on some other programme It right away un pauses.

Last login: Tue Dec 16 20:29:52 on ttys000
Pathums-MacBook-Pro:~ Admin$ open -a FontForge NidahasFMAbhaya-Regular.ttf;
Pathums-MacBook-Pro:~ Admin$ ps aux | grep fontforge | head -n1 | awk '{print $2}'
6091
Pathums-MacBook-Pro:~ Admin$ (lldb)
(lldb) attach 6091
Process 6091 stopped
Executable module set to "/Applications/FontForge.app/Contents/Resources/opt/local/bin/fontforge".
Architecture set to: x86_64h-apple-macosx.
(lldb) c
Process 6091 resuming
Process 6091 stopped
* thread #1: tid = 0x747d3, 0x00007fff8b81952e libsystem_kernel.dylib`mach_msg_trap + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x00007fff8b81952e libsystem_kernel.dylib`mach_msg_trap + 10
libsystem_kernel.dylib`mach_msg_trap + 10:
-> 0x7fff8b81952e:  retq   
   0x7fff8b81952f:  nop    

libsystem_kernel.dylib`mach_msg_overwrite_trap:
   0x7fff8b819530:  movq   %rcx, %r10
   0x7fff8b819533:  movl   $0x1000020, %eax
(lldb) bt
error: libfontforgeexe.2.dylib debug map object file '/Applications/FontForge.app/Contents/Resources/opt/local/var/////////////////////////////////////////////////////////////////////fontforgeexe/.libs/libfontforgeexe_la-startui.o' has changed (actual time is 0x548c6c8c, debug map time is 0x548c2d43) since this executable was linked, file will be ignored
* thread #1: tid = 0x747d3, 0x00007fff8b81952e libsystem_kernel.dylib`mach_msg_trap + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007fff8b81952e libsystem_kernel.dylib`mach_msg_trap + 10
    frame #1: 0x00007fff8b81869f libsystem_kernel.dylib`mach_msg + 55
    frame #2: 0x00007fff971bdb14 CoreFoundation`__CFRunLoopServiceMachPort + 212
    frame #3: 0x00007fff971bcfdb CoreFoundation`__CFRunLoopRun + 1371
    frame #4: 0x00007fff971bc838 CoreFoundation`CFRunLoopRunSpecific + 296
    frame #5: 0x00007fff8f1fd43f HIToolbox`RunCurrentEventLoopInMode + 235
    frame #6: 0x00007fff8f1fd1ba HIToolbox`ReceiveNextEventCommon + 431
    frame #7: 0x00007fff8f351537 HIToolbox`_AcquireNextEvent + 50
    frame #8: 0x00007fff8f346951 HIToolbox`RunApplicationEventLoop + 170
    frame #9: 0x000000010015da40 libfontforgeexe.2.dylib`fontforge_main + 6642
    frame #10: 0x000000010000ef2c fontforge`start + 52
(lldb)

pathumego · 2014-12-16T20:10:22Z

@davelab6 FontForge.app 201411 26_0425.zip which is on downloads page has this issue on my mac.

pathumego · 2014-12-16T20:14:55Z

Using FontForge.app 201411 26_0425.zip on OSx 10.10.1

Last login: Wed Dec 17 01:41:05 on ttys001
Pathums-MacBook-Pro:~ Admin$ open -a FontForge NidahasFMAbhaya-Regular.ttf;
Pathums-MacBook-Pro:~ Admin$ ps aux | grep fontforge | head -n1 | awk '{print $2}'
8334
Pathums-MacBook-Pro:~ Admin$ (lldb)
(lldb) attach 8334
Process 8334 stopped
Executable module set to "/Applications/FontForge.app/Contents/Resources/opt/local/bin/fontforge".
Architecture set to: x86_64h-apple-macosx.
(lldb) c
Process 8334 resuming
Process 8334 stopped
* thread #1: tid = 0xb193a, 0x00007fff8b81952e libsystem_kernel.dylib`mach_msg_trap + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x00007fff8b81952e libsystem_kernel.dylib`mach_msg_trap + 10
libsystem_kernel.dylib`mach_msg_trap + 10:
-> 0x7fff8b81952e:  retq   
   0x7fff8b81952f:  nop    

libsystem_kernel.dylib`mach_msg_overwrite_trap:
   0x7fff8b819530:  movq   %rcx, %r10
   0x7fff8b819533:  movl   $0x1000020, %eax
(lldb) bt
* thread #1: tid = 0xb193a, 0x00007fff8b81952e libsystem_kernel.dylib`mach_msg_trap + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007fff8b81952e libsystem_kernel.dylib`mach_msg_trap + 10
    frame #1: 0x00007fff8b81869f libsystem_kernel.dylib`mach_msg + 55
    frame #2: 0x00007fff971bdb14 CoreFoundation`__CFRunLoopServiceMachPort + 212
    frame #3: 0x00007fff971bcfdb CoreFoundation`__CFRunLoopRun + 1371
    frame #4: 0x00007fff971bc838 CoreFoundation`CFRunLoopRunSpecific + 296
    frame #5: 0x00007fff8f1fd43f HIToolbox`RunCurrentEventLoopInMode + 235
    frame #6: 0x00007fff8f1fd1ba HIToolbox`ReceiveNextEventCommon + 431
    frame #7: 0x00007fff8f351537 HIToolbox`_AcquireNextEvent + 50
    frame #8: 0x00007fff8f346951 HIToolbox`RunApplicationEventLoop + 170
    frame #9: 0x000000010015e9e0 libfontforgeexe.2.dylib`fontforge_main + 6642
    frame #10: 0x000000010000ef2c fontforge`start + 52
(lldb)

frank-trampe · 2014-12-16T21:14:54Z

It seems fairly clear from the backtraces that the problem is in RunApplicationEventLoop as called in fontforge_main.

@DomT4, I am curious as to whether the homebrew build defines the __Mac macro. If not, that would explain the different outcomes (since, otherwise, FontForge just uses the GDrawEventLoop). The code comments also suggest that listen_to_apple_events, another prerequisite to using the Apple event handler, only gets set if FontForge is running from the .app bundle, but I cannot confirm that.

According to Apple's documentation, RunApplicationEventLoop is part of the Carbon A. P. I., which is still supported in Yosemite (despite being deprecated with Mountain Lion). But I wonder whether there is some problem in the Yosemite Carbon implementation. Does anybody on an older version of the operating system have this problem?

Another possible problem is that, according to the FontForge source code, RunApplicationEventLoop is not provided by Apple's 64-bit headers (since Carbon is 32-bit-only). FontForge works around this by declaring the function prototype, and it links successfully, but I am inclined to think that there is some compelling reason that Apple did not want this 32-bit function called from 64-bit code. (I don't understand the Mach-O system and how inter-architectural function calls work, if they do.) Perhaps there are inconsistent word sizes somewhere?

There is another possibility. The documentation states that RunApplicationEventLoop waits on relevant events. It could be that we are misconfiguring the event filter and that FontForge only wakes on certain classes of events.

I would appreciate advice from any low-level Macintosh platform experts on this (and any referrals to such experts).

adrientetar · 2014-12-16T21:37:16Z

Would it be hard to switch to Cocoa?

davelab6 · 2014-12-16T21:37:18Z

On 16 December 2014 at 16:14, Frank Trampe notifications@github.com wrote:

Does anybody on an older version of the operating system have this problem?

I'm on 10.9

frank-trampe · 2014-12-16T21:39:37Z

I probably need to figure out how much Carbon FontForge presently uses.

davelab6 · 2014-12-16T21:40:48Z

I found these 2 commits with maybe relevant notes:

4371784

3506ff3

frank-trampe · 2014-12-16T21:48:24Z

This suggests that the main reason for using the Apple event loop was a desire to add Finder integration. So maybe the dependencies are not too deep.

frank-trampe · 2014-12-17T04:40:29Z

I'm currently working through the open Coverity defects in order to rule out stack-smashing problems, but I will plan to look into a Cocoa equivalent to the event loop soon.

frank-trampe · 2014-12-17T05:24:52Z

I decided to cap that work for now. I looked through the remainder of the [very long] defect list and did not see any obvious sources of stack-smashing or over-running problems.

I would be curious as to whether those who experience the pausing problem get it even without opening a typeface. If so, that would rule out a large portion of the codebase in our hunt for any memory corruption problem.

pathumego · 2014-12-17T07:12:07Z

It happens for me without opening anything. It is there from the start-up
itself.

On Wed, Dec 17, 2014, 10:54 AM Frank Trampe notifications@github.com
wrote:

I decided to cap that work for now. I looked through the remainder of the
[very long] defect list and did not see any obvious sources of
stack-smashing or over-running problems.

I would be curious as to whether those who experience the pausing problem
get it even without opening a typeface. If so, that would rule out a large
portion of the codebase in our hunt for any memory corruption problem.

—
Reply to this email directly or view it on GitHub
#1956 (comment)
.

davelab6 · 2014-12-17T13:58:31Z

Yes it happens for me on an Untitled 1 font that is empty, just clicking
menus and such

frank-trampe · 2014-12-17T21:38:55Z

There is not much precedent for using Cocoa in a standard X11 + C application. The GIMP has ported completely to a native Macintosh interface. Inkscape (as far as I know) uses no Cocoa and makes very little use of Carbon. As a result, it does not support file drops onto a running application. So, given the limited loss of functionality that would result and the heavy cost of adding a Cocoa wrapper, I'm inclined to fix this problem by dropping Carbon. Any objections?

I have a few questions for Homebrew folks (@DomT4). Is __Mac defined (in confdefs.h) when building from Homebrew? And does drag + drop work for opening files?

larsenwork · 2014-12-17T21:49:14Z

I would choose increased stability and code simplicity over limited "nice to have" functionality any day

DomT4 · 2014-12-18T00:09:34Z

Is __Mac defined (in confdefs.h)

Not sure to be honest, I'm afraid.

And does drag + drop work for opening files?

From the Finder? Nope. Have to manually do File => Open.

davelab6 · 2014-12-18T00:40:30Z

People, this is crazy. The builds from 2 months back don't do this so it's
something recent we did.

frank-trampe · 2014-12-18T00:48:27Z

If this is some sort of word size or byte alignment problem from calling a 32-bit function from 64-bit code, it is difficult to determine exactly what change triggered the visible problem. When operating on the outer fringes of the specification, the behavior of other parties' code is undefined, and seemingly unrelated adjustments to our code (that change the exact locations of other things in the binary) can make these things appear and disappear. It could even be from an update to Apple's development tools rather than from something that we did. (I would be curious to see whether a new build of old code on @monkeyiq's machine would demonstrate the problem or not.)

That may not be the source of the problem, but I do not have any other leads.

If I could reproduce the problem, I would try bisecting it. Would anybody like to try?

frank-trampe · 2014-12-29T18:25:46Z

@adrientetar, were you able to reproduce the problem previously? Can you test now?

adrientetar · 2014-12-29T18:38:09Z

Well, I have never used FontForge on a Mac so far.

frank-trampe · 2014-12-29T18:39:22Z

Hmm.

frank-trampe · 2014-12-29T18:39:51Z

@davelab6, do you know of anybody who is able to reproduce this problem on a Macintosh and is able to build from source?

davelab6 · 2014-12-30T00:07:18Z

When I built from source using homebrew it didn't happen. I'll try the daily @monkeyiq build tomorrow

pathumego · 2014-12-30T03:56:17Z

Just checked 201412/27_0429/FontForge.app.dmg dose not have this problem :)

frank-trampe · 2014-12-30T04:11:14Z

Excellent. It's time for a release!

davelab6 added the I-perf label Nov 27, 2014

davelab6 assigned monkeyiq Nov 28, 2014

larsenwork mentioned this issue Dec 6, 2014

FF lagging and response time is 5-10 seconds #1977

Closed

frank-trampe mentioned this issue Dec 17, 2014

Fix a few things about which Coverity complained. #2001

Merged

frank-trampe mentioned this issue Dec 19, 2014

Disable the Carbon event loop. #2004

Merged

frank-trampe closed this as completed Dec 30, 2014

adrientetar mentioned this issue Mar 7, 2015

Bring back breakpad on osx #2120

Closed

2014-11 Mac strange pausing issue #1956

2014-11 Mac strange pausing issue #1956

Comments

davelab6 commented Nov 27, 2014

DomT4 commented Nov 27, 2014

frank-trampe commented Nov 27, 2014

DomT4 commented Nov 27, 2014

monkeyiq commented Nov 27, 2014

davelab6 commented Nov 27, 2014

davelab6 commented Nov 27, 2014

DomT4 commented Nov 27, 2014

frank-trampe commented Nov 27, 2014

adrientetar commented Nov 27, 2014

davelab6 commented Nov 27, 2014

davelab6 commented Nov 27, 2014

monkeyiq commented Nov 27, 2014

larsenwork commented Nov 30, 2014

frank-trampe commented Nov 30, 2014

larsenwork commented Nov 30, 2014

frank-trampe commented Nov 30, 2014

larsenwork commented Nov 30, 2014

frank-trampe commented Nov 30, 2014

larsenwork commented Nov 30, 2014

monkeyiq commented Dec 3, 2014

monkeyiq commented Dec 3, 2014

larsenwork commented Dec 6, 2014

pathumego commented Dec 7, 2014

BenTalagan commented Dec 7, 2014

pathumego commented Dec 8, 2014

davelab6 commented Dec 10, 2014

davelab6 commented Dec 10, 2014

davelab6 commented Dec 15, 2014

pathumego commented Dec 16, 2014

pathumego commented Dec 16, 2014

pathumego commented Dec 16, 2014

frank-trampe commented Dec 16, 2014

adrientetar commented Dec 16, 2014

davelab6 commented Dec 16, 2014

frank-trampe commented Dec 16, 2014

davelab6 commented Dec 16, 2014

frank-trampe commented Dec 16, 2014

frank-trampe commented Dec 17, 2014

frank-trampe commented Dec 17, 2014

pathumego commented Dec 17, 2014

davelab6 commented Dec 17, 2014

frank-trampe commented Dec 17, 2014

larsenwork commented Dec 17, 2014

DomT4 commented Dec 18, 2014

davelab6 commented Dec 18, 2014

frank-trampe commented Dec 18, 2014

frank-trampe commented Dec 29, 2014

adrientetar commented Dec 29, 2014

frank-trampe commented Dec 29, 2014

frank-trampe commented Dec 29, 2014

davelab6 commented Dec 30, 2014

pathumego commented Dec 30, 2014

frank-trampe commented Dec 30, 2014