BSOD with Dokan 0.7.3 RC on Windows 8.1 Pro x64 #27

mapomme1108 · 2015-06-28T06:16:58Z

Hello,

I use Dokan 0.7.3 RC (latest) for FISSH on Windows 8.1 Pro x64.

I have BSOD everytime I shutdown the computer.
If I uninstall Dokan, the BSOD dissapears.

Is there à fix for this?

Thanks.

Liryna · 2015-06-28T06:37:27Z

Hello

Can you give me some more informations ?
Do you still have mounted driver when you shutdown ?
Can you make a test with mirror ?

mapomme1108 · 2015-06-28T06:55:05Z

Yes I still have mounted drive when I shutdown.
What is "a test with mirror" ?

Liryna · 2015-06-28T06:57:01Z

In your dokan folder you should have a mirror.exe
https://github.com/dokan-dev/dokany/blob/master/dokan_mirror/mirror.c#L1065

mapomme1108 · 2015-06-28T07:33:08Z

When I type this command
mirror.exe /r d:/ test /l M

Drive M is mounted

Liryna · 2015-06-28T07:34:02Z

What happen if you shutdown when it is mounted ?

mapomme1108 · 2015-06-28T08:01:03Z

When I exit FISSH, mount the drive with mirror.exe and shutdown, there is BSOD

Liryna · 2015-06-28T08:29:29Z

Ok
And if only run mirror.exe without launching at all FISSH ?

marinkobabic · 2015-06-29T18:35:48Z

Please download the driver from this post BenjaminKim/dokanx#47 and test it. Should work also in your case. If it works fine I will make a pull request and dokany can merge it.

Liryna · 2015-06-29T18:41:33Z

Thank you @marinkobabic but I am not a big fan to installing something that I do not know 😢
Is it possible to commit the changes in your repository ?

mapomme1108 · 2015-06-30T08:50:17Z

Hello

I have installed the driver (right click on the INF file and "Install") and there is still BSOD
I have tested it with both x86 and x64 drivers
I still use Dokan 0.7.3 RC : should I try with other versions?

I forgot to mention it but I have a message when FiSSH start :
"Cannot connect with IPC-Server
Application will now close"

marinkobabic · 2015-06-30T21:36:05Z

@mapomme1108
Have now made a new version, the download link remains. If you still have a BSOD would be great to get the memory.dmp or some BSOD details. Maybe you can also provide steps to reproduce, even if I must install some software provided by you.

The change I have made now is related to deletion of device object, when keep alive timeout is reached, is not used or if your application was not closed properly and somebody still tries to access the device.

@Liryna
If I provide the changes now, some discussions will start. I would like to fix the BSOD first and you will get the source for sure. You can also try to investigate the BSOD and maybe we will have the same solution at the end.

mapomme1108 · 2015-07-01T04:27:56Z

I have tried the new version but the problem is the same.
Should I install Dokan 0.7.3 RC if I use your driver or is it useless?

BSOD message is : "SYSTEM_SERVICE_EXECPTION = Dokanx.sys"

I have installed WinSSHFS and it is working (I have used FiSSH so far)
I have just had to install WinSSHFS in Windows 7 compatibility mode and it works
WinSSHFS install its own Dokan Library = 0.60 version

marinkobabic · 2015-07-01T05:55:33Z

You can use the newest dokan library, but my driver. So I have to install https://github.com/tuiSSE/win-sshfs and then run the software. Close it and shutdown the machine to get the BSOD?

mapomme1108 · 2015-07-01T06:06:22Z

Yes you can try but you have to install the Dokan Library 0.7.3 RC first

EDIT : I don't close the software before to shutdown

marinkobabic · 2015-07-01T20:17:17Z

I'm not able to reproduce your issue with https://github.com/dimov-cz/win-sshfs/releases 1.5.12.5 and dokany version 0.7.1. So please provide me all links of the installers and the steps so that I can reproduce the problem. If I can reproduce it, I will fix the BSOD :-)

marinkobabic · 2015-07-02T23:36:55Z

So I was able to reproduce the BSOD using FISSH. Please download this driver and perform the test again and let me know if all works as expected. Let's see if you can break this version of the driver.

What here the special case is, is that we have two dokan devices. That is the reason why it's so hard to reproduce this issue.

landrix · 2015-07-08T20:42:25Z

FYI, with v0.7.3-RC just for a test. i mount folder with the delphi-mirror-sample and shutdown my pc (Win8.1 x64) windows crash (no BSOD). after restart i get a memory dump.

i think, if i would manage WM_QUERYENDSESSION / WM_ENDSESSION, it should unmount correctly, i will check this tomorrow

i don't know, whether there are other ways in the driver, to prevent this behavior

marinkobabic · 2015-07-09T09:18:42Z

You should never get a BSOD even if your application crashes. Did you try the driver above?

Maxhy · 2015-07-09T09:28:45Z

@marinkobabic thank you for your implication, you're helping a lot on Dokan community but if only few people are testing your driver here I believe it is because you just posted binaries without any explanation of your changes. Proceeding like this is not community oriented, and more important it even violates Dokan sys project LGPL license (=> distributing a modified version of the project without the source code). This is definitely not the way it should be done. You should fork the repositories and create a different branch for each of your test.

landrix · 2015-07-09T09:30:50Z

@marinkobabic i will test only drivers from this repo

mapomme1108 · 2015-07-09T09:40:32Z

I got it to work this way :

1- install Dokan 0.6 (required by winsshfs)
2- install winsshfs 0.0.1.5 from Google code
3- install dokan x86 driver provided by marinkobacic
At this point, there is still BSOD

4- Uninstall Dokan 0.6
5- Install Dokan 0.7.3 RC

no more BSOD on my Win 8.1 Pro x64 PC

landrix · 2015-07-09T09:46:31Z

? whats the diffent between

1- install Dokan 0.6 (required by winsshfs)

and

3- install dokan x86 driver provided by marinkobacic

isn't Dokan 0.6 a driver?

mapomme1108 · 2015-07-09T15:03:10Z

1- Install Dokan 0.6
= install Dokan library and Dokan x64 driver; the x86 driver won't install because my OS is x64 (Win 8.1 Pro)
2- install winsshfs
3- install Dokan x86 driver provided by marinkobacic
= there are no conflict with other driver because they were not installed in step 1
4- uninstall Dokan 0.6
5- Install Dokan 0.7.3 RC

It works!

marinkobabic · 2015-07-09T23:59:02Z

@mapomme1108
As you can see in the comment above winsshfs worked from beginning on with latest Rc. Try the same using FISSH and you will fail. No idea why you install a x86 driver on x64 machine.

@Maxhy
I have a fork of dokanx. My commits are just not synced yet. Makes no sense to make a pull request because we have still a not finished discussion about my latest pull request.
If you are not able to reproduce the issues above it will become hard to discuss about the changes because you can't prove them. Don't think the changes would be merged in this case.

Liryna · 2015-07-10T06:27:09Z

@marinkobabic About your pull request, DokanY has sync the part of your pull request that keep the current logic. For us, we think your request is closed.
But as you can see DokanX is practically unmaintained so it gonna be open for a while.

If you want to have feedback and test in your new changes, I highly suggest you to provide a pull request to DokanY.

marinkobabic · 2015-07-10T10:40:44Z

@Liryna
You are fine. I hate it to make questions and to get no feedback. This was the case in the discussion with Mathy and Benjamin.

By the way my changes related to BSOD are now synced to my fork of dokanx. You can investigate the changes. Feel free to ask if something is not clear. I'm still working on a special BSOD which is hard to reproduce. This one was not reported by community.

landrix · 2015-07-10T10:47:07Z

i think this one

https://github.com/marinkobabic/dokanx/tree/Windows8DeleteIssue

Liryna · 2015-07-10T10:48:32Z

@marinkobabic Good news ! Thank you alot :) !
@landrix Thank for the quick link !

I will be glad to look at it in the next days 😃

Maxhy · 2015-07-10T12:20:06Z

Ok I took a look on your changes @marinkobabic. These look really good, you even removed few code duplication on irp completion and few bugs will be fixed indeed.
I'm just wondering about one fix, the one related to DokanStopCheckThreadInternal / DokanStopEventNotificationThreadInternal functions. I believe you did these changes and use IoQueueWorkItem routine to fix OS shutdown BSOD? Is that because IRQL is higher at shutdown? What's your investigation result here? Thank you.

marinkobabic · 2015-07-10T14:41:09Z

CopleteRequest was sent even if the state was STATUS_PENDING. Using one single method which completes the Irp I can ensure that this does not happen again.
The problem which happens rare is that the KeWaitForSingleObject hangs forever. Using the IoAllocateWorkItem I was able to fix the problem.

Liryna · 2015-07-10T14:56:37Z

2- Ok so if I understand well, DokanStopEventNotificationThread will not block but KeWaitForSingleObject in DokanStopEventNotificationThreadInternal will still hangs forever ?

marinkobabic · 2015-07-10T15:00:44Z

Correct. You can test it using FISSH. Simple way to reproduce is:

Run FISSH
Drive Z is mounted. Mount a second drive so that we have minimum two dokan devices
Restart dokan mounter service
Kill FISSH using task manager

Maxhy · 2015-07-10T17:09:24Z

Ok I'm confident with your changes now, thank you for the publication and explanations. I ported your fixes to Dokany with commit 837633a
Thanks @marinkobabic.

Maxhy · 2015-07-10T20:33:24Z

@mapomme1108 could you try with https://github.com/dokan-dev/dokany/releases/tag/0.7.3-RC2? I wasn't able to reproduce your BSOD with previous version so if this is fixed with this pre-release which include @marinkobabic, please close this issue.

mapomme1108 · 2015-07-10T21:13:59Z

Ok, I will try tomorrow

mapomme1108 · 2015-07-11T15:00:23Z

I have tried FISSH package and Winsshfs with 0.7.3 RC2 but both dont work

There are BSOD on unmount and shutdown with this error message :
"PAGE_FAULT_IN_NONPAGE_AREA"

Maybe I can try on a clean Windows 8.1 install?

marinkobabic · 2015-07-11T17:30:43Z

Please try it on clean Windows 8.1 and let me know how I can reproduce it.

entropin · 2015-07-13T07:59:17Z

I'm still having masive problems with BSOD during the development of my app. Whether I close my app or the debug stops it dont seam to matter.

I have an onClose calleback on my c# app where I call dokan.revomeMountPoint('s:') dokan.unmount('s') before exiting but it dose no differens any tip on how do you handle the development cykle? Calling dokanclt.exe /u s return 0 and the app still crache the computer when closing it.

marinkobabic · 2015-07-13T08:47:41Z

Pleaso do the following:

Download the WinDbg
File -> Open crash dump
Navigate to c:\windows\memory.dmp and open it
Type !analyze -v and confirm it with enter
Post the content here of the output whenever it changes

Thanks
Marinko

marinkobabic · 2015-07-13T08:55:58Z

Don't forget to rename the memory dump because the system overwrites it every time you get BSOD. Maybe we need one of them.

landrix · 2015-07-16T16:40:13Z

According to my modest knowledge: Would it bring something to take a look at the stable system driver of TrueCrypt?

Liryna · 2015-07-16T18:00:50Z

It is right that TrueCrypt have a windows driver but dokan work differently :(

jbrads · 2015-07-16T20:48:33Z

I am experiencing the same issue with the C# mirror sample using 0.7.3 RC2 on Windows 8.1 pro x64.

It does not matter how I exit the app, e.g just close it, or unmount the drive first it causes a BSOD with "PAGE_FAULT_IN_NONPAGE_AREA"

Here's is the output of !analyze -v on my memory dump

5: kd> !analyze -v

                   Bugcheck Analysis                                    *

PAGE_FAULT_IN_NONPAGED_AREA (50)
Invalid system memory was referenced. This cannot be protected by try-except,
it must be protected by a Probe. Typically the address is just plain bad or it
is pointing at freed memory.
Arguments:
Arg1: ffffffffffffffd0, memory referenced.
Arg2: 0000000000000001, value 0 = read operation, 1 = write operation.
Arg3: fffff8030f267fb3, If non-zero, the instruction address which referenced the bad memory
address.
Arg4: 0000000000000000, (reserved)

Debugging Details:

***** Kernel symbols are WRONG. Please fix symbols to do analysis.

ADDITIONAL_DEBUG_TEXT:
You can run '.symfix; .reload' to try to fix the symbol path and load symbols.

MODULE_NAME: dokan

FAULTING_MODULE: fffff8030f215000 nt

DEBUG_FLR_IMAGE_TIMESTAMP: 55a020f0

WRITE_ADDRESS: ffffffffffffffd0

FAULTING_IP:
nt!ObfDereferenceObject+23
fffff803`0f267fb3 f0480fc15ed0 lock xadd qword ptr [rsi-30h],rbx

MM_INTERNAL_CODE: 0

DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT

BUGCHECK_STR: AV

CURRENT_IRQL: 0

ANALYSIS_VERSION: 6.3.9600.17336 (debuggers(dbg).150226-1500) amd64fre

LAST_CONTROL_TRANSFER: from fffff8030f39505e to fffff8030f365ca0

STACK_TEXT:
ffffd00021a81368 fffff8030f39505e : 0000000000000050 ffffffffffffffd0 0000000000000001 ffffd00021a815d0 : nt!KeBugCheckEx
ffffd00021a81370 fffff8030f268839 : 0000000000000001 ffffe0014963c080 ffffd00021a815d0 ffffe001528f9080 : nt!CcTestControl+0x2210e
ffffd00021a81410 fffff8030f36ff2f : 0000000000000001 ffffffffffffffff 0000000000000000 fffff8030f2c6226 : nt!ExReleasePushLockEx+0x7d9
ffffd00021a815d0 fffff8030f267fb3 : ffffe0014d860b38 ffffe0014950cb68 ffffe0014950cbd0 ffffe0014d860a00 : nt!setjmpex+0x1f1f
ffffd00021a81760 fffff80174a71af7 : ffffe0014d860da0 ffffe0014d860da0 ffffe00100000000 ffffe00100000000 : nt!ObfDereferenceObject+0x23
ffffd00021a817a0 fffff80174a70afe : ffffe0014d860b38 ffffe0014d860da0 ffffe0014d860a00 ffffe0014d860b38 : dokan+0x5af7
ffffd00021a817e0 fffff80174a6df01 : 00000000c0000002 ffffd00021a81b80 ffffe0014d8608b0 0000000000000000 : dokan+0x4afe
ffffd00021a81830 fffff8030f63a77f : 0000000000000001 ffffe001537fb3e0 ffffe001537fb3e0 0000000000000000 : dokan+0x1f01
ffffd00021a81880 fffff8030f639d22 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!NtDeviceIoControlFile+0xab3
ffffd00021a81a20 fffff8030f3714b3 : ffffd00021a81b10 0000000000000000 ffffd00000000001 00000000005ae4a8 : nt!NtDeviceIoControlFile+0x56
ffffd00021a81a90 0000000077aa2352 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!setjmpex+0x34a3
00000000005aed48 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : 0x77aa2352

STACK_COMMAND: kb

FOLLOWUP_IP:
dokan+5af7
fffff801`74a71af7 4883a39801000000 and qword ptr [rbx+198h],0

SYMBOL_STACK_INDEX: 5

SYMBOL_NAME: dokan+5af7

FOLLOWUP_NAME: MachineOwner

IMAGE_NAME: dokan.sys

BUCKET_ID: WRONG_SYMBOLS

FAILURE_BUCKET_ID: WRONG_SYMBOLS

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:wrong_symbols

FAILURE_ID_HASH: {70b057e8-2462-896f-28e7-ac72d4d365f8}

Followup: MachineOwner

Maxhy · 2015-07-17T05:46:16Z

Thanks for the analyze @jbrads. Is that a complete/kernel memory dump or a small memory dump (minidump)? If minidump, could you share your dump file for analyze on my side too? Otherwise here is the pdb file for Dokan 0.7.3 RC2 Windows 8.1 x64 driver file: http://wikisend.com/download/541738/dokan.pdb (I tried to use SymbolSource.org to automate symbols publication but it is a bit complicated if you're not using NuGet...). My source files are stored on F:\dev\dokan\dokany.

marinkobabic · 2015-07-17T10:37:55Z

Your dump would be very interesting if you could provide it. Is it always the same stack trace? Please send us the link to the dump.

jbrads · 2015-07-18T13:09:49Z

Hi, apologies for the delay. It is a complete memory dump at 2.4GB.

Here is the updated output with the symbols loaded, I'll repro the issue and crash my PC now and see if it changes ;)

ADDITIONAL_DEBUG_TEXT:
You can run '.symfix; .reload' to try to fix the symbol path and load symbols.

MODULE_NAME: dokan

FAULTING_MODULE: fffff8030f215000 nt

DEBUG_FLR_IMAGE_TIMESTAMP: 55a020f0

WRITE_ADDRESS: ffffffffffffffd0

FAULTING_IP:
nt!ObfDereferenceObject+23
fffff803`0f267fb3 f0480fc15ed0 lock xadd qword ptr [rsi-30h],rbx

MM_INTERNAL_CODE: 0

DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT

BUGCHECK_STR: AV

CURRENT_IRQL: 0

ANALYSIS_VERSION: 6.3.9600.17336 (debuggers(dbg).150226-1500) amd64fre

LAST_CONTROL_TRANSFER: from fffff8030f39505e to fffff8030f365ca0

STACK_TEXT:
ffffd00021a81368 fffff8030f39505e : 0000000000000050 ffffffffffffffd0 0000000000000001 ffffd00021a815d0 : nt!KeBugCheckEx
ffffd00021a81370 fffff8030f268839 : 0000000000000001 ffffe0014963c080 ffffd00021a815d0 ffffe001528f9080 : nt!CcTestControl+0x2210e
ffffd00021a81410 fffff8030f36ff2f : 0000000000000001 ffffffffffffffff 0000000000000000 fffff8030f2c6226 : nt!ExReleasePushLockEx+0x7d9
ffffd00021a815d0 fffff8030f267fb3 : ffffe0014d860b38 ffffe0014950cb68 ffffe0014950cbd0 ffffe0014d860a00 : nt!setjmpex+0x1f1f
ffffd00021a81760 fffff80174a71af7 : ffffe0014d860da0 ffffe0014d860da0 ffffe00100000000 ffffe00100000000 : nt!ObfDereferenceObject+0x23
ffffd00021a817a0 fffff80174a70afe : ffffe0014d860b38 ffffe0014d860da0 ffffe0014d860a00 ffffe0014d860b38 : dokan!DokanStopCheckThread+0x73 [f:\dev\dokan\dokany\sys\timeout.c @ 372]
ffffd00021a817e0 fffff80174a6df01 : 00000000c0000002 ffffd00021a81b80 ffffe0014d8608b0 0000000000000000 : dokan!DokanEventRelease+0xf2 [f:\dev\dokan\dokany\sys\notification.c @ 531]
ffffd00021a81830 fffff8030f63a77f : 0000000000000001 ffffe001537fb3e0 ffffe001537fb3e0 0000000000000000 : dokan!DokanDispatchDeviceControl+0x139 [f:\dev\dokan\dokany\sys\device.c @ 485]
ffffd00021a81880 fffff8030f639d22 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!NtDeviceIoControlFile+0xab3
ffffd00021a81a20 fffff8030f3714b3 : ffffd00021a81b10 0000000000000000 ffffd00000000001 00000000005ae4a8 : nt!NtDeviceIoControlFile+0x56
ffffd00021a81a90 0000000077aa2352 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : nt!setjmpex+0x34a3
00000000005aed48 0000000000000000 : 0000000000000000 0000000000000000 0000000000000000 0000000000000000 : 0x77aa2352

STACK_COMMAND: kb

FOLLOWUP_IP:
dokan!DokanStopCheckThread+73 [f:\dev\dokan\dokany\sys\timeout.c @ 372]
fffff801`74a71af7 4883a39801000000 and qword ptr [rbx+198h],0

FAULTING_SOURCE_LINE: f:\dev\dokan\dokany\sys\timeout.c

FAULTING_SOURCE_FILE: f:\dev\dokan\dokany\sys\timeout.c

FAULTING_SOURCE_LINE_NUMBER: 372

SYMBOL_STACK_INDEX: 5

SYMBOL_NAME: dokan!DokanStopCheckThread+73

FOLLOWUP_NAME: MachineOwner

IMAGE_NAME: dokan.sys

BUCKET_ID: WRONG_SYMBOLS

FAILURE_BUCKET_ID: WRONG_SYMBOLS

ANALYSIS_SOURCE: KM

FAILURE_ID_HASH_STRING: km:wrong_symbols

FAILURE_ID_HASH: {70b057e8-2462-896f-28e7-ac72d4d365f8}

Followup: MachineOwner

jbrads · 2015-07-18T13:24:13Z

Hi, I can confirm in my case it is the same stack trace and location each time

marinkobabic · 2015-07-18T18:16:46Z

It's a merge issue. Can be fixed very easy. DokanStopCheckThread shoul remove the following lines
KeSetEvent(&Dcb->KillEvent, 0, FALSE);

    if (Dcb->TimeoutThread) {
        KeWaitForSingleObject(Dcb->TimeoutThread, Executive,
            KernelMode, FALSE, NULL);
        ObDereferenceObject(Dcb->TimeoutThread);
        Dcb->TimeoutThread = NULL;
    }

Those lines are part of DokanStopCheckThreadInternal.

Liryna · 2015-07-18T20:37:48Z

@marinkobabic Well find 😄 !
I removed it, will make a RC monday.

jbrads · 2015-07-19T09:25:55Z

Thanks, if you update this issue when the RC is available I will verify.

Maxhy · 2015-07-19T09:30:26Z

Thanks for the analyze @jbrads, the issue identification @marinkobabic and the fix @Liryna.
My mistake, I failed the merge and didn't encounter the problem on my tests...

I just published signed drivers @jbrads if you want to test again with latest changes: https://github.com/dokan-dev/dokany/releases/tag/0.7.3-RC3

jbrads · 2015-07-19T10:06:21Z

Thanks, I can confirm I can no longer reproduce this issue with RC3

alexpmorris · 2015-07-19T13:31:58Z

I can confirm that it also solved the problem for me as well! I tried breaking out of the application over and over again without a hitch. great work! 👍

entropin · 2015-07-19T13:33:13Z

Works here as well, on win10
On Jul 19, 2015 3:32 PM, "Alexander Morris" notifications@github.com
wrote:

I can confirm that it also solved the problem for me as well! I tried
breaking out of the application over and over again without a hitch. great
work! [image: 👍]

—
Reply to this email directly or view it on GitHub
#27 (comment).

Liryna added a commit that referenced this issue Jul 18, 2015

Fix merge - duplicate DokanStopCheckThreadInternal #27

24ff442

marinkobabic mentioned this issue Jul 18, 2015

mirror causes blue screen on 64-bit Windows7 on ctrl-c when trying to dismount #31

Closed

Maxhy added the Bug label Jul 19, 2015

Maxhy closed this as completed Jul 20, 2015

BSOD with Dokan 0.7.3 RC on Windows 8.1 Pro x64 #27

BSOD with Dokan 0.7.3 RC on Windows 8.1 Pro x64 #27

Comments

mapomme1108 commented Jun 28, 2015

Liryna commented Jun 28, 2015

mapomme1108 commented Jun 28, 2015

Liryna commented Jun 28, 2015

mapomme1108 commented Jun 28, 2015

Liryna commented Jun 28, 2015

mapomme1108 commented Jun 28, 2015

Liryna commented Jun 28, 2015

marinkobabic commented Jun 29, 2015

Liryna commented Jun 29, 2015

mapomme1108 commented Jun 30, 2015

marinkobabic commented Jun 30, 2015

mapomme1108 commented Jul 1, 2015

marinkobabic commented Jul 1, 2015

mapomme1108 commented Jul 1, 2015

marinkobabic commented Jul 1, 2015

marinkobabic commented Jul 2, 2015

landrix commented Jul 8, 2015

marinkobabic commented Jul 9, 2015

Maxhy commented Jul 9, 2015

landrix commented Jul 9, 2015

mapomme1108 commented Jul 9, 2015

landrix commented Jul 9, 2015

mapomme1108 commented Jul 9, 2015

marinkobabic commented Jul 9, 2015

Liryna commented Jul 10, 2015

marinkobabic commented Jul 10, 2015

landrix commented Jul 10, 2015

Liryna commented Jul 10, 2015

Maxhy commented Jul 10, 2015

marinkobabic commented Jul 10, 2015

Liryna commented Jul 10, 2015

marinkobabic commented Jul 10, 2015

Maxhy commented Jul 10, 2015

Maxhy commented Jul 10, 2015

mapomme1108 commented Jul 10, 2015

mapomme1108 commented Jul 11, 2015

marinkobabic commented Jul 11, 2015

entropin commented Jul 13, 2015

marinkobabic commented Jul 13, 2015

marinkobabic commented Jul 13, 2015

landrix commented Jul 16, 2015

Liryna commented Jul 16, 2015

jbrads commented Jul 16, 2015

Debugging Details:

Maxhy commented Jul 17, 2015

marinkobabic commented Jul 17, 2015

jbrads commented Jul 18, 2015

jbrads commented Jul 18, 2015

marinkobabic commented Jul 18, 2015

Liryna commented Jul 18, 2015

jbrads commented Jul 19, 2015

Maxhy commented Jul 19, 2015

jbrads commented Jul 19, 2015

alexpmorris commented Jul 19, 2015

entropin commented Jul 19, 2015