Add spawnProcessDetached #5483

FreeSlave · 2017-06-14T17:25:01Z

spawnProcessDetached is function similar to spawnProcess, but user does not need to wait on spawned process to avoid zombies (resource leak). Instead of returning Pid instance spawnProcessDetached returns integer pid via pointer. Rationale: user should not be able to wait on detached process.

Currently only Posix version is implemented. It uses double fork technique and setsid to detach the new process from its caller. Since this implementation reuses a lot of stuff from spawnProcess I moved some implementation details to separate private functions to share them between spawnProcess and spawnProcessDetached.

THIS IS WIP
I'll add more unittests and Windows version later. For now say if you like the idea.

CyberShadow · 2017-06-14T18:38:31Z

I believe that on Windows, you simply have to close the process handle as soon as the process is created to effectively "detach" the process. (This is not the same as a detached console window.)

Can this be implemented as a Config flag instead? I'm not sure about adding an entire overload set. Then it can be used for spawnShell as well. Pid will likely need a few changes to make waiting on a detached processes an explicit error.

FreeSlave · 2017-06-14T20:14:59Z

I believe that on Windows, you simply have to close the process handle as soon as the process is created to effectively "detach" the process.

I believe so. This is another reason why I did not want to use Pid class. Process handle is part of Pid class on Windows. Closing it will make osHandle be able to be null. So users will need to check for osHandle not being null if they want to use it. That differs from the current situation when osHandle is always not null. And it's also inconsistent with Posix, where osHandle is same as pid and therefore always valid. Do you think it is ok?

CyberShadow · 2017-06-14T20:19:16Z

So users will need to check for osHandle not being null if they want to use it.

I think it's more of not doing a thing that doesn't make sense to do.

And it's also inconsistent with Posix, where osHandle is same as pid and therefore always valid.

If you spawn a detached process, technically there is no guarantee that the PID remains valid for any amount of time - the process may exit on its own and the OS may assign the PID to another process. So, if we are to reuse Pid, probably the best course is a private bool owned field (which is false for detached processes), and kill and wait require that owned is true.

CC @schveiguy @kyllingstad

FreeSlave · 2017-06-14T20:27:43Z

kill and wait require that owned is true

Why would kill require owned is true? You can kill any process via task manager or "kill" program which are clearly do not own this process.

CyberShadow · 2017-06-14T20:29:58Z

Because it might not be the process you actually intend to kill - that process may have exited a long time ago.

FreeSlave · 2017-06-14T22:34:56Z

Got it.
I'm still not sure that Pid class can be re-used here. There was that PR #5086 that introduced possibility to have non-owned Pid instances. And it was reverted later. Probably because it complicates the usage of other functions (like already said wait and kill).

CyberShadow · 2017-06-14T23:08:30Z

#5086 had more problems than that, though (killing processes being an intrinsically platform-dependent operation; needing to call OpenProcess on Windows). I think overloading Pid here makes more sense than adding an overload set.

kyllingstad · 2017-06-15T09:27:32Z

I think overloading Pid here makes more sense than adding an overload set.

I very much agree with this.

FreeSlave · 2017-06-15T18:11:48Z

Added Pid.owned property and Config.detached constant.
Also added separate unittest to check non-ability of waiting/killing not owned process. It runs OK, but I get this message at terminal

/bin/sh: 0: Can't open /tmp/std.process temporary file 76b5fac2-57f8-4f5b-88dc-9fd27c32ae73

CyberShadow

Would like to review the implementation after the refactorings are undone.

CyberShadow · 2017-06-15T21:34:59Z

std/process.d

+version(Posix)
+{
+private:
+    const(char)[] getExecutablePath(const(char)[] name)


Since there is no longer an overload set, these refactorings aren't needed any more, right? If so, can they be undone?

CyberShadow · 2017-06-15T21:35:56Z

std/process.d

+
+    /**
+    Spawn process in detached state. This removes the need in calling
+    $(LREF wait) to clean up the process resources.


Should have a note here that calling wait or kill with the resulting Pid is invalid.

CyberShadow · 2017-06-15T21:36:59Z

std/process.d

+    This is $(D true) if process was spawned without $(LREF Config.detached), $(D false)
+    otherwise. You can't $(LREF wait) for or $(LREF kill) process you don't own.
+    */
+    @property bool owned() const @safe pure nothrow


No need to make this public. I can't think of many cases that programs would mix owned and non-owned Pids and require distinguishing them. Making it an implementation detail also allows changing the underlying implementation in the future, if necessary.

CyberShadow · 2017-06-15T21:38:04Z

std/process.d

@@ -1374,6 +1501,8 @@ private:
    version (Posix)
    int performWait(bool block) @trusted
    {
+        import std.exception : enforceEx;
+        enforceEx!ProcessException(_owned, "Attempt to wait for the not owned process");


Consider "Can't wait on a detached process"

CyberShadow · 2017-06-15T21:38:09Z

std/process.d

@@ -1422,6 +1551,8 @@ private:
    {
        int performWait(bool block) @trusted
        {
+            import std.exception : enforceEx;
+            enforceEx!ProcessException(_owned, "Attempt to wait for the not owned process");


CyberShadow · 2017-06-15T21:39:20Z

std/process.d

@@ -1503,7 +1647,7 @@ Signal codes are defined in the $(D core.sys.posix.signal) module
 (which corresponds to the $(D signal.h) POSIX header).

 Throws:
-$(LREF ProcessException) on failure.
+$(LREF ProcessException) on failure or on attempt to wait for not owned process.


Consider "detached" or "non-child" instead of "not owned", the "owned" term should be an implementation detail

CyberShadow · 2017-06-15T21:39:29Z

std/process.d

@@ -1558,7 +1702,7 @@ Returns:
 An $(D std.typecons.Tuple!(bool, "terminated", int, "status")).

 Throws:
-$(LREF ProcessException) on failure.
+$(LREF ProcessException) on failure or on attempt to wait for not owned process.


CyberShadow · 2017-06-15T21:39:33Z

std/process.d

@@ -1636,6 +1780,7 @@ assert(wait(pid) == -SIGKILL); // Negative return value on POSIX!

 Throws:
 $(LREF ProcessException) on error (e.g. if codeOrSignal is invalid).
+    or on attempt to kill not owned process.


CyberShadow · 2017-06-15T21:39:40Z

std/process.d

@@ -1652,6 +1797,8 @@ void kill(Pid pid)
 /// ditto
 void kill(Pid pid, int codeOrSignal)
 {
+    import std.exception : enforceEx;
+    enforceEx!ProcessException(pid.owned(), "Attempt to kill not owned process");


CyberShadow · 2017-06-15T21:41:20Z

It runs OK, but I get this message at terminal

Is this an expected and unavoidable effect of the unit test, or do you need time/assistance in tracking it down?

FreeSlave · 2017-06-15T21:50:20Z

Is this an expected and unavoidable effect of the unit test, or do you need time/assistance in tracking it down?

It should not happen. It's simple TestScript usage. But I can't understand why it happens.

FreeSlave · 2017-06-15T23:40:23Z

Ok, I think I got it. TestScript destructor runs (effectively removing the file) before file execution. Looks like race condition between /bin/sh running with script file and removing this file by unittest.
We can overcome this by adding Thread.sleep. What do you think?

CyberShadow · 2017-06-18T02:19:51Z

std/process.d

+        else
+            throw ProcessException.newFromErrno("Could not create pipe to get process pid");
+    }
+    scope(exit) if (config & Config.detached) close(pidPipe[0]);


Any reason to not reuse the same pipe used for communicating the result (forkPipe)?

After the second fork started the first and the second forks run in parallel. I can't ensure that pid will be written to the pipe before possible errors.

I see. Please add this explanation as a comment somewhere.

I guess calling getpid from the second child is also a possibility, but I guess it's not worth the change.

There's also a possibility that the second fork fails. In this case we would need to pass fictitious pid from the first fork to adjust.

make these considerations a nice comment

Already considered in the comment before int[2] pidPipe;

CyberShadow · 2017-06-18T02:21:19Z

std/process.d

            }
-            close(workDirFD);
        }


Why change the working directory in the second fork? Wouldn't it be simpler to change it in the first fork regardless of Config.detached?

CyberShadow · 2017-06-18T02:46:44Z

std/process.d

-                pfds[i].fd = i + 3;
-                pfds[i].events = 0;
-                pfds[i].revents = 0;
+                setsid();


Hmm, isn't setsid supposed to be called from the first fork (so that the session leader is killed)?

You're right. Will change.

I found that setsid makes some apps run incorrectly when starting from terminal without setting stdin to /dev/null.
E.g. run setsid openarena from terminal. It shows a window, but does render anything.
Should we do anything about that?

Unfortunately I am not sufficiently familiar with the subject to confidently answer that.

From looking online, I noticed that setsid is something that is often done when double-forking, but not prescriptively so. Perhaps it only makes sense for non-interactive daemon processes?

If we don't setsid, the spawned process will belong to the same session so pkill -s '$sid' will send signal to this process too.
However on my system all my desktop applications belong to the same session, so it's ok.
We can add another flag to config to control whether the forked child should call setsid.

OK, so correct me if I'm wrong, but as I understand the most commonly-observed difference will be that ^C will ~~not~~ kill the spawned process if it's in the same session.

In which case, it does sound like it's worthy of a Flag addition, with the Windows implementation being mapped to the CREATE_NEW_PROCESS_GROUP process creation flag.

It may be worth leaving it for a separate PR.

^C is sent only to the foreground processes of session. And session has distinction between foreground and background processes only if it has connection to controlling terminal.
Even if doing setsid via flag it's still not clear if user want the new process to be a session leader (i.e. call setsid in the second fork) or not (call setsid in the first fork and let it die).

It may be worth leaving it for a separate PR.

So shall I remove setsid for now?

Thanks for the clarification. I think I'm out of my depth here. Perhaps someone more knowledgeable than me about POSIX process sessions can chime in, @kyllingstad @schveiguy @jmdavis ?

I've done some experiments. Login via ssh (just plain login without tmux, screen or similar) and spawned some script process in detached state without calling to setsid. Logout and re-login again, the process still ran. So I guess setsid is not necessary after all.

CyberShadow · 2017-06-18T02:50:17Z

std/process.d

@@ -1359,6 +1459,11 @@ final class Pid
    }

 private:
+    @property bool owned() const @safe pure nothrow


No point in a private property, just name the field owned.

CyberShadow · 2017-06-18T02:51:27Z

std/process.d

+    This leads to the annoying message like "/bin/sh: 0: Can't open /tmp/std.process temporary file" to appear when running tests.
+    It does not happen in unittests with non-detached processes because we always wait() for them to finish.
+    */
+    Thread.sleep(dur!"msecs"(5));


I think this is fine, but I'd give it a more generous timeout of 100 msecs or so just to exclude race conditions.

I used 5 msecs, because there's the same sleep in the other unittest under version(Android), having a comment about Android killing sleeping applications.

That's fine, we don't have CI for Android; I'm more concerned about Windows, where process creation is very slow. If the Android maintainer finds that this test is causing them trouble then they can fix it as needed. (Also, I was pretty sure Android kills idle apps, not processes...)

dlang-bot · 2017-06-25T09:54:26Z

Thanks for your pull request, @FreeSlave! We are looking forward to reviewing it, and you should be hearing from a maintainer soon.

Some tips to help speed things up:

smaller, focused PRs are easier to review than big ones
try not to mix up refactoring or style changes with bug fixes or feature enhancements
provide helpful commit messages explaining the rationale behind each change

Bear in mind that large or tricky changes may require multiple rounds of review and revision.

Please see CONTRIBUTING.md for more information.

Bugzilla references

Fix	Bugzilla	Description
✓	17519	RedBlackTree doesn't like const/immutable elements

CyberShadow · 2017-06-26T07:12:01Z

Could you please rebase on top of the latest master? That should fix CircleCI.

FreeSlave · 2017-06-26T16:54:56Z

Rebased. Is any other change needed?

CyberShadow · 2017-06-26T17:07:28Z

Ah, a changelog entry or linked bug as per @dlang-bot's comment. I think that's all if there is no further feedback.

…toring

…ge directory in the first fork

FreeSlave · 2017-06-26T17:53:22Z

I've added changelog FreeSlave@fa8a73f, but github does not see the changes. Probably because of the previous rebase.
Upd: it was resolved.

CyberShadow · 2017-06-26T19:36:57Z

@wilzbach Any idea why changelog is not showing up in DAutoTest diff here?

wilzbach · 2017-06-26T19:45:01Z

@wilzbach Any idea why changelog is not showing up in DAutoTest diff here?

Yes, it seems like Martin has added this file and thus it doesn't get generated:

https://github.com/dlang/dlang.org/blob/master/changelog/2.075.0_pre.dd

Should we bump the version at dlang.org? Probably a good idea to do this for all version files after the master -> stable branch-off?

CyberShadow · 2017-06-27T06:56:46Z

Should we bump the version at dlang.org? Probably a good idea to do this for all version files after the master -> stable branch-off?

Sounds like a good idea.

CyberShadow · 2017-07-14T02:59:15Z

changelog/std-process-config-detached.dd

+
+Config.detached allows to $(REF_ALTTEXT spawn process, spawnProcess, std, process) independently from the caller.
+No need to wait on the process spawned that way - no zombie will left after process exits.
+Attempts to $(REF wait, std, process) on or $(REF kill, std, process) detached processes will throw.


Some copyediting:

Config.detached allows $(REF_ALTTEXT spawning processes, spawnProcess, std, process) which run independently from the current process. There is no need to wait on the processes created with this flag, and no $(HTTPS en.wikipedia.org/wiki/Zombie_process, zombie process) will be left after they exit.
Attempts to call $(REF wait, std, process) or $(REF kill, std, process) on detached processes will throw.

JackStouffer · 2017-07-14T11:44:56Z

Does this really need Andrei's approval? It doesn't really add a new symbol.

wilzbach · 2017-07-14T12:00:25Z

Does this really need Andrei's approval? It doesn't really add a new symbol.

Well there's Config.detached, but yeah I'm a bit frustrated about the huge bottleneck that this is causing as well. Maybe

we should let the bot send @andralex a mail every week about open PRs which require his approval
move away from this single point of failure and redefine this policy to "at least two codeowners need to approve a new symbol" (or require the @andralex label only for modules he cares about, i.e. for which he is a code owner)

andralex · 2017-07-14T12:32:20Z

std/process.d

@@ -339,7 +340,8 @@ version(Posix) private enum InternalError : ubyte
    noerror,
    exec,
    chdir,
-    getrlimit
+    getrlimit,
+    doubleFork


add a comma so the next addition blames properly

andralex · 2017-07-14T12:32:41Z

std/process.d

+    // but we don't have a direct access to the second fork pid from the caller side thus use a pipe.
+    // We also can't reuse forkPipe for that purpose
+    // because we can't predict the order in which pid and possible error will be written
+    // since the first and the second forks will run in parallel.


looks like a multiline comment

andralex · 2017-07-14T12:33:28Z

std/process.d

+        if (core.sys.posix.unistd.pipe(pidPipe) == 0)
+            setCLOEXEC(pidPipe[1], true);
+        else
+            throw ProcessException.newFromErrno("Could not create pipe to get process pid");


Simplify flow:

if (core.sys.posix.unistd.pipe(pidPipe) != 0) throw ...; setCLOEXEC(pidPipe[1], true);

andralex · 2017-07-14T12:34:15Z

std/process.d

+        else
+            throw ProcessException.newFromErrno("Could not create pipe to get process pid");
+    }
+    scope(exit) if (config & Config.detached) close(pidPipe[0]);


make these considerations a nice comment

andralex · 2017-07-14T12:35:27Z

std/process.d

+            // In the case that stderr is redirected to stdout, we need
+            // to backup the file descriptor since stdout may be redirected
+            // as well.
+            if (stderrFD == STDOUT_FILENO)  stderrFD = dup(stderrFD);


double space?

It was like this in previous implementation

andralex · 2017-07-14T12:36:04Z

std/process.d

+                immutable maxToClose = maxDescriptors - 3;
+
+                // Call poll() to see which ones are actually open:
+                pollfd* pfds = cast(pollfd*) malloc(pollfd.sizeof * maxToClose);


Again, you're asking to change things that were like this before me. Well, I can, for sure.

Yes and much appreciated!

andralex · 2017-07-14T12:37:26Z

std/process.d

        auto status = InternalError.noerror;
        auto readExecResult = core.sys.posix.unistd.read(forkPipe[0], &status, status.sizeof);
+        // Save error number just in case if subsequent "waitpid" fails and overrides errno
+        auto lastError = .errno;


andralex · 2017-07-14T12:38:15Z

std/process.d

                case InternalError.noerror:
                    assert(false);
            }
-
            if (readExecResult == error.sizeof)
                throw ProcessException.newFromErrno(error, errorMsg);
            else


andralex · 2017-07-14T12:38:29Z

std/process.d

+            if (read(pidPipe[0], &actualPid, id.sizeof) == id.sizeof)
+                id = actualPid;
+            else
+                throw ProcessException.newFromErrno("Could not read from pipe to get detached process id");


simplify flow

andralex · 2017-07-14T12:38:45Z

std/process.d

+        CloseHandle(pi.hProcess);
+        return new Pid(pi.dwProcessId);
+    }
+    else


Delete what? Can you elaborate? Oh, you mean just else. Got it.

I mean delete the else line.

andralex · 2017-07-14T12:44:42Z

move away from this single point of failure and redefine this policy to "at least two codeowners need to approve a new symbol" (or require the @andralex label only for modules he cares about, i.e. for which he is a code owner)

Please let me be the point of failure for the sake of better checks and balances of library consistency. In several prior cases it has happened that design sensibilities somewhat foreign to D/Phobos have entered the standard library, to everybody's displeasure. Thanks.

schveiguy

LGTM, I would just tweak the timing a bit. There's no guarantee on how long a process takes to finish, especially on a busy OS, and losing a few seconds in the time to test phobos isn't going to kill us.

schveiguy · 2017-07-14T13:22:26Z

std/process.d

+        CloseHandle(pi.hProcess);
+        return new Pid(pi.dwProcessId);
+    }
+    else


schveiguy · 2017-07-14T13:24:12Z

std/process.d

+            wait(pid);
+        else
+            // We need to wait a little to ensure that the process has finished and data was written to files
+            Thread.sleep(dur!"msecs"(200));


Wait longer. There's no need to vie for performance here. Wait 2 seconds instead of .2

Also, stylistic change, you can put 2.seconds instead of dur!"seconds"(2)

I would prefer unittests run as fast as they could. But ok.

schveiguy · 2017-07-14T13:27:43Z

std/process.d

+    This leads to the annoying message like "/bin/sh: 0: Can't open /tmp/std.process temporary file" to appear when running tests.
+    It does not happen in unittests with non-detached processes because we always wait() for them to finish.
+    */
+    Thread.sleep(dur!"msecs"(100));


Longer time here too.

andralex

I'm ok with the addition, leaving detailed reviews to @schveiguy and others

schveiguy · 2017-07-25T20:24:19Z

ping @CyberShadow any idea why the DAutoTest fails?

schveiguy · 2017-07-25T20:25:47Z

Bugzilla references

@wilzbach this looks invalid to me, this has nothing to do with RedBlackTree. I don't actually see a bugzilla reference in any commit, so not sure why this showed up.

CyberShadow · 2017-07-25T20:28:11Z

ping @CyberShadow any idea why the DAutoTest fails?

Please see #5639 (comment) / #5626 (comment) / #5625 (comment) / #5619 (comment) / #5617 (comment).

TL;DR: GitHub bug.

schveiguy · 2017-07-25T21:09:29Z

@CyberShadow thanks, here's a followup: while this bug exists, what is the correct response, seeing as that test is required for merging? Should I just ping you/@wilzbach, or is there a way I can fix it myself?

wilzbach · 2017-07-25T21:17:09Z

while this bug exists, what is the correct response, seeing as that test is required for merging? Should I just ping you/@wilzbach, or is there a way I can fix it myself?

AFAICT the only safe way is to create a new PR.
It also needs to be a different commit sha :/

We can overwrite it manually if the auto-tester passes, but there is always the slight risk of breaking Jenkins or CircleCI though. However, at least for CircleCi we get a warning mail afterwards, so I think we can start to risk it. Creating a new PR all the time, is really annoying.

wilzbach · 2017-07-25T21:20:55Z

@wilzbach this looks invalid to me, this has nothing to do with RedBlackTree. I don't actually see a bugzilla reference in any commit, so not sure why this showed up.

I would guess that it's due to the GitHub API sending rubbish.

wilzbach · 2017-07-25T21:23:09Z

We can overwrite it manually if the auto-tester passes, but there is always the slight risk of breaking Jenkins or CircleCI though. However, at least for CircleCi we get a warning mail afterwards, so I think we can start to risk it. Creating a new PR all the time, is really annoying.

Tried it this time - let's hope for the best. We really need to get GitHub to fix their bugs though..

CyberShadow requested changes Jun 15, 2017

View reviewed changes

FreeSlave force-pushed the spawnProcessDetached branch 2 times, most recently from 45c0286 to 827dcd5 Compare June 16, 2017 00:19

CyberShadow requested changes Jun 18, 2017

View reviewed changes

FreeSlave force-pushed the spawnProcessDetached branch from 07578c0 to fb3ff62 Compare June 26, 2017 07:15

CyberShadow approved these changes Jun 26, 2017

View reviewed changes

FreeSlave added 7 commits June 26, 2017 20:49

Add spawnProcessDetached for Posix

aca6b04

Add Config.detached flag

1481dc6

Update documentation comments for std.process. Remove redundant refac…

2ce1891

…toring

Add comments about pipes. Move setsid to the first process fork. Chan…

d6225c8

…ge directory in the first fork

Don't call setsid when spawning detached process

19c9f2b

Add stream redirection unittests for detached processes

0f357f0

Add changelog for Config.detached

fa8a73f

dlang-bot added the Severity:Bug Fix label Jun 26, 2017

FreeSlave force-pushed the spawnProcessDetached branch from 2a32acd to fa8a73f Compare June 26, 2017 18:13

wilzbach added the @andralex label Jun 26, 2017

JackStouffer requested a review from andralex July 3, 2017 18:28

CyberShadow reviewed Jul 14, 2017

View reviewed changes

Update spawnProcess Config.detached changelog

6d91b1e

FreeSlave requested review from kyllingstad and schveiguy as code owners July 14, 2017 09:57

andralex approved these changes Jul 14, 2017

View reviewed changes

schveiguy reviewed Jul 14, 2017

View reviewed changes

Style changes for std.process

2169c19

andralex approved these changes Jul 25, 2017

View reviewed changes

andralex removed the @andralex label Jul 25, 2017

schveiguy added the Merge:auto-merge label Jul 25, 2017

wilzbach merged commit fe89346 into dlang:master Jul 25, 2017

Add spawnProcessDetached #5483

Add spawnProcessDetached #5483

Conversation

FreeSlave commented Jun 14, 2017 • edited Loading

CyberShadow commented Jun 14, 2017 • edited Loading

FreeSlave commented Jun 14, 2017

CyberShadow commented Jun 14, 2017

FreeSlave commented Jun 14, 2017 • edited Loading

CyberShadow commented Jun 14, 2017

FreeSlave commented Jun 14, 2017

CyberShadow commented Jun 14, 2017

kyllingstad commented Jun 15, 2017

FreeSlave commented Jun 15, 2017 • edited Loading

CyberShadow left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CyberShadow Jun 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CyberShadow commented Jun 15, 2017

FreeSlave commented Jun 15, 2017

FreeSlave commented Jun 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FreeSlave Jun 18, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FreeSlave Jun 19, 2017 • edited Loading

Choose a reason for hiding this comment

CyberShadow Jun 19, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FreeSlave Jun 19, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dlang-bot commented Jun 25, 2017 • edited Loading

Bugzilla references

CyberShadow commented Jun 26, 2017

FreeSlave commented Jun 26, 2017

CyberShadow commented Jun 26, 2017

FreeSlave commented Jun 26, 2017 • edited Loading

CyberShadow commented Jun 26, 2017

wilzbach commented Jun 26, 2017

CyberShadow commented Jun 27, 2017

Choose a reason for hiding this comment

JackStouffer commented Jul 14, 2017

wilzbach commented Jul 14, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FreeSlave commented Jun 14, 2017 •

edited

Loading

CyberShadow commented Jun 14, 2017 •

edited

Loading

FreeSlave commented Jun 14, 2017 •

edited

Loading

FreeSlave commented Jun 15, 2017 •

edited

Loading

CyberShadow Jun 15, 2017 •

edited

Loading

FreeSlave commented Jun 15, 2017 •

edited

Loading

FreeSlave Jun 18, 2017 •

edited

Loading

FreeSlave Jun 19, 2017 •

edited

Loading

CyberShadow Jun 19, 2017 •

edited

Loading

FreeSlave Jun 19, 2017 •

edited

Loading

dlang-bot commented Jun 25, 2017 •

edited

Loading

FreeSlave commented Jun 26, 2017 •

edited

Loading

FreeSlave Jul 14, 2017 •

edited

Loading

schveiguy Jul 14, 2017 •

edited

Loading