Issue 12090 - Make std.concurrency compatible with fibers as threads #1910

Merged
merged 2 commits into from Oct 22, 2014

Conversation

Projects
None yet
@DmitryOlshansky

This comment has been minimized.

Show comment
Hide comment
Member

DmitryOlshansky commented Feb 18, 2014

@complexmath Needs rebase

@John-Colvin

This comment has been minimized.

Show comment
Hide comment
@John-Colvin

John-Colvin May 2, 2014

Contributor

ping. Still needs rebasing.

Contributor

John-Colvin commented May 2, 2014

ping. Still needs rebasing.

@andralex

This comment has been minimized.

Show comment
Hide comment
@andralex

andralex May 20, 2014

Member

test failures, rebase

Member

andralex commented May 20, 2014

test failures, rebase

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath May 21, 2014

Member

rebased

On May 19, 2014, at 5:15 PM, Andrei Alexandrescu notifications@github.com wrote:

test failures, rebase


Reply to this email directly or view it on GitHub.

Member

complexmath commented May 21, 2014

rebased

On May 19, 2014, at 5:15 PM, Andrei Alexandrescu notifications@github.com wrote:

test failures, rebase


Reply to this email directly or view it on GitHub.

@mihails-strasuns

This comment has been minimized.

Show comment
Hide comment
@mihails-strasuns

mihails-strasuns May 22, 2014

Seems to include some unrelated commits right now.

Seems to include some unrelated commits right now.

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath May 22, 2014

Member

Well how the heck did that happen? Should I submit a new pull request, or does someone know how to remove those?

Member

complexmath commented May 22, 2014

Well how the heck did that happen? Should I submit a new pull request, or does someone know how to remove those?

@yebblies

This comment has been minimized.

Show comment
Hide comment
Member

yebblies commented May 22, 2014

Try this: #1153 (comment)

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath May 23, 2014

Member

That worked. Thanks!

Member

complexmath commented May 23, 2014

That worked. Thanks!

@mihails-strasuns

This comment has been minimized.

Show comment
Hide comment
@mihails-strasuns

mihails-strasuns May 23, 2014

Now it has two commits with identical title (first one being trivial changes) ;) You probably want to squash it all together later. Anyway, going to review this soon!

Also pinging @s-ludwig

Now it has two commits with identical title (first one being trivial changes) ;) You probably want to squash it all together later. Anyway, going to review this soon!

Also pinging @s-ludwig

std/concurrency.d
+
+interface Scheduler
+{
+ void start( void delegate() op );

This comment has been minimized.

@JakobOvrum

JakobOvrum May 25, 2014

Member

This doesn't appear to be used anywhere but in the implementations of spawn, maybe it should be removed?

@JakobOvrum

JakobOvrum May 25, 2014

Member

This doesn't appear to be used anywhere but in the implementations of spawn, maybe it should be removed?

This comment has been minimized.

@complexmath

complexmath May 26, 2014

Member

That's what it's for. The Scheduler allows different concurrency mechanisms to back the threading in std.concurrency. The start function is for creating these threads.

@complexmath

complexmath May 26, 2014

Member

That's what it's for. The Scheduler allows different concurrency mechanisms to back the threading in std.concurrency. The start function is for creating these threads.

This comment has been minimized.

@JakobOvrum

JakobOvrum May 26, 2014

Member

I mean the implementations of ThreadScheduler.spawn and FiberScheduler.spawn. If it was used in std.concurrency.spawn then that might serve as an example of its utility, but as it is, it looks like Scheduler.spawn is the only required primitive.

@JakobOvrum

JakobOvrum May 26, 2014

Member

I mean the implementations of ThreadScheduler.spawn and FiberScheduler.spawn. If it was used in std.concurrency.spawn then that might serve as an example of its utility, but as it is, it looks like Scheduler.spawn is the only required primitive.

This comment has been minimized.

@complexmath

complexmath May 26, 2014

Member

Oh yes, this definitely needs to be documented because it isn't evident from the code. start() isn't strictly required in many cases. That's intended to be called in Dmain so everything, even the main thread, is run by the scheduler. This is necessary for vibe.d, for example, but many other types of schedulers will work without it.

@complexmath

complexmath May 26, 2014

Member

Oh yes, this definitely needs to be documented because it isn't evident from the code. start() isn't strictly required in many cases. That's intended to be called in Dmain so everything, even the main thread, is run by the scheduler. This is necessary for vibe.d, for example, but many other types of schedulers will work without it.

@JakobOvrum

This comment has been minimized.

Show comment
Hide comment
@JakobOvrum

JakobOvrum May 25, 2014

Member

This adds public but undocumented types along with the apparently user-facing scheduler variable. Is there a rationale for them being undocumented? It makes the intention of the PR slightly unclear.

Member

JakobOvrum commented May 25, 2014

This adds public but undocumented types along with the apparently user-facing scheduler variable. Is there a rationale for them being undocumented? It makes the intention of the PR slightly unclear.

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath May 26, 2014

Member

Scheduler should really be documented. Though the two implemented examples were meant to serve as documentation of their own.

Member

complexmath commented May 26, 2014

Scheduler should really be documented. Though the two implemented examples were meant to serve as documentation of their own.

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath Jun 5, 2014

Member

I've added some documentation. Let me know how it looks.

Member

complexmath commented Jun 5, 2014

I've added some documentation. Let me know how it looks.

@mihails-strasuns

This comment has been minimized.

Show comment
Hide comment
@mihails-strasuns

mihails-strasuns Jun 6, 2014

Can you please add some examples of using FiberScheduler? After reading docs and API I have tried something like this and failed:

import std.concurrency;
import core.thread;
import std.stdio;

void spawned(Tid tid)
{
    for (;;)
    {   
        writeln("waiting for numbers");

        scheduler.yield();

        receive(
            (int i) { writefln("received number %s", i); }
        );
    }   
}

void main()
{
    scheduler = new FiberScheduler();

    auto tid = spawn(&spawned, thisTid);

    writefln("main after spawn");

    foreach (i; 0..5)
    {
        writefln("sending %s", i); 
        tid.send(i);

        scheduler.yield();
    }   
}

It works as I expected it to work when ThreadScheduler is used.

Another thing is that I am worried about scheduler being null by default. That means task implementation needs always to check its state before yielding which results in unnecessary boilerplate. Should be probably hidden behind free function, similar to spawn.

Sorry for "soon" being so long :)

Can you please add some examples of using FiberScheduler? After reading docs and API I have tried something like this and failed:

import std.concurrency;
import core.thread;
import std.stdio;

void spawned(Tid tid)
{
    for (;;)
    {   
        writeln("waiting for numbers");

        scheduler.yield();

        receive(
            (int i) { writefln("received number %s", i); }
        );
    }   
}

void main()
{
    scheduler = new FiberScheduler();

    auto tid = spawn(&spawned, thisTid);

    writefln("main after spawn");

    foreach (i; 0..5)
    {
        writefln("sending %s", i); 
        tid.send(i);

        scheduler.yield();
    }   
}

It works as I expected it to work when ThreadScheduler is used.

Another thing is that I am worried about scheduler being null by default. That means task implementation needs always to check its state before yielding which results in unnecessary boilerplate. Should be probably hidden behind free function, similar to spawn.

Sorry for "soon" being so long :)

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath Jun 6, 2014

Member

Try

import std.concurrency;
import core.thread;
import std.stdio;

void spawned(Tid tid)
{
    for (;;)
    {   
        writeln("waiting for numbers");

        scheduler.yield(); // explicit yield is unnecessary as receive will yield if it blocks, but can't hurt

        receive(
            (int i) { writefln("received number %s", i); }
        );
    }   
}

void main()
{
    scheduler = new FiberScheduler();
    scheduler.start({
        auto tid = spawn(&spawned, thisTid);

        writefln("main after spawn");

        foreach (i; 0..5)
        {
            writefln("sending %s", i); 
            tid.send(i);

            scheduler.yield();
        }
    });   
}

scheduler.start() starts the dispatcher, which is necessary for FiberScheduler. yield() will start it as well, but that could lead to weird behavior as execution won't (currently) return to the yielding thread until all fibers terminate. I'll see about adding an example.

Member

complexmath commented Jun 6, 2014

Try

import std.concurrency;
import core.thread;
import std.stdio;

void spawned(Tid tid)
{
    for (;;)
    {   
        writeln("waiting for numbers");

        scheduler.yield(); // explicit yield is unnecessary as receive will yield if it blocks, but can't hurt

        receive(
            (int i) { writefln("received number %s", i); }
        );
    }   
}

void main()
{
    scheduler = new FiberScheduler();
    scheduler.start({
        auto tid = spawn(&spawned, thisTid);

        writefln("main after spawn");

        foreach (i; 0..5)
        {
            writefln("sending %s", i); 
            tid.send(i);

            scheduler.yield();
        }
    });   
}

scheduler.start() starts the dispatcher, which is necessary for FiberScheduler. yield() will start it as well, but that could lead to weird behavior as execution won't (currently) return to the yielding thread until all fibers terminate. I'll see about adding an example.

@mihails-strasuns

This comment has been minimized.

Show comment
Hide comment
@mihails-strasuns

mihails-strasuns Jun 6, 2014

as execution won't (currently) return to the yielding thread until all fibers terminate

Ah this is what has confused me. is this because of implementation difficulties or intentional? Anyway, should be mentioned in FiberScheduler docs clearly.

This works and should be probably added as documented unittest to module description as it makes obvious difference between fiber and thread schedulers:

import core.thread;
import std.stdio;

void spawned(Tid tid)
{
    bool finished = false;

    for (;;)
    {
        receive(
            (int i) { writefln("received number %s", i); },
            (Tid t) { finished = true; }
        );

        if (finished)
            return;
    }
}

void main()
{
//    scheduler = new FiberScheduler();
    scheduler = new ThreadScheduler();

    scheduler.start({
        auto tid = spawn(&spawned, thisTid);

        foreach (i; 0..5)
        {
            writefln("sending %s", i); 

            tid.send(i);

            scheduler.yield();
        }

        tid.send(thisTid);
    });
}

as execution won't (currently) return to the yielding thread until all fibers terminate

Ah this is what has confused me. is this because of implementation difficulties or intentional? Anyway, should be mentioned in FiberScheduler docs clearly.

This works and should be probably added as documented unittest to module description as it makes obvious difference between fiber and thread schedulers:

import core.thread;
import std.stdio;

void spawned(Tid tid)
{
    bool finished = false;

    for (;;)
    {
        receive(
            (int i) { writefln("received number %s", i); },
            (Tid t) { finished = true; }
        );

        if (finished)
            return;
    }
}

void main()
{
//    scheduler = new FiberScheduler();
    scheduler = new ThreadScheduler();

    scheduler.start({
        auto tid = spawn(&spawned, thisTid);

        foreach (i; 0..5)
        {
            writefln("sending %s", i); 

            tid.send(i);

            scheduler.yield();
        }

        tid.send(thisTid);
    });
}
@mihails-strasuns

This comment has been minimized.

Show comment
Hide comment
@mihails-strasuns

mihails-strasuns Jun 6, 2014

Also, @s-ludwig - your feedback is still necessary :P

Also, @s-ludwig - your feedback is still necessary :P

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath Jun 6, 2014

Member

Your comment about having a standalone yield got me thinking. I have another pull request blocked by this one that was going to add a standalone yield for another purpose, and I've realized that for it to work correctly, the user should never call scheduler.yield() explicitly. So I think it's a good idea to get the standalone yield in right now, and I may just merge the two pull requests for this reason even though they aren't strictly otherwise related. I'll see about improving the documentation for Scheduler as well. I think scheduler.start() should always be called in main even if some schedulers don't require it just for the sake of simplicity.

Member

complexmath commented Jun 6, 2014

Your comment about having a standalone yield got me thinking. I have another pull request blocked by this one that was going to add a standalone yield for another purpose, and I've realized that for it to work correctly, the user should never call scheduler.yield() explicitly. So I think it's a good idea to get the standalone yield in right now, and I may just merge the two pull requests for this reason even though they aren't strictly otherwise related. I'll see about improving the documentation for Scheduler as well. I think scheduler.start() should always be called in main even if some schedulers don't require it just for the sake of simplicity.

@s-ludwig

This comment has been minimized.

Show comment
Hide comment
@s-ludwig

s-ludwig Jun 6, 2014

Member

I'll try to find some time to build a custom DMD/Phobos and make a demo integration for vibe.d over the next days, as that's the only definitive way to find out the last possible issues, I guess.

Just a little technical documentation comment: There are some doc comments without a short summary paragraph (see "Sections" at http://dlang.org/ddoc.html), which will result in some crowded overview tables in the DDOX documentation layout.

Member

s-ludwig commented Jun 6, 2014

I'll try to find some time to build a custom DMD/Phobos and make a demo integration for vibe.d over the next days, as that's the only definitive way to find out the last possible issues, I guess.

Just a little technical documentation comment: There are some doc comments without a short summary paragraph (see "Sections" at http://dlang.org/ddoc.html), which will result in some crowded overview tables in the DDOX documentation layout.

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath Jun 6, 2014

Member

Thanks @s-ludwig. From previous discussion, I've also merged in my other pull request with this one. This seemed appropriate since it affects the yield() implementation. I've also added a unittest that tests both the FiberScheduler and this new Generator type.

Member

complexmath commented Jun 6, 2014

Thanks @s-ludwig. From previous discussion, I've also merged in my other pull request with this one. This seemed appropriate since it affects the yield() implementation. I've also added a unittest that tests both the FiberScheduler and this new Generator type.

@mihails-strasuns

This comment has been minimized.

Show comment
Hide comment
@mihails-strasuns

mihails-strasuns Jun 6, 2014

Test failure because of "statement is not reachable" warning
(new stuff looks awesome on the first glance)

Test failure because of "statement is not reachable" warning
(new stuff looks awesome on the first glance)

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath Jun 6, 2014

Member

Random Win64 breakage? That errorlevel seems weird.

Member

complexmath commented Jun 6, 2014

Random Win64 breakage? That errorlevel seems weird.

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath Jun 8, 2014

Member

I guess this means that fibers are broken on win64? Is there anyone that can look into this?

Member

complexmath commented Jun 8, 2014

I guess this means that fibers are broken on win64? Is there anyone that can look into this?

@s-ludwig

This comment has been minimized.

Show comment
Hide comment
@s-ludwig

s-ludwig Jun 8, 2014

Member

They definitely were broken until recently: http://issues.dlang.org/show_bug.cgi?id=12800

Member

s-ludwig commented Jun 8, 2014

They definitely were broken until recently: http://issues.dlang.org/show_bug.cgi?id=12800

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath Jun 8, 2014

Member

Hrm. Then I'm at a loss to explain two successive test failures on Win64 only. It doesn't appear to be displaying any output at all from the test run on that platform. Could the failure be elsewhere?

Member

complexmath commented Jun 8, 2014

Hrm. Then I'm at a loss to explain two successive test failures on Win64 only. It doesn't appear to be displaying any output at all from the test run on that platform. Could the failure be elsewhere?

@s-ludwig

This comment has been minimized.

Show comment
Hide comment
@s-ludwig

s-ludwig Jun 8, 2014

Member

Getting vibe.d to compile on DMD master was a bit more involved than I had hoped for due to some regressions, but now I've got something that seems to work (start and yield are just implemented as no-ops). Now there is just one major issue, assuming I didn't misunderstand something. In vibe.d, there are two flavors of spawn aka runTask and runWorkerTask. The former starts a new task in a fiber of the current thread, while the latter starts it in a fiber of a different thread.

But If I see it right, tasks currently have to be started using spawn, if they are supposed to be usable for message passing (for populating the ThreadInfo of the new task). In that case, there would be no possibility to choose between the two forms of starting a task. What about adding a way to perform the setup* manually from the outside, so that runTask could also be made to work? The question would just be how to ideally avoid creating another public API method that is only really useful for Scheduler implementations and not for the normal API user...

*:

auto spawnTid = Tid( new MessageBox ); // just this line would be enough in theory
auto ownerTid = thisTid;
thisInfo.ident = spawnTid;
thisInfo.owner = ownerTid;

BTW, I'll try to reproduce the Win64 issue tomorrow.

Member

s-ludwig commented Jun 8, 2014

Getting vibe.d to compile on DMD master was a bit more involved than I had hoped for due to some regressions, but now I've got something that seems to work (start and yield are just implemented as no-ops). Now there is just one major issue, assuming I didn't misunderstand something. In vibe.d, there are two flavors of spawn aka runTask and runWorkerTask. The former starts a new task in a fiber of the current thread, while the latter starts it in a fiber of a different thread.

But If I see it right, tasks currently have to be started using spawn, if they are supposed to be usable for message passing (for populating the ThreadInfo of the new task). In that case, there would be no possibility to choose between the two forms of starting a task. What about adding a way to perform the setup* manually from the outside, so that runTask could also be made to work? The question would just be how to ideally avoid creating another public API method that is only really useful for Scheduler implementations and not for the normal API user...

*:

auto spawnTid = Tid( new MessageBox ); // just this line would be enough in theory
auto ownerTid = thisTid;
thisInfo.ident = spawnTid;
thisInfo.owner = ownerTid;

BTW, I'll try to reproduce the Win64 issue tomorrow.

@s-ludwig

This comment has been minimized.

Show comment
Hide comment
@s-ludwig

s-ludwig Jun 9, 2014

Member

For Win64, I'm getting a different error code (-1073741701, which corresponds to 0xc000007b aka STATUS_INVALID_IMAGE_FORMAT), maybe that's related to the curl.lib that I had to get from somewhere. But -1073741819 should correspond to a STATUS_ACCESS_VIOLATION.

Member

s-ludwig commented Jun 9, 2014

For Win64, I'm getting a different error code (-1073741701, which corresponds to 0xc000007b aka STATUS_INVALID_IMAGE_FORMAT), maybe that's related to the curl.lib that I had to get from somewhere. But -1073741819 should correspond to a STATUS_ACCESS_VIOLATION.

@s-ludwig

This comment has been minimized.

Show comment
Hide comment
@s-ludwig

s-ludwig Jun 9, 2014

Member

Got it to run finally. The crash is in Fiber.call of the FiberScheduler, but there is no obvious relation to the scheduler code. Looks like #809 probably was only a partial fix.

Member

s-ludwig commented Jun 9, 2014

Got it to run finally. The crash is in Fiber.call of the FiberScheduler, but there is no obvious relation to the scheduler code. Looks like #809 probably was only a partial fix.

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath Jun 9, 2014

Member

Makes me wonder if I should conditionally disable the unittest I added.

Member

complexmath commented Jun 9, 2014

Makes me wonder if I should conditionally disable the unittest I added.

UplinkCoder added a commit to UplinkCoder/vibe.d that referenced this pull request Jun 12, 2014

@mihails-strasuns

This comment has been minimized.

Show comment
Hide comment
@mihails-strasuns

mihails-strasuns Jun 16, 2014

In general current state looks good to me.

I'd love to see std.concurrency turned into package with smaller modules. For example, Generator is perfect match for separate module utility. But this is subject to follow-up pull requests.

In general current state looks good to me.

I'd love to see std.concurrency turned into package with smaller modules. For example, Generator is perfect match for separate module utility. But this is subject to follow-up pull requests.

@Geod24 Geod24 referenced this pull request in vibe-d/vibe.d Jun 28, 2014

Closed

dmd ~master cannot compile vibe #703

@WalterBright

This comment has been minimized.

Show comment
Hide comment
@WalterBright

WalterBright Jul 10, 2014

Member

There's a lot of new code, and only a handful of lines of unittests. Can you try compiling it with -cov and try for 100% coverage?

Member

WalterBright commented Jul 10, 2014

There's a lot of new code, and only a handful of lines of unittests. Can you try compiling it with -cov and try for 100% coverage?

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath Jul 10, 2014

Member

Does -cov even work with multithreaded code?

Member

complexmath commented Jul 10, 2014

Does -cov even work with multithreaded code?

@WalterBright

This comment has been minimized.

Show comment
Hide comment
@WalterBright

WalterBright Jul 10, 2014

Member

Does -cov even work with multithreaded code?

It should. Try it and see! It's easy:

dmd -unittest -main -cov std/concurrency.d

will do the trick.

There will be race conditions as the counters for the lines are incremented without synchronization, but that doesn't matter if you just are looking for "did this line execute" rather than "exactly how many times did this line execute".

Member

WalterBright commented Jul 10, 2014

Does -cov even work with multithreaded code?

It should. Try it and see! It's easy:

dmd -unittest -main -cov std/concurrency.d

will do the trick.

There will be race conditions as the counters for the lines are incremented without synchronization, but that doesn't matter if you just are looking for "did this line execute" rather than "exactly how many times did this line execute".

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath Jul 10, 2014

Member

Okay.

Regarding the Win64 test failure. This pretty much has to be caused by a bug in the Win64 fiber code (in core.thread) that the unittest for Generator is triggering. I don't have a working Win64 dev environment to look into this. Should the unittest be conditionally disabled? Or maybe there's someone who wants to check it out and see if they know what's going on? I know that a patch was applied to the Win64 fiber code recently, but I'm guessing it's insufficient to fix whatever is going on there. I don't want to have this pull request blocked by code in Druntime that's broken only for Win64.

Member

complexmath commented Jul 10, 2014

Okay.

Regarding the Win64 test failure. This pretty much has to be caused by a bug in the Win64 fiber code (in core.thread) that the unittest for Generator is triggering. I don't have a working Win64 dev environment to look into this. Should the unittest be conditionally disabled? Or maybe there's someone who wants to check it out and see if they know what's going on? I know that a patch was applied to the Win64 fiber code recently, but I'm guessing it's insufficient to fix whatever is going on there. I don't want to have this pull request blocked by code in Druntime that's broken only for Win64.

@WalterBright

This comment has been minimized.

Show comment
Hide comment
@WalterBright

WalterBright Jul 10, 2014

Member

I'm not competent to review this PR. But I will pull it on faith after the -cov thing is done. I agree with your proposal to simply disable it for Win64 for the time being, with the proviso that existing behavior on Win64 not being negatively affected.

Member

WalterBright commented Jul 10, 2014

I'm not competent to review this PR. But I will pull it on faith after the -cov thing is done. I agree with your proposal to simply disable it for Win64 for the time being, with the proviso that existing behavior on Win64 not being negatively affected.

@s-ludwig

This comment has been minimized.

Show comment
Hide comment
@s-ludwig

s-ludwig Jul 10, 2014

Member

@complexmath Do you have a good idea for allowing external creation of Tids? That would be the only missing piece for proper vibe.d integration. To keep it visible only to the Scheduler implementation, one idea would be to simply pass an additional Tid function() factory parameter to Scheduler.start().

Member

s-ludwig commented Jul 10, 2014

@complexmath Do you have a good idea for allowing external creation of Tids? That would be the only missing piece for proper vibe.d integration. To keep it visible only to the Scheduler implementation, one idea would be to simply pass an additional Tid function() factory parameter to Scheduler.start().

@DmitryOlshansky

This comment has been minimized.

Show comment
Hide comment
@DmitryOlshansky

DmitryOlshansky Sep 23, 2014

Member

I hate to say but "merge changes" commit is strange and suspisiouly looks like squashed version of all commits since last rebase. 11K additions and 8K deletions is not acceptable, please do a proper rebase e.g. by cherry-picking relevant commits on top of recent master branch.

Member

DmitryOlshansky commented Sep 23, 2014

I hate to say but "merge changes" commit is strange and suspisiouly looks like squashed version of all commits since last rebase. 11K additions and 8K deletions is not acceptable, please do a proper rebase e.g. by cherry-picking relevant commits on top of recent master branch.

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath Sep 23, 2014

Member

Ugh... "git up" is apparently evil, as that's the only thing I did other than the usual fetch and rebase. It must have screwed something up.

Member

complexmath commented Sep 23, 2014

Ugh... "git up" is apparently evil, as that's the only thing I did other than the usual fetch and rebase. It must have screwed something up.

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath Sep 23, 2014

Member

Hm, rebasing against upstream/master is doing something horrible, but I fixed the pull request history.

Member

complexmath commented Sep 23, 2014

Hm, rebasing against upstream/master is doing something horrible, but I fixed the pull request history.

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath Sep 24, 2014

Member

Well... the best I could do to get the pull request up to date was apparently to re-apply the file as a commit on top of the previous one. Not ideal, but I'm out of ideas.

Member

complexmath commented Sep 24, 2014

Well... the best I could do to get the pull request up to date was apparently to re-apply the file as a commit on top of the previous one. Not ideal, but I'm out of ideas.

@DmitryOlshansky

This comment has been minimized.

Show comment
Hide comment
@DmitryOlshansky

DmitryOlshansky Sep 24, 2014

Member

There is still a merge commit 803ed7b, please try to avoid them. It may be even simpler to just create a temporary branch from up-to-date master and reconstruct changes to std.concurrency there, then hard reset this branch to the temporary one.

Member

DmitryOlshansky commented Sep 24, 2014

There is still a merge commit 803ed7b, please try to avoid them. It may be even simpler to just create a temporary branch from up-to-date master and reconstruct changes to std.concurrency there, then hard reset this branch to the temporary one.

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath Sep 24, 2014

Member

Missed that one. I'll sort it out. I think part of the problem is that these changes leaked into my local master somehow. I'll rewind that and start over.

Member

complexmath commented Sep 24, 2014

Missed that one. I'll sort it out. I think part of the problem is that these changes leaked into my local master somehow. I'll rewind that and start over.

@mihails-strasuns

This comment has been minimized.

Show comment
Hide comment
@mihails-strasuns

mihails-strasuns Sep 24, 2014

It may be even simpler to just create a temporary branch from up-to-date master and reconstruct changes

Actually you can just hard reset PR branch to upstream/master and then cherry-pick needed old commits - git does not delete unreferenced commits for a long time so as long as hash is written down somewhere (reflog) those can be cherry-picked.

It may be even simpler to just create a temporary branch from up-to-date master and reconstruct changes

Actually you can just hard reset PR branch to upstream/master and then cherry-pick needed old commits - git does not delete unreferenced commits for a long time so as long as hash is written down somewhere (reflog) those can be cherry-picked.

+ throw t;
+ if( m_fibers[m_pos].state() == Fiber.State.TERM )
+ {
+ if( m_pos >= (m_fibers = remove( m_fibers, m_pos )).length )

This comment has been minimized.

@DmitryOlshansky

DmitryOlshansky Sep 26, 2014

Member

A bit too clever code to save an extra line.

@DmitryOlshansky

DmitryOlshansky Sep 26, 2014

Member

A bit too clever code to save an extra line.

This comment has been minimized.

@DmitryOlshansky

DmitryOlshansky Sep 26, 2014

Member

Also consider remove with "unstable" policy i.e. swapping with the last one.

@DmitryOlshansky

DmitryOlshansky Sep 26, 2014

Member

Also consider remove with "unstable" policy i.e. swapping with the last one.

@DmitryOlshansky

This comment has been minimized.

Show comment
Hide comment
@DmitryOlshansky

DmitryOlshansky Sep 26, 2014

Member

One question still bothers me (as per NG discussion). I can't yet see the code that shares fibers across thread pool, but Sean indicates that fibers will silently move across threads.

This is unacceptable. First because it breaks everything TL idom right there as Fiber is not marked as shared explicitly, secondly it opens a door to countless bugs because in 3rd party libraries relying on persistent TL (some though would work, if TL is used for caches, connection pools etc. but suboptimally).

As moving fibers across threads is a useful thing to do so as to balance the load (although there are other IMHO far better ways to do it) it should NOT be the default and has to be enabled explicitly with big red button.

Member

DmitryOlshansky commented Sep 26, 2014

One question still bothers me (as per NG discussion). I can't yet see the code that shares fibers across thread pool, but Sean indicates that fibers will silently move across threads.

This is unacceptable. First because it breaks everything TL idom right there as Fiber is not marked as shared explicitly, secondly it opens a door to countless bugs because in 3rd party libraries relying on persistent TL (some though would work, if TL is used for caches, connection pools etc. but suboptimally).

As moving fibers across threads is a useful thing to do so as to balance the load (although there are other IMHO far better ways to do it) it should NOT be the default and has to be enabled explicitly with big red button.

@DmitryOlshansky

This comment has been minimized.

Show comment
Hide comment
@DmitryOlshansky

DmitryOlshansky Sep 26, 2014

Member

(Just to clarify) My last comment doesn't block this PR in any capacity.

Member

DmitryOlshansky commented Sep 26, 2014

(Just to clarify) My last comment doesn't block this PR in any capacity.

@mihails-strasuns

This comment has been minimized.

Show comment
Hide comment
@mihails-strasuns

mihails-strasuns Oct 3, 2014

@complexmath do you need any help with finalizing the commits?

@complexmath do you need any help with finalizing the commits?

@klickverbot

This comment has been minimized.

Show comment
Hide comment
@klickverbot

klickverbot Oct 4, 2014

Member

@DmitryOlshansky: FYI, there is also this problem: ldc-developers/ldc#666 (very fitting issue number, given the time it took me to track it down)

Member

klickverbot commented Oct 4, 2014

@DmitryOlshansky: FYI, there is also this problem: ldc-developers/ldc#666 (very fitting issue number, given the time it took me to track it down)

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath Oct 7, 2014

Member

I've mostly been really busy with other stuff, but some help would be great.

Member

complexmath commented Oct 7, 2014

I've mostly been really busy with other stuff, but some help would be great.

@mihails-strasuns

This comment has been minimized.

Show comment
Hide comment
@mihails-strasuns

mihails-strasuns Oct 11, 2014

@complexmath I have rebased your PR against current master and squashed everything in two commits : https://github.com/Dicebot/phobos/tree/complexmath-concurrency-pr

You can reset your PR to that branch by doing this:

git remote add dicebot https://github.com/Dicebot/phobos.git
git fetch dicebot
git checkout addConcurrencyScheduler
git reset --hard dicebot/complexmath-concurrency-pr
git push -f origin addConcurrencyScheduler

@complexmath I have rebased your PR against current master and squashed everything in two commits : https://github.com/Dicebot/phobos/tree/complexmath-concurrency-pr

You can reset your PR to that branch by doing this:

git remote add dicebot https://github.com/Dicebot/phobos.git
git fetch dicebot
git checkout addConcurrencyScheduler
git reset --hard dicebot/complexmath-concurrency-pr
git push -f origin addConcurrencyScheduler
@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath Oct 19, 2014

Member

Done. Thanks for the help!

Member

complexmath commented Oct 19, 2014

Done. Thanks for the help!

@DmitryOlshansky

This comment has been minimized.

Show comment
Hide comment
@DmitryOlshansky

DmitryOlshansky Oct 21, 2014

Member

Hm, it can't merge again. Needs yet another git pull --rebase upstream master ...

Member

DmitryOlshansky commented Oct 21, 2014

Hm, it can't merge again. Needs yet another git pull --rebase upstream master ...

@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath Oct 21, 2014

Member

Oh FFS. So I did this:

git fetch upstream master
git rebase upstream master
git checkout addConcurrencyScheduler
git merge master
... resolved conflict in std/concurrency.d
git push origin addConcurrencyScheduler

And this created a million changes. Instead of the "git merge master" step I guess I should have done something else?

Member

complexmath commented Oct 21, 2014

Oh FFS. So I did this:

git fetch upstream master
git rebase upstream master
git checkout addConcurrencyScheduler
git merge master
... resolved conflict in std/concurrency.d
git push origin addConcurrencyScheduler

And this created a million changes. Instead of the "git merge master" step I guess I should have done something else?

@DmitryOlshansky

This comment has been minimized.

Show comment
Hide comment
@DmitryOlshansky

DmitryOlshansky Oct 21, 2014

Member

@complexmath Yeah.. instead just this would have worked:

git checkout addConcurrencyScheduler
git pull --rebase upstream master

No need to re-fetch or anything.
Now to reset back to your old state to try the above, do:

git checkout addConcurrencyScheduler
git reset --hard 1b73a4e

That should put your branch at the right commit as it was.

Member

DmitryOlshansky commented Oct 21, 2014

@complexmath Yeah.. instead just this would have worked:

git checkout addConcurrencyScheduler
git pull --rebase upstream master

No need to re-fetch or anything.
Now to reset back to your old state to try the above, do:

git checkout addConcurrencyScheduler
git reset --hard 1b73a4e

That should put your branch at the right commit as it was.

complexmath and others added some commits Feb 6, 2014

Make std.concurrency compatible with fibers
https://d.puremagic.com/issues/show_bug.cgi?id=12090

With this commit fibers can have own message boxes and act similar
to threads in that regard.
Add Generator to std.concurrency
Generator is an extension to a Fiber that yields return values of specific type
@complexmath

This comment has been minimized.

Show comment
Hide comment
@complexmath

complexmath Oct 21, 2014

Member

Thanks. For some reason I thought "git pull" would always result in the huge merge we were trying to avoid.

Member

complexmath commented Oct 21, 2014

Thanks. For some reason I thought "git pull" would always result in the huge merge we were trying to avoid.

@DmitryOlshansky

This comment has been minimized.

Show comment
Hide comment
@DmitryOlshansky

DmitryOlshansky Oct 21, 2014

Member

@complexmath You are welcome. Going to auto-merge it unless something spooky happens.

Member

DmitryOlshansky commented Oct 21, 2014

@complexmath You are welcome. Going to auto-merge it unless something spooky happens.

@klickverbot

This comment has been minimized.

Show comment
Hide comment
@klickverbot

klickverbot Oct 21, 2014

Member

So Generator is accepted now? (I personally don't care too much either way.)

Member

klickverbot commented Oct 21, 2014

So Generator is accepted now? (I personally don't care too much either way.)

@mihails-strasuns

This comment has been minimized.

Show comment
Hide comment
@mihails-strasuns

mihails-strasuns Oct 22, 2014

I don't see any truly compelling reason to not merge it. If there are any issues with it will be easier to create separate PR's for those before release and figure those out on case by case basis. This has taken wa too long already, some sense of accomplishment is necessary :)

I don't see any truly compelling reason to not merge it. If there are any issues with it will be easier to create separate PR's for those before release and figure those out on case by case basis. This has taken wa too long already, some sense of accomplishment is necessary :)

@mihails-strasuns

This comment has been minimized.

Show comment
Hide comment
@mihails-strasuns

mihails-strasuns Oct 22, 2014

Auto-merge toggled on

Auto-merge toggled on

mihails-strasuns pushed a commit that referenced this pull request Oct 22, 2014

Михаил Страшун
Merge pull request #1910 from complexmath/addConcurrencyScheduler
Issue 12090 - Make std.concurrency compatible with fibers as threads

@mihails-strasuns mihails-strasuns merged commit b956f68 into dlang:master Oct 22, 2014

1 check passed

default Pass: 10
Details
@mihails-strasuns

This comment has been minimized.

Show comment
Hide comment
@mihails-strasuns

mihails-strasuns Oct 22, 2014

Couldn't resists to press the button myself :P

Couldn't resists to press the button myself :P

@mihails-strasuns

This comment has been minimized.

Show comment
Hide comment
@mihails-strasuns

mihails-strasuns Oct 22, 2014

For some reason I thought "git pull" would always result in the huge merge we were trying to avoid

It is other way around :) git merge will always result in huge merge commit. git pull may or may not result in merge commit depending on branch state and flags used (with --rebase it never will)

For some reason I thought "git pull" would always result in the huge merge we were trying to avoid

It is other way around :) git merge will always result in huge merge commit. git pull may or may not result in merge commit depending on branch state and flags used (with --rebase it never will)

@complexmath

This comment has been minimized.

Show comment
Hide comment
Member

complexmath commented Oct 22, 2014

Yay!

@DmitryOlshansky

This comment has been minimized.

Show comment
Hide comment
@DmitryOlshansky

DmitryOlshansky Oct 22, 2014

Member

Awesome. Oh, the strange pleasure of having 6+ month old PR merged :)

Member

DmitryOlshansky commented Oct 22, 2014

Awesome. Oh, the strange pleasure of having 6+ month old PR merged :)

@IgorStepanov

This comment has been minimized.

Show comment
Hide comment
@IgorStepanov

IgorStepanov Oct 22, 2014

Contributor

@DmitryOlshansky May be my new AA implementation will be merged somewhen=)
Remaining about 4 months? :o)

Contributor

IgorStepanov commented Oct 22, 2014

@DmitryOlshansky May be my new AA implementation will be merged somewhen=)
Remaining about 4 months? :o)

@DmitryOlshansky

This comment has been minimized.

Show comment
Hide comment
@DmitryOlshansky

DmitryOlshansky Oct 22, 2014

Member

@IgorStepanov other common durations are 8 month, and a year ;)
Sadly I still lack knowledge of druntime to merge big pulls there.

Member

DmitryOlshansky commented Oct 22, 2014

@IgorStepanov other common durations are 8 month, and a year ;)
Sadly I still lack knowledge of druntime to merge big pulls there.

@IgorStepanov

This comment has been minimized.

Show comment
Hide comment
@IgorStepanov

IgorStepanov Oct 22, 2014

Contributor

@DmitryOlshansky

other common durations are 8 month, and a year ;)

Ok, I have to be patient:)

Sadly I still lack knowledge of druntime to merge big pulls there.
BTW, I you have any time and energy, you may see the only dlang/druntime#934.
This PR implements AA backend, it haven't any dependence and it may be seen simply as the implementation of the abstract container.

Contributor

IgorStepanov commented Oct 22, 2014

@DmitryOlshansky

other common durations are 8 month, and a year ;)

Ok, I have to be patient:)

Sadly I still lack knowledge of druntime to merge big pulls there.
BTW, I you have any time and energy, you may see the only dlang/druntime#934.
This PR implements AA backend, it haven't any dependence and it may be seen simply as the implementation of the abstract container.

@MartinNowak

This comment has been minimized.

Show comment
Hide comment
@MartinNowak

MartinNowak Nov 7, 2014

Member

Awesome. Oh, the strange pleasure of having 6+ month old PR merged :)

Write smaller pull requests and discuss design decisions upfront.
I think it was a failure to merge this and disregarding outstanding design concerns.
#1910 (comment)
#1910 (comment)
#1910 (comment)
#1910 (comment)
It remains open how FiberScheduler can be integrated with event loops and it's somewhat unclear what the exact design goals are.

Please never start a complex feature by implementing a pull request.
The whole discussion is littered with Fiber bug reports, git noise and style comments, but very little is concerned with the API and how this integrates with the rest of D's infrastructure.

So for the next time, please start with a DIP or a newsgroup discussion telling people what you want to achieve and how you intend to implement it.

Member

MartinNowak commented Nov 7, 2014

Awesome. Oh, the strange pleasure of having 6+ month old PR merged :)

Write smaller pull requests and discuss design decisions upfront.
I think it was a failure to merge this and disregarding outstanding design concerns.
#1910 (comment)
#1910 (comment)
#1910 (comment)
#1910 (comment)
It remains open how FiberScheduler can be integrated with event loops and it's somewhat unclear what the exact design goals are.

Please never start a complex feature by implementing a pull request.
The whole discussion is littered with Fiber bug reports, git noise and style comments, but very little is concerned with the API and how this integrates with the rest of D's infrastructure.

So for the next time, please start with a DIP or a newsgroup discussion telling people what you want to achieve and how you intend to implement it.

- auto t = new Thread( &exec ); t.start();
- links[spawnTid] = linked;
+ if( scheduler !is null )
+ scheduler.spawn( &exec );

This comment has been minimized.

@MartinNowak

MartinNowak Nov 7, 2014

Member

Couldn't you simply initialize scheduler with a default 1:1 threading scheduler?
It's now possible to create and store classes in globals at CT.

@MartinNowak

MartinNowak Nov 7, 2014

Member

Couldn't you simply initialize scheduler with a default 1:1 threading scheduler?
It's now possible to create and store classes in globals at CT.

This comment has been minimized.

@complexmath

complexmath Nov 7, 2014

Member

Yes, but this would add unnecessary overhead for the default case.

@complexmath

complexmath Nov 7, 2014

Member

Yes, but this would add unnecessary overhead for the default case.

This comment has been minimized.

@MartinNowak

MartinNowak Nov 7, 2014

Member

Just a vtable right? More a matter of taste.

@MartinNowak

MartinNowak Nov 7, 2014

Member

Just a vtable right? More a matter of taste.

@mihails-strasuns

This comment has been minimized.

Show comment
Hide comment
@mihails-strasuns

mihails-strasuns Nov 11, 2014

@MartinNowak I disagree that this specific feature is complex enough to keep bashing it again and again. Most of the linked issues are not really related to adding fiber support to std.concurrency but to possible system built on top of this. This specific addition was small and was working as intended, I see no point in keeping it stalling for 1 more year while discussing more abstract things.

@MartinNowak I disagree that this specific feature is complex enough to keep bashing it again and again. Most of the linked issues are not really related to adding fiber support to std.concurrency but to possible system built on top of this. This specific addition was small and was working as intended, I see no point in keeping it stalling for 1 more year while discussing more abstract things.

@complexmath complexmath deleted the complexmath:addConcurrencyScheduler branch Feb 19, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment