Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for channel selection #27800

Closed
alexcrichton opened this issue Aug 13, 2015 · 91 comments
Closed

Tracking issue for channel selection #27800

alexcrichton opened this issue Aug 13, 2015 · 91 comments

Comments

@alexcrichton
Copy link
Member

@alexcrichton alexcrichton commented Aug 13, 2015

This is a tracking issue for the unstable mpsc_select feature in the standard library. Known points:

  • The API is unsafe, and should probably never be stabilized.
  • Servo apparently makes very heavy use of this API, so it can't be removed without replacement currently.
  • The correctness of the implementation is somewhat dubious, it's incredibly complicated.
  • The macro-based approach for selection is dubious.
  • This is not a scalable implementation of selection (e.g. "poll everything each time wait() is called")
  • This drags down some other channel-related code with a few mutexes in otherwise lock-free code.
@jdm
Copy link
Contributor

@jdm jdm commented Aug 13, 2015

cc me

@steveklabnik
Copy link
Member

@steveklabnik steveklabnik commented Aug 13, 2015

Is it worth investigating @BurntSushi 's https://github.com/BurntSushi/chan ?

@steveklabnik
Copy link
Member

@steveklabnik steveklabnik commented Aug 13, 2015

I also think @carllerche was supporting the idea of removing channels from the stdlib

@alexcrichton
Copy link
Member Author

@alexcrichton alexcrichton commented Aug 13, 2015

Note that I'm not advocating removing channels from the standard library (plus they're stable). Removing selection is also probably not an option as it's so closely tied to the implementation details of channels.

One of the major points of complexity of the current implementation is that it's all basically 99% lock free (giving us a nice performance boost in theory). I believe @BurntSushi's implementation (please correct me if I'm wrong!) is largely based on mutexes and condition variables.

@BurntSushi
Copy link
Member

@BurntSushi BurntSushi commented Aug 13, 2015

@alexcrichton Correct. I doubt very much that chan will ever be lock free because I'm not convinced the semantics I want can be done with them. It's also much easier to implement. :-)

@nagisa
Copy link
Contributor

@nagisa nagisa commented Aug 13, 2015

I personally prefer the epoll-like design as implemented in comm.

@carllerche
Copy link
Member

@carllerche carllerche commented Aug 13, 2015

Streams in Eventual are lock free :) There is also stream selection, which is lock free (and definitely is a pain to implement).

Re: removing channels from stdlib, as @alexcrichton points out, they are stable. I would not worry about adding select to them. They are good as a starting point. If somebody requires more complex concurrency semantics, they can use a lib. I'm not sure why @alexcrichton thinks that removing select from std is hard though...

@tripped
Copy link

@tripped tripped commented Aug 20, 2015

Just hopping on to mention #12902 as one of the current weaknesses of the macro approach to select!.

@softprops
Copy link

@softprops softprops commented Sep 17, 2015

Fwiw, channel selection with golang channels is part of what makes golang concurrency story so alluring to newcomers to go. For a newcomer to a safe concurrent language like rust, needing to mull over which 3rd party concurrency crate is popular this week to pick for actually writing concurrent code feels onerous. Having primatives like channels and channel selection in the std lib make rust more attractive for concurrent applications than alternatives like go and make crates a little more interoperable than resolving conflicting implementation issues betwwen 3rd party options.

@BurntSushi
Copy link
Member

@BurntSushi BurntSushi commented Sep 17, 2015

Fwiw, channel selection with golang channels is part of what makes golang concurrency story so alluring to newcomers to go.

@softprops This doesn't address your primary concerns, but FYI, chan_select! in the chan crate was built to specifically match the semantics of Go's select.

@softprops
Copy link

@softprops softprops commented Sep 17, 2015

@BurntSushi nice!

@jonhoo
Copy link
Contributor

@jonhoo jonhoo commented May 1, 2016

Correct me if I'm wrong, but doesn't the macro approach also have the drawback that selecting over a dynamic set of channels isn't easily expressible?

@szagi3891
Copy link

@szagi3891 szagi3891 commented May 1, 2016

Best to get rid of the macro select.
I made a working prototype that shows that it is possible:
https://github.com/szagi3891/channels_async-rs/blob/master/examples/select.rs#L76
Without any section unsafe.

@canndrew
Copy link
Contributor

@canndrew canndrew commented Jun 2, 2016

Here's a wacky idea: add support for generically sized tuples and anonymous enums. Then it will be possible to select across an arbitrary number of channels of different types without macros:

let ch0: Receiver<i32> = ...;
let ch1: Receiver<String> = ...;
match select((ch0, ch1)) {
    (x|!) => println!("Got an i32: {}", x),
    (!|y) => println!("Got a String: {}", y),
}
@BurntSushi
Copy link
Member

@BurntSushi BurntSushi commented Nov 2, 2016

Is there anyone still using this API? I don't see any obvious path to stabilization. It would be nice to just remove it.

@alexcrichton
Copy link
Member Author

@alexcrichton alexcrichton commented Nov 2, 2016

@jdm is this still being used in Servo?

@SimonSapin
Copy link
Contributor

@SimonSapin SimonSapin commented Nov 2, 2016

grep finds 4 instances of select! in servo master.

@mattgreen
Copy link

@mattgreen mattgreen commented Nov 4, 2016

I'd like to remove this from watchexec, but I'm unsure what to move to. One of my crates already streams data to me via a channel, so I hopped onto that bandwagon.

I'm usually watching for either control-c or file system inputs at all times, since I have cleanup to do when the user hits control-c. select! might be unstable but I can at least accomplish this.

@softprops
Copy link

@softprops softprops commented Nov 4, 2016

I've had only good experiences with https://github.com/BurntSushi/chan

@szagi3891
Copy link

@szagi3891 szagi3891 commented Nov 4, 2016

@mattgreen :
I think you should be able to replace this selecta! macoro of one channel.

This channel type will be :

enum Message {
    CtrlC,
    NormalMessage(....),
}

The main thread would send in theright time message Message::CtrlC.
The remainder of the program would send messages Message::NormalMessage

It makes sense ?

@antrik
Copy link
Contributor

@antrik antrik commented Nov 12, 2016

Servo's ipc-channel library ( https://github.com/servo/ipc-channel ) also uses mpsc_select (for its inprocess back-end) -- which is a bit of a pain, since platforms relying on this back-end can't use ipc-channel with stable Rust. (See servo/ipc-channel#118 )

It should be noted though that this back-end is not a real implementation, but rather just a partial workaround for platforms that don't have a "real" inter-process implementation yet. While it was originally created for Windows, most likely Windows will be getting a proper implementation soonish. ( servo/ipc-channel#108 )

Right now the inprocess back-end is also being used for Android -- though I'm not sure why really...

So while this back-end might continue to be useful during bring-up of new platforms (depending on whether Servo keeps the option of running in a single process), it's not exactly a first-class citizen I'd say.

@antrik
Copy link
Contributor

@antrik antrik commented Nov 12, 2016

I'd like to point out that https://github.com/BurntSushi/chan -- which is being touted as a replacement for mpsc here -- doesn't really seem like a valid alternative to me: unlike mpsc, it has clonable and sharable receivers.

This might seem convenient for some use cases; but I'm not convinced it's a real win over just wrapping the receiver in a Mutex<> or Arc<Mutex<>> yourself when you know it's save and necessary for your application. (As mutexes are safe and easy to use in Rust -- unlike in Go -- I don't see any motivation for channels to double as a locking substitute...)

More importantly though, for the vast majority of use cases, sharing the receiver is actually not desirable: rather, it often indicates bugs -- and thus not rejecting this by default is indeed more error prone!

(Not even considering implementation overhead for something that's not needed at all in most cases...)

@BurntSushi
Copy link
Member

@BurntSushi BurntSushi commented Nov 12, 2016

@antrik Please see the README for chan. Notably:

The purpose of this crate is to provide a safe concurrent abstraction for communicating sequential processes. Performance takes a second seat to semantics and ergonomics. If you're looking for high performance synchronization, chan is probably not the right fit.

There are a number of reasons why it doesn't necessarily replace std::sync::mpsc. Nevertheless, if you want to write in a CSP style in Rust, then I do think it's your only option on stable Rust because it gives you chan_select!. I suspect that's why people are bringing it up (which seems reasonable to me).

@antrik
Copy link
Contributor

@antrik antrik commented Nov 12, 2016

@BurntSushi as long as there is a general understanding that chan is a special-purpose library rather than a general replacement, that's perfectly fine -- I'm just concerned that there seems to be an air here of "we don't need to fix mpsc since there is chan", which I find rather disconcerting...

@SimonSapin
Copy link
Contributor

@SimonSapin SimonSapin commented Sep 29, 2018

Next is landing a deprecation warning.

Mark-Simulacrum added a commit to Mark-Simulacrum/rust that referenced this issue Nov 9, 2018
…apin

Deprecate mpsc channel selection

Rationale: rust-lang#27800 (comment)

r? @SimonSapin
@SimonSapin
Copy link
Contributor

@SimonSapin SimonSapin commented Jan 29, 2019

Triage: this unstable feature was deprecated in 1.32, so it is fine to remove any time.

@stjepang
Copy link
Contributor

@stjepang stjepang commented Mar 5, 2019

I published a blog post titled Proposal: New channels for Rust's standard library discussing what to do with channels in the standard library.

How should we proceed about resolving this issue?

Bug #39364 is still present, which @stepancheg attempted to fix in PR #42883 but got stuck waiting on removal of mpsc_select. Perhaps now is a good time to remove mpsc_select and revive that PR.

Another option is to switch the whole implementation to https://github.com/stjepang/new-channel. As a bonus, we get performance improvements, impl Sync for Sender, SyncSender::send_timeout, more tests, lower compilation time, and fewer unsafe blocks.

Some people have suggested we just deprecate the whole mpsc module and leave channels out of the standard library.

Finally, the last option is to write a RFC for std::sync::channel that would eventually replace std::sync::mpsc. This would also allow us to fix #42883 and bring all the other improvements to std::sync::mpsc in the meantime.

@SimonSapin
Copy link
Contributor

@SimonSapin SimonSapin commented Mar 5, 2019

Oh I didn’t realize there was already work blocked on the removal of selection. If you want to send a removal PR feel free to ping me for review. We’re already good to go process-wise.

Replacing the internals of std::sync::mpsc and/or deprecating it is also interesting, but not relevant to this issue. It should probably be discussed on internals.rlo and then turned into an RFC.

@stjepang
Copy link
Contributor

@stjepang stjepang commented Mar 5, 2019

@stepancheg Would you be interested in giving PR #42883 another shot and removing selection along the way?

For followers of this thread, here's the the last status report on the bug fix.

@theronic
Copy link

@theronic theronic commented Mar 12, 2019

What's the succession story for waiting on two channels and taking from the first one that yields a message?

@SimonSapin
Copy link
Contributor

@SimonSapin SimonSapin commented Mar 12, 2019

Consider using another library such as https://crates.io/crates/crossbeam-channel

@havelund
Copy link

@havelund havelund commented Apr 1, 2019

How can one have channels and no selection between channels in the core language.

@steveklabnik
Copy link
Member

@steveklabnik steveklabnik commented Apr 1, 2019

Channels are not in the core language; they are in the standard library.

@havelund
Copy link

@havelund havelund commented Apr 1, 2019

Sure, but still: being able to select which channel to read from seems to be a fundamental feature that should follow a message passing paradigm based on channels. I tried using mutexes and that was a disaster since they are not reentrant.

@BurntSushi
Copy link
Member

@BurntSushi BurntSushi commented Apr 1, 2019

@havelund As linked above, use a library: https://crates.io/crates/crossbeam-channel

@havelund
Copy link

@havelund havelund commented Apr 1, 2019

Yes thanks. I found it. It seems to work fine.

bors added a commit that referenced this issue May 20, 2019
Remove the unstable and deprecated mpsc_select

This removes macro `select!` and `std::sync::mpsc::{Handle, Select}`,
which were all unstable and have been deprecated since 1.32.

Closes #27800
r? @SimonSapin
@bors bors closed this in #60921 May 20, 2019
@sdegutis
Copy link
Contributor

@sdegutis sdegutis commented Mar 3, 2020

For what it's worth, I ended up at this issue because I was trying to find out what happened to the select! macro in std::sync::mpsc which would be useful to me right about now. And only through this issue did I find out about crossbeam-channel which I'm switching to for its select!. Would be nice to have something like that in the standard library.

@dralley
Copy link
Contributor

@dralley dralley commented Apr 15, 2020

Is there still any discussion about deprecating std::sync::mpsc in favor of something based on crossbeam-channel? I haven't seen any discussion since just after the blog post came out.

@SimonSapin
Copy link
Contributor

@SimonSapin SimonSapin commented Apr 15, 2020

std::sync::mpsc (or at least its public API) is here to stay since it’s #[stable]. At most we could make its usage emit deprecation warnings, but what positive outcome would that achieve? That would be to balance against ecosystem churn.

@jonhoo
Copy link
Contributor

@jonhoo jonhoo commented Apr 15, 2020

@SimonSapin I read @dralley's proposal as replacing the internals, not the API, but could be wrong?

@dralley
Copy link
Contributor

@dralley dralley commented Apr 15, 2020

Not my proposal just to be clear, I'm just curious as to whether any action items had ever come out of that discussion.

@acestronautical
Copy link

@acestronautical acestronautical commented Apr 27, 2020

As someone who came to love Go, coded in it professionally for almost a year, and subsequently fell out of love with it (generics, performance, package management, code gen hell), the lack of Select in the standard library in Rust is what is preventing me from adopting it.

If I need high performance and I'm going to have to rely on external code for concurrency support anyways, I might as well go with the path of least resistance and write in C.

@BurntSushi
Copy link
Member

@BurntSushi BurntSushi commented Apr 27, 2020

@Ace-Cassidy You just need to add crossbeam-channel = "0.4" to your Cargo.toml and you should be good to go.

@ssokolow
Copy link

@ssokolow ssokolow commented Apr 27, 2020

@Ace-Cassidy Rust is designed to have a minimal standard library so that different components can evolve their APIs and bump their versions independently from each other.

You'll also need external crates (maintained by Rust team members) for things like random number generation (rand), regular expression support (regex), or pre-built bindings for the parts of the POSIX API that don't have equivalents on Win32 (libc).

astonbitecode added a commit to astonbitecode/j4rs that referenced this issue Sep 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

You can’t perform that action at this time.