Add vectored read and write support #58357

sfackler · 2019-02-10T17:29:48Z

This functionality has lived for a while in the tokio ecosystem, where
it can improve performance by minimizing copies.

sfackler · 2019-02-10T17:30:09Z

Unix also supports vectored IO on files/stdout/etc, but I haven't plumbed that through yet.

sfackler · 2019-02-10T17:30:20Z

carllerche · 2019-02-10T19:11:22Z

Seems a bit early to commit this to std. It looks like it is copying the API from iovec that has not yet been released (and has not yet gotten real world usage yet).

sfackler · 2019-02-10T20:50:40Z

It'll be landing as unstable so we can continue to tweak as needed, but I don't think there'll really need to be all that much. The design space here is pretty constrained, and when talking about this at the all hands last week, we were all in agreement that the current iovec approach of DST hackery to avoid separate immutable and mutable types wasn't worth it. The other possibilities are this and IoVec<&[u8]>/IoVec<&mut [u8]>.

alexcrichton · 2019-02-11T12:50:12Z

This looks great to me, thanks @sfackler! I think I personally prefer the current approach over IoVec<T>.

One thing I think though we'll still want to game out is the extension traits for Unix/Windows perhaps? Ideally those would be used internally as well to avoid pointer casts

alexcrichton · 2019-02-11T12:43:34Z

src/libstd/io/mod.rs

+    /// # Panics
+    ///
+    /// Panics on Windows if the slice is larger than 4GB.
+    #[unstable(feature = "iovec", issue = "0")]


Just a reminder to fill out the issue before merging

src/libstd/io/mod.rs

sfackler · 2019-02-11T15:40:10Z

The extension trait would be on [IoVec] and [IoVecMut], right?

sfackler · 2019-02-11T15:42:56Z

Oh, we may not actually want those extension traits, since we'd need to expose iovec and WSABUF in the std public API.

alexcrichton · 2019-02-11T16:22:33Z

We don't necessarily need an extension trait but rather a free function. Thinking about it though a pointer cast is likely required anyway because the libc types on crates.io won't unify with the ones from libstd. In that case documenting this is probably the best way to go

sfackler · 2019-02-12T03:32:42Z

I just changed them to implement Deref/DerefMut. It makes the code a bit nicer.

rust-highfive · 2019-02-12T05:08:26Z

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:0fe6e128:start=1549942434637732500,finish=1549942508737713414,duration=74099980914
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-6.0
---
travis_time:start:test_debuginfo
Check compiletest suite=debuginfo mode=debuginfo-both (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:07:12] 
[01:07:12] running 119 tests
[01:07:36] .iiiii...i.....i..i...i..i.i..i.ii...i.....i..i....i..........iiii..........i...ii...i.......ii.i.i. 100/119
[01:07:40] i......iii.i.....ii
[01:07:40] 
[01:07:40]  finished in 28.435
[01:07:40] travis_fold:end:test_debuginfo

---
[01:32:55] travis_fold:end:stage0-linkchecker

[01:32:55] travis_time:end:stage0-linkchecker:start=1549948090938071448,finish=1549948093129082597,duration=2191011149

[01:32:57] std/io/struct.IoVec.html:393: broken link fragment `#method.sort_by_key` pointing to `std/io/struct.IoVec.html`
[01:32:57] std/io/struct.IoVec.html:505: broken link fragment `#method.make_ascii_uppercase` pointing to `std/io/struct.IoVec.html`
[01:32:57] std/io/struct.IoVec.html:510: broken link fragment `#method.make_ascii_lowercase` pointing to `std/io/struct.IoVec.html`
[01:33:03] thread 'main' panicked at 'found some broken links', src/tools/linkchecker/main.rs:41:9
[01:33:03] 
[01:33:03] 
[01:33:03] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/linkchecker" "/checkout/obj/build/x86_64-unknown-linux-gnu/doc"
[01:33:03] expected success, got: exit code: 101
[01:33:03] expected success, got: exit code: 101
[01:33:03] 
[01:33:03] 
[01:33:03] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test
[01:33:03] Build completed unsuccessfully in 0:37:12
[01:33:03] Makefile:48: recipe for target 'check' failed
[01:33:03] make: *** [check] Error 1
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:0852ae14
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Tue Feb 12 05:08:21 UTC 2019
---
travis_time:end:1303481a:start=1549948103091812517,finish=1549948103146636412,duration=54823895
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:0074ab6a
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:121e15f4
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

sfackler · 2019-02-12T05:26:49Z

Uh, why are doc links broken?

alexcrichton · 2019-02-12T13:01:41Z

I suspect that has to do with deref inlining methods or something like that, although beyond that I'm not sure why the links are broken

In any case r=me with a tracking issue filed

sfackler · 2019-02-13T01:41:49Z

@QuietMisdreavus do you know what's going on with the link checker up above?

This functionality has lived for a while in the tokio ecosystem, where it can improve performance by minimizing copies.

Returning &'a mut [u8] was unsound, and we may as well just have them directly deref to their slices to make it easier to work with them.

rust-highfive · 2019-02-14T04:25:14Z

The job x86_64-gnu-llvm-6.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

travis_time:end:02c7b877:start=1550117567041456961,finish=1550117567964715854,duration=923258893
$ git checkout -qf FETCH_HEAD
travis_fold:end:git.checkout

Encrypted environment variables have been removed for security reasons.
See https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions
$ export SCCACHE_BUCKET=rust-lang-ci-sccache2
$ export SCCACHE_REGION=us-west-1
Setting environment variables from .travis.yml
$ export IMAGE=x86_64-gnu-llvm-6.0
---
travis_time:start:tidy
tidy check
[00:03:50] * 565 error codes
[00:03:50] * highest error code: E0722
[00:03:50] tidy error: /checkout/src/libstd/io/mod.rs:285: mismatches the `issue` in previous
[00:03:51] some tidy checks failed
[00:03:51] 
[00:03:51] 
[00:03:51] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/tidy" "/checkout/src" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "--no-vendor" "--quiet"
[00:03:51] 
[00:03:51] 
[00:03:51] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test src/tools/tidy
[00:03:51] Build completed unsuccessfully in 0:00:45
[00:03:51] Build completed unsuccessfully in 0:00:45
[00:03:51] make: *** [tidy] Error 1
[00:03:51] Makefile:68: recipe for target 'tidy' failed
The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:005b4447
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)
Thu Feb 14 04:16:49 UTC 2019
---
travis_time:end:066b627a:start=1550117809913271018,finish=1550117809917839753,duration=4568735
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:00678750
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:2386af50
travis_time:start:2386af50
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:144d6b9c
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

sfackler · 2019-02-14T04:57:59Z

@bors r=alexcrichton

bors · 2019-02-14T04:58:01Z

📌 Commit 5ca3b00 has been approved by alexcrichton

rust-highfive · 2019-02-24T21:40:08Z

The job dist-various-2 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[01:04:38] [RUSTC-TIMING] panic_unwind test:false 0.305
[01:04:38] warning: dropping unsupported crate type `dylib` for target `x86_64-fortanix-unknown-sgx`
[01:04:38] 
[01:04:41] error[E0308]: mismatched types
[01:04:41]    --> src/libstd/sys/sgx/net.rs:111:19
[01:04:41] 111 |         self.read(buf)
[01:04:41]     |                   ^^^ types differ in mutability
[01:04:41]     |
[01:04:41]     = note: expected type `&mut [u8]`
[01:04:41]     = note: expected type `&mut [u8]`
[01:04:41]                found type `&io::IoVecMut<'_>`
[01:04:41] error[E0308]: mismatched types
[01:04:41] error[E0308]: mismatched types
[01:04:41]    --> src/libstd/sys/sgx/net.rs:123:19
[01:04:41] 123 |         self.read(buf)
[01:04:41]     |                   ^^^ types differ in mutability
[01:04:41]     |
[01:04:41]     = note: expected type `&mut [u8]`
[01:04:41]     = note: expected type `&mut [u8]`
[01:04:41]                found type `&io::IoVec<'_>`
[01:04:44] error: aborting due to 2 previous errors
[01:04:44] 
[01:04:44] For more information about this error, try `rustc --explain E0308`.
[01:04:44] [RUSTC-TIMING] std test:false 5.752
---
travis_time:end:027323f4:start=1551044304323950905,finish=1551044304362673927,duration=38723022
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0e9acc63
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:10ebc355
travis_time:start:10ebc355
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:0182a9cc
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

sfackler · 2019-02-24T21:47:45Z

@bors r=alexcrichton

bors · 2019-02-24T21:47:45Z

📌 Commit c5b87a2 has been approved by alexcrichton

bors · 2019-02-25T00:39:11Z

⌛ Testing commit c5b87a2 with merge dc00640b2c9aaeedc49ac03d44e710037d8a8285...

bors · 2019-02-25T02:02:22Z

💔 Test failed - checks-travis

rust-highfive · 2019-02-25T02:02:23Z

The job dist-various-1 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

  Downloading https://files.pythonhosted.org/packages/2d/99/b2c4e9d5a30f6471e410a146232b4118e697fa3ffc06d6a65efde84debd0/futures-3.2.0-py2-none-any.whl
Collecting urllib3<1.25,>=1.20; python_version == "2.7" (from botocore==1.12.101->awscli)
  Downloading https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl (118kB)
Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.12.101->awscli)
  Downloading https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
  Downloading https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl (226kB)
Requirement already satisfied: pyasn1>=0.1.3 in /usr/lib/python2.7/dist-packages (from rsa<=3.5.0,>=3.1.2->awscli) (0.1.9)
Requirement already satisfied: six>=1.5 in /usr/lib/python2.7/dist-packages (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore==1.12.101->awscli) (1.10.0)
Installing collected packages: futures, urllib3, jmespath, python-dateutil, docutils, botocore, s3transfer, rsa, colorama, awscli
---
[01:20:23] 
[01:20:24] error[E0425]: cannot find value `buf` in this scope
[01:20:24]   --> src/libstd/sys/redox/net/tcp.rs:49:15
[01:20:24]    |
[01:20:24] 49 |         match buf.iter().find(|b| !b.is_empty()) {
[01:20:24]    |               ^^^ help: a local variable with a similar name exists: `bufs`
[01:20:26] error: aborting due to previous error
[01:20:26] 
[01:20:26] For more information about this error, try `rustc --explain E0425`.
[01:20:26] [RUSTC-TIMING] std test:false 3.623
---
travis_time:end:0dcac805:start=1551060064925381961,finish=1551060064942960225,duration=17578264
travis_fold:end:after_failure.3
travis_fold:start:after_failure.4
travis_time:start:0029bcaa
$ ln -s . checkout && for CORE in obj/cores/core.*; do EXE=$(echo $CORE | sed 's|obj/cores/core\.[0-9]*\.!checkout!\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb --batch -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.4
travis_fold:start:after_failure.5
travis_time:start:25f380c0
travis_time:start:25f380c0
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:06b38ca1
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

sfackler · 2019-02-25T02:49:25Z

@bors r=alexcrichton

bors · 2019-02-25T02:49:25Z

📌 Commit 4785c74 has been approved by alexcrichton

bors · 2019-02-26T02:48:21Z

⌛ Testing commit 4785c74 with merge fb162e6...

@alexcrichton

Add vectored read and write support This functionality has lived for a while in the tokio ecosystem, where it can improve performance by minimizing copies. r? @alexcrichton

bors · 2019-02-26T05:47:47Z

☀️ Test successful - checks-travis, status-appveyor
Approved by: alexcrichton
Pushing fb162e6 to master...

bors · 2019-02-26T05:47:57Z

☀️ Test successful - checks-travis, status-appveyor
Approved by: alexcrichton
Pushing fb162e6 to master...

rust-highfive assigned alexcrichton Feb 10, 2019

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Feb 10, 2019

sfackler force-pushed the vectored-io branch 2 times, most recently from 8932b55 to 03fc79a Compare February 10, 2019 18:18

alexcrichton reviewed Feb 11, 2019

View reviewed changes

sfackler added 3 commits February 13, 2019 19:40

Add vectored read and write support

31bcec6

This functionality has lived for a while in the tokio ecosystem, where it can improve performance by minimizing copies.

impl Deref/DerefMut for IoVec types

596f182

Returning &'a mut [u8] was unsound, and we may as well just have them directly deref to their slices to make it easier to work with them.

Whitelist iovec types in linkchecker

034de8d

sfackler force-pushed the vectored-io branch from b3b5eaf to 034de8d Compare February 14, 2019 04:07

Add a tracking issue

5ca3b00

sfackler force-pushed the vectored-io branch from 9c76f7b to 5ca3b00 Compare February 14, 2019 04:27

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 14, 2019

jonas-schievink mentioned this pull request Feb 14, 2019

Tracking issue for vectored IO support #58452

Closed

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Feb 24, 2019

Fix sgx

c5b87a2

sfackler force-pushed the vectored-io branch from 3c8c835 to c5b87a2 Compare February 24, 2019 21:45

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 24, 2019

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Feb 25, 2019

Fix redox

4785c74

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 25, 2019

bors added the merged-by-bors This PR was explicitly merged by bors. label Feb 26, 2019

bors merged commit 4785c74 into rust-lang:master Feb 26, 2019

bors mentioned this pull request Feb 26, 2019

Use the parking_lot locking primitives #56410

Closed

sfackler deleted the vectored-io branch February 26, 2019 05:54

bors mentioned this pull request Feb 26, 2019

Deprecate Read::initializer in favor of ptr::freeze #58363

Closed

jeikabu mentioned this pull request Mar 2, 2019

Add support for IoVec to iov jeikabu/runng#48

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add vectored read and write support #58357

Add vectored read and write support #58357

sfackler commented Feb 10, 2019

sfackler commented Feb 10, 2019

sfackler commented Feb 10, 2019

carllerche commented Feb 10, 2019

sfackler commented Feb 10, 2019

alexcrichton commented Feb 11, 2019

alexcrichton Feb 11, 2019

sfackler commented Feb 11, 2019

sfackler commented Feb 11, 2019

alexcrichton commented Feb 11, 2019

sfackler commented Feb 12, 2019

rust-highfive commented Feb 12, 2019

sfackler commented Feb 12, 2019

alexcrichton commented Feb 12, 2019

sfackler commented Feb 13, 2019

rust-highfive commented Feb 14, 2019

sfackler commented Feb 14, 2019

bors commented Feb 14, 2019

rust-highfive commented Feb 24, 2019

sfackler commented Feb 24, 2019

bors commented Feb 24, 2019

bors commented Feb 25, 2019

bors commented Feb 25, 2019

rust-highfive commented Feb 25, 2019

sfackler commented Feb 25, 2019

bors commented Feb 25, 2019

bors commented Feb 26, 2019

bors commented Feb 26, 2019

bors commented Feb 26, 2019

Add vectored read and write support #58357

Add vectored read and write support #58357

Conversation

sfackler commented Feb 10, 2019

sfackler commented Feb 10, 2019

sfackler commented Feb 10, 2019

carllerche commented Feb 10, 2019

sfackler commented Feb 10, 2019

alexcrichton commented Feb 11, 2019

alexcrichton Feb 11, 2019

Choose a reason for hiding this comment

sfackler commented Feb 11, 2019

sfackler commented Feb 11, 2019

alexcrichton commented Feb 11, 2019

sfackler commented Feb 12, 2019

rust-highfive commented Feb 12, 2019

sfackler commented Feb 12, 2019

alexcrichton commented Feb 12, 2019

sfackler commented Feb 13, 2019

rust-highfive commented Feb 14, 2019

sfackler commented Feb 14, 2019

bors commented Feb 14, 2019

rust-highfive commented Feb 24, 2019

sfackler commented Feb 24, 2019

bors commented Feb 24, 2019

bors commented Feb 25, 2019

bors commented Feb 25, 2019

rust-highfive commented Feb 25, 2019

sfackler commented Feb 25, 2019

bors commented Feb 25, 2019

bors commented Feb 26, 2019

bors commented Feb 26, 2019

bors commented Feb 26, 2019