[WIP] Add size-limited command interface #74549

Artoria2e5 · 2020-07-20T10:47:49Z

As described in #40384, most systems put an upper limit to the size of commands. The std::process::Command interface should therefore be updated to handle this problem and detect when the OS finds the command-line too big.

Progress on this stuff:

rust-highfive · 2020-07-20T10:47:52Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @LukasKalbertodt (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

bors · 2020-07-23T11:03:57Z

☔ The latest upstream changes (presumably #74667) made this pull request unmergeable. Please resolve the merge conflicts.

LukasKalbertodt · 2020-07-25T10:42:13Z

Hi @Artoria2e5 and thanks for the PR so far.

Would you like to have a review/feedback of the state so far? Or are you still actively working on it? In any case, just let me know!

bors · 2020-07-28T03:13:48Z

☔ The latest upstream changes (presumably #73265) made this pull request unmergeable. Please resolve the merge conflicts.

Artoria2e5 · 2020-07-28T18:19:26Z

A few rebases may happen given the recent change in file layout, so I think I am not ready yet.

LukasKalbertodt · 2020-07-28T22:59:44Z

Sure. Just let me know if you need any help or if I should take a look at the changes.

I still need to come up witth a place for the trait to go. Honestly I don't know where to put the file yet. This thing being quadratic is a problem too. I guess I will make a little struct with the current length and argc in it to keep track. (And yeah it only works as long as envp is unchanged.)

It looks like I really can't get away with the whole storing error thing as they don't have Copy. Maybe I will end up using the ugly problem enum thing like I did with Unix too. Grrr...

Artoria2e5 · 2020-07-29T12:31:28Z

@LukasKalbertodt I think I can ask for a very early review now. Most of my questions are in the commit messages, but generic ~~pointers~~ hints with Rust will help a lot too.

The main problem I have seems to be I really don't know where to put the trait.

I am bad at this...

Artoria2e5 · 2020-07-29T12:38:09Z

library/std/src/sys/unix/process/process_common.rs

+    pub fn get_size(&mut self) -> io::Result<usize> {
+        use crate::mem;
+        let argv = &self.argv.0;
+        let argv_size: usize = argv.iter().map(|x| unsafe { strlen(*x) + 1 }).sum::<usize>()


Oh great I forgot that the last element is a nullptr!

But seriously I should this without strlen... like accumulating a length in add when it's still an OsStr.

Artoria2e5 · 2020-07-29T14:53:30Z

library/std/src/sys/windows/process.rs

+        let program = rprogram.as_ref().unwrap_or(&self.program);
+
+        // Prepare and terminate the application name and the cmdline
+        // XXX: this won't work for 16-bit, might be preferable to do a extend_from_slice


Better do an append to the cmdline in new(). Oh I forgot it won't work because of the PATH resolving above. Gotta leave it for now...

it wants tight lines!

bjorn3 · 2020-08-23T09:13:28Z

library/std/src/sys/unix/process/process_common.rs

+                sys::os::set_errno(0);
+                let limit = libc::sysconf(libc::_SC_ARG_MAX);
+                let errno = sys::os::errno();
+                sys::os::set_errno(old_errno);


This is not necessary, right? I think it is expected that any function in libstd that interacts with the OS in some way can clobber errno.

Hmm, I didn't know that!

pickfire · 2020-08-31T15:24:49Z

library/std/src/sys/unix/process/process_common.rs

@@ -75,11 +76,14 @@ pub struct Command {
    args: Vec<CString>,
    argv: Argv,
    env: CommandEnv,
+    env_size: Option<usize>,
+    arg_max: Option<isize>,


Can arg_max possibly be Some(0)?

It wouldn't make sense to be that. Are you suggesting a naked isize?

Either that or NonZero types.

That looks cool. Let me get it changed.

pickfire · 2020-08-31T15:26:24Z

library/std/src/sys/unix/process/process_common.rs

+                self.arg_max = limit.try_into().ok();
+            }
+        }
+        Ok(self.arg_max.unwrap() < 0 || self.get_size()? < (self.arg_max.unwrap() as usize))


Shouldn't arg_max be usize so it won't be less than zero?

No, a negative value indicates that the arg size is unlimited. If my reading of POSIX is right, that is.

IIRC -0 of usize is still -1 for unlimited.

pickfire · 2020-08-31T15:27:15Z

library/std/src/sys/unix/process/process_common.rs

+                let old_errno = sys::os::errno();
+                sys::os::set_errno(0);


Why should we clear existing errno?

I don't think the no-error case automatically clears errno. Does it?

Dylan-DPC-zz · 2020-09-11T13:37:11Z

r? @Amanieu

bors · 2020-10-02T10:07:49Z

☔ The latest upstream changes (presumably #77029) made this pull request unmergeable. Please resolve the merge conflicts.

Note that reviewers usually do not review pull requests until merge conflicts are resolved! Once you resolve the conflicts, you should change the labels applied by bors to indicate that your PR is ready for review. Post this as a comment to change the labels:

@rustbot modify labels: +S-waiting-on-review -S-waiting-on-author

Artoria2e5 · 2020-10-02T11:51:46Z

I don't really understand why it's trying to compile a Windows thing for Unix…

Dylan-DPC-zz · 2020-10-16T01:21:42Z

@Artoria2e5 any updates on this?

bors · 2020-10-16T10:01:17Z

☔ The latest upstream changes (presumably #77666) made this pull request unmergeable. Please resolve the merge conflicts.

Note that reviewers usually do not review pull requests until merge conflicts are resolved! Once you resolve the conflicts, you should change the labels applied by bors to indicate that your PR is ready for review. Post this as a comment to change the labels:

@rustbot modify labels: +S-waiting-on-review -S-waiting-on-author

jyn514

error[E0432]: unresolved imports `crate::sys::process::Arg`, `crate::sys::process::Problem`, `crate::sys::process::RawArg`
  --> library/std/src/sys/windows/ext/process.rs:11:31
   |
11 | pub use crate::sys::process::{Arg, Problem, RawArg};
   |                               ^^^  ^^^^^^^  ^^^^^^ no `RawArg` in `sys::unix::process`
   |                               |    |
   |                               |    no `Problem` in `sys::unix::process`
   |                               no `Arg` in `sys::unix::process`

I don't really understand why it's trying to compile a Windows thing for Unix…

@Artoria2e5 the documentation for the standard library has a very cursed build system and shows all platforms even though the host is linux: #43348.

// FIXME: This really should be somewhere else, since it will be duplicated for unix. sys_common? I have no idea.
// The implementations should apply to unix, but describing it to the type system is another thing.

That sounds about right. The approach I would take is to have CommandSized in sys_common, but make the trait implementations specific to the OS.

library/std/src/sys/windows/ext/process.rs

jyn514 · 2020-10-30T06:15:27Z

library/std/src/sys/windows/ext/process.rs

+    /// Build many commands.
+    fn xargs<I, S, A>(program: S, args: I, before: Vec<A>, after: Vec<A>) -> io::Result<Vec<Self>>
+    where
+        I: IntoIterator<Item = A>,
+        S: AsRef<OsStr> + Copy,
+        A: Arg;


What does this do that maybe_args doesn't?

Like its namesake, xargs is supposed to create a new command to fit in additional parameters when there's too many of them. As a result it returns possibly many commands.

It can still fail from a single oversized parameter being impossible to fit.

jyn514 · 2020-10-30T06:16:39Z

library/std/src/sys/windows/process.rs

+pub enum Problem {
+    SawNul,
+    Oversized,
+}


I'd make the name of the error more descriptive.

Suggested change

pub enum Problem {

SawNul,

Oversized,

}

pub enum ArgumentError {

SawNul,

Oversized,

}

jyn514 · 2020-10-30T06:17:01Z

library/std/src/sys/windows/process.rs

+    fn append_to(&self, cmd: &mut Vec<u16>, force_quotes: bool) -> Result<usize, Problem>;
+    #[unstable(feature = "command_sized", issue = "74549")]
+    fn arg_size(&self, force_quotes: bool) -> Result<usize, Problem>;
+    fn to_os_string(&self) -> OsString;


The other functions need annotations too, I think.

jyn514 · 2020-10-30T06:17:18Z

library/std/src/sys/windows/process.rs

@@ -133,6 +190,13 @@ impl Command {
    pub fn creation_flags(&mut self, flags: u32) {
        self.flags = flags;
    }
+    #[allow(dead_code)]


What is this for?

There seem to be a lot of these, some pre-existing 🤷

Artoria2e5 · 2020-11-15T01:59:51Z

I can't seem to get the nonzero stuff working: I got hit with E0423 where I want to construct it.

That sounds about right. The approach I would take is to have CommandSized in sys_common, but make the trait implementations specific to the OS.

I think I can get away with that. The unix version can call the normal maybe_arg() while the windows can go on the ext functions. The outward signature is going to stay as the Arg trait, I think?

jyn514 · 2020-11-25T22:05:45Z

@Artoria2e5 yes, the Arg trait is going to be the same, but moved into one place so we can be sure it's not different between platforms.

Dylan-DPC-zz · 2021-01-29T16:49:30Z

@Artoria2e5 thanks for taking the time to contribute. I have to close this due to inactivity. If you wish and you have the time you can open a new PR with these changes and we'll take it from there. Thanks

rust-highfive assigned LukasKalbertodt Jul 20, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 20, 2020

Artoria2e5 added 2 commits July 29, 2020 02:15

Allow set_errno on Linux

095829a

[WIP] size limit for command line

191e319

Artoria2e5 force-pushed the command-size branch from ecc111d to 191e319 Compare July 29, 2020 02:06

Artoria2e5 added 3 commits July 29, 2020 10:15

fixup! CI complaint

0aa20e7

windows/process: fix some obvious issues, create some more

ff126ba

It looks like I really can't get away with the whole storing error thing as they don't have Copy. Maybe I will end up using the ugly problem enum thing like I did with Unix too. Grrr...

Artoria2e5 commented Jul 29, 2020

View reviewed changes

Artoria2e5 added 2 commits July 29, 2020 21:03

unix: incremental size, no strlen

dd5e440

fixup! unix: argv[0] has a size too

312dcd7

Artoria2e5 commented Jul 29, 2020

View reviewed changes

Artoria2e5 added 9 commits July 31, 2020 18:06

windows: sort the error type out

d190d3e

unix, windows: add temporary dead_code

fea9cef

unix: cache envp

29d21a8

Windows: draft traits for raw args and custom escapers

82502b9

Windows: raw args trait

0357037

Windows: minor rename while I figure out where to put sized

c7e5302

unix: minor error stuff

a9c93ff

windows: very rough xargs

18ec8e3

minor: run formatter

cb30a5c

it wants tight lines!

bjorn3 reviewed Aug 23, 2020

View reviewed changes

jyn514 added the T-libs-api Relevant to the library API team, which will review and decide on the PR/issue. label Aug 30, 2020

pickfire reviewed Aug 31, 2020

View reviewed changes

rust-highfive assigned Amanieu and unassigned LukasKalbertodt Sep 11, 2020

Merge branch 'master' into command-size

99ec627

Merge branch 'master' into command-size

fc52a05

jyn514 reviewed Oct 30, 2020

View reviewed changes

jyn514 added the S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. label Oct 30, 2020

Artoria2e5 added 2 commits November 15, 2020 09:56

fixup! remove dup stdarch in 82502b9

d11f795

Windows commandext: apply review suggestions

c945ef4

Dylan-DPC-zz removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Nov 21, 2020

Dylan-DPC-zz closed this Jan 29, 2021

camelid added the S-inactive Status: Inactive and waiting on the author. This is often applied to closed PRs. label Jan 29, 2021

Artoria2e5 mentioned this pull request Mar 9, 2021

[WIP] Add size-limited command interface, take 2 #82930

Closed

3 tasks

[WIP] Add size-limited command interface #74549

[WIP] Add size-limited command interface #74549

Conversation

Artoria2e5 commented Jul 20, 2020 • edited Loading

rust-highfive commented Jul 20, 2020

bors commented Jul 23, 2020

LukasKalbertodt commented Jul 25, 2020

bors commented Jul 28, 2020

Artoria2e5 commented Jul 28, 2020

LukasKalbertodt commented Jul 28, 2020

Artoria2e5 commented Jul 29, 2020 • edited Loading

Artoria2e5 Jul 29, 2020 • edited Loading

Choose a reason for hiding this comment

Artoria2e5 Jul 29, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pickfire Aug 31, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dylan-DPC-zz commented Sep 11, 2020

bors commented Oct 2, 2020

Artoria2e5 commented Oct 2, 2020

Dylan-DPC-zz commented Oct 16, 2020

bors commented Oct 16, 2020

jyn514 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Artoria2e5 commented Nov 15, 2020

jyn514 commented Nov 25, 2020

Dylan-DPC-zz commented Jan 29, 2021

Artoria2e5 commented Jul 20, 2020 •

edited

Loading

Artoria2e5 commented Jul 29, 2020 •

edited

Loading

Artoria2e5 Jul 29, 2020 •

edited

Loading

Artoria2e5 Jul 29, 2020 •

edited

Loading

pickfire Aug 31, 2020 •

edited

Loading