Use shell parsing for response files

[P1n3appl3]

It turns out that #116610 is inadequate, we actually need rustc to perform shell parsing on the lines in response files. This will require adding a dep to the compiler: https://crates.io/crates/shlex

Both clang and gcc do shlex parsing, so it seems reasonable to make rustc do the same. However this would technically be a breaking change if some build system was relying on being able to pass unquoted arguments with spaces in them in a response file and having rustc interpret them as a single arg. To me that scenario seems pretty unlikely, but I'd understand if people want to be more cautious.

How should we proceed? We could do any of:

• Add some tests and land as-is assuming that no one relies on the current lack of shell parsing
• Perform a crater run with this change and then do ^ with more confidence. Unfortunately crater doesn't do a good job of detecting breakages due to this change because cargo doesn't use response files. We'd expect ninja/make/bazel/buck/etc. build systems to be the ones affected by this. We could do a survey of build tools with rust support but I'd guess that because all of those tools interface with c compilers that do perform shlex parsing they'd be quoting their args correctly in the first place.
• Write an MCP so that the compiler maintainers can sign off on the breaking change before we merge it.
• Add a new unstable flag such as -Z response_file_shlex that enables this new behavior for all response files on the command line
  • If it's necessary to be able to mix both types of response file handling, it could work like -Bstatic/-Bdynamic where they toggle it for any response files after the flag
  • We could also have the flag apply to a specific file

[rustbot]

r? @wesleywiser

(rustbot has picked a reviewer for you, use r? to override)

[rustbot]

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

Third-party dependency whitelist may have been modified! You must ensure that any new dependencies have compatible licenses before merging.

cc @davidtwco, @wesleywiser

[Nilstrieb]

I think crater would be useless. Given that clang and GCC do this, it seems unlikely that any build system would rely on it not being done. At least to me this seems reasonable, but you should create an MCP it.
Doing a quick survey of existing users of response files would be great if possible, just to make sure. Maybe git blame around the relevant code will reveal some?

[apiraino]

Switching to S-waiting-on-author to take action after compiler-team#684 finishes (or advance a new MCP?)

@rustbot author

[bors]

☔ The latest upstream changes (presumably #119777) made this pull request unmergeable. Please resolve the merge conflicts.

[Dylan-DPC]

@P1n3appl3 any updates on this? thanks

[P1n3appl3]

#118680 superseded this PR and was landed