Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid leaking doskey macros from internal vcvarsall runs. #1295

Merged
merged 6 commits into from Jan 10, 2024

Conversation

BillyONeal
Copy link
Member

This is a resolution for microsoft/vcpkg#32641 .

First, I transform all the process launch things to use a struct instead, because this needs to add yet another parameter and if I had to touch all callers to think about this parameter anyway, it made sense to just do the struct transformation now.

Second, this adds a new field to that struct CreateNewConsole, which if true runs things in a new console, which insulates us from doskey. Notably, there is a performance cost for creating new consoles, so we only want to do it when we think it is likely to matter, like here.

Below is a demonstration of the bug, and that the bug is fixed after this change:

D:\vcpkg>.\bootstrap-vcpkg.bat
Downloading https://github.com/microsoft/vcpkg-tool/releases/download/2023-11-16/vcpkg.exe -> D:\vcpkg\vcpkg.exe... done.
Validating signature... done.

vcpkg package management program version 2023-11-16-4c1df40a3c5c5e18de299a99e9accb03c2a82e1e

See LICENSE.txt for license information.
Telemetry
---------
vcpkg collects usage data in order to help us improve your experience.
The data collected by Microsoft is anonymous.
You can opt-out of telemetry by re-running the bootstrap-vcpkg script with -disableMetrics,
passing --disable-metrics to vcpkg on the command line,
or by setting the VCPKG_DISABLE_METRICS environment variable.

Read more about vcpkg telemetry at docs/about/privacy.md

D:\vcpkg>.\vcpkg.exe install zlib:x64-windows --binarysource=clear;
Computing installation plan...
The following packages will be built and installed:
  * vcpkg-cmake:x64-windows -> 2023-05-04
    zlib:x64-windows -> 1.3
Additional packages (*) will be modified to complete this operation.
Detecting compiler hash for triplet x64-windows...
Installing 1/2 vcpkg-cmake:x64-windows...
Building vcpkg-cmake:x64-windows...
-- Installing: D:/vcpkg/packages/vcpkg-cmake_x64-windows/share/vcpkg-cmake/vcpkg_cmake_configure.cmake
-- Installing: D:/vcpkg/packages/vcpkg-cmake_x64-windows/share/vcpkg-cmake/vcpkg_cmake_build.cmake
-- Installing: D:/vcpkg/packages/vcpkg-cmake_x64-windows/share/vcpkg-cmake/vcpkg_cmake_install.cmake
-- Installing: D:/vcpkg/packages/vcpkg-cmake_x64-windows/share/vcpkg-cmake/vcpkg-port-config.cmake
-- Installing: D:/vcpkg/packages/vcpkg-cmake_x64-windows/share/vcpkg-cmake/copyright
-- Performing post-build validation
Elapsed time to handle vcpkg-cmake:x64-windows: 33.9 ms
Installing 2/2 zlib:x64-windows...
Building zlib:x64-windows...
-- Using cached madler-zlib-v1.3.tar.gz.
-- Cleaning sources at D:/vcpkg/buildtrees/zlib/src/v1.3-8825ee792f.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source D:/vcpkg-downloads/madler-zlib-v1.3.tar.gz
-- Applying patch 0001-Prevent-invalid-inclusions-when-HAVE_-is-set-to-0.patch
-- Applying patch 0002-skip-building-examples.patch
-- Applying patch 0003-build-static-or-shared-not-both.patch
-- Applying patch 0004-android-and-mingw-fixes.patch
-- Using source at D:/vcpkg/buildtrees/zlib/src/v1.3-8825ee792f.clean
-- Configuring x64-windows
-- Building x64-windows-dbg
-- Building x64-windows-rel
-- Installing: D:/vcpkg/packages/zlib_x64-windows/share/zlib/vcpkg-cmake-wrapper.cmake
-- Fixing pkgconfig file: D:/vcpkg/packages/zlib_x64-windows/lib/pkgconfig/zlib.pc
-- Using cached mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst.
-- Using cached msys2-msys2-runtime-3.4.9-3-x86_64.pkg.tar.zst.
-- Using msys root at D:/vcpkg-downloads/tools/msys2/023cdb3ca06f77f2
-- Fixing pkgconfig file: D:/vcpkg/packages/zlib_x64-windows/debug/lib/pkgconfig/zlib.pc
-- Installing: D:/vcpkg/packages/zlib_x64-windows/share/zlib/copyright
-- Performing post-build validation
Elapsed time to handle zlib:x64-windows: 2.6 s
Total install time: 2.6 s
The package zlib is compatible with built-in CMake targets:

    find_package(ZLIB REQUIRED)
    target_link_libraries(main PRIVATE ZLIB::ZLIB)

D:\vcpkg>doskey /macros
vcpkg="C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\vcpkg\vcpkg-cmd.cmd" $*

== broken

D:\vcpkg>doskey vcpkg=

D:\vcpkg>doskey /macros

D:\vcpkg>copy ..\vcpkg-tool\out\build\Win-x64-Debug-WithArtifacts\vcpkg.exe .
Overwrite .\vcpkg.exe? (Yes/No/All): y
        1 file(s) copied.

D:\vcpkg>.\vcpkg.exe x-set-installed
The following packages will be removed:
    vcpkg-cmake:x64-windows
    zlib:x64-windows
Removing 1/2 zlib:x64-windows
Elapsed time to handle zlib:x64-windows: 4.59 ms
Removing 2/2 vcpkg-cmake:x64-windows
Elapsed time to handle vcpkg-cmake:x64-windows: 2.29 ms
Total install time: 7.07 ms

D:\vcpkg>.\vcpkg.exe install zlib:x64-windows --binarysource=clear;
Computing installation plan...
The following packages will be built and installed:
  * vcpkg-cmake:x64-windows -> 2023-05-04
    zlib:x64-windows -> 1.3
Additional packages (*) will be modified to complete this operation.
Detecting compiler hash for triplet x64-windows...
Installing 1/2 vcpkg-cmake:x64-windows...
Building vcpkg-cmake:x64-windows...
-- Installing: D:/vcpkg/packages/vcpkg-cmake_x64-windows/share/vcpkg-cmake/vcpkg_cmake_configure.cmake
-- Installing: D:/vcpkg/packages/vcpkg-cmake_x64-windows/share/vcpkg-cmake/vcpkg_cmake_build.cmake
-- Installing: D:/vcpkg/packages/vcpkg-cmake_x64-windows/share/vcpkg-cmake/vcpkg_cmake_install.cmake
-- Installing: D:/vcpkg/packages/vcpkg-cmake_x64-windows/share/vcpkg-cmake/vcpkg-port-config.cmake
-- Installing: D:/vcpkg/packages/vcpkg-cmake_x64-windows/share/vcpkg-cmake/copyright
-- Performing post-build validation
Elapsed time to handle vcpkg-cmake:x64-windows: 35.9 ms
Installing 2/2 zlib:x64-windows...
Building zlib:x64-windows...
-- Using cached madler-zlib-v1.3.tar.gz.
-- Cleaning sources at D:/vcpkg/buildtrees/zlib/src/v1.3-8825ee792f.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source D:/vcpkg-downloads/madler-zlib-v1.3.tar.gz
-- Applying patch 0001-Prevent-invalid-inclusions-when-HAVE_-is-set-to-0.patch
-- Applying patch 0002-skip-building-examples.patch
-- Applying patch 0003-build-static-or-shared-not-both.patch
-- Applying patch 0004-android-and-mingw-fixes.patch
-- Using source at D:/vcpkg/buildtrees/zlib/src/v1.3-8825ee792f.clean
-- Configuring x64-windows
-- Building x64-windows-dbg
-- Building x64-windows-rel
-- Installing: D:/vcpkg/packages/zlib_x64-windows/share/zlib/vcpkg-cmake-wrapper.cmake
-- Fixing pkgconfig file: D:/vcpkg/packages/zlib_x64-windows/lib/pkgconfig/zlib.pc
-- Using cached mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst.
-- Using cached msys2-msys2-runtime-3.4.9-3-x86_64.pkg.tar.zst.
-- Using msys root at D:/vcpkg-downloads/tools/msys2/023cdb3ca06f77f2
-- Fixing pkgconfig file: D:/vcpkg/packages/zlib_x64-windows/debug/lib/pkgconfig/zlib.pc
-- Installing: D:/vcpkg/packages/zlib_x64-windows/share/zlib/copyright
-- Performing post-build validation
Elapsed time to handle zlib:x64-windows: 2.6 s
Total install time: 2.7 s
The package zlib is compatible with built-in CMake targets:

    find_package(ZLIB REQUIRED)
    target_link_libraries(main PRIVATE ZLIB::ZLIB)


D:\vcpkg>doskey /macros

D:\vcpkg>

== fixed

Copy link
Contributor

@ras0219-msft ras0219-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Publishing a partial review through commands.export.cpp

};

extern const WorkingDirectory default_working_directory;
extern const Environment default_environment;
struct RedirectedProcessLaunchSettings : ProcessLaunchSettings
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm concerned about inheriting here due to implicit conversion. E.g. cmd_execute(redirected) will do the wrong thing and it isn't terribly obvious.

Alternatively, cmd_execute() should take a different inherited type UnredirectedProcessLaunchSettings that doesn't have additional fields over the base.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

E.g. cmd_execute(redirected) will do the wrong thing and it isn't terribly obvious.

It isn't clear to me that it does the wrong thing. The only difference between these structs is that the settings in the redirected block are about redirection. The only reason they aren't the same struct is so that cmd_execute isn't claiming to respect settings it does not.

However, I also don't see any reason for it to ever convert so I changed it by duplicating the members.

ProcessLaunchSettings(const Command& cmd) : cmd(cmd) { }
ProcessLaunchSettings(Command&& cmd) : cmd(cmd) { }

ProcessLaunchSettings& string_arg(StringView s) &
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this fluent style isn't needed much -- most callers already (or could with small tweaks) use it to construct the Command before constructing the ProcessLaunchSettings.

I could see this making a lot of sense if there were common APIs which wanted to return an XYZProcessLaunchSettings that we'd like to chain off of.

src/vcpkg-test/system.process.cpp Outdated Show resolved Hide resolved
int code = 0;
auto res = cmd_execute_and_stream_lines(cmd, [&code](StringView line) {
auto res = cmd_execute_and_stream_lines(settings, [&code](StringView line) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does every call site need to change, compared to having an overload that takes a Command and uses the other defaults from RedirectedProcessLaunchSettings?

{
case Encoding::Utf8:
raw_cb = [&](char* buf, size_t bytes_read) {
std::replace(buf, buf + bytes_read, '\0', '?');
data_cb(StringView{buf, bytes_read});
if (settings.echo_in_debug == EchoInDebug::Show && Debug::g_debugging)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we echo to debug before processing? If processing takes a long time or crashes, that would ensure the debug output is available.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense to me. Note though that this is the same order in the existing product:

Strings::append(output, sv);
if (echo_in_debug == EchoInDebug::Show && Debug::g_debugging)
{
msg::write_unlocalized_text_to_stdout(Color::none, sv);

@@ -573,7 +576,7 @@ namespace
});
}

return cmd_execute_and_capture_output(cmdline).then([&](ExitCodeAndOutput&& res) -> ExpectedL<Unit> {
return cmd_execute_and_capture_output(settings).then([&](ExitCodeAndOutput&& res) -> ExpectedL<Unit> {
if (Debug::g_debugging)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this replaced by your "echo in debug" parameter?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it can be yes

@@ -1078,7 +1081,7 @@ namespace
return msg::format(msgRestoredPackagesFromGCS, msg::count = count, msg::elapsed = ElapsedTime(elapsed));
}

Command command() const { return m_tool; }
RedirectedProcessLaunchSettings command() const { return RedirectedProcessLaunchSettings{Command{m_tool}}; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should continue to return a Command. This function doesn't need to prescribe any of the other parameters of RedirectedProcessLaunchSettings like "echo in debug" or "working directory". If there are things that this should prescribe, that sounds like there's an intermediate type that should be returned instead.


const auto& abi_info = action.abi_info.value_or_exit(VCPKG_LINE_INFO);
auto env = paths.get_action_env(*abi_info.pre_build_info, abi_info.toolset.value_or_exit(VCPKG_LINE_INFO));
auto& env = settings.environment.emplace(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why emplace over assign?

const auto& env = paths.get_action_env(pre_build_info, toolset);
RedirectedProcessLaunchSettings settings;
settings.cmd = vcpkg::make_cmake_cmd(paths, paths.ports_cmake, std::move(cmake_args));
settings.environment = paths.get_action_env(pre_build_info, toolset);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why assign over emplace?

@@ -127,31 +127,31 @@ namespace vcpkg
}
}

auto env = get_modified_clean_environment(extra_env);
ProcessLaunchSettings settings;
auto& env = settings.environment.emplace(get_modified_clean_environment(extra_env));
if (!build_env_cmd.empty())
{
#if defined(_WIN32)
env = cmd_execute_and_capture_environment(build_env_cmd, env);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need adjusting to avoid doskey macros?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this one is fixable, since the intent here is for the user to be able to use the terminal inside, that means it has to be in their terminal window.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, this caller does not, because cmd_execute_and_capture_environment itself is what passes CreateNewConsole::Yes.

Copy link
Contributor

@ras0219-msft ras0219-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finished review.


auto env = default_environment;
auto& env = git_archive.environment.emplace();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is default_environment the same as a default constructed environment?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const WorkingDirectory default_working_directory;
const Environment default_environment;

Yes

# Conflicts:
#	src/vcpkg/base/system.process.cpp
#	src/vcpkg/commands.build.cpp
#	src/vcpkg/commands.export.cpp
#	src/vcpkg/commands.integrate.cpp
#	src/vcpkg/configure-environment.cpp
#	src/vcpkg/vcpkgpaths.cpp
@@ -127,34 +127,26 @@ namespace vcpkg
}
}

auto env = get_modified_clean_environment(extra_env);
#if defined(_WIN32)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that the old code on non-Win32 was:

        auto env = get_modified_clean_environment(extra_env);
        if (!build_env_cmd.empty())
        {
            Checks::msg_exit_with_message(VCPKG_LINE_INFO, msgEnvPlatformNotSupported); // <-- !!!
        }

        Command cmd("");
        Checks::msg_exit_with_message(VCPKG_LINE_INFO, msgEnvPlatformNotSupported); // <-- !!!

        if (!options.command_arguments.empty())
        {
            cmd.string_arg("/c").raw_arg(options.command_arguments[0]);
        }
        auto rc = cmd_execute(cmd, default_working_directory, env);
        Checks::exit_with_code(VCPKG_LINE_INFO, rc.value_or_exit(VCPKG_LINE_INFO));

.string_arg(".vcpkg-root"),
settings),
Tools::GIT)
.value_or_exit(VCPKG_LINE_INFO);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is a behavior change from previous -- if the git command failed, we would silently attempt to continue.

Agreed with the change, but highlighting it in case we see issues later.

@BillyONeal BillyONeal merged commit 48ba35a into microsoft:main Jan 10, 2024
5 checks passed
BillyONeal added a commit to BillyONeal/vcpkg-tool that referenced this pull request Feb 3, 2024
BillyONeal added a commit to BillyONeal/vcpkg-tool that referenced this pull request Feb 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants