[rollup:2021-07-06] Rollup PR (#18838)

* [rollup:2021-07-06 1/8] PR #18272 (@strega-nil)

[scripts-audit] vcpkg_from_*

* [rollup:2021-07-06 2/8] PR #18319 (@strega-nil)

[scripts-audit] add guidelines for cmake

* [rollup 2021-07-06 3/8] PR #18410 (@mheyman)

[vcpkg-cmake-config] documentation fix

* [rollup:2021-07-06 4/8] PR #18488 (@strega-nil)

[scripts-audit] vcpkg_execute_*

* [rollup:2021-07-06 5/8] PR #18517 (@strega-nil)

[scripts-audit] vcpkg_extract_source_archive

* [rollup:2021-07-06 6/8] PR #18674 (@NancyLi1013)

[vcpkg doc] Update examples

* [rollup:2021-07-06 7/8] PR #18695 (@JackBoosY)

[vcpkg] Update the minimum version of vcpkg

* [rollup:2021-07-06 8/8] PR #18758 (@ras0219-msft)

[vcpkg_from_git] Fix error if downloads folder does not exist

* build docs!

* fix bond:*-windows

* fix nmap

Co-authored-by: nicole mazzuca <mazzucan@outlook.com>
Co-authored-by: Michael Heyman <Michael.Heyman@jhuapl.edu>
Co-authored-by: NancyLi1013 <lirui09@beyondsoft.com>
Co-authored-by: JackBoosY <yuzaiyang@beyondsoft.com>
Co-authored-by: Robert Schumacher <ras0219@outlook.com>

docs/README.md

@@ -33,6 +33,7 @@ Vcpkg helps you manage C and C++ libraries on Windows, Linux and MacOS. This too
 - [Common CMake definitions](maintainers/vcpkg_common_definitions.md)
 - [Maintainer Guidelines](maintainers/maintainer-guide.md)
 - [Creating Registries](maintainers/registries.md)
+- [CMake Guidelines](maintainers/cmake-guidelines.md)
 
 ### [Vcpkg-Tool](https://github.com/microsoft/vcpkg-tool) Maintainer Help
 

docs/maintainers/cmake-guidelines.md

@@ -0,0 +1,127 @@
+# CMake Guidelines
+
+We expect that all CMake scripts that are either:
+
+- In the `scripts/` directory, or
+- In a `vcpkg-*` port
+
+should follow the guidelines laid out in this document.
+Existing scripts may not follow these guidelines yet;
+it is expected that we will continue to update old scripts
+to fall in line with these guidelines.
+
+These guidelines are intended to create stability in our scripts.
+We hope that they will make both forwards and backwards compatibility easier.
+
+## The Guidelines
+
+- Except for out-parameters, we always use `cmake_parse_arguments()`
+  rather than function parameters or referring to `${ARG<N>}`.
+  - This doesn't necessarily need to be followed for "script-local helper functions"
+    - In this case, positional parameters should be put in the function
+      declaration (rather than using `${ARG<N>}`),
+      and should be named according to local rules (i.e. `snake_case`).
+    - Exception: positional parameters that are optional should be
+      given a name via `set(argument_name "${ARG<N>}")`, after checking `ARGC`.
+  - Out-parameters should be the first parameter to a function. Example:
+  ```cmake
+  function(format out_var)
+    cmake_parse_arguments(PARSE_ARGV 1 "arg" ...)
+    # ... set(buffer "output")
+    set("${out_var}" "${buffer}" PARENT_SCOPE)
+  endfunction()
+  ```
+- There are no unparsed or unused arguments.
+  Always check for `ARGN` or `arg_UNPARSED_ARGUMENTS`.
+  `FATAL_ERROR` when possible, `WARNING` if necessary for backwards compatibility.
+- All `cmake_parse_arguments` must use `PARSE_ARGV`.
+- All `foreach` loops must use `IN LISTS` and `IN ITEMS`.
+- The variables `${ARGV}` and `${ARGN}` are unreferenced,
+  except in helpful messages to the user.
+  - (i.e., `message(FATAL_ERROR "blah was passed extra arguments: ${ARGN}")`)
+- We always use functions, not macros or top level code.
+  - Exception: "script-local helper macros". It is sometimes helpful to define a small macro.
+    This should be done sparingly, and functions should be preferred.
+  - Exception: `vcpkg.cmake`'s `find_package`.
+- Scripts in the scripts tree should not be expected to need observable changes
+  as part of normal operation.
+  - Example violation: `vcpkg_acquire_msys()` has hard-coded packages and versions that need updating over time due to the MSYS project dropping old packages.
+  - Example exception: `vcpkg_from_sourceforge()` has a list of mirrors which needs maintenance but does not have an observable behavior impact on the callers.
+- All variable expansions are in quotes `""`,
+  except those which are intended to be passed as multiple arguments.
+  - Example:
+  ```cmake
+  set(working_directory "")
+  if(DEFINED arg_WORKING_DIRECTORY)
+    set(working_directory "WORKING_DIRECTORY" "${arg_WORKING_DIRECTORY}")
+  endif()
+  # calls do_the_thing() if NOT DEFINED arg_WORKING_DIRECTORY,
+  # else calls do_the_thing(WORKING_DIRECTORY "${arg_WORKING_DIRECTORY}")
+  do_the_thing(${working_directory})
+  ```
+- There are no "pointer" or "in-out" parameters
+  (where a user passes a variable name rather than the contents),
+  except for simple out-parameters.
+- Variables are not assumed to be empty.
+  If the variable is intended to be used locally,
+  it must be explicitly initialized to empty with `set(foo "")`.
+- All variables expected to be inherited from the parent scope across an API boundary (i.e. not a file-local function) should be documented. Note that all variables mentioned in triplets.md are considered documented.
+- Out parameters are only set in `PARENT_SCOPE` and are never read.
+  See also the helper `z_vcpkg_forward_output_variable()` to forward out parameters through a function scope.
+- `CACHE` variables are used only for global variables which are shared internally among strongly coupled
+  functions and for internal state within a single function to avoid duplicating work.
+  These should be used extremely sparingly and should use the `Z_VCPKG_` prefix to avoid
+  colliding with any local variables that would be defined by any other code.
+  - Examples:
+    - `vcpkg_cmake_configure`'s `Z_VCPKG_CMAKE_GENERATOR`
+    - `z_vcpkg_get_cmake_vars`'s `Z_VCPKG_GET_CMAKE_VARS_FILE`
+- `include()`s are only allowed in `ports.cmake` or `vcpkg-port-config.cmake`.
+- `foreach(RANGE)`'s arguments _must always be_ natural numbers,
+  and `<start>` _must always be_ less than or equal to `<stop>`.
+  - This must be checked by something like:
+  ```cmake
+  if(start LESS_EQUAL end)
+    foreach(RANGE start end)
+      ...
+    endforeach()
+  endif()
+  ```
+- All port-based scripts must use `include_guard(GLOBAL)`
+  to avoid being included multiple times.
+- `set(var)` should not be used. Use `unset(var)` to unset a variable,
+  and `set(var "")` to set it to the empty value. _Note: this works for use as a list and as a string_
+
+### CMake Versions to Require
+
+- All CMake scripts, except for `vcpkg.cmake`,
+  may assume the version of CMake that is present in the
+  `cmake_minimum_required` of `ports.cmake`.
+  - This `cmake_minimum_required` should be bumped every time a new version
+    of CMake is added to `vcpkgTools.xml`, as should the
+    `cmake_minimum_required` in all of the helper `CMakeLists.txt` files.
+- `vcpkg.cmake` must assume a version of CMake back to 3.1 in general
+  - Specific functions and options may assume a greater CMake version;
+    if they do, make sure to comment that function or option
+    with the required CMake version.
+
+
+### Changing Existing Functions
+
+- Never remove arguments in non-internal functions;
+  if they should no longer do anything, just take them as normal and warn on use.
+- Never add a new mandatory argument.
+
+### Naming Variables
+
+- `cmake_parse_arguments`: set prefix to `"arg"`
+- Local variables are named with `snake_case`
+- Internal global variable names are prefixed with `Z_VCPKG_`.
+- External experimental global variable names are prefixed with `X_VCPKG_`.
+
+- Internal functions are prefixed with `z_vcpkg_`
+  - Functions which are internal to a single function (i.e., helper functions)
+    are named `[z_]<func>_<name>`, where `<func>` is the name of the function they are
+    a helper to, and `<name>` is what the helper function does.
+    - `z_` should be added to the front if `<func>` doesn't have a `z_`,
+      but don't name a helper function `z_z_foo_bar`.
+- Public global variables are named `VCPKG_`.

docs/maintainers/internal/z_vcpkg_forward_output_variable.md

@@ -0,0 +1,38 @@
+# z_vcpkg_forward_output_variable
+
+The latest version of this document lives in the [vcpkg repo](https://github.com/Microsoft/vcpkg/blob/master/docs/).
+
+This macro helps with forwarding values from inner function calls,
+through a local function scope, into pointer out parameters.
+
+```cmake
+z_vcpkg_forward_output_variable(ptr_to_parent_var var_to_forward)
+```
+
+is equivalent to
+
+```cmake
+if(DEFINED ptr_to_parent_var)
+    if(DEFINED value_var)
+        set("${ptr_to_parent_var}" "${value_var}" PARENT_SCOPE)
+    else()
+        unset("${ptr_to_parent_var}" PARENT_SCOPE)
+    endif()
+endif()
+```
+
+Take note that the first argument should be a local variable that has a value of the parent variable name.
+Most commonly, this local is the result of a pointer-out parameter to a function.
+If the variable in the first parameter is not defined, this function does nothing,
+simplifying functions with optional out parameters.
+Most commonly, this should be used in cases like:
+
+```cmake
+function(my_function out_var)
+    file(SHA512 "somefile.txt" local_var)
+    z_vcpkg_forward_output_variable(out_var local_var)
+endfunction()
+```
+
+## Source
+[scripts/cmake/z\_vcpkg\_forward\_output\_variable.cmake](https://github.com/Microsoft/vcpkg/blob/master/scripts/cmake/z_vcpkg_forward_output_variable.cmake)

docs/maintainers/portfile-functions.md

@@ -58,6 +58,7 @@
 
 - [vcpkg\_internal\_get\_cmake\_vars](internal/vcpkg_internal_get_cmake_vars.md)
 - [z\_vcpkg\_apply\_patches](internal/z_vcpkg_apply_patches.md)
+- [z\_vcpkg\_forward\_output\_variable](internal/z_vcpkg_forward_output_variable.md)
 - [z\_vcpkg\_function\_arguments](internal/z_vcpkg_function_arguments.md)
 - [z\_vcpkg\_prettify\_command\_line](internal/z_vcpkg_prettify_command_line.md)
 

docs/maintainers/ports/vcpkg-cmake-config/vcpkg_cmake_config_fixup.md

@@ -10,7 +10,7 @@ Additionally corrects common issues with targets, such as absolute paths and inc
 vcpkg_cmake_config_fixup(
     [PACKAGE_NAME <name>]
     [CONFIG_PATH <config-directory>]
-    [DO_NOT_DELETE_CONFIG_PATH_PARENT]
+    [DO_NOT_DELETE_PARENT_CONFIG_PATH]
     [NO_PREFIX_CORRECTION]
 )
 ```

docs/maintainers/vcpkg_clean_msbuild.md

@@ -11,7 +11,7 @@ vcpkg_clean_msbuild()
 
 ## Examples
 
-* [xalan-c](https://github.com/Microsoft/vcpkg/blob/master/ports/xalan-c/portfile.cmake)
+* [python3](https://github.com/Microsoft/vcpkg/blob/master/ports/python3/portfile.cmake)
 
 ## Source
 [scripts/cmake/vcpkg\_clean\_msbuild.cmake](https://github.com/Microsoft/vcpkg/blob/master/scripts/cmake/vcpkg_clean_msbuild.cmake)

docs/maintainers/vcpkg_download_distfile.md

@@ -13,6 +13,7 @@ vcpkg_download_distfile(
     URLS <http://mainUrl> <http://mirror1>...
     FILENAME <output.zip>
     SHA512 <5981de...>
+    [ALWAYS_REDOWNLOAD]
 )
 ```
 ## Parameters
@@ -38,6 +39,11 @@ Skip SHA512 hash check for file.
 
 This switch is only valid when building with the `--head` command line flag.
 
+### ALWAYS_REDOWNLOAD
+Avoid caching; this is a REST call or otherwise unstable.
+
+Requires `SKIP_SHA512`.
+
 ### HEADERS
 A list of headers to append to the download request. This can be used for authentication during a download.
 

docs/maintainers/vcpkg_execute_in_download_mode.md

@@ -7,28 +7,13 @@ Execute a process even in download mode.
 ## Usage
 ```cmake
 vcpkg_execute_in_download_mode(
-    COMMAND <cmd> [<arguments>]
-    [WORKING_DIRECTORY <dir>]
-    [TIMEOUT <seconds>]
-    [RESULT_VARIABLE <variable>]
-    [OUTPUT_VARIABLE <variable>]
-    [ERROR_VARIABLE <variable>]
-    [INPUT_FILE <file>]
-    [OUTPUT_FILE <file>]
-    [ERROR_FILE <file>]
-    [OUTPUT_QUIET]
-    [ERROR_QUIET]
-    [OUTPUT_STRIP_TRAILING_WHITESPACE]
-    [ERROR_STRIP_TRAILING_WHITESPACE]
-    [ENCODING <name>]
+    ...
 )
 ```
 
-The signature of this function is identical to `execute_process()` except that
-it only accepts one COMMAND argument, i.e., does not support chaining multiple
-commands with pipes.
+The signature of this function is identical to `execute_process()`.
 
-See [`execute_process()`] for a detailed description of the parameters.
+See [`execute_process()`] for more details.
 
 [`execute_process()`]: https://cmake.org/cmake/help/latest/command/execute_process.html
 

docs/maintainers/vcpkg_execute_required_process_repeat.md

@@ -7,10 +7,11 @@ Execute a process until the command succeeds, or until the COUNT is reached.
 ## Usage
 ```cmake
 vcpkg_execute_required_process_repeat(
-    COUNT <num>
     COMMAND <cmd> [<arguments>]
+    COUNT <num>
     WORKING_DIRECTORY <directory>
     LOGNAME <name>
+    [ALLOW_IN_DOWNLOAD_MODE]
 )
 ```
 

docs/maintainers/vcpkg_extract_source_archive.md

@@ -2,27 +2,74 @@
 
 The latest version of this document lives in the [vcpkg repo](https://github.com/Microsoft/vcpkg/blob/master/docs/maintainers/vcpkg_extract_source_archive.md).
 
-Extract an archive into the source directory. Deprecated in favor of [`vcpkg_extract_source_archive_ex`](vcpkg_extract_source_archive_ex.md).
+Extract an archive into the source directory.
 
 ## Usage
+There are two "overloads" of this function. The first is deprecated:
+
 ```cmake
-vcpkg_extract_source_archive(
-    <${ARCHIVE}> [<${TARGET_DIRECTORY}>]
+vcpkg_extract_source_archive(<${ARCHIVE}> [<${TARGET_DIRECTORY}>])
+```
+
+This overload should not be used.
+
+The latter is suggested to use for all future `vcpkg_extract_source_archive`s.
+
+```cmake
+vcpkg_extract_source_archive(<out-var>
+    ARCHIVE <path>
+    [NO_REMOVE_ONE_LEVEL]
+    [PATCHES <patch>...]
+    [SOURCE_BASE <base>]
+    [BASE_DIRECTORY <relative-path> | WORKING_DIRECTORY <absolute-path>]
 )
 ```
-## Parameters
-### ARCHIVE
-The full path to the archive to be extracted.
 
-This is usually obtained from calling [`vcpkg_download_distfile`](vcpkg_download_distfile.md).
+`vcpkg_extract_source_archive` takes an archive and extracts it.
+It replaces existing uses of `vcpkg_extract_source_archive_ex`.
+The simplest use of it is:
+
+```cmake
+vcpkg_download_distfile(archive ...)
+vcpkg_extract_source_archive(source_path ARCHIVE "${archive}")
+```
+
+The general expectation is that an archives are laid out with a base directory,
+and all the actual files underneath that directory; in other words, if you
+extract the archive, you'll get something that looks like:
+
+```
+zlib-1.2.11/
+    doc/
+        ...
+    examples/
+        ...
+    ChangeLog
+    CMakeLists.txt
+    README
+    zlib.h
+    ...
+```
 
-### TARGET_DIRECTORY
-If specified, the archive will be extracted into the target directory instead of `${CURRENT_BUILDTREES_DIR}/src/`.
+`vcpkg_extract_source_archive` automatically removes this directory,
+and gives you the items under it directly. However, this only works
+when there is exactly one item in the top level of an archive.
+Otherwise, you'll have to pass the `NO_REMOVE_ONE_LEVEL` argument to
+prevent `vcpkg_extract_source_archive` from performing this transformation.
 
-This can be used to mimic git submodules, by extracting into a subdirectory of another archive.
+If the source needs to be patched in some way, the `PATCHES` argument
+allows one to do this, just like other `vcpkg_from_*` functions.
 
-## Notes
-This command will also create a tracking file named <FILENAME>.extracted in the TARGET_DIRECTORY. This file, when present, will suppress the extraction of the archive.
+`vcpkg_extract_source_archive` extracts the files to
+`${CURRENT_BUILDTREES_DIR}/<base-directory>/<source-base>-<hash>.clean`.
+When in editable mode, no `.clean` is appended,
+to allow for a user to modify the sources.
+`base-directory` defaults to `src`,
+and `source-base` defaults to the stem of `<archive>`.
+You can change these via the `BASE_DIRECTORY` and `SOURCE_BASE` arguments
+respectively.
+If you need to extract to a location that is not based in `CURRENT_BUILDTREES_DIR`,
+you can use the `WORKING_DIRECTORY` argument to do the same.
 
 ## Examples