Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs] Add maintainer guidelines #6871

Merged
1 change: 1 addition & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Vcpkg helps you manage C and C++ libraries on Windows, Linux and MacOS. This too

- [Control files](maintainers/control-files.md)
- [Portfile functions](maintainers/portfile-functions.md)
- [Maintainer Guidelines](maintainers/maintainer-guide.md)

### Specifications

Expand Down
17 changes: 14 additions & 3 deletions docs/maintainers/control-files.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,16 +43,27 @@ The port version.

This field is an alphanumeric string that may also contain `.`, `_`, or `-`. No attempt at ordering versions is made; all versions are treated as bit strings and are only evaluated for equality.

By convention, if a portfile is modified without incrementing the "upstream" version, a `-#` is appended to create a unique version string.
For tagged-release ports, we follow the following convention:

Some projects do not have named releases. In these cases use the date of the version do not have labeled releases, in these cases use the date of the last commit in `YYYY-MM-DD` format. See the `abseil` port as an example.
1. If the port follows a scheme like `va.b.c`, we remove the leading `v`. In this case, it becomes `a.b.c`.
2. If the port includes its own name in the version like `curl-7_65_1`, we remove the leading name: `7_65_1`
3. If the port has been modified, we append a `-N` to distinguish the versions: `1.2.1-4`

For rolling-release ports, we use the date that the _commit was accessed by you_, formatted as `YYYY-MM-DD`. Stated another way: if someone had a time machine and went to that date, they would see this commit as the latest master.

For example, given:
1. The latest commit was made on 2019-04-19
2. The current version string is `2019-02-14-1`
3. Today's date is 2019-06-01.

Then if you update the source version today, you should give it version `2019-06-01`. If you need to make a change which doesn't adjust the source version, you should give it version `2019-02-14-2`.

Example:
```no-highlight
Version: 1.0.5-2
```
```no-highlight
Version: 2019-3-21
Version: 2019-03-21
```

#### Description
Expand Down
178 changes: 178 additions & 0 deletions docs/maintainers/maintainer-guide.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
# Maintainer Guidelines and Policies

This document lists a set of policies that you should apply when adding or updating a port recipe. It is intended to serve the role of [Debian's Policy Manual](https://www.debian.org/doc/debian-policy/), [Homebrew's Maintainer Guidelines](https://docs.brew.sh/Maintainer-Guidelines), and [Homebrew's Formula Cookbook](https://docs.brew.sh/Formula-Cookbook).

## PR Structure

### Make separate Pull Requests per port

Whenever possible, separate changes into multiple PR's. This makes them significantly easier to review and prevents issues with one set of changes from holding up every other change.

### Avoid trivial changes in untouched files

For example, avoid reformatting or renaming variables in portfiles that otherwise have no reason to be modified for the issue at hand. However, if you need to modify the file for the primary purpose of the PR (updating the library), then obviously beneficial changes like fixing typos are appreciated!

### Check names against other repositories

A good service to check many at once is [Repology](https://repology.org/). If the library you are adding could be confused with another one, consider renaming to make it clear.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A link to the control-file.md documentation that already talks about this would be useful.


### Use GitHub Draft PRs

GitHub Draft PRs are a great way to get CI or human feedback on work that isn't yet ready to merge. Most new PRs should be opened as drafts and converted to normal PRs once the CI passes.

More information about GitHub Draft PRs: https://github.blog/2019-02-14-introducing-draft-pull-requests/

## Portfiles
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you perhaps add notes on the recommended way to do OS checks?

In my past few submitted PRs, I have received suggestions to use:

Some formal guidelines on the matter would be great

Copy link
Contributor

@Neumann-A Neumann-A Jun 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that this is required. You simply have to learn to differentiate between HOST and TARGET correctly and wehter you need to check the HOST or the TARGET in the cases you are looking at. Also you need to differentiate between the circumstance if you are in cmakes script or configure/generator mode. The portfile are consumed in script mode while the CMakeLists.txt are consumed via normal configure/generator mode. In #6856 you were patching a CMakeLists.txt where you wanted to check the TARGET. Here CMAKE_SYSTEM_NAME is set correctly and can be and should be used. It is not set in script mode. In #6813 you want to figure out the correct build command on the HOST machine in script mode so you need to use CMAKE_HOST_(WIN32|UNIX|APPLE).
So in the end you just need to read/learn what certain variables in cmake mean and when to use them correctly and that has nothing to do with VCPKG.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great explanation and I think it would also be good to include this in the document for future maintainers!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added an ascii art diagram to illustrate the difference. Let me know if it helps!


### Avoid deprecated helper functions

At this time, the following helpers are deprecated:

1. `vcpkg_extract_archive()` should be replaced by `vcpkg_extract_archive_ex()`
2. `vcpkg_apply_patches()` should be replaced by the `PATCHES` arguments to the "extract" helpers (e.g. `vcpkg_from_github()`)
3. `vcpkg_build_msbuild()` should be replaced by `vcpkg_install_msbuild()`

### Avoid excessive comments in portfiles

Ideally, portfiles should be short, simple, and as declarative as possible. Remove any helper comments introduced by the `create` command before submitting a PR.

## Build Techniques

### Do not use vendored dependencies

Do not use embedded copies of libraries. All dependencies should be split out and packaged separately so they can be updated and maintained.

### Prefer using CMake

When multiple buildsystems are available, prefer using CMake. Additionally, when appropriate, it can be easier and more maintainable to rewrite alternative buildsystems into CMake using `file(GLOB)` directives.

Examples: [abseil](../../ports/abseil/portfile.cmake)

### Choose either static or shared binaries

By default, `vcpkg_configure_cmake()` will pass in the appropriate setting for `BUILD_SHARED_LIBS`, however for libraries that don't respect that variable, you can switch on `VCPKG_LIBRARY_LINKAGE`:

```cmake
string(COMPARE EQUAL "${VCPKG_LIBRARY_LINKAGE}" "static" KEYSTONE_BUILD_STATIC)
string(COMPARE EQUAL "${VCPKG_LIBRARY_LINKAGE}" "dynamic" KEYSTONE_BUILD_SHARED)

vcpkg_configure_cmake(
SOURCE_PATH ${SOURCE_PATH}
PREFER_NINJA
OPTIONS
-DKEYSTONE_BUILD_STATIC=${KEYSTONE_BUILD_STATIC}
-DKEYSTONE_BUILD_SHARED=${KEYSTONE_BUILD_SHARED}
)
```

### When defining features, explicitly control dependencies

When defining a feature that captures an optional dependency, ensure that the dependency will not be used accidentally when the feature is not explicitly enabled. For example:

```cmake
set(CMAKE_DISABLE_FIND_PACKAGE_ZLIB ON)
if("zlib" IN_LIST FEATURES)
set(CMAKE_DISABLE_FIND_PACKAGE_ZLIB OFF)
endif()

vcpkg_configure_cmake(
SOURCE_PATH ${SOURCE_PATH}
PREFER_NINJA
OPTIONS
-DCMAKE_DISABLE_FIND_PACKAGE_ZLIB=${CMAKE_DISABLE_FIND_PACKAGE_ZLIB}
)
```

Note that `ZLIB` in the above is case-sensitive. See the [cmake documentation](https://cmake.org/cmake/help/v3.15/variable/CMAKE_DISABLE_FIND_PACKAGE_PackageName.html) for more details.

## Versioning

### Follow common conventions for the `Version:` field

See our [CONTROL files document](control-files.md#version) for a full explanation of our conventions.

### Update the `Version:` field in the `CONTROL` file of any modified ports

Vcpkg uses this field to determine whether a given port is out-of-date and should be changed whenever the port's behavior changes.

Our convention for this field is to append a `-N` to the upstream version when changes need to be made.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#3707

This -N approach could be confusing since it is not able to tell if the suffix comes from the upstream or vcpkg port. I recommended to use +N conventions from SemVer.

Copy link
Contributor Author

@ras0219-msft ras0219-msft Jun 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is interesting, however:

Build metadata SHOULD be ignored when determining version precedence. Thus two versions that differ only in the build metadata, have the same precedence.

If we are using SemVer as the driving design, this would imply that +2 is not to be considered "better" than +1 -- which we do intend.

Based on my experience will all the current libraries, the number of cases where a trailing -N is potentially confusing has been extremely small. The vast majority of libraries prefer to use .s or _s for numeric/semver-style versions or tend to have alphabetic terms when trailing (-rc1, -beta, -apr2018, etc). It also helps that we avoid shipping prerelease versions as much as possible :).


For Example:

- Zlib's package version is currently `1.2.1`.
- You've discovered that the wrong copyright file has been deployed, and fixed that in the portfile.
- You should update the `Version:` field in the control file to `1.2.1-1`.

See our [CONTROL files document](control-files.md#version) for a full explanation of our conventions.

## Patching

### Prefer options over patching

It is preferable to set options in a call to `vcpkg_configure_xyz()` over patching the settings directly.

Common options that allow avoiding patching:
1. [MSBUILD] `<PropertyGroup>` settings inside the project file can be overridden via `/p:` parameters
2. [CMAKE] Calls to `find_package(XYz)` in CMake scripts can be disabled via [`-DCMAKE_DISABLE_FIND_PACKAGE_XYz=ON`](https://cmake.org/cmake/help/v3.15/variable/CMAKE_DISABLE_FIND_PACKAGE_PackageName.html)
3. [CMAKE] Cache variables (declared as `set(VAR "value" CACHE STRING "Documentation")` or `option(VAR "Documentation" "Default Value")`) can be overriden by just passing them in on the command line as `-DVAR:STRING=Foo`. One notable exception is if the `FORCE` parameter is passed to `set()`. See also the [CMake `set` documentation](https://cmake.org/cmake/help/v3.15/command/set.html)

### Minimize patches

When making changes to a library, strive to minimize the final diff. This means you should _not_ reformat the upstream source code when making changes that affect a region. Also, when disabling a conditional, it is better to add a `AND FALSE` or `&& 0` to the condition than to delete every line of the conditional.

This helps to keep the size of the vcpkg repository down as well as improves the likelihood that the patch will apply to future code versions.

### Do not implement features in patches

The purpose of patching in vcpkg is to enable compatibility with compilers, libraries, and platforms. It is not to implement new features in lieu of following proper Open Source procedure (submitting an Issue/PR/etc).

## Do not build tests/docs/examples by default

When submitting a new port, check for any options like `BUILD_TESTS` or `WITH_TESTS` or `POCO_ENABLE_SAMPLES` and ensure the additional binaries are disabled. This minimizes build times and dependencies for the average user.

Optionally, you can add a `test` feature which enables building the tests, however this should not be in the `Default-Features` list.

## Enable existing users of the library to switch to vcpkg

### Do not add `CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS`

Unless the author of the library is already using it, we should not use this CMake functionality because it interacts poorly with C++ templates and breaks certain compiler features. Libraries that don't provide a .def file and do not use __declspec() declarations simply do not support shared builds for Windows and should be marked as such with `vcpkg_check_linkage(ONLY_STATIC_LIBRARY)`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it Ok to do this for C libraries?


### Do not rename binaries outside the names given by upstream

This means that if the upstream library has different names in release and debug (libx versus libxd), then the debug library should not be renamed to `libx`. Vice versa, if the upstream library has the same name in release and debug, we should not introduce a new name.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing:
If CMakes find_package expects a different name for the debug libraries and the library does not provide this by default use CMAKE_DEBUG_POSTFIX in vcpkg_configure_cmake to "rename" the library. This is required so that downstream projects can correctly find the debug configuration of the library using CMake. Do not set CMAKE_DEBUG_POSTFIX if CMakes find_package does not require it.

Copy link
Contributor Author

@ras0219-msft ras0219-msft Jun 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that in general, we should not be doing this.

The reasoning is twofold:

  1. CMake (i.e. Kitware) doesn't have the authority to tell open source projects what their binaries should be called. Just because someone at Kitware merged a change to a FindXYZ script does not mean that zlib/expat/boost/openssl should now globally have their binaries renamed without the consent of the upstream maintainers.
  2. Existing users of the libraries who aren't using CMake will have existing expectations. For example, Makefiles will often just use -lfoo because that's what the binaries are called in all existing package managers and systems. We want to enable these users to switch to vcpkg seamlessly, which means we need to use the same names that they are used to using -- renaming the debug name to libfood.a would break them.

When using single configuration generators with CMake, this should be a non-issue -- find_library(the_one_name) will find the right library.

However, we do need to handle the case of Multi-configuration generators. I believe the right way to proceed for this case will be to add vcpkg-cmake-wrapper.cmake helpers which fixup the built-in cmake FindXYZ modules by (for instance) modifying XYZ_LIBRARIES to include debug;debug/lib/foo;optimized;lib/foo. There is a finite number of FindXYZ scripts in CMake and they've publicly stated that they do not want to add any more, so I'm not concerned with the costs of this approach ballooning out of control.

This appears (to me) to be the ONLY way to:

  1. Work with existing non-CMake build scripts we don't control
  2. Work with CMake multi-config generators (like Visual Studio) when using built-in FindXYZ modules
  3. Avoid alienating upstream authors by introducing names for their binaries that they have not "blessed"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ras0219-msft:

CMake (i.e. Kitware) doesn't have the authority to tell open source projects what their binaries should be called. Just because someone at Kitware merged a change to a FindXYZ script does not mean that zlib/expat/boost/openssl should now globally have

That is in general a correct stance with one important exception: If the project uses CMake the maintainers of the project implicitly agree to renaming the library because passing CMAKE_<config>_POSTFIX is a valid - and i would even say, common - option for the users to pass to the configure step which the libraries authors have to expect. Unless they explicitly unset CMAKE_<config>_POSTFIX they agreed to such a renaming.
To apply this to the mentioned examples: zlib, expat are CMake based projects so no issue there. boost probably has a build option to define library suffixes. OpenSSL is somewhat special since the build does not generate the suffixed version but "officially" distributed binaries have the additional suffix. So do with that whatever you like; I don't mind to much in this particular case because it is just a mess. They probably just do a library renaming like I have seen in the openblas CMakeLists.txt for shared build but I won't investigate that further.

CMake (i.e. Kitware) doesn't have the authority to tell open source projects what their binaries should be called

Does VCPKG have that authority? e.g.: #6698

Existing users of the libraries who aren't using CMake will have existing expectations. For example, Makefiles will often just use -lfoo because that's what the binaries are called in all existing package managers and systems. We want to enable these users to switch to vcpkg seamlessly, which means we need to use the same names that they are used to using -- renaming the debug name to libfood.a would break them.

I think your arguments is based on a wrong assumption here. Do package managers and system even supply debug libraries? Do Makefiles care about external debug libraries? Did the writer of the Makefile even consider external debug libraries? I think the answer to those questions are probably all "No" because I read somewhere that on linux debug/release are not so well defined as on Windows due to the iterator debug levels. Due to that VCPKG should not care about them because we still deliver the release libraries and it should still work for external users. Your main concern here seems to be the ports within VCPKG which use Makefiles instead of CMake. To patch these Makefiles is the work VCPKG has to do internally.
Furthermore there has been examples/requests where users copied both configurations into the same folder and here the different naming is of course mandatory. Without the renaming you would brake those users. (also I agree that this is probably a bad reason and different configurations should go into different folders)
To further add to this: Adding a debug suffix is the only way for CMake to pickup the debug libraries with 100% certainty due to how find_library works. It also avoids the silly case of doing a dropin replacement of debug/release shared libraries. So instead of removing debug suffixes VCPKG should probably rather add them (but that is not something I am willing to actively promote).

However, we do need to handle the case of Multi-configuration generators. I believe the right way to proceed for this case will be to add vcpkg-cmake-wrapper.cmake helpers which fixup the built-in cmake FindXYZ modules by (for instance) modifying XYZ_LIBRARIES to include debug;debug/lib/foo;optimized;lib/foo. There is a finite number of FindXYZ scripts in CMake and they've publicly stated that they do not want to add any more, so I'm not concerned with the costs of this approach ballooning out of control.

From my experience with PR 5543. debug;debug/lib/foo;optimized;lib/foo does not work everywhere. Generator expressions do not work everywhere. finite number around 150 in CMake and probably a factor 3 to 10 more in all ports either derived from the CMake ones or customized to satisfy the libraries demands. So i would at least expect 10-30% of all ports in VCPKG require a wrapper. vcpkg-cmake-wrapper.cmake can we please rename that to vcpkg_find_xyz.cmake because that is what it is actually doing and add a function vcpkg_install_find_wrapper.cmake to install them.

TL;DR: My opinion

  • If the project is using CMake, renaming via configure options is ok because library authors implicitly agreed due to the usage of CMake configure options.
  • We should not care about external Makefiles not using renamed debug libraries because they probably did not care about the debug case anyway. VCPKG internally used Makefiles must be patched to account for the debug case correctly.
  • We need to figure out a way for Multi-configuration generators to work correctly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First, given that a library "officially" uses the same name in release and debug, we can expect that FindXYZ scripts will be written with a single find_library() call.

As I see it, our options are:

  1. Do nothing -> single configuration works perfectly. Multi config does not work. External non-cmake buildsystems work (because we didn't change anything compared to "raw" upstream).
  2. Add debug suffix -> All existing FindXYZ scripts are broken in all configurations, because they find the release library in debug mode. I think that attempting to hack up find_library() to "automatically" find the new names is too problematic.
  3. Add debug suffix AND a vcpkg-cmake-wrapper -> Single config works. Multi config works. Non-CMake buildsystems are all broken because none of them are designed to expect the new name.
  4. Add vcpkg-cmake-wrapper -> single configuration still works perfectly (no changes). Multi config also now works. Existing non-cmake buildsystems work as well (no changes).

Given this, it seems like only adding a vcpkg-cmake-wrapper is the obviously correct thing to do. I'll also note that Multi-config generators, while we do care about making them work, are going to become less and less common since VS now has built-in cmake support (which runs in Single Configuration mode :)).

I'll also note that within Vcpkg we always build in single-configuration style (even if using MSBuild!), so as long as we just don't add a suffix, everything will tend to work fine (including existing FindXYZ scripts embedded in other libraries). In this case, we actually tend to hit the opposite problem where libraries might add a suffix in their official binaries but FindXYZ scripts aren't written to handle them correctly.

option for the users to pass to the configure step which the libraries authors have to expect.

I have not seen it to be conventional to pass this value in with other package managers (apt, brew, etc). If you have evidence to the contrary, I'd love to see it!

Just because CMake has a crazy option which can mess with your library, doesn't mean that you've implicitly approved of it.

Does VCPKG have that authority?

At the end of the day, we do have to ship something, and we have the power to patch buildsystems to decide what the binaries are called. My primary point in the guidelines is that we should seek to exercise this power as little as possible while achieving compatibility. As much as possible, we should use the names that the upstream buildsystem(s) use. The one common exception is where we do choose to exercise this is static vs dynamic -- I'll omit talking more about that case for brevity.

boost probably has a build option to define library suffixes

Boost specifically has an enumeration of valid library naming schemes -- we use one of them (with asterisks).

Do Makefiles care about external debug libraries? Did the writer of the Makefile even consider external debug libraries?

You're right that Makefiles on Linux may not commonly handle this, but other cases will. Qt/QMake does, MSBuild does, and NMake Makefiles on Windows will (for exactly the reason you stated).

Your main concern here seems to be the ports within VCPKG which use Makefiles instead of CMake.

No, I'm actually primarily concerned with users outside vcpkg, using buildsystems that we can't know or fix. Within vcpkg's curated set, I'm certain that we can patch our way to victory no matter what strategy we choose here -- however, we can't patch external software like firefox (for example). Therefore, we should choose names that are maximally likely to be compatible with existing practice. There's an obvious choice for that: it's whatever the current buildsystems produce by default with no interference/non-default options.

examples/requests where users copied both configurations into the same folder

Links? Vcpkg must be able to handle libraries that use the same name for release and debug (in the general case). Therefore, we always use separate folders.

Adding a debug suffix is the only way for CMake to pickup the debug libraries with 100% certainty due to how find_library works.

That only applies if you are using find_library(), and only if you are using multi-config generators. If you are using -config.cmake files, everything works fine. If you are using a single configuration, then having debug and release named the same actually improves things, because it becomes impossible to accidentally find the wrong library (since we list debug before release).

debug;debug/lib/foo;optimized;lib/foo does not work everywhere.

This is the outcome for FindXYZ.cmake scripts that do handle both libraries at once; if that doesn't work, we have bigger problems!

vcpkg-cmake-wrapper.cmake can we please rename that to vcpkg_find_xyz.cmake

Not a bad idea, I agree this would be clearer. Though I think the primary problem with vcpkg-cmake-wrapper is simply that it's largely undocumented -- thanks for bringing this up and I should add it to this doc!


Important caveat:
- Static and shared variants often should be renamed to a common scheme. This enables consumers to use a common name and be ignorant of the downstream linkage. This is safe because we only make one at a time available.

Note that if a library generates CMake integration files (`foo-config.cmake`), renaming must be done through patching the CMake build itself instead of simply calling `file(RENAME)` on the output archives/LIBs.

Finally, DLL files on Windows should never be renamed post-build because it breaks the generated LIBs.

## Useful implementation notes

### Portfiles are run in Script Mode

While `portfile.cmake`'s and `CMakeLists.txt`'s share a common syntax and core CMake language constructs, portfiles run in "Script Mode", whereas `CMakeLists.txt` files run in "Build Mode" (unofficial term). The most important difference between these two modes is that "Script Mode" does not have a concept of "Target" -- any behaviors that depend on the "target" machine (`CMAKE_CXX_COMPILER`, `CMAKE_EXECUTABLE_SUFFIX`, `CMAKE_SYSTEM_NAME`, etc) will not be correct.

Portfiles have direct access to variables set in the triplet file, but `CMakeLists.txt`s do not (though there is often a translation that happens -- `VCPKG_LIBRARY_LINKAGE` versus `BUILD_SHARED_LIBS`).

Finally, portfiles and CMake builds invoked by portfiles are run in different processes. Conceptually:

```no-highlight
+----------------------------+ +------------------------------------+
| CMake.exe | | CMake.exe |
+----------------------------+ +------------------------------------+
| Triplet file | ====> | Toolchain file |
| (x64-windows.cmake) | | (scripts/buildsystems/vcpkg.cmake) |
+----------------------------+ +------------------------------------+
| Portfile | ====> | CMakeLists.txt |
| (ports/foo/portfile.cmake) | | (buildtrees/../CMakeLists.txt) |
+----------------------------+ +------------------------------------+
```

To determine the host in a portfile, the standard CMake variables are fine (`CMAKE_HOST_WIN32`).

To determine the target in a portfile, the vcpkg triplet variables should be used (`VCPKG_CMAKE_SYSTEM_NAME`).

See also our [triplet documentation](../users/triplets.md) for a full enumeration of possible settings.