Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[vcpkg] Add commands to maintain and verify versions db integrity #14999

Merged
merged 61 commits into from Jan 8, 2021

Conversation

vicroms
Copy link
Member

@vicroms vicroms commented Dec 8, 2020

This PR will add new commands to help maintain the version files up-to-date.

  • Command to verify version files.
  • Command to verify baseline file.
  • Command to update versions and baseline files.
  • Add end-to-end tests
  • e2e test for ci-verify-versions to catch changes to the git sha within a single version
  • Rename $x-default-baseline to "builtin-baseline" and check it is not used without the versions feature enabled
  • Rename the field in baseline files to "baseline"
  • Direct actionable messages for any failures reported by ci-verify-versions
  • Check versions/ directory in
  • Enable new CI checks

Name of commands is up for discussion.

vcpkg x-ci-verify-versions

This command verifies that each port:

  • Has a corresponding version file in ${VCPKG_ROOT}/port_versions/${prefix}-/${port_name}.json
  • The file contains a versions array.
  • The top entry in the versions array matches the current version of the port.
  • The local SHA matches the one in the git-tree property.
  • The baseline file exists.
  • The baseline version matches the current local version and the latest version in the db files.

This command will run as part of CI once the port_versions files have been merged.

vcpkg x-add-version ${port name}

Sets the current version in baseline.json to the version of the port in the ports directory; also adds the version to the corresponding versions file.

Drive-by changes:

  • Changes to BaselineDeserializer
    • Changed return type from VersionT to SchemedVersion. This is necessary for updating the baseline file correctly with x-add-version.
  • Changes to x-history:
    • Removed port version splitting.
    • Output to file with --output.
  • Changes to generatePortVersions.py:
    • Parallelized generation. NOTE: Requires setting GIT_OPTIONAL_LOCKS=0 to avoid crashes.
    • Removed baseline.json generation.
  • Changes to generateBaseline.py:
    • Generate from existing files in port_versions since calling x-package-info is infinitely slow.
    • Sped up execution 200-fold.

@vicroms vicroms changed the title [vcpkg] Add commands to maintain and verify versions db integrity [WIP][vcpkg] Add commands to maintain and verify versions db integrity Dec 8, 2020
@vicroms vicroms marked this pull request as draft December 8, 2020 09:10
@JackBoosY JackBoosY self-assigned this Dec 9, 2020
@JackBoosY JackBoosY added category:vcpkg-feature The issue is a new capability of the tool that doesn’t already exist and we haven’t committed info:internal This PR or Issue was filed by the vcpkg team. labels Dec 9, 2020
@vicroms
Copy link
Member Author

vicroms commented Dec 9, 2020

We now check that local git-tree objects match those in the db files.

FAIL: zziplib
Found the following errors:
Error: Git tree-ish object for version `0.13.71` in `D:\vcpkg\port_versions\z-\zziplib.json` does not match local port files.
        Local SHA: 8171b5d16b817dabe3a1e0dfc711f2aa556b7a4e
         File SHA: 46d8c438267ece7a8e8ca0079896b25e579a845d

scripts/generateBaseline.py Outdated Show resolved Hide resolved
toolsrc/src/vcpkg-test/commands.cpp Show resolved Hide resolved
toolsrc/src/vcpkg/vcpkgcmdarguments.cpp Outdated Show resolved Hide resolved
"Couldn't find explicitly specified baseline `\"default\"` in the baseline file.",
baseline_identifier);
return Strings::format("Couldn't find explicitly specified baseline `\"default\"` in the baseline file.",
baseline_identifier);
}

// attempt to check out the baseline:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I may be missing some subtleties w.r.t. the interesting return value from load_baseline_versions() and I haven't looked at the consumers of this function yet, however:

Why does the processing of port_versions/baseline.json potentially cause this section to be short-circuited? It seems that this function should:

  1. if baseline_identifier is empty, return {}
  2. if baseline_identifier is "default", parse the builtin baseline json and return that (or fail)
  3. otherwise, try to check out baseline_identifier and parse from that.

If so, I'd expect a top-level if() to make that clearer.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what is the actual use case for empty baseline_identifier. I just tried to preserve existing behavior.

toolsrc/src/vcpkg/commands.add-version.cpp Outdated Show resolved Hide resolved
@ras0219-msft ras0219-msft merged commit 2a42024 into microsoft:master Jan 8, 2021
@vicroms vicroms deleted the versioning/version-files branch January 14, 2021 00:06
strega-nil pushed a commit to strega-nil/vcpkg that referenced this pull request May 5, 2021
* [vcpkg] Refactor end-to-end tests

* [vcpkg] Cherry-pick x-builtin-ports-root from microsoft#14999

* [vcpkg] Move create test from unit tests to e2e

Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
strega-nil pushed a commit to strega-nil/vcpkg that referenced this pull request May 5, 2021
…crosoft#14999)

* [vcpkg] Add x-ci-verify-versions command

* Code cleanup

* Remove port version splitting from x-history

* Fix wrong message on success

* Parallelize versions file generator

* Use cpu_count()/2 to avoid crashes

* Check db SHA against local SHA

* Check baseline version with x-ci-verify-versions and make baseline generator much faster

* Implement x-add-version to update version db files

* Better checks for x-add-info and make x-ci-verify-versions silent on success

* Use find() instead of [] on maps

* Create version file if does not exist

* Allow redirection of ports/ and port_versions/

* add test ports

* WIP end-to-end tests

* Change pats in e2e tests

* Fix e2e args

* e2e once more

* Pass �ersions feature flag to e2e

* Exit with code 1 if there are errors

* Files to test for failure cases

* Update test files

* Add test for x-add-version

* fix redirected ports in last test

* Add CI check (use dummy data)

* Add feature-flags=versions

* Ignore subdirectories inside ports

* Add --verify-git-trees switch

* [vcpkg] Fix build breaks

* [x-ci-verify-versions] PR comments

* [x-add-version] PR comments

* Fix merge conflicts

* Modify tests and pipeline

* Baselines should only have version-string

* Refactor x-add-version

* [vcpkg] Fix help message

* [vcpkg] Fix minor warnings

* `x-add-version --all` doesn't stop on first failure and reduced default verbosity

* [vcpkg] Fix default-baseline

* Load file instead of using paths provider

* Format

* Remove ci test

* Add fish port for testing

* Update version files

* Update fish port to cause SHA discrepancy

* Test for discrepancy between local SHA and declared SHA

* Missing = operator

* Check for error message since x-add-version exits with code 0

* Make x-add-version fail with non-zero exit code if not run with --all

Co-authored-by: Robert Schumacher <roschuma@microsoft.com>
Co-authored-by: Robert Schumacher <ras0219@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category:vcpkg-feature The issue is a new capability of the tool that doesn’t already exist and we haven’t committed info:internal This PR or Issue was filed by the vcpkg team.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants