[vcpkg] Implement versions db generator #13777

vicroms · 2020-09-28T04:14:47Z

Reduced scope of this PR to only generating version files from our Git history

Modify x-history to allow JSON output and add more version related information
Make x-history actually parse CONTROL and vcpkg.json files to fix several edge cases
Create script that generates version database files for each port
Detect version scheme when loading CONTROL files
Separate version string from port version in old CONTROL files

vicroms · 2020-09-28T04:27:39Z

I'm not sure whether to check in the db files, there are too many of them.
Right now the DB files are one per port (located inside scripts/port_versions_db and use JSON format. Generating them took about 1 hour on my computer.

Also, it seems like I forgot to clean up some unrelated code.

ras0219-msft

Thanks for opening this PR!

I haven't finished reading more deeply into the algorithm, but here are some more trivial comments.

toolsrc/src/vcpkg/commands.porthistory.cpp

toolsrc/include/vcpkg/versions.h

toolsrc/src/vcpkg/commands.porthistory.cpp

toolsrc/src/vcpkg/versions.cpp

…et-versions-from-history

* Introduce FeatureFlagSettings struct to more easily access feature flags throughout the program * To avoid users accidentally starting to write "version" instead of "version-string" in their manifests, vcpkg explicitly detects and prevents usage of ports with schemes other than "String" * Drive-by fix of copiable SourceControlFileLocation and an exposed use-after-move bug This code is largely extracted from PR microsoft#13777 Co-authored-by: Victor Romero <romerosanchezv@gmail.com>

…et-versions-from-history

toolsrc/src/vcpkg/sourceparagraph.cpp

ras0219

I've lightly reviewed the changes to port history since that's an x- command.

toolsrc/src/vcpkg/paragraphs.cpp

toolsrc/src/vcpkg/sourceparagraph.cpp

toolsrc/src/vcpkg/vcpkgcmdarguments.cpp

#14079) * [vcpkg] Add `versions` feature flag and version field manifest parsing * Introduce FeatureFlagSettings struct to more easily access feature flags throughout the program * To avoid users accidentally starting to write "version" instead of "version-string" in their manifests, vcpkg explicitly detects and prevents usage of ports with schemes other than "String" * Drive-by fix of copiable SourceControlFileLocation and an exposed use-after-move bug This code is largely extracted from PR #13777 Co-authored-by: Victor Romero <romerosanchezv@gmail.com> * [vcpkg] Address CR Comments. Fix test crash on Linux. Co-authored-by: Robert Schumacher <roschuma@microsoft.com> Co-authored-by: Victor Romero <romerosanchezv@gmail.com>

strega-nil

I haven't looked at the rest of the code, but here are the things I see that should be corrected with the python.

scripts/generatePortVersionsDb.py

…et-versions-from-history

…to versioning/get-versions-from-history

strega-nil

Thanks for the changes victor!

[vcpkg] Implement versions db generator (microsoft#13777)

microsoft#14079) * [vcpkg] Add `versions` feature flag and version field manifest parsing * Introduce FeatureFlagSettings struct to more easily access feature flags throughout the program * To avoid users accidentally starting to write "version" instead of "version-string" in their manifests, vcpkg explicitly detects and prevents usage of ports with schemes other than "String" * Drive-by fix of copiable SourceControlFileLocation and an exposed use-after-move bug This code is largely extracted from PR microsoft#13777 Co-authored-by: Victor Romero <romerosanchezv@gmail.com> * [vcpkg] Address CR Comments. Fix test crash on Linux. Co-authored-by: Robert Schumacher <roschuma@microsoft.com> Co-authored-by: Victor Romero <romerosanchezv@gmail.com>

* [vcpkg] Add script to generate ports versions history * [vcpkg] Fix formatting * Fetch port versions from commit ID * Use global --x-json switch * Use --no-checkout when cloning secondary instance * Clone from local repository instead of from GitHub * Use CmdLineBuilder to build git commands * Use CmdLineBuilder and reduce repeated code * Fetch version at baseline and code cleanup * Guess version scheme from old CONTROL files * Rename version db generator script * Simplify x-history json output * Use CONTROL/manifest parsers on x-history * Use git-tree instaed of commit-id * Remove 'ports' field from root object * Clean up code * More code cleanup * Improve port version detection * Improve generator logging * Do not ignore parsing errors in CONTROL files * PR review comments in Python script * Fix subprocess.run() calls * Make `canonicalize()` return error instead of terminating * [vcpkg] Add tests for new test_parse_control_file paths * Remove unnecessary std::move() calls * Fix formatting * Python formatting Co-authored-by: Robert Schumacher <roschuma@microsoft.com>

[vcpkg] Add script to generate ports versions history

2c310bb

[vcpkg] Fix formatting

5e40036

PhoebeHui assigned PhoebeHui and JackBoosY Sep 28, 2020

JackBoosY added category:vcpkg-feature The issue is a new capability of the tool that doesn’t already exist and we haven’t committed info:internal This PR or Issue was filed by the vcpkg team. labels Sep 28, 2020

Fetch port versions from commit ID

59926f1

vicroms changed the title ~~[vcpkg] Add script to generate version database files~~ [vcpkg] Add script to generate version database files and fetch sources Sep 28, 2020

ras0219-msft requested changes Sep 28, 2020

View reviewed changes

vicroms added 11 commits September 28, 2020 13:38

Use global --x-json switch

394dccf

Use --no-checkout when cloning secondary instance

9fabe51

Clone from local repository instead of from GitHub

baa05ef

Use CmdLineBuilder to build git commands

037471c

Use CmdLineBuilder and reduce repeated code

9a7f298

Merge branch 'master' of github.com:microsoft/vcpkg into versioning/g…

c0117eb

…et-versions-from-history

Fetch version at baseline and code cleanup

b07099c

Guess version scheme from old CONTROL files

87fc6af

Rename version db generator script

5ed80e0

Merge branch 'master' of github.com:microsoft/vcpkg into versioning/g…

61043f5

…et-versions-from-history

Simplify x-history json output

2fe2a61

ras0219 mentioned this pull request Oct 15, 2020

[vcpkg] Add versions feature flag and version field manifest parsing #14079

Merged

vicroms added 4 commits October 19, 2020 13:51

Merge branch 'master' of github.com:microsoft/vcpkg into versioning/g…

b98c92a

…et-versions-from-history

Use CONTROL/manifest parsers on x-history

5c02c5b

Use git-tree instaed of commit-id

ee191eb

Remove 'ports' field from root object

7248364

vicroms mentioned this pull request Oct 20, 2020

[vcpkg registries] Registries: Take 2 #13590

Merged

vicroms changed the title ~~[vcpkg] Add script to generate version database files and fetch sources~~ [vcpkg] Implement versions db generator Oct 20, 2020

vicroms added 2 commits October 20, 2020 23:03

Improve port version detection

91cb16b

Improve generator logging

42105e1

ras0219 suggested changes Oct 21, 2020

View reviewed changes

toolsrc/src/vcpkg/sourceparagraph.cpp Show resolved Hide resolved

ras0219 suggested changes Oct 21, 2020

View reviewed changes

ras0219-msft and others added 2 commits October 21, 2020 15:02

Merge branch 'master' of https://github.com/microsoft/vcpkg into HEAD

52559f4

Do not ignore parsing errors in CONTROL files

8ca07da

vicroms requested a review from BillyONeal October 22, 2020 03:22

strega-nil suggested changes Oct 22, 2020

View reviewed changes

ras0219 approved these changes Oct 22, 2020

View reviewed changes

vicroms and others added 7 commits October 22, 2020 14:20

Merge branch 'master' of github.com:microsoft/vcpkg into versioning/g…

e400b5a

…et-versions-from-history

PR review comments in Python script

08c5685

Fix subprocess.run() calls

38504e7

Make canonicalize() return error instead of terminating

d27f294

[vcpkg] Add tests for new test_parse_control_file paths

d664855

Remove unnecessary std::move() calls

0d077e3

Merge branch 'dev/roschuma/cr' of https://github.com/ras0219/vcpkg in…

8a9115c

…to versioning/get-versions-from-history

strega-nil reviewed Oct 22, 2020

View reviewed changes

vicroms added 2 commits October 22, 2020 17:50

Fix formatting

c312541

Python formatting

0c7079e

vicroms merged commit e9f8cc6 into microsoft:master Oct 26, 2020

Jimmy-Hu added a commit to Jimmy-Hu/vcpkg that referenced this pull request Oct 27, 2020

Merge pull request #374 from microsoft/master

1251af9

[vcpkg] Implement versions db generator (microsoft#13777)

vicroms deleted the versioning/get-versions-from-history branch October 27, 2020 17:01

vicroms mentioned this pull request Nov 26, 2020

[vcpkg] x-history command fails for boost package #11422

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[vcpkg] Implement versions db generator #13777

[vcpkg] Implement versions db generator #13777

vicroms commented Sep 28, 2020 •

edited

Loading

vicroms commented Sep 28, 2020

ras0219-msft left a comment

ras0219 left a comment

strega-nil left a comment

strega-nil left a comment

[vcpkg] Implement versions db generator #13777

[vcpkg] Implement versions db generator #13777

Conversation

vicroms commented Sep 28, 2020 • edited Loading

vicroms commented Sep 28, 2020

ras0219-msft left a comment

Choose a reason for hiding this comment

ras0219 left a comment

Choose a reason for hiding this comment

strega-nil left a comment

Choose a reason for hiding this comment

strega-nil left a comment

Choose a reason for hiding this comment

vicroms commented Sep 28, 2020 •

edited

Loading